The Cron Job That Lied to You

Your nightly backup ran at 2 AM. The ping arrived on schedule. No alerts, no incidents, nothing in your dashboard but a row of green checkmarks. The backup file was empty. The job ran. It checked in. It lied to you. Basic heartbeat monitoring solves one problem: knowing whether your job ran at all. If the ping stops arriving, you get alerted. That is genuinely useful and it catches a whole class of failures. But there is a quieter category of failure that heartbeat monitoring alone does not cover. The job shows up, does its ping, and something is still wrong. Here are the four ways that happens. The job finished, but so did another copy of it Your sync job runs every five minutes and usually completes in about 90 seconds. One night the database gets slow. The job starts taking six minutes. Cron does not know this. At the five minute mark it fires a new instance. Now you have two copies of the same job running at the same time, both reading and writing to the same tables. Each one event