Six ways a pipeline says "success" while writing nothing

Green is not a status. Six real incidents where a job reported success and had written nothing, and the six questions I now ask any green dashboard.

By Dasha. Updated November 6 2026. https://heardofyou.io/blog/six-ways-a-pipeline-says-success

At my day job I look after a few dozen scheduled jobs that copy data from one place to another every night. In August I found that three of the daily ones had been reporting "succeeded" every morning while none of them was finishing. One had been dropped quietly every day for 66 days. The log was green the whole time. Nobody had lied to me. I had built a system in which "success" meant six different things, and not one of them was "the data arrived".

Here is what the status column said, next to what had actually happened.

| What the log said | What had happened |
|---|---|
| succeeded | The request was queued. The function died five seconds in. |
| success, 87 runs in a row | About 18% of the rows had arrived. |
| 200, ok: true, upserted: 0 | The upstream API was down. |
| success | Three writes aimed at columns that do not exist. |
| exit 0 | A backup with today's date and March's contents. |
| no alert | A sync that had been dead for 33 days. |

1. The scheduler logs the enqueue, not the result

The mechanism is boring, which is why it worked on me. The Postgres scheduler extension fires an HTTP request at a function and logs "succeeded" when the request has been handed off, not when the function returns. The HTTP helper underneath gives up after 5 seconds by default. One of my jobs takes 34.8 seconds. So every morning the request went out, the scheduler wrote success, and the function died a few seconds in. Two sibling jobs had a status code of NULL, which is the response table's way of shrugging.

The log I should have been reading was that response table, which nobody reads because it has a terrible name and no dashboard. The fix was two lines: an explicit timeout based on the measured run time, and a check on the destination table instead of the scheduler's opinion. Sixty-six days to learn two lines.

2. The filter that accepts everything except "error"

I had 62 freshness checks, one per job, and I was quite proud of them. All 62 filtered on status not equal to "error". None mentioned "partial" or "truncated". None required "success". And all 62 pointed at the log table rather than the table the data was supposed to land in. Every check verified that a log row had been written, never that rows had.

Over 60 days the log held 299 partial runs across five jobs, 78 truncated runs on one job, and 46 runs flagged "issues found". That is 423 non-success runs that passed their watchdog by construction. One dimension table sat at roughly 18% of its true size, about 75,000 rows against 404,000, across 87 consecutive runs marked success. Another job stopped at exactly 800 rows, its own page cap, on 11 of 21 days, because the loop exited while the cursor still said there was more.

The tell is a row count that lands on a round number repeatedly, or a daily count trending down while the status stays green. The rule is to record rows expected beside rows written, and let the status be derived from the comparison rather than asserted by the writer. A job that cannot say what it expected should report "unknown", never "success".

3. Errors that come back as values

Two flavours of the same thing. The first is .catch(() => []). During an outage of an upstream API, a sync returned 200 {ok: true, upserted: 0} and the outage became a quiet day. The fix was three states instead of two, success, partial and error, plus a test that simulates the failure and proves the error state actually fires. A state you have never seen fire is a drawing of a state.

The second is the client library that returns {error} instead of throwing. Three writes targeted columns that did not exist. All three reported success. Zero rows were recorded against twelve real events, and I found it only because an unrelated type check was already red and I went looking. Same family: a delete-then-reload job that timed out after the delete had committed and logged "success, 0 rows" over an empty table, which is technically accurate.

4. The backup with today's date

A nightly job packed a git clone into cloud storage. It exited 0 every night from March to September. The last line of the script ended in 2>/dev/null, so the exit code was always the exit code of nothing in particular. Both source repositories had been returning 404 since March 6. For six months the job faithfully re-packed a frozen March clone, and because it wrote a new file each night, the file always carried today's date. A backup you check by timestamp passes this test forever. A backup dated today that contains March is worse than no backup, because it stops you looking. Verify by contents, never by timestamp or exit code.

5. What a monitor must be able to say

One weekend in August my alerting produced 9 alerts. Seven were false. The one thing that was genuinely broken, a sync that had been dead for 33 days, never alerted at all. The monitor derived "stale" and "rows recently written" from the same column of the same output table, so its guard for "ran and produced nothing" could never trigger. Another check compared a business date to the current time and fired every morning about nothing.

A monitor has to be able to say three different things: fine, broken, and I could not look. If what it prints when looking at nothing matches what it prints when everything is fine, it is decoration. The questions I now ask any green dashboard, including my own:

1. What does this check print when it looks at nothing?
2. Is it reading the log or the destination?
3. What did the run expect, and where is that written down?
4. Which statuses other than "error" can this job emit, and which of them pass?
5. If the credential died tonight, what would tomorrow's dashboard show?
6. When did this last fail for real, and did a person see it?

The rule I follow now: verify at the destination, never at the caller. A status is a claim the writer makes about itself. A row count in the table the data was meant to reach is a measurement. Only one of those is allowed to turn a light green.

None of this is clever. All six are ordinary Postgres, HTTP and shell behaviour that I knew in the abstract and did not check in the particular. It is also why my side project runs every question three times and puts a person at the end of the pipeline: a confident "success" from a machine is not a measurement until something dull has counted what landed.