We spent a day auditing our own portfolio and found four production paths that had been reporting success while producing nothing. One had been dead for eighteen days. Another had published zero articles on every scheduled run for weeks. A third made a routing fix a no-op for three weeks while looking correct in the source. Every one of them logged its own failure, accurately, to a file nobody read.

This is a field report rather than a product argument. The failures are ours, the evidence is checkable, and the useful part is the pattern they share.

What were we actually trying to do?

The starting task had nothing to do with reliability. We wanted to know which of roughly forty documented business ideas deserved attention, so we went looking for demand data: search impressions, sessions, which pages ranked. The audit was supposed to take an hour. Finding the data was what took the day.

Failure 1: a job that stamped its own success marker

A local job pulls Google Analytics and Search Console data for four properties each morning, writes a JSON artifact per site, and publishes a summary into the work graph. It runs from a launchd schedule with a minimal PATH.

It had been failing since July 13 with FileNotFoundError: [Errno 2] No such file or directory: 'gcloud'. The Google Cloud SDK lives outside the directories launchd puts on PATH. Every run raised, was caught, was logged, and then the job wrote today’s date into its .last_run_date marker anyway. The marker exists to make the job idempotent within a day. It was recording that the job had run, which was true, rather than that the job had produced anything, which was false.

Eighteen days of no search data, with a file on disk asserting the job had run that morning.

The fix was one entry on a PATH. The defect was that the marker recorded execution instead of output.

Failure 2: a publisher that published nothing, on schedule

A bridge pulls approved article drafts out of the work graph and publishes them to two live sites. Its log line, on every single cycle for weeks:

done: 0 published, 0 gated, 0 consumed, 100 fetched

That line is accurate. It is also a description of total failure, printed at INFO, four times a day, into a file with no reader.

Forty-two article drafts had accumulated since July 20. The bridge classified every one of them as skip. It fetched the first 100 work items from a queue that had grown to 272, so it could not even see most of them, and the ones it did see were rejected for reasons that were individually correct and collectively wrong.

There is a watchdog for exactly this condition. It is supposed to page when clean drafts age unconsumed. It did not page. We have not yet determined whether it was misconfigured or watching a condition these drafts never entered, and we are not going to guess in public.

Failure 3: a routing fix that was a no-op for three weeks

This is the one worth reading carefully, because nothing about it looks wrong.

A content pack writes for three different sites. Each brief carries the site it targets, and the pack uses that to select the right editorial voice and the right publishing tag. A commit on July 10 added exactly this, with a comment explaining the bug it prevented.

The site never survived the round trip. Findings are written into the work graph through an envelope that keeps its own keys and discards the rest. Reading the metadata back off a live brief returns:

body, confidence, cost_estimate, escalated, executed,
mandate, pending_id, proposed_action, receipt, rung

No site. All 26 research briefs in the queue read back with site unset, so every article fell through to the default directive, which belonged to a divorce-inventory product. Road-trip weather topics and home-inventory topics were written with divorce framing, tagged for the wrong site, and then correctly rejected by that site’s brand gate at publish time.

The code was right. The write was silently dropped, and the read silently defaulted. Three weeks of a fix that existed and did nothing.

Failure 4: thirty-five ranking pages serving 404

The last one is the most expensive and the least technical.

Ten days earlier, a commit titled “Improve AdSense readiness and editorial trust” had added robots: {index: false, follow: false} to 35 of a site’s 40 blog posts and trimmed the sitemap from 48 URLs to 13. That was a deliberate decision and a defensible one at the time.

What the audit surfaced is that those 35 posts were the pages that ranked. Search Console, 28 days:

Page	Position	Status
best-time-drive-route66	5.5	hidden
driving-i70-rockies	5.7	hidden
weather-i95-by-month	6.8	hidden
driving-i40-cross-country	7.6	hidden
best-times-cross-country-drive	9.0	visible
mountain-pass-snow-checklist	10.0	hidden
driving-i75-michigan-florida	11.0	hidden

Six of the seven best-ranking pages on the site had been switched off. The homepage sat at position 36.

The hold had also been implemented twice. Beyond the robots directive, the hosting config excluded all 35 directories from deployment, so the pages were not merely deindexed. They were returning 404. A 404 is a stronger removal signal than noindex, and these URLs were still ranking in the top ten while serving one.

Removing the robots line alone would have shipped a fix that changed nothing. We caught the second half only by reading the deploy config before deploying.

What do these four have in common?

Each one had a mechanism that reported on itself, and each mechanism reported on the wrong thing.

The search job asserted that it ran. It should have asserted that it wrote four files.
The publisher asserted that a cycle completed. It should have asserted that a cycle that fetched 100 candidates and published none is an anomaly, not a result.
The routing code asserted nothing. The write succeeded because dropping unknown keys is not an error, and the read succeeded because a missing key has a default.
The deploy asserted that files uploaded. It should have asserted that the pages a sitemap advertises return 200.

In every case a log line contained the truth. In no case did anything act on it.

Which of these would a receipt have caught?

The honest answer is: not most of them, as things stood.

Three of the four ran outside the receipt surface entirely. They are local scheduled scripts, not crew actions, so they emitted no receipt at all. The fourth was inside it and still passed, because the outcome assertion recorded that the pack completed rather than what the pack produced.

That is the lesson worth taking, and it sharpens something we have written about before. A receipt is only as good as its outcome assertion, and an outcome assertion that names completion instead of an artifact is a tool-success log wearing better clothes. “The run finished” is transport. “The run wrote four site files, published nine drafts, and left zero unconsumed” is an outcome.

The second lesson is cheaper and just as important. A log that records failure is not an alert. All four of these failures were discoverable by reading a file. None of them were surfaced to anyone. Detection you have to go looking for is detection you do not have.

What we changed

Put the SDK on the scheduled job’s PATH, then reran it. Four sites of search data flowing again for the first time since July 13.
Fixed the publisher’s routing to read topic from the content rather than trusting a tag that means nothing, and raised its fetch ceiling. Zero publishable drafts became nine.
Recovered the target site from the brief title, so the three-week no-op stops being one, and filed the envelope’s silent key-dropping as a defect in its own right. Any pack putting routing data in metadata has the same latent bug.
Removed the robots directives and the hosting exclusions together, then verified against production: 404 to 200 on all 35, index, follow present, sitemap serving 48 URLs.

Every one of those is a small change. The interesting part was never the fixes. It was that a system with cost tracking, run events, a dashboard, and a watchdog ran for three weeks in a state where four of its output paths produced nothing, and reported that everything was fine.

If you run agents in production, the question worth asking is not whether your runs succeed. It is what your success assertion is actually asserting.

Four silent failures in our own fleet, and what a receipt would have caught