Field report · 3 min read

Four jobs looked healthy and delivered nothing

Missing analytics, unpublished drafts, misrouted content, and broken pages revealed four ways a completed run can conceal failed work.

An audit of our business portfolio found four output paths that looked operational but were not delivering their intended results. One had failed for eighteen days.

Grex is our system for coordinating AI teams and checking their work. Some of these paths ran outside Grex, and the one inside its receipt system still recorded the wrong success condition.

That distinction matters. This is a report of failures in our own operation, including what Grex’s controls could not catch at the time.

The analytics job recorded that it ran

A scheduled job was supposed to collect analytics and search data for four sites. It could not find a required command-line tool in the scheduler’s environment.

The job logged the error but still wrote the date into its “already ran today” marker. For eighteen days, that marker confirmed execution without confirming any data had been produced.

Adding the tool to the scheduled environment restored collection. The larger lesson was to make completion depend on the four expected output files.

The publishing bridge could not see its queue

A bridge was meant to publish approved drafts to two websites. For weeks, each run reported a hundred fetched items and zero publications.

Forty-two drafts had accumulated. The queue had grown to 272 items, but the bridge only fetched its first hundred. Routing and acceptance rules also skipped the drafts it did see.

A watchdog should have warned about aging drafts. The audit had not yet established why it failed to alert.

Correcting routing and increasing the fetch limit made nine drafts publishable. A completed polling cycle had not meant the publishing job was working.

A destination field disappeared in storage

A content team wrote for three sites. Its briefs identified the destination so that articles could use the correct subject framing and publishing route.

The storage envelope silently dropped that field. All 26 inspected briefs returned without a destination and fell back to the divorce-inventory site’s defaults.

Weather and general inventory articles therefore reached the wrong brand checks and were correctly rejected there. A routing fix had existed in source for three weeks without surviving the write-and-read path.

A temporary repair recovered the destination from brief titles. Silent loss of the field was recorded as a separate defect.

Thirty-five ranking pages returned errors

An earlier editorial hold removed 35 of forty blog posts from search indexing. It also excluded their directories from deployment, so visitors received 404 errors.

Six of the site’s seven strongest ranking pages were among those hidden. Removing only the indexing restriction would not have restored them.

The repair removed both restrictions. Production checks confirmed all 35 pages returned successfully and the sitemap again contained 48 URLs. That proved availability was restored, not that search rankings recovered immediately.

What a receipt would have caught

A receipt is a saved record of work, cost, permission, and claimed result. Three of these paths emitted no Grex receipts. The fourth recorded that a team completed its run, rather than what it produced.

The existing receipt design therefore would not have caught most of these failures.

A useful success condition names the result: four data files written, approved drafts published, or advertised pages reachable. It also needs an alert when that result is missing.

All four failures left evidence in logs. None reliably brought that evidence to someone who could act on it.

← Back to blog