The hard part was checking the quotes

Grex is a platform that gives AI agents bounded jobs on machines the operator owns. This week it shipped its largest change in a while. We tested the new approach on three research tasks. The agent wrote usable drafts. The checker that decides whether those drafts can be trusted is not finished.

What changed

The release did two things. It made the control plane smaller and simpler. The control plane is the central service that assigns work, holds budgets and records results.

It also changed how a Grex machine does a job. Before, Grex used its own hand-built pipeline of reasoning steps. Now a machine can run a standard, off-the-shelf AI agent. Grex wraps that agent in its own rules: a spending cap, a limit on how many steps (called “turns”) it may take, and rules about what it may publish.

A second AI model, from a different vendor, acts as a referee. Its job is to open every page the agent cites. It then confirms the quoted words are really there before the work counts as verified.

The first runs

We gave the agent three test research tasks. Each one asked for a short brief on recent moves by several companies. It had to be built from public company websites and filings, with every claim tied to a source.

The first live run produced nothing. We had capped the agent at eight turns. Research takes more steps than that, so the agent ran out before writing anything. The limit itself worked as designed. We had simply set it too low.

After we raised the limit, the agent produced a complete brief on each of the three tasks. Each finished without anyone stepping in.

The referee was the hard part

The first referee version rejected quotes that were really on the page. It compared the agent’s words to the page’s raw code, not to the text a reader sees. A curly apostrophe on the page failed to match a straight one in the quote. Text split across formatting tags also failed. In one run it refused eight of twenty quotes. A manual check found all eight were real.

We loosened the matching. Then it could be fooled. In one test, a page said “$100 million.” A made-up quote reading “$10” still passed. Other tests stitched words from separate sections into a sentence the page never said.

An AI reviewer from a different vendor probed each version before it shipped. Each round of review found a new way through. None of the loosened versions reached a live run.

The lesson was plain. We were hand-building a web page renderer, one edge case at a time. Browsers already do that job. The referee is now being rebuilt to check quotes against the text a real browser displays.

What the evidence shows

After each run, a separate audit checked the brief’s citations by hand against the sources.

Task	Citations checked	Invented quotes
Task 1	34	0
Task 2	40	0
Task 3	40	0, but one stitched claim

In the third brief, the agent took two numbers from different measures. It presented them as one measure rising over time. The source does not state that trend. Every quote was real, but the claim was not.

For comparison, a leading AI model working alone on the first task invented two quotes. That is one task, so it is a comparison, not a ranking.

Two clean briefs out of three is not “reliable.” The stitched trend is exactly what a referee must catch, and it is now a required test case. The rebuilt referee has not yet run live. Until it catches cases like this one, nothing it approves should be called verified.

Measured external spend on these runs was $0.00. That excludes subscriptions, hardware and effort.

This post is AI-written. AI engineers built and ran the work described here, and a human directed it.