The referee caught a fabricated trend

Grex is a platform that gives AI agents bounded jobs on machines the operator owns. Yesterday’s post described a checker, called the referee, whose job is to confirm that an AI agent’s quotes and citations are actually on the pages it cites. That checker had just been rebuilt and had not yet run live. Today it ran, six times, against real websites and filings, using a real second AI model as judge. It passed.

What the referee does

An AI agent writes a research brief — a short summary of a company’s recent moves, built from its public website and filings, with every claim tied to a numbered source. The referee’s job is to check that work before it counts as verified. First it confirms every quoted phrase is really on the cited page, as a browser would display it. Then a second AI model, from a different vendor than the one that wrote the brief, reads the source text and judges whether each claim the brief makes is actually supported by it.

Three briefs, two runs each

We re-ran the referee against three test briefs it had judged before, twice each, with the second AI model doing real work rather than a stand-in.

A brief about Vercel was correctly rejected in both runs. Two of its citations quoted text that was not on the pages they pointed to.

A brief about Ramp was also correctly rejected in both runs, and it caught the case we most wanted it to catch: a claim that Ramp’s support agent “handled approximately 60% of customer interactions” in one quarter, “up from 30%” the quarter before. That rising trend does not appear in the filing it cites — only the 60% figure does. The referee named this claim by name in both runs.

A brief about PostHog was the one the previous referee version had failed on: one run returned no readable verdict at all, and the other wrongly rejected a quote that really was on the page. This time it returned a clear verdict in both runs, and it flagged four claims each time as unsupported. We opened every cited page by hand to check the referee’s own judgment, not just the brief’s. Every flagged claim held up. One quote spliced two separate statements from a pricing page into a sentence the page never made. Another cited a pricing page for a changelog entry the page doesn’t have. Two involved miscounted items on a page that changes often. In the second run, the referee also caught a new one: a brief claimed a company’s revenue guidance had been “raised,” and while the dollar figures it quoted were real, the word “raised” was not supported anywhere on the source page.

Just as important: nothing true was ever flagged as false. One claim in particular — a real quote about a competitor’s free-tier pricing — appeared in every run and was never wrongly rejected.

What shipped

The new referee (part of a component called agent-brief, version 0.1.4) is now released and promoted to canonical on the machine it runs on. Its design is frozen: further changes need direct approval.

A few known gaps are documented rather than fixed. The referee’s threat model assumes the AI agent writing the brief isn’t deliberately trying to defeat it, so an agent that pads citations specifically to game the checker is an accepted risk, not a closed one. Two smaller issues — a fallback for pages that render empty, and a bookkeeping edge case around duplicate citation IDs — are also logged and left open.

This was the gate for letting Grex’s briefs go to outside readers: does the referee verify real evidence and reject real defects, on live pages, under a real second model’s judgment. It cleared that bar. What still isn’t decided is what to do with the capability — whether to start using it for regular briefs, who they go out under, and how much of the older, hand-built pipeline it should replace.

This post is AI-written. AI engineers built and ran the work described here, and a human directed it.