Grex 2.0
Did review improve the result?
The first AutomationBench pilot tied the direct runner on timely correctness and exposed a broken correction loop.
Completed · diagnostic evidence
Both arms finished 13 of 30 tasks strictly correctly, on time and without assistance. Grex took about 2.1 times the total elapsed time. The later audit found that corrections never received reviewer feedback, so this pilot cannot establish whether a functioning review-and-correction loop helps.
The experiment, explained
A narrated explanation of recorded results, with synthetic narration and visible English captions. Diagrams explain the method; they are not recordings of execution.
Read the video transcript
Does review improve the result?
Does adding review make an AI worker complete more business tasks correctly? We tested Grex against a direct runner using the same model and simulated tools.
The hypothesis
The original protocol asked whether Grex adds value beyond its worker model. Our reconstructed hypothesis was that review and one correction might improve outcomes under shared limits.
Thirty matched tasks
Thirty tasks covered six business domains. Each arm received an identical starting world. An independent evaluator graded completed states after execution, without feeding answers to the workers.
Thirteen out of thirty, each
Both arms completed thirteen tasks correctly, on time and without assistance. Grex took about twice the elapsed time. No correction rescued a failed task.
A broken feedback path
The audit found a crucial defect: corrections never received reviewer feedback. Reviews also exceeded limits, and corrections repeated simulated writes. This did not cleanly test the intended loop.
Repair, then measure again
We kept the result and repaired the mechanics. The November fourth rerun is a separate experiment, with its own outcome and limitations. Neither study used live customer accounts.
What were we trying to find out?
Does Grex improve completed business tasks beyond what the same worker model achieves in a direct run?
Hypothesis
Independent review and one correction might increase correct outcomes under shared episode limits.
This directional hypothesis is reconstructed from the question and comparison recorded in the November 2 protocol. It is not a quotation or a registered numerical prediction. The protocol preceded execution; implementation defects prevented a clean test of the intended correction loop.
How the experiment worked
Run the same task in two separate simulated business environments, then grade the final state after execution ends. The direct arm uses the same Grex assignment infrastructure without review; this isolates the added review path more closely than comparing unrelated products.
Direct runner
- Worker attempts the task
- Retain the final world
Grex pilot
- Worker attempts the task
- Separate review
- Feedback not delivered to correction
- Correction repeats the original request; review if completed
Independent evaluation checks the retained worlds against the same upstream grader.
Broken feedback path: the correction received the original request again, without reviewer feedback. Review rejection did not mean the benchmark failed.
Download the method illustration
Sample: 30 matched tasks: five each from sales, marketing, operations, support, finance and HR. There were 60 primary episodes and 90 attempts including corrections, all retained.
Freeze the comparison
Pin the public benchmark, task selection, initial worlds, instructions and episode limits before the scored run.
Run both arms
Use MiniMax-M3 on the same worker. Balance which arm runs first across the 30 pairs.
Review and correct
The Grex arm receives independent Claude Opus review and at most one correction. The intended feedback path was later found broken.
Seal, then grade
An independent evaluator checks the final simulated state with the unchanged upstream assertions after the attempts are sealed.
Audit side effects
Inspect action records separately for duplicate writes, policy breaches and reviewer disagreements the benchmark score may miss.
Controls and differences
- Both arms used identical initial worlds, the same worker model, tool loop and offered tools.
- Both had an 80-tool-call episode ceiling and a 30-minute episode limit. Individual Grex attempts had a shorter seven-minute limit.
- Worker and operational reviewer could not access hidden assertions or expected answers. Known-good and known-bad controls covered all 64 assertion families used.
- No real email, CRM or customer accounts were connected. All business effects happened in the simulation.
What happened
| Measure | Direct runner | Grex pilot |
|---|---|---|
| Strictly correct, timely and unassisted | 13 / 30 | 13 / 30 |
| Strict correctness alone | 13 / 30 | 14 / 30 |
| Total episode elapsed time | 68.7 minutes | 144.7 minutes |
| Median episode elapsed time | 120 seconds | 258 seconds |
| Reviewer acceptances | No reviewer | 0 / 58 reviews |
| Corrections that rescued a failed task | Not applicable | 0 / 30 corrections |
| Duplicate simulated effects | 2 in 2 episodes | 34 in 11 episodes |
No timely-correctness advantage
Grex won two task pairs, lost two and tied 26 on timely, unassisted strict correctness. One additional strictly correct Grex world was excluded because its correction ended in a provider timeout.
The correction loop was not actually tested
All 30 corrections received the original task again, without the reviewer's observations or notice that earlier actions persisted. Their first model requests matched the first attempts. This defect limits what the zero-rescue result means.
Rejection was not the same as an incorrect world
The reviewer rejected all 58 submissions it saw. Of 27 rejections where the benchmark passed, the audit classified 15 as justified operational objections, six as incomplete evidence and six as unsupported or immaterial. Agreement with the benchmark alone would miss legitimate concerns.
The audit changed the duplicate count
The complete audit found 34 duplicate simulated effects in the Grex arm, replacing the initial count of 19. Thirty-two occurred during corrections. The direct arm had two. These effects did not change the upstream correctness score.
Limits were breached and evidence was missing
Thirty-three of 58 reviews exceeded the reviewer token reserve, and the driver continued instead of halting. Thirty-three review packets omitted calls; 25 omitted at least one write. No blind operator disposition was recorded.
What this does not establish
- This is diagnostic evidence, not a protocol-conformant estimate of Grex's intended review-and-correction capability.
- Thirty tasks and one run per arm cannot establish a general quality advantage. The study used one worker model and Claude-family review and evaluation.
- The public benchmark contains scoring defects. One selected task required an action its offered tools could not perform; it remains in the denominator.
- Strict world-state correctness is separate from operational appropriateness, reviewer acceptance and an operator's decision to use the result. Blind operator usability cannot be reconstructed after scores are revealed.
- Execution used existing subscriptions with reported zero incremental charges. Subscription allocation, owned-compute cost and a provider-invoice reconciliation were not established; the work was not free.
The decision
Retain the negative result and repair the mechanics before evaluating tuned instructions or claiming benefit. Do not describe rejecting everything as evidence of safety.
What followed
The November 4 mechanics rerun delivered the missing feedback and tested other repairs. The study was later closed before its unseen-task test. Read the linked follow-up for those results and the stopping decision.
Evidence and further reading
Public summaries retain the reported counts and limitations. Private execution traces and infrastructure details are not published.
- Download the pilot evidence summary
Sanitized aggregate transcription of the independently authored report and its diagnostic addendum. Private task answers and operational traces are excluded.
- Pinned AutomationBench source
The public benchmark used for this study. These results are not an official leaderboard score.