0 / 58
Reviews accepted
Every completed review rejected the submission. Of 27 rejections of grader-correct worlds, the evaluator classified 15 as operationally justified, 6 as lacking decisive evidence and 6 as unsupported or immaterial.
Grex 2.0 · Evaluation progress · November 3, 2026
Does review improve the work?
We tested Grex on 30 simulated business tasks, each run with and without its review-and-correction loop. Both approaches completed 13 tasks correctly, on time and without intervention. Grex took about twice as long. The first pilot shows where the platform needs to improve.
First pilot · Completed November 3
Equal timely correctness. More work for Grex.
All 60 assigned runs are included. Each approach used the same 30 tasks and the same MiniMax-M3 worker model.
| Measure | Direct runner | Grex with review |
|---|---|---|
| Correct, on time, unassisted | 13 / 30 | 13 / 30 |
| Strict benchmark pass | 13 / 30 | 14 / 30 |
| Total elapsed execution time | 68.7 minutes | 144.7 minutes |
| Tool calls | 1,270 | 1,865 |
| Duplicate simulated effects | 2 in 2 episodes | 34 in 11 episodes |
On timely, unassisted correctness, Grex won 2 paired tasks, lost 2 and tied 26. One additional Grex world passed the grader, but its correction ended in a timeout and did not count as timely completion. A benchmark pass describes the final simulated state; it does not mean Grex's reviewer accepted the work.
What the pilot exposed
A review loop must help the operator finish.
0 / 30
Episodes rescued by correction
No correction changed a strict failure into a pass. Eight first attempts spent the entire shared tool-call budget, leaving their correction no calls.
34
Duplicate simulated effects
The Grex approach produced 34 duplicate effects across 11 episodes, versus 2 across 2 episodes for direct execution. Thirty-two of Grex’s duplicates came from corrections. These were simulated actions.
33 / 58
Reviews exceeded their reserve
Token accounting missed part of the reviewer's usage. The driver continued despite its stop rule. This is a protocol deviation, even without an incremental charge.
A follow-up audit found omitted calls in 33 review packets, including hidden writes in 25. Required operator dispositions were not recorded before grading, so we cannot report how much work an operator accepted for use. These gaps prevent a claim of improved delivery or proven safety.
Updated from the evaluator’s November 3 addendum: the full action-log audit replaces the initial count of 19 duplicate sends or creates in 10 Grex episodes. The revised count includes 19 simulated messages and 15 record effects; it changes no upstream score.
Method
A controlled comparison on simulated business work
AutomationBench is Zapier's public benchmark. Agents use simulated business tools to complete tasks in finance, operations, support, HR, marketing and sales. We selected five tasks per domain and kept the task set fixed.
Direct runner
One attempt, with up to 30 minutes and 80 tool calls. It used the same worker model, tools and execution bridge as the Grex approach.
Grex with review
A seven-minute attempt, a separate Claude Opus 5.5 review, and at most one correction followed by review. The whole episode shared the 30-minute and 80-call limits.
A separate evaluator applied the upstream grader after each run was finalized. It checked all 90 retained attempts, including corrections, and verified their recorded-state integrity. The evaluator did not build, run or tune the study. Engineering and study preparation were frontier-agent assisted; this was not an autonomously designed experiment.
How to read these results
- A small diagnostic pilot. Thirty paired tasks, one worker model and one run per task cannot establish broad superiority or equivalence.
- Grader limitations. Twenty-four tasks were flagged as potentially affected by registered grader issues. The evaluator found no observed result flipped by a registered issue, but unresolved false-pass risk remains. Duplicate effects also escape the score.
- Unequal attempt lengths. Both approaches had the same episode deadline; individual Grex attempts had shorter limits. Review reserves were breached. This was not a clean comparison under every intended resource constraint.
- Shared model-family limitations. The reviewer, evaluator and independent acceptance roles used Claude-family models. Independent roles do not remove shared blind spots.
- Public tasks. The sample may overlap model training data. This is not a leaderboard submission or a test against live customer accounts.
The recorded model routes used existing subscriptions, with no metered fallback observed. Subscription allocation, hardware and preparation effort were not fully costed, and no invoice reconciliation was performed. Zero observed incremental model charges does not mean free execution or demonstrated savings.
Progress record · As of November 3
What is complete, and what comes next
- Complete: execution integration. Isolated simulated worlds, per-attempt records and separate grading were exercised through Grex. This does not establish general-purpose access to the real services represented by the benchmark.
- Complete: first pilot. Sixty primary runs and 90 total attempts were evaluated. The unfavorable results and protocol deviations remain the baseline.
- Next: repair and prove the mechanics. Improve review visibility, preserve capacity for corrections, handle prior effects, enforce limits and record operator decisions. A mechanics-only comparison precedes instruction tuning.
- Planned: bounded tuning and unseen tasks. Compare untuned direct execution, direct execution with tuned instructions, and tuned Grex. Freeze the unseen test set before tuning and measure preparation effort. No tuning result is claimed here.
The next study must distinguish better instructions from benefits contributed by review and correction. Its readiness and independent acceptance gates still apply. A better benchmark score would not by itself establish customer demand or unattended business reliability.