What Grex's first business-task pilot revealed

Grex runs bounded AI tasks on computers its operator controls, retaining the work and its review. We tested whether adding a reviewer and a correction improved simulated business work. In the first pilot, both Grex and a direct runner completed 13 of 30 tasks correctly, on time and without intervention. Grex took about twice as long.

This is a diagnostic result. It identifies problems to fix before claiming that the review loop improves delivery.

The same tasks, two approaches

We used Zapier’s AutomationBench, which supplies simulated business tools and a grader that checks the resulting state. The fixed sample contained five tasks each in finance, operations, support, HR, marketing and sales. No real customer accounts or messages were involved.

Both approaches used MiniMax-M3 and the same tools. The direct runner had one attempt. Grex added a separate Claude Opus 5.5 reviewer and allowed one correction after rejection.

Each episode had a 30-minute deadline and 80 tool calls. Individual Grex attempts had seven-minute limits. A separate evaluator graded all 90 retained attempts after completion, including corrections. Engineering and study preparation were frontier-agent assisted.

The results

Measure	Direct	Grex
Correct, on time, unassisted	13/30	13/30
Strict benchmark pass	13/30	14/30
Total elapsed execution time	68.7 minutes	144.7 minutes
Tool calls	1,270	1,865
Duplicate simulated effects	2 in 2 episodes	34 in 11 episodes

On the timely-correct measure, Grex won two paired tasks, lost two and tied 26. One additional Grex world passed the grader, but its correction timed out. It therefore did not count as timely completion.

No correction rescued a strict failure. In eight episodes, the first attempt used every available tool call, leaving the correction none.

A correct world is not an accepted result

The reviewer rejected all 58 submissions it reviewed. Twenty-seven rejected worlds passed the benchmark grader. The evaluator’s follow-up audit classified these 27 rejections: 15 were operationally justified, 6 lacked decisive evidence, and 6 were unsupported or immaterial. The reviewer considered policies and effects that the grader sometimes misses.

The reviewer also lacked evidence. The follow-up audit found omitted calls in 33 review packets, including hidden writes in 25. Required operator decisions were not recorded before grading, so we cannot calculate how much work an operator accepted for use.

The most consequential finding was repeated action. The full audit found 34 duplicate simulated effects across 11 Grex episodes, compared with 2 across 2 direct episodes. Thirty-two of Grex’s duplicates came from corrections. The grader did not penalize them.

These revised counts replace the initial finding of 19 duplicate sends or creates in 10 Grex episodes. The fuller count includes 19 simulated messages and 15 record effects. It changes no upstream score. A useful correction must establish what already happened before repeating work.

Limits we are carrying forward

Thirty tasks with one worker model cannot establish broad superiority or equivalence. The tasks are public, and the reviewer and evaluator came from the same model family. This is not a leaderboard submission.

Twenty-four tasks were flagged as potentially affected by registered grader issues. The evaluator found no observed result flipped by a registered issue, but unresolved false-pass risk remains.

The experiment also breached its own resource contract. Thirty-three reviews exceeded their declared token reserve, and the driver continued despite the stop rule. Review visibility, missing operator decisions and unequal attempt lengths further limit the comparison.

The recorded routes used existing subscriptions, with no metered fallback observed. Subscription allocation, hardware and preparation effort were not fully costed, and no invoice reconciliation was performed. This result does not establish free execution or savings.

What happens next

The next stage separates repairs to execution mechanics from instruction tuning. It calls for complete review evidence, usable correction budgets, handling of prior effects, enforced limits and recorded operator decisions. A mechanics-only comparison comes first.

The planned tuning study compares three approaches: untuned direct execution, direct execution with tuned instructions, and tuned Grex. The unseen test set must be frozen before tuning. Preparation effort and remaining failures belong in the result.

There are no completed tuning results to report here. The first pilot’s failures remain part of the record. Follow the benchmark progress page for the method, current status and downloadable result summary.