Grex 2.0
What changed after the repairs?
A mechanics-only rerun reduced duplicate effects but completed fewer tasks correctly.
Completed · study closed
Grex completed 8 of 30 tasks strictly correctly, on time and without assistance, compared with 13 in the previous day's pilot. Duplicate simulated effects fell from 34 to 15 and token-bound breaches fell to zero. Correctness did not improve, and no correction rescued a failed task.
The experiment, explained
A narrated explanation of recorded results, with synthetic narration and visible English captions. Diagrams explain the method; they are not recordings of execution.
Read the video transcript
What changed after the repairs?
Our first AutomationBench pilot had a broken feedback path. The next question was what would change if we repaired the mechanics and repeated the same thirty tasks.
Measure before tuning
The plan recorded a mechanics-only measurement round before tuning. The reconstructed hypothesis was that better feedback and evidence might improve outcomes. It did not promise a numerical gain.
One rerun, no concurrent control
Grex received complete review evidence, a protected correction budget and duplicate guards. We kept the original instructions, then compared the result with the previous day's Grex run.
Fewer duplicates. Fewer correct tasks.
Timely, unassisted correctness fell from thirteen to eight tasks. Duplicate effects fell from thirty-four to fifteen. No correction rescued a failure, and five of nine reviewer acceptances failed the benchmark.
This is tuning data
These tasks were already known tuning data. There was no concurrent direct control. Changed limits, provider failures and model variation prevent a clean causal explanation of the drop.
The study was closed
Four later tuning missions were rejected for evidence grounding. The owner closed the study before the unseen-task test. The observed repairs remain useful evidence, without establishing a review-loop benefit.
What were we trying to find out?
What changes when we repair the pilot's runtime defects while keeping its worker instructions and review criteria?
Hypothesis
Delivering correction feedback, preserving review evidence and protecting correction resources might improve outcomes before any instruction tuning.
This directional statement is reconstructed from the prospectively recorded mechanics-only round in the November 3 protocol amendment. The original plan specified a measurement round, not a guaranteed improvement or numerical prediction.
How the experiment worked
Rerun the same 30 tasks through Grex after repairing the mechanics. Keep the original instructions and review criteria. Compare with the prior day's result; there is no concurrent direct-runner control.
Historical pilot
- Retain the original 30 Grex results
- Use as a descriptive baseline
Mechanics rerun
- Run the same 30 tuning tasks
- Deliver feedback to corrections
- Independently grade retained worlds
Compare correctness, duplicate effects, review decisions and enforced limits.
No fresh direct-runner control. This is a before-and-after diagnostic, not a causal estimate or an unseen-test result.
Download the method illustration
Sample: 30 previously scored tasks, now designated the tuning set. The run produced 54 sealed attempts, including 24 corrections, and 52 completed reviews. Every episode remains in the denominator.
Repair the feedback path
Send the reviewer observations and a record of earlier actions to corrections. Preserve complete call evidence for review.
Protect bounded work
Cap the first attempt at 60 calls within an 80-call episode ceiling, reserve at least 20 for correction, and apply duplicate guards.
Repeat the original tasks
Run Grex on the 30 tuning tasks with the original worker instructions and review criteria, without a tuned bundle.
Grade and audit
An independent evaluator checks all sealed attempts and classifies reviewer disagreements, duplicate effects and guard refusals.
Controls and differences
- The task set and original instructions and criteria were retained. The worker remained MiniMax-M3.
- Readiness received independent acceptance before this rerun. Some protections were proven synthetically rather than exercised in the live readiness tasks.
- All 64 grader assertion families passed their controls; all 54 attempt-integrity checks passed without evaluation errors.
- The 80-call total remained fixed, while review capacity increased to accommodate complete packets. These are changed mechanics, not an equal-compute comparison with the pilot.
What happened
| Measure | November 3 Grex | November 4 rerun |
|---|---|---|
| Strictly correct, timely and unassisted | 13 / 30 | 8 / 30 |
| Strict correctness alone | 14 / 30 | 9 / 30 |
| Corrections that rescued a failed task | 0 / 30 | 0 / 24 |
| Duplicate simulated effects | 34 in 11 episodes | 15 in 5 episodes |
| Reviewer acceptances | 0 / 58 reviews | 9 / 52 reviews |
| Acceptances on benchmark-failing worlds | 0; no acceptances | 5 / 9 acceptances |
| Reviewer token-bound breaches | 33 | 0 |
| Total episode elapsed time | 144.7 minutes | 150.2 minutes |
Fewer correct outcomes, with uncertain cause
Against the previous Grex run, the rerun won no tasks, lost five and tied 25 on timely correctness. Two losses hit the new first-attempt call cap, one was a provider timeout and two had similar call counts but different outputs. A single rerun cannot isolate mechanics from model variation.
Feedback arrived, but corrections still rescued nothing
None of the 24 corrections changed a failing world into a passing one. Four episodes received acceptance after correction; two had already passed the benchmark before correction, and two still failed it.
Acceptance became possible, not reliably correct
The reviewer issued nine acceptances across 52 reviews. Four were on passing worlds and five on failing worlds. Three of those five missed defects visible in the packet; two depended on hidden assertion detail. This is not a validated acceptance gate.
The guards helped and also blocked legitimate work
Duplicate effects fell to 15 in five episodes; twelve came from corrections. Of 11 guard refusals, ten prevented duplicates and one blocked a legitimate correction notice. Reworded messages and records still bypassed some guards.
The repaired boundaries held in this run
No reviewer token-bound breaches were recorded. Complete packets removed the earlier rejections caused by omitted evidence. These are operational improvements, separate from task correctness or a broad safety claim.
The study stopped before the unseen test
After this rerun, four MiniMax tuning missions were rejected for insufficient evidence grounding. The owner chose to close the study before the signed unseen-task test ran. Across the pilot and rerun, 54 correction attempts were made; none converted a strict failure into a pass. Some corrections began from worlds that already passed.
What this does not establish
- These tasks had already been scored and became tuning data. Results are optimistically biased for later tuning and cannot demonstrate performance on unseen work.
- There was no same-day direct arm. The comparison changes both mechanics and time, with one run per task. The five paired losses do not establish that the repairs caused the drop.
- The benchmark's known scoring defects remain. Passing its assertions is not equivalent to correct, appropriate business behavior.
- The worker used one model; review and evaluation used Claude-family models, which may share blind spots.
- Existing subscriptions supplied the execution, with reported zero incremental charges. That does not mean zero resource cost or an apportioned subscription cost.
- An independent evaluator authored the report and the lead recorded its outcome. A separate acceptance of the final report has not been established in this public summary.
The decision
The study was closed on November 4 after the mechanics rerun and four rejected tuning missions. No quality or safety benefit of the review-and-correct loop was established on this workload. Fewer observed duplicates remain a separate operational finding.
What followed
The planned unseen-task test was never run. Its signed manifest remains unused. A separate outcome-verification study was proposed; it is not a result of this experiment and is not reported as executed here.
Evidence and further reading
Public summaries retain the reported counts and limitations. Private execution traces and infrastructure details are not published.
- Download the mechanics-rerun evidence summary
Sanitized aggregates from the independent November 4 report. Preserves the cross-day comparison and tuning-set limitations.
- Pinned AutomationBench source
Unchanged upstream task assertions. This is not an official leaderboard result.