Grex 2.0 · October 2026
Did a planner improve small code changes?
Three Grex 2 pairs measured no correctness gain from adding a planner—and found a bug both review paths missed.
Completed · diagnostic case uncertain
Worker-only and planner-plus-worker submissions passed the same selected cases on all three tasks. Planning added about nine seconds per pair. All six model reviews accepted, but a separate source-level probe found an untested fallback bug in both diagnostic submissions. That task’s broader correctness remained uncertain.
The experiment, explained
A narrated explanation of recorded results, with synthetic narration and visible English captions. Diagrams explain the method; they are not recordings of execution.
Read the video transcript
Did a planner help?
Does a planner improve a small code change? We tested three task pairs in Grex 2, changing planner presence while keeping the worker fixed.
The hypothesis
The pre-run plan fixed the task contract, worker and reviewer. Our directional hypothesis is retrospective: planning might improve correctness enough to justify its extra time. No numerical gain was promised.
Two paths, shared checks
Each task ran with a worker alone and with a planner followed by that same worker. The exact submissions received the same model review and separate held-out behavior checks.
Equal selected-case scores
Both arms passed the selected cases on all three tasks. Planning added about nine seconds per pair. Every model review accepted, but a later probe found a shared diagnostic fallback bug.
The missed bug matters
The evaluator had not tested inherited object keys. That left the broader diagnostic requirement uncertain despite passing checks. One pair per task and unknown costs also prevent a general conclusion.
Keep the decision narrow
The supported result is no measured planner benefit on these small tasks. We retained the uncertainty and setup failures. The experiment did not establish a universal rule for choosing agent teams.
What were we trying to find out?
Does adding a planning agent before the same coding worker improve small, well-specified code changes?
Hypothesis
A separate planner might improve correctness enough to justify its added execution time.
Retrospective directional interpretation. The pre-run pair plan explicitly changed planner presence while fixing the worker, reviewer and task contract; no numerical improvement prediction is claimed.
How the experiment worked
Create two missions for each isolated code fixture: a coding worker alone, and a planner followed by that same worker. Keep the reviewer and acceptance contract fixed. Reassess exact retained submissions with an independent deterministic evaluator.
Worker only
- Coding worker
- Independent model review
- Held-out behavior checks
Planner + worker
- Planner then same worker
- Same model review
- Same behavior checks
Same selected-case scores; planner added 8.6–9.2 seconds
Both paths missed a diagnostic fallback bug. Six acceptances are not six proven-correct submissions.
Download the method illustration
Sample: Three task pairs, six submissions: HTTP retry-delay parsing, per-key request coalescing and exact-output diagnostic refactoring. One pair per task.
Pin the code task
Use the same fixture base, allowed files, requirements and deadline in both arms.
Change planner presence
Run a worker alone or add a planner before that same worker.
Review the exact submission
Apply the same independent engineering reviewer to both outputs.
Check behavior separately
Run held-out cases against the retained revisions and record source-level follow-up findings.
Controls and differences
- The same worker and reviewer revisions, ordered criteria, paths and 15-minute deadline were pinned within each pair.
- The worker and planner used the same Codex model with different role settings; Claude supplied the independent engineering review. This was a role-structure comparison, not a model-vendor bake-off.
- Pair order varied: two tasks ran worker-only first and one ran the planner arm first.
- Evaluator cases and reference material were excluded from worker and planner inputs.
What happened
| Measure | Worker only | Planner + same worker |
|---|---|---|
| Retry-delay parsing | 10 / 10 cases | 10 / 10 cases |
| Request coalescing | 7 / 7 cases | 7 / 7 cases |
| Diagnostic refactor, selected cases | 5 / 5 cases | 5 / 5 cases |
| Retry-delay elapsed time | 40.288 seconds | 48.948 seconds |
| Request-coalescing elapsed time | 29.671 seconds | 38.835 seconds |
| Diagnostic elapsed time | 40.517 seconds | 49.136 seconds |
| Model review acceptance | 3 / 3 | 3 / 3 |
No measured benefit on the selected cases
Both arms passed ten retry-delay cases, seven request-coalescing cases and five selected diagnostic cases. The planner added 8.660, 9.164 and 8.619 seconds respectively.
Both gates missed an edge case
The diagnostic outputs inherited a lookup behavior that treated certain object-prototype keys as known severity levels. The broad requirement called for unknown levels to fall back. Neither the selected cases nor the model reviewer caught this.
Review acceptance did not settle correctness
All six model reviews accepted. The later diagnostic finding left operator assessments uncertain. Passing a bounded evaluator and receiving an approving review did not establish the whole requirement.
Setup failures were retained
The retry-delay baseline required a corrected setup retry. Final evaluator reassessment also encountered a missing executable path and a report-path collision. Earlier failed evidence stayed intact; no experimental code output was integrated.
What this does not establish
- One small pair per task cannot show that planning generally helps or does not help.
- Selected-case scores are not exhaustive correctness scores; diagnostic correctness remains unresolved in this record.
- Cost is unknown for every arm. Available token observations do not establish a complete cost comparison.
- The historical evaluator used an externally supplied compiler, differing from the workspace-local command declared in the manifests.
- This isolates planner presence within a fixed model arrangement, not the merits of different models or larger teams.
The decision
Record no measured correctness improvement from adding the planner on these tasks. Preserve the diagnostic uncertainty and improve coverage before treating review acceptance as sufficient.
What followed
Use a planner when a concrete task warrants it, then test that decision with task-appropriate evidence. This study did not establish a general team-selection rule.
Evidence and further reading
Public summaries retain the reported counts and limitations. Private execution traces and infrastructure details are not published.
- Download the sanitized evidence summary
Aggregate transcription of retained historical source records. Not a raw-data release or independently reproduced experiment.