Grex 2.0
Could better instructions beat retries?
One frozen Harness² candidate failed independent review where the unchanged instructions and ordinary retries passed.
Completed · adoption on hold
The unchanged instructions produced an accepted brief. Ordinary retries produced an accepted brief on attempt two. The frozen refined candidate passed mechanical checks but failed independent review with two unsupported claims. This was one source-window group, not a success-rate study.
The experiment, explained
A narrated explanation of recorded results, with synthetic narration and visible English captions. Diagrams explain the method; they are not recordings of execution.
Read the video transcript
Could better instructions beat retries?
Can automatically improved instructions produce a better research brief than trying again? We tested a Harness squared candidate around Grex's research loop in a bounded external experiment.
The original question
That comparison question was recorded before execution. The reconstructed hypothesis was that refined instructions might improve the brief. The final candidate was frozen before held-out evaluation.
One held-out source group
We used one held-out group of three official release pages. The conditions shared source material, models and criteria. An independent evaluator judged the outputs without knowing their condition.
The refined candidate was rejected
Unchanged instructions passed. Fresh retries passed on the second attempt. The refined candidate passed mechanical checks but failed independent review with two unsupported claims. Every attempt remained recorded.
A narrow result
This is not a success-rate study. Development adapter defects gave the refinement process noisy feedback. Shared ceilings also did not mean equal actual spend or complete cost accounting.
Adoption stayed on hold
We held production adoption and stopped expansion from this pilot. The result rejects this candidate on this group, not instruction refinement in general. A broader transfer experiment was not run.
What were we trying to find out?
Can automatic instruction refinement improve a research brief beyond ordinary retries under shared resource ceilings?
Hypothesis
A Harness²-refined instruction set might produce a better supported brief than unchanged instructions or fresh retries.
The comparison question was recorded before model execution. This directional wording is a reconstruction, not a quotation or registered numerical prediction. The final held-out design froze a development-generated candidate before evaluation; it did not refine instructions on the held-out material.
How the experiment worked
Adopt the upstream Harness² proposal, probe and composition process in an isolated controller around Grex's research loop. Compare a frozen candidate with unchanged instructions and fresh retries using the same retained source material and independent judging criteria.
Unchanged instructions
- One fresh attempt
- Accepted
Ordinary retries
- Fresh attempts; no feedback
- Accepted on attempt two
Frozen candidate
- One fresh attempt
- Rejected: unsupported claims
Judge all four outputs against the same criteria.
One held-out group and faulty development feedback prevent a general claim about refinement.
Download the method illustration
Sample: One held-out source-window group containing three official release pages. Four evaluation attempts total: one unchanged, two retries and one refined. The three pages are not three independent tasks.
Freeze the source material
Retain exact source text, task criteria and a fixed clock. Keep development outputs separate from held-out inputs.
Refine on development data
Run the actual upstream process with a restricted instruction-editing surface. Retain failed integration attempts and their costs.
Freeze one candidate
Freeze the composed instructions before any held-out output exists. No refinement or evaluator feedback reaches the held-out run.
Compare three conditions
Run unchanged instructions, fresh retries that stop at first acceptance, and the frozen candidate under declared resource ceilings.
Judge the exact outputs
A blinded independent evaluator checks each brief against the same sources and criteria after Grex's mechanical checks.
Controls and differences
- The final comparison used the same MiniMax-M3 worker, Haiku synthesis, source bytes, clock, criteria, permissions and empty initial memory.
- The editable candidate surface was limited to worker instructions and tool guidance. Models, tools, permissions and acceptance rules were not tuned.
- Retries received no prior output or reviewer feedback and stopped at the first accepted result. The first failed retry was retained.
- The candidate was frozen before evaluation. The final direct-API comparison used the same transport across conditions and retained all usage.
- An independent result review accepted the integrity and accounting of the comparison and its hold decision; it did not accept the candidate's output quality.
What happened
| Measure | Unchanged | Ordinary retries | Refined candidate |
|---|---|---|---|
| Selected result | Accepted | Accepted | Rejected |
| Held-out attempts | 1 | 2 | 1 |
| Mechanical checks | Passed | Both passed | Passed |
| Unsupported claims in selected output | 0 | 0 | 2 |
| Observed elapsed time | 227 seconds | 590 seconds | 214 seconds |
| Held-out Anthropic API estimate | $0.160 | $0.360 | $0.181 |
Mechanical checks were insufficient
All four evaluation attempts passed mechanical checks. Independent review rejected the first retry and the refined candidate for unsupported claims. Traceable quotes alone did not guarantee faithful synthesis.
The candidate did not transfer successfully
The refined brief broadened a source's model-update claim beyond the products it covered. It also called an item unverifiable even though the retained source contained it. The unchanged brief and second fresh retry passed.
Development feedback was noisy
Adapter defects affected proposal parsing, delivery checks and finalization during development. The composer saw false failure signals. Later checks established that the exact frozen candidate ran as recorded, but they could not turn this into a clean estimate of Harness²'s usual effectiveness.
Costs include unsuccessful work
The new Anthropic API list-price estimate was $1.083277 including development failures and evaluation, under an approved $10 ceiling. It is not an observed invoice or credit debit. MiniMax subscription usage and engineering effort are separate; no complete all-in cost was established.
What this does not establish
- One convenience-selected source group cannot establish general effectiveness, a success rate or superiority to retries. The baseline already passed, leaving limited room for improvement.
- The source pages were fetched in October and carried August release metadata. They were not proven historical August snapshots. This tested synthesis of retained pages, not live discovery.
- Faulty development feedback materially limits attribution to Harness² itself. The supported conclusion concerns this candidate and this recorded process.
- Conditions shared declared resource ceilings, not equal actual compute or spend. Earlier native-CLI attempts included unmeasured usage, retained as unknown rather than zero.
- This ran in an external experimental controller using Grex's pinned research implementation. It was not a production-schedule trial, and no refinement subsystem was adopted into Grex.
- Independent model judging is fallible. No JobBench transfer experiment or customer-value result was established.
The decision
Keep production adoption on hold and stop expansion from this pilot. The frozen candidate failed the quality gate while the unchanged instructions and ordinary retries passed.
What followed
Reconsider only for a demonstrated recurring failure and a clean study across multiple untouched source windows. The next design would need to distinguish refinement from retries and a direct instruction fix. No follow-up is claimed as launched.
Evidence and further reading
Public summaries retain the reported counts and limitations. Private execution traces and infrastructure details are not published.
- Download the refinement-pilot evidence summary
Sanitized aggregate transcription of the completed report and independent result acceptance. Source bodies, private traces and operational identifiers are excluded.
- Pinned Harness² source
The upstream instruction-refinement implementation adopted for the experiment.