Historical Grex 1 · August 2026
What the first benchmark runs revealed
Four public benchmarks exposed tool, scoring and runtime gaps in the earlier Grex architecture.
Historical · mixed completion
The archived runs produced task results and several actionable failures. GAIA and retail-task samples completed more fully than some tool and terminal tasks. Different model mixes, harness revisions and samples prevent a platform-lift claim. These are historical attempts from the architecture that preceded Grex 2.
The experiment, explained
A narrated explanation of recorded results, with synthetic narration and visible English captions. Diagrams explain the method; they are not recordings of execution.
Read the video transcript
What did the first runs reveal?
Before Grex 2, we tried a small standing suite across research, transactional, tool-use and terminal work. The question was whether one framework could carry these different tasks.
An engineering question
The original objective was reusable execution and evidence. Our hypothesis wording is retrospective: one framework might support several task types. It was not a registered claim of superiority.
Small subsets, recorded gaps
The later suite planned seventy-five tasks across four families. Solving and scoring ran separately. Tool availability was recorded, and terminal tasks used an explicitly adapted environment.
Results and missing results
The later GAIA sample passed twenty-four of thirty tasks. Terminal work recorded six passes, five failures and one unscored task. Three tool-use tasks were unavailable rather than scored.
What the comparison cannot show
These were small historical samples. Models, tools and harness behavior changed between runs. The results are not current platform scores, official leaderboard submissions or evidence that orchestration caused improvement.
A more useful next question
We retained the failures and narrowed later studies around specific mechanisms. Direct-execution comparisons and AutomationBench asked whether added process improved outcomes, rather than treating any benchmark score as platform proof.
What were we trying to find out?
Could the earlier Grex framework carry different kinds of benchmark work through the same execution and evidence process?
Hypothesis
A shared framework might support research, transactional, tool-use and terminal tasks without a separate execution system for each.
Retrospective interpretation of the original standing-suite objective. These runs were not a registered claim that Grex would beat a direct agent or a particular model.
How the experiment worked
Small pinned subsets were assigned to the earlier Grex benchmark runner. A solving process produced answers or work artifacts; a separate scoring process used the benchmark-specific checks. The later run changed both tool access and model allocation.
Research and retail
- Load task and inputs
- Solve with recorded model mix
- Apply task checks
Tool and terminal work
- Check environment availability
- Execute with available tools
- Record scores or missing results
Per-task results, failures and unavailable work
Historical architecture; changed harnesses and adapted environments prevent a platform score.
Download the method illustration
Sample: The August 31 suite planned 75 tasks: 30 GAIA, 20 retail tasks, 13 MCP-Atlas and 12 adapted terminal tasks. Earlier subsets differed, especially MCP-Atlas.
Select bounded subsets
Pin small public-task subsets that the available environment can attempt.
Run through the framework
Execute tasks on enrolled machines and retain the model mix and tool availability.
Score separately
Use benchmark-specific scoring and record failed, unavailable and unscored tasks.
Inspect the gaps
Separate task failures from missing tools, timeouts and scoring defects.
Controls and differences
- Task identities were retained for the 30 GAIA and 20 retail tasks across the two runs.
- Solving and scoring were separate processes, with benchmark-specific task environments where required.
- The terminal subset was adapted to the machine operating system rather than the official container environment; it was explicitly not leaderboard-comparable.
- The second run was budget-model-first but still used a frontier reviewer. It was not a single-model comparison.
What happened
| Measure | August 30 run | August 31 run |
|---|---|---|
| GAIA Level 1 | 21 / 27 scored; 3 not attempted | 24 / 30 |
| Retail transactional tasks | 16 / 20 | 16 / 20 |
| MCP-Atlas | 0 / 3; mean partial 0.4 | 3 / 10 scored; 3 unavailable |
| Adapted terminal tasks | Not run in this suite | 6 pass; 5 fail; 1 unscored |
The denominator changes the story
The first GAIA headline counted 27 scored tasks out of 30 selected. Three were not attempted because of a safeguards refusal or timeouts. Reporting only the scored denominator would hide those outcomes.
Tool availability was part of the failure
The tool-use runs included missing server support and later admission failures. The August 31 record lists three unavailable tasks separately from the ten scored tasks.
Terminal work did not fully settle
Six adapted terminal tasks passed and five failed. One remained unscored because of a scorer-settlement defect. The attempted environment differed from the official benchmark setup.
The later score does not isolate the model
Tools, harness behavior and model allocation changed together. The recorded gains cannot show that one model was better or that orchestration caused the difference.
What this does not establish
- Historical Grex architecture, not a score for the current Grex 2 platform.
- Small subsets and changed conditions prevent causal comparisons or official leaderboard claims.
- Missing and unscored tasks remain visible; the table does not pool incompatible denominators.
- These figures are transcribed from the archived program record. This publication does not reproduce the raw benchmark runs.
- No complete, reconciled all-in cost comparison is established here.
The decision
Keep these attempts as engineering evidence about task coverage and failure handling. Do not present them as general autonomy scores or proof that the platform improves its models.
What followed
Later experiments asked narrower questions with explicit direct-runner comparisons. The direct-execution study and AutomationBench records show how the evaluation changed.
Evidence and further reading
Public summaries retain the reported counts and limitations. Private execution traces and infrastructure details are not published.
- Download the sanitized evidence summary
Aggregate transcription of retained historical source records. Not a raw-data release or independently reproduced experiment.