Volume
524 model calls
3.34M tokens across the full suite on the larger of the two nodes.
Benchmarks
Public agent benchmarks, run by the fleet being measured.
Grex runs GAIA, τ-bench, MCP-Atlas and Terminal-Bench as ordinary missions on its own nodes. The nodes load a pinned task sample, solve it with their own tools, and a separate scorer applies each benchmark's own checker. Every result on this page carries the model mix that produced it.
These are regression baselines for our releases. They are not leaderboard submissions, and the sample sizes are small and fixed. The caveats are written out below the numbers, not under them.
Suite v1 · 2026-08-31 · digest 399c8e674265
Seventy-five pinned tasks, budget model in the solver seat.
| Benchmark | Sample | v1 result — budget-first | Mean | v0 reference — frontier ceiling |
|---|---|---|---|---|
| GAIA validation, level 1 | 30 | 24 pass / 6 fail | 0.80 | 21 / 27 scored · mean 0.778 (3 not attempted) |
| τ-bench retail | 20 | 16 pass / 4 fail | 0.80 | 16 / 20 · mean 0.80 |
| MCP-Atlas | 13 | 3 / 10 scored (3 unattemptable — see below) | 0.60 | 0 / 3 |
| Terminal-Bench, bare metal | 12 | 6 pass / 5 fail · 1 score pending* | — | not in v0 |
* One Terminal-Bench task was still awaiting a score when this page was published. It is reported as pending rather than dropped from the denominator.
The v0 column was produced the day before with a frontier model doing the solving. It is kept here as a ceiling reference, not as a target: v1 matches it on τ-bench, beats it on GAIA, and scores the first MCP-Atlas passes we have recorded.
The three unattemptable MCP-Atlas tasks were refused by our own task-admission check, which read a static server registry and did not see servers supplied by the mission contract. That is a defect in our harness, not a benchmark result. It is filed and being fixed, and the tasks stay in the sample count so the loss is visible.
The model mix
We test Grex, not the model inside it.
A benchmark score is only interesting if you know what produced it. The v1 run was metered end to end, and the mix is stamped on every record the run published.
Solver
83% budget model
The strategist doing the actual work was the cheap model, for the large majority of calls.
Referee
82 frontier calls
The frontier model appeared only as a judge, never as the solver.
Fallback
Zero
No call silently escalated to the expensive model. The meter, not the model list, proves it.
Method
One mission per suite run. The node does the work.
- The mission is authored, not scripted. Each suite run is a single operator-authored mission with a fixed budget and a pinned task sample. No bespoke benchmark harness sits beside the platform.
- The node loads the pinned sample. The task set is pinned by digest, so a rerun is comparable to the run before it.
- It solves with its own tools. Shell, browser, MCP servers and the shared toolbelt — the same capabilities any mission gets, every call metered to a receipt.
- A separate scorer applies the benchmark's own checker. The scorer runs as its own actor with none of the solver's context, and uses each benchmark's published checker rather than a scoring rubric of ours.
- The pack publishes its report and concludes the mission. Per-task answers and scores, then an aggregate report, then a conclusion — through the ordinary mission path, with the model mix stamped on every record.
Read this before quoting a number
What these scores are not.
- The samples are small and fixed. Thirty GAIA tasks, twenty τ-bench tasks, a handful of MCP-Atlas and Terminal-Bench tasks. A few tasks either way moves the mean.
- Terminal-Bench here is a bare-metal adaptation of a benchmark designed to run in containers. Our nodes are whole operating systems, not Docker images. The result is comparable release to release; it is not comparable to the published leaderboard.
- MCP-Atlas is judged by our referee model, not by the official judge. Read it as our measurement, not as the benchmark's.
- These are regression baselines for our own releases. They exist so a score drop between versions is caught like a failing test. They are not leaderboard submissions and should not be compared with published leaderboard entries.
What the suite actually found
Every run surfaced real defects in our own platform.
The most useful output of the benchmark program so far has not been the scores. Running seventy-five unfamiliar tasks against the platform, under budget, on real hardware, flushed out failure modes that our own missions had never provoked.
Budget
Silent stalls after a budget stop
A run stopping on budget left its worker disabled, so every later slot refused to start without saying why. A budget stop is a checkpoint, not a verdict.
Lifecycle
Zombie leases on withdrawn missions
Work claims stamped at materialization time survived the withdrawal of the mission they belonged to, and bound later runs to a mission that no longer existed.
Cost
A 1.49M-token orphan
A spawned child kept answering questions for a withdrawn mission until it hit its own budget — twice the day's entire benchmark spend, spent on nothing.
Tooling
Tool paths that worked only in a shell
Package runners that resolved fine interactively were missing from the environment the actors actually ran in, so whole classes of tool task were quietly unreachable.
All four were fixed and re-proven live in the same week, and the fixed platform is what produced the v1 column above. The day-by-day record of both runs, including the failures, is in the log:
The first benchmark day, and the bugs it found The budget-first rerun
Standing measurement
This page changes when the suite is rerun.
The suite is pinned by digest so that each release can be measured against the same tasks. When a release moves a number — up or down — the number here moves with it, and the losses stay on the page.