Field report · 3 min read
Testing a cheaper model on real tasks
A budget model matched the prior retail score, but changing tools and inconsistent scoring denominators limit what the comparison proves.
A budget model completed sixteen of twenty simulated retail tasks inside Grex today. That matched the previous frontier-model run on the same benchmark sample.
Grex supplies the mission process around an AI worker: tools, limits, evidence, and checking. The question was whether useful performance would hold with a cheaper model doing the work.
The recorded results
| Benchmark sample | Result | Caveat |
|---|---|---|
| GAIA | 24 passes out of thirty | The prior report counted 21 passes among 27 scored, from thirty selected |
| tau-bench retail | Sixteen passes out of twenty | Same pass count as the previous frontier-model run |
| MCP-Atlas | Three passes among ten scored | Three further tasks were not attempted because of an admission defect |
| Terminal-Bench | Six passes, five failures | One score was still pending |
The MCP report also recorded a mean score of 0.60. That is a separate scoring measure, not a 60% pass rate; three passes among ten tasks is 30%.
The larger worker’s ledger recorded 524 model calls and 3.34 million tokens. About 83% of calls used the budget model. The frontier model appeared in the reviewer role for 82 calls, with no recorded silent fallback.
A promising comparison, with limits
On one GAIA question, the budget model found the correct penguin species in four steps and 47 seconds. The frontier model had taken nine steps and 55 seconds the day before.
That is an encouraging example. It does not isolate Grex’s contribution: the surrounding tools and execution support had changed too. A clean measurement of that contribution requires holding the model and other conditions constant.
More tools also exposed more failures
Workers gained a command shell, a browser, and access to tool servers through the Model Context Protocol, or MCP. Terminal tasks ran on a dedicated operating-system account, rather than in a container.
The shell had time and output limits but was not a security sandbox. The record showed a library installation and a filesystem-wide search. Tighter installation controls were specified.
Of 107 MCP calls, 27 failed. An admission check also overlooked servers provided by the mission itself, making three tasks unattemptable.
Budget exhaustion incorrectly disabled later work, delaying completion by about ninety minutes. A Terminal-Bench scorer kept revisiting one task without settling it, leaving its report incomplete.
Business work used the same mission process
Two business experiments also began: a competitive watch and a territory comparison across three cities and three trades. Both had 48-hour windows, evidence targets, and spending limits.
Earlier manually assembled research became their comparison baseline. The next question was whether Grex could produce useful business evidence under the same discipline as its benchmark work.