WEBVTT

00:00:00.300 --> 00:00:01.170
Before Grex 2,

00:00:01.170 --> 00:00:02.075
we tried a small

00:00:02.075 --> 00:00:04.058
standing suite across research,

00:00:04.058 --> 00:00:06.807
transactional, tool-use and terminal work.

00:00:07.107 --> 00:00:08.327
The question was whether

00:00:08.327 --> 00:00:09.605
one framework could carry

00:00:09.605 --> 00:00:10.883
these different tasks.

00:00:11.842 --> 00:00:13.550
The original objective was

00:00:13.550 --> 00:00:15.852
reusable execution and evidence.

00:00:16.152 --> 00:00:18.777
Our hypothesis wording is retrospective:

00:00:18.777 --> 00:00:20.503
one framework might support

00:00:20.503 --> 00:00:21.870
several task types.

00:00:22.170 --> 00:00:22.755
It was not a

00:00:22.755 --> 00:00:24.772
registered claim of superiority.

00:00:25.758 --> 00:00:26.576
The later suite

00:00:26.576 --> 00:00:28.022
planned 75 tasks

00:00:28.022 --> 00:00:29.342
across 4 families.

00:00:29.642 --> 00:00:31.690
Solving and scoring ran separately.

00:00:31.990 --> 00:00:34.027
Tool availability was recorded,

00:00:34.027 --> 00:00:35.456
and terminal tasks used

00:00:35.456 --> 00:00:37.814
an explicitly adapted environment.

00:00:38.800 --> 00:00:39.583
The later GAIA

00:00:39.583 --> 00:00:41.020
sample passed 24

00:00:41.020 --> 00:00:42.064
of 30 tasks.

00:00:42.364 --> 00:00:44.700
Terminal work recorded 6 passes,

00:00:44.700 --> 00:00:45.849
5 failures and

00:00:45.849 --> 00:00:47.228
one unscored task.

00:00:47.528 --> 00:00:48.846
3 tool-use tasks were

00:00:48.846 --> 00:00:50.728
unavailable rather than scored.

00:00:51.717 --> 00:00:53.807
These were small historical samples.

00:00:54.107 --> 00:00:55.692
Models, tools and harness

00:00:55.692 --> 00:00:57.734
behavior changed between runs.

00:00:58.034 --> 00:00:59.074
The results are not

00:00:59.074 --> 00:01:00.537
current platform scores,

00:01:00.537 --> 00:01:02.488
official leaderboard submissions

00:01:02.488 --> 00:01:03.398
or evidence that

00:01:03.398 --> 00:01:05.543
orchestration caused improvement.

00:01:06.508 --> 00:01:07.897
We retained the failures

00:01:07.897 --> 00:01:09.417
and narrowed later studies

00:01:09.417 --> 00:01:11.202
around specific mechanisms.

00:01:11.502 --> 00:01:14.748
Direct-execution comparisons and AutomationBench

00:01:14.748 --> 00:01:16.519
asked whether added process

00:01:16.519 --> 00:01:17.810
improved outcomes,

00:01:17.810 --> 00:01:19.138
rather than treating

00:01:19.138 --> 00:01:20.392
any benchmark score

00:01:20.392 --> 00:01:21.720
as platform proof.