WEBVTT

00:00:00.300 --> 00:00:01.846
Our first AutomationBench

00:00:01.846 --> 00:00:02.451
pilot had a

00:00:02.451 --> 00:00:03.863
broken feedback path.

00:00:04.163 --> 00:00:05.362
The next question was

00:00:05.362 --> 00:00:06.495
what would change if

00:00:06.495 --> 00:00:07.961
we repaired the mechanics

00:00:07.961 --> 00:00:09.160
and repeated the same

00:00:09.160 --> 00:00:10.093
30 tasks.

00:00:11.050 --> 00:00:11.994
The plan recorded

00:00:11.994 --> 00:00:13.567
a mechanics-only measurement

00:00:13.567 --> 00:00:14.826
round before tuning.

00:00:15.126 --> 00:00:17.089
The reconstructed hypothesis was

00:00:17.089 --> 00:00:18.510
that better feedback and

00:00:18.510 --> 00:00:20.609
evidence might improve outcomes.

00:00:20.909 --> 00:00:21.909
It did not promise

00:00:21.909 --> 00:00:23.042
a numerical gain.

00:00:24.008 --> 00:00:26.395
Grex received complete review evidence,

00:00:26.395 --> 00:00:28.143
a protected correction budget

00:00:28.143 --> 00:00:29.555
and duplicate guards.

00:00:29.855 --> 00:00:31.787
We kept the original instructions,

00:00:31.787 --> 00:00:33.117
then compared the result

00:00:33.117 --> 00:00:34.384
with the previous day's

00:00:34.384 --> 00:00:35.018
Grex run.

00:00:36.008 --> 00:00:37.827
Timely, unassisted correctness

00:00:37.827 --> 00:00:38.848
fell from 13

00:00:38.848 --> 00:00:39.806
to 8 tasks.

00:00:40.106 --> 00:00:41.608
Duplicate effects fell from

00:00:41.608 --> 00:00:42.986
34 to 15.

00:00:43.286 --> 00:00:45.210
No correction rescued a failure,

00:00:45.210 --> 00:00:45.818
and 5 of

00:00:45.818 --> 00:00:47.371
9 reviewer acceptances

00:00:47.371 --> 00:00:48.790
failed the benchmark.

00:00:49.758 --> 00:00:51.033
These tasks were already

00:00:51.033 --> 00:00:52.126
known tuning data.

00:00:52.426 --> 00:00:52.983
There was no

00:00:52.983 --> 00:00:54.432
concurrent direct control.

00:00:54.732 --> 00:00:55.803
Changed limits,

00:00:55.803 --> 00:00:57.577
provider failures and model

00:00:57.577 --> 00:00:59.202
variation prevent a clean

00:00:59.202 --> 00:01:00.828
causal explanation of the

00:01:00.828 --> 00:01:01.345
drop.

00:01:02.300 --> 00:01:03.232
4 later tuning

00:01:03.232 --> 00:01:04.476
missions were rejected

00:01:04.476 --> 00:01:05.905
for evidence grounding.

00:01:06.205 --> 00:01:07.145
The owner closed

00:01:07.145 --> 00:01:08.084
the study before

00:01:08.084 --> 00:01:09.427
the unseen-task test.

00:01:09.727 --> 00:01:10.925
The observed repairs

00:01:10.925 --> 00:01:12.355
remain useful evidence,

00:01:12.355 --> 00:01:15.017
without establishing a review-loop benefit.