WEBVTT

00:00:00.300 --> 00:00:01.213
For a task that

00:00:01.213 --> 00:00:03.154
needs one solver invocation,

00:00:03.154 --> 00:00:04.828
do extra execution layers

00:00:04.828 --> 00:00:06.273
improve the result?

00:00:06.573 --> 00:00:07.718
We compared a direct

00:00:07.718 --> 00:00:08.863
path with the longer

00:00:08.863 --> 00:00:10.008
chain in the earlier

00:00:10.008 --> 00:00:11.288
Grex architecture.

00:00:12.258 --> 00:00:13.883
The recorded question was

00:00:13.883 --> 00:00:15.802
whether direct execution with

00:00:15.802 --> 00:00:17.722
frozen inputs and independent

00:00:17.722 --> 00:00:19.272
verification could do as

00:00:19.272 --> 00:00:19.789
well.

00:00:20.089 --> 00:00:20.576
This is a

00:00:20.576 --> 00:00:22.768
retrospective hypothesis summary,

00:00:22.768 --> 00:00:25.273
not a registered numerical prediction.

00:00:26.258 --> 00:00:27.267
5 pairs covered

00:00:27.267 --> 00:00:28.150
3 task types

00:00:28.150 --> 00:00:29.096
and 2 models.

00:00:29.396 --> 00:00:30.951
Each pair shared its task,

00:00:30.951 --> 00:00:32.126
model and prompt.

00:00:32.426 --> 00:00:34.054
Local rubric judging used

00:00:34.054 --> 00:00:35.608
3 judgments per item

00:00:35.608 --> 00:00:37.162
and the median result.

00:00:38.133 --> 00:00:39.808
Both paths passed 17

00:00:39.808 --> 00:00:41.483
of 41 judged items.

00:00:41.783 --> 00:00:43.187
These are rubric items,

00:00:43.187 --> 00:00:45.175
not 17 completed tasks.

00:00:45.475 --> 00:00:46.365
In the holdout,

00:00:46.365 --> 00:00:47.487
both produced files

00:00:47.487 --> 00:00:48.213
but timed out

00:00:48.213 --> 00:00:49.400
and scored poorly.

00:00:50.383 --> 00:00:52.080
Repeated development tasks,

00:00:52.080 --> 00:00:54.742
environment differences and provenance-label

00:00:54.742 --> 00:00:56.805
failures limited the comparison.

00:00:57.105 --> 00:00:58.358
The judging was local,

00:00:58.358 --> 00:00:59.579
with known defects.

00:00:59.879 --> 00:01:00.716
This was not an

00:01:00.716 --> 00:01:02.388
official JobBench result or

00:01:02.388 --> 00:01:04.061
a current-platform score.

00:01:05.050 --> 00:01:06.310
The decision was to

00:01:06.310 --> 00:01:08.043
simplify execution of one

00:01:08.043 --> 00:01:09.146
bounded task.

00:01:09.446 --> 00:01:11.115
Scheduling, persistence,

00:01:11.115 --> 00:01:12.888
authorization, recovery and

00:01:12.888 --> 00:01:14.348
cooperating agents were

00:01:14.348 --> 00:01:16.017
outside this comparison.

00:01:16.317 --> 00:01:17.063
The result did

00:01:17.063 --> 00:01:17.809
not show those

00:01:17.809 --> 00:01:19.922
responsibilities were unnecessary.