WEBVTT

00:00:00.300 --> 00:00:01.431
Does a planner improve

00:00:01.431 --> 00:00:02.561
a small code change?

00:00:02.861 --> 00:00:03.969
We tested 3 task

00:00:03.969 --> 00:00:04.849
pairs in Grex 2,

00:00:04.849 --> 00:00:06.673
changing planner presence while

00:00:06.673 --> 00:00:08.237
keeping the worker fixed.

00:00:09.217 --> 00:00:10.457
The pre-run plan fixed

00:00:10.457 --> 00:00:11.593
the task contract,

00:00:11.593 --> 00:00:12.971
worker and reviewer.

00:00:13.271 --> 00:00:16.225
Our directional hypothesis is retrospective:

00:00:16.225 --> 00:00:18.486
planning might improve correctness

00:00:18.486 --> 00:00:19.799
enough to justify its

00:00:19.799 --> 00:00:20.674
extra time.

00:00:20.974 --> 00:00:22.979
No numerical gain was promised.

00:00:23.967 --> 00:00:24.992
Each task ran with

00:00:24.992 --> 00:00:26.017
a worker alone and

00:00:26.017 --> 00:00:27.383
with a planner followed

00:00:27.383 --> 00:00:28.681
by that same worker.

00:00:28.981 --> 00:00:30.810
The exact submissions received

00:00:30.810 --> 00:00:32.029
the same model review

00:00:32.029 --> 00:00:33.790
and separate held-out behavior

00:00:33.790 --> 00:00:34.400
checks.

00:00:35.383 --> 00:00:36.622
Both arms passed the

00:00:36.622 --> 00:00:37.934
selected cases on all

00:00:37.934 --> 00:00:38.882
3 tasks.

00:00:39.182 --> 00:00:40.530
Planning added about 9

00:00:40.530 --> 00:00:41.571
seconds per pair.

00:00:41.871 --> 00:00:43.658
Every model review accepted,

00:00:43.658 --> 00:00:44.639
but a later probe

00:00:44.639 --> 00:00:46.181
found a shared diagnostic

00:00:46.181 --> 00:00:47.162
fallback bug.

00:00:48.133 --> 00:00:49.452
The evaluator had not

00:00:49.452 --> 00:00:51.504
tested inherited object keys.

00:00:51.804 --> 00:00:52.999
That left the broader

00:00:52.999 --> 00:00:55.456
diagnostic requirement uncertain despite

00:00:55.456 --> 00:00:56.519
passing checks.

00:00:56.819 --> 00:00:57.876
One pair per task

00:00:57.876 --> 00:00:59.312
and unknown costs also

00:00:59.312 --> 00:01:01.427
prevent a general conclusion.

00:01:02.383 --> 00:01:03.717
The supported result is

00:01:03.717 --> 00:01:05.317
no measured planner benefit

00:01:05.317 --> 00:01:06.650
on these small tasks.

00:01:06.950 --> 00:01:08.462
We retained the uncertainty

00:01:08.462 --> 00:01:09.659
and setup failures.

00:01:09.959 --> 00:01:11.304
The experiment did not

00:01:11.304 --> 00:01:12.932
establish a universal rule

00:01:12.932 --> 00:01:14.631
for choosing agent teams.