We asked two frontier LLM agents, Claude Fable 5.2 and GPT-6 Astra, to make a robot do ten LIBERO manipulation tasks in three ways: by writing ordinary Python code, by writing the weights of a small neural network (an MLP) by hand in NumPy, and by driving the robot live, choosing every action from the camera image. When writing policies, neither could run a simulator or simulate the robot in any other way, train, or see a single demonstration. We scored every episode two ways: with LIBERO's standard success check, and with a stricter one that also requires the robot to let go.
With LIBERO's standard check, Fable's hand-built network solved 414 of 500 held-out episodes (82.8%) on its first try, and 92.8% after one revision. For context, fine-tuned π0.5's published score on the same LIBERO suites is about 97%. Fable's code solved 88.8% and then 89.2%, once we exclude one change made with a search our rules forbid. Under the strict check, the network scores 74.8% and 84.2%, and the code 87.8% and 88.2%. Astra wrote its first versions about 6 times faster, for a quarter to a half of the cost, but its written policies solved 10% to 38%, and, unlike Fable, it wrote a better network than program on its first try.
Driving live, scored with LIBERO's standard check, was Fable's weakest approach: it solved 50% of 50 held-out episodes, at $5.44 per success against 7 to 10 cents for its written policies. Astra's live control solved 32%, level with its written policies, but still cost about 30 times more per success. This is a first glance, with one authoring run per cell.
Figure 1. The four revised policies on one held-out scene
playground/LLMs for robot control
Held-out scene 0 of “put both moka pots on the stove”, run by the four revised (v2) policies, shown at 2× speed. Fable's code (top left), Fable's network (top right) and Astra's code (bottom left) set both pots on the stove, let go and back away. Each clip runs until the robot has let go for one second. Astra's network (bottom right) stacks the second pot on top of the first and is still holding it when time runs out at step 520. Across the 50 held-out scenes these four policies scored 50, 48, 31 and 0. Each clip replays the scored episode exactly.
Three competing paradigms
We consider three primary ways for an LLM to control a robot. First, it can write a program that runs the robot (code as policy). Second, it can write the weights of a neural network to complete a task directly (weight-space programming). Third, it can skip the policy creation component, and drive the robot itself, looking at each camera frame and choosing the next few actions.
The first two produce a frozen artifact that runs on its own at 20 Hz with no LLM in the loop, so the LLM's time and cost are paid once. The third puts the LLM in the loop for every action, so they are paid on every episode. In our first post, agents edited and built networks for small tasks and for π0.5. Here we use the same tasks, source materials and scoring for every approach, and ask two questions. How do the three approaches compare on success, time and cost? And how do Fable and Astra compare within each?
Figure 2. Three ways to control a robot with an LLM
CODE AS POLICY
Task + LIBERO sourceread as text, no simulator
→
The LLM writes a programPython, may keep memory between steps
→
A frozen policy.pyruns on its own at 20 Hz
WEIGHT-SPACE PROGRAMMING
Task + LIBERO sourceread as text, no simulator
→
The LLM computes weightsin NumPy, by reasoning, no training
→
A frozen MLPruns on its own at 20 Hz, no memory
LLM AGENT CONTROL
Camera image + arm statereturned after every move
→
The LLM picks the next movesup to 20 actions, 1 s of robot time
→
Repeat until doneevery action comes from the LLM, live
playground/LLMs for robot control
In code as policy and weight-space programming, the LLM writes a policy once, and the frozen policy then drives the robot on scenes the LLM never saw. In LLM agent control, the LLM is the policy and chooses every action live.
The setup
We drew ten tasks from LIBERO's four suites with a fixed random seed: three spatial tasks, three object tasks, two goal tasks and two long-horizon tasks (Figure 3). Each model worked alone, once per format, in its own clean room: a folder with the LIBERO and robosuite source as text, the scene and object assets, Python with NumPy, and a validator. It could not run the simulator, or simulate the robot any other way. Everything it knew about the scenes it had to work out by reading code, such as where LIBERO places each object, how the arm's controller responds to a command, and where the camera looks.
Figure 3. The ten tasks
spatialBowl in top drawer → plate
spatialBowl next to plate → plate
spatialBowl on ramekin → plate
objectAlphabet soup → basket
objectMilk → basket
objectTomato sauce → basket
goalWine bottle → rack
goalTurn on the stove
longBoth moka pots → stove
longSoup and sauce → basket
playground/LLMs for robot control
The ten tasks, at the first frame of held-out scene 0. Object positions change from scene to scene. The long-horizon tasks have to move two objects within one episode.
The author gets
The LIBERO and robosuite source code, read as text
The task files and the scene and object assets
The observation and action spec, and a validator for its submission
Python with NumPy
Not allowed
Running or rendering the simulator
Any simulation of its own, such as stepping its own equations of the arm or feeding a policy's actions back into a model of the scene
LIBERO's init-state files, demonstrations or any recorded rollout before v2
Training, fitting or any search over parameters
Pretrained models and the internet
Tasks
10 LIBERO tasks drawn with a fixed seed: 3 spatial, 3 object, 2 goal, 2 long-horizon
Authors
Claude Fable 5.2 in Claude Code and GPT-6 Astra in the Codex CLI, both at high reasoning effort, with identical prompts
Two formats
A Python policy that may keep memory between steps, or one frozen feed-forward network (affine layers with ReLU activations)
Scoring
LIBERO's 50 standard held-out scenes per task, 500 episodes per policy, hidden from both authors. An episode succeeds when the goal holds with the gripper touching nothing but the robot for one second; LIBERO's usual check is reported separately
Both formats share one interface. A policy receives the 256 × 256 front camera image, the arm's state now and one step earlier, and the instruction. It returns a 7-D action: a small move and turn of the gripper, and open or close.
Each author made two versions. Version 1 is zero-shot: designed, checked on made-up inputs and frozen without any feedback. We then ran 20 development rollouts of v1 (2 per task, on fresh random scenes), handed back the recordings, and allowed one revision, v2. Constants could come from the source code but not from the recordings. Every script an author ran was saved, and we read all of them, and the session logs, before accepting a policy. We scored both versions on LIBERO's 50 standard held-out scenes per task.
For the third approach, each model drove the robot itself. It had one tool, act, which sends up to 20 actions (one second of robot time) and returns the next camera image and arm state. Every action came from the model, live, on held-out scenes 0 to 4 of each task: 50 episodes per model, on the same scenes the written policies also faced. We had planned 100 episodes per model and stopped at 50 for budget reasons, after we had seen interim success counts.
The agent gets
One tool, act: send 0 to 20 actions, get back the 256 × 256 camera image and the arm's state
The same observation and action as the policies above
The instruction, the action format and its limits
Not allowed
Any other tool, files or shell
The internet
Any training. Every action comes from the model, live
Agents
Claude Fable 5.2 in Claude Code, headless, and GPT-6 Astra in the Codex CLI, both at high reasoning effort, with identical prompts
Limits
The task's usual step limit, at most 40 act calls with actions, and 60 minutes of wall time per episode
Scoring
Held-out scenes 0 to 4 of each task, 50 episodes per model, paired scene by scene with the code and network policies, with LIBERO's usual check. We planned 100 episodes per model and stopped at 50 for budget reasons, after seeing interim success counts
These are our second-round results. In the first round, the rules let authors build their own renderers and simplified simulators and test policies in them. An adversarial review flagged this, along with a way to pass LIBERO's success check without finishing the task (see Strict scoring). We rebuilt the clean rooms with the rules above and had both models write all eight policies again from scratch. The live-control arm did not use a simulator of its own, so we kept its episodes. Astra's policies changed most between rounds: its revised code fell from 56.6% in the first round to 33.2% here. Round two removed the homemade simulators but also changed the brief, so the drop has more than one possible cause. The first-round policies and numbers are in the downloads.
Results
For Fable, writing a policy clearly beat driving the robot live (Figure 4). Its code solved 439 of 500 held-out episodes (87.8%) on its first try and 441 (88.2%) after one revision, with one disputed change excluded (see One round of feedback). Its hand-built network reached 74.8% and then 84.2%. Driving live, it solved 25 of 50 episodes (50%) under LIBERO's usual check.
Figure 4. Fable: held-out success by approach
playground/LLMs for robot control
Fable's held-out success for each approach. Written policies use the strict check on 500 episodes (50 scenes for each of 10 tasks), with whiskers showing Wilson 95% confidence intervals for the revised versions. *Live control uses LIBERO's usual check on 50 episodes (scenes 0 to 4 of each task), because it ran before the strict check and has no strict score of its own. The revised code leaves the soup-and-sauce task at its first version; as submitted it scored 94.2%.
For Astra, the three approaches were close (Figure 5). Its code solved 10.0% on the first try and 33.2% after one revision, its network 35.2% and then 38.0%, and driving live it solved 16 of 50 episodes (32%). Live control's numbers rest on 50 episodes rather than 500 and use LIBERO's more lenient check, so they are less certain: the 95% confidence interval runs from 37% to 63% for Fable and from 21% to 46% for Astra, against ±2 to ±4 points for the written policies.
Figure 5. Astra: held-out success by approach
playground/LLMs for robot control
Astra's held-out success for each approach, scored the same way as Fable's in Figure 4. *Live control uses LIBERO's usual check.
The per-task view shows where the points went (Figure 6). Fable's code solved five of ten tasks in at least 49 of 50 scenes on its first try; its weakest task was soup and sauce into the basket (15). Its network matched the code on most tasks, but scored 1 on the wine rack in both versions and 0 on the stove in v1. Astra's code solved only the stove on its first try, and its network beat it on four tasks.
Figure 6. Strict successes out of 50 held-out scenes, per task
Task
Fable · code
Fable · network
Astra · code
Astra · network
v1
v2
v1
v2
v1
v2
v1
v2
spatialBowl in top drawer → plate
41
43
49
49
0
0
27
0
spatialBowl next to plate → plate
43
43
42
44
0
41
40
40
spatialBowl on ramekin → plate
50
50
41
48
0
44
0
0
objectAlphabet soup → basket
50
50
50
50
0
0
9
50
objectMilk → basket
49
49
50
50
0
0
0
0
objectTomato sauce → basket
50
50
50
50
0
0
50
50
goalWine bottle → rack
47
47
1
1
0
0
0
0
goalTurn on the stove
44
44
0
35
50
50
50
50
longBoth moka pots → stove
50
50
49
48
0
31
0
0
longSoup and sauce → basket
15
15
42
46
0
0
0
0
Total, out of 500
439
441
374
421
50
166
176
190
playground/LLMs for robot control
Strict successes out of 50 held-out scenes, for each task and written policy. Darker cells are higher. Every policy saw the same 50 scenes per task, so the columns can be compared scene by scene. Fable's revised code shows soup and sauce at its first version (15); as submitted it scored 45 there.
Code against weights
For Fable, code beat weights by 13 points on the first try (p < 0.001). After one revision the gap was 4 points, 88.2% against 84.2%, which is not statistically resolved (p = 0.06). Most of it is two tasks: on the wine rack the network carries the bottle into the rack and lets go before it is seated (Figure 14), and its first version never turned the stove on. The code keeps a list of steps and checks that each is done before starting the next. The network has no memory beyond the arm's previous state, so every decision, including when to let go, must come from what it sees at that moment.
For Astra the result flips. Its network beat its code on the first try, 35.2% against 10.0%, and the two were close after one revision, 38.0% against 33.2% (p = 0.09). Both networks are sparse. Fable's first network has 14 layers and about 100,000 nonzero parameters (84,000 weights and 15,000 biases) in weight matrices with 3.8 billion entries, small enough to run on a laptop CPU in about 0.3 seconds per episode. Astra's is larger: 24 layers and about 230,000 nonzero parameters (201,000 weights).
Driving live
Live control was Fable's weakest approach. Using LIBERO's check for both, on the 50 scenes both approaches faced, Fable's revised code solved 22 that live control failed, and live control solved 2 that the code failed (p < 0.001). Its revised network won 20 to 2 (p < 0.001). Astra's live control was level with its revised policies: 11 to 11 against its code and 8 to 11 against its network (p = 1.0 and 0.65).
We suspect part of the gap is information. The policy authors could read the LIBERO source: where each object is placed, how high the table is, where the camera looks and how the controller responds. The live agents had none of that. They saw only the camera image and the arm's state, and had to infer the scene's layout and depth from a single front view.
The harness may also play a part. Claude Code and Codex are built for software work, not for controlling a robot step by step, so some of the live-driving shortfall may come from the agent harness rather than the models themselves. Written policies avoid this: once frozen, they run without the harness.
Driving live does give the model eyes and memory, and Fable drove all five moka-pot scenes to success (Figure 7). Most live failures were running out of time. The agents misjudged depth from the single front camera, knocked objects over, and spent the rest of the episode recovering. All 34 of Astra's failures and 19 of Fable's 25 hit the step or call limit; in the other six, Fable stopped itself with too few steps left to finish.
Figure 7. Fable and Astra driving the robot live
playground/LLMs for robot control
Held-out scene 2 of “put both moka pots on the stove”, driven live by Fable (left) and Astra (right), shown at 2× speed. The counter shows the simulator step and the act call that sent it. Fable grasps each pot by its handle and sets both on the stove; LIBERO's goal check passes at step 202, on its 17th act call, after 4.3 minutes of wall time. Astra knocks the first pot over within 40 steps and the second soon after, and runs out of time at step 520. Each clip replays the scored episode's recorded actions exactly.
One round of feedback
One look at 20 development rollouts clearly improved two of the four written policies: +9.4 points for Fable's network and +23.2 for Astra's code. Fable changed only two tasks in its code. For one of them, soup and sauce, it chose between candidate soup detectors by their accuracy on synthetic images it drew itself. That is the kind of search our rules forbid, so we report Fable's revised code with that task left at its first version: 88.2%, 0.4 points above its first version. As submitted, with the new detector, it scored 94.2%. Neither that 0.4-point gain nor Astra's network's 2.8 points is statistically resolved (p = 0.75 and 0.11). Most of Astra's code gain came from one fix: its phases moved on after a fixed number of steps, whether or not the hand had arrived (Figure 8).
Figure 8. Astra's code before and after feedback
playground/LLMs for robot control
Held-out scene 0 of “pick up the black bowl next to the plate and place it on the plate”, run by Astra's code before (left) and after (right) its revision, at real speed. In v1, each phase moves on after a fixed number of steps whether or not the hand has arrived, the hand never gets hold of the bowl, and time runs out at step 220. In v2, each phase waits for the hand to arrive, and the bowl is on the plate at step 161. The clip runs on until the robot has let go for one second.
Cost and time
A written policy is paid for once. Fable spent about $45 and two and a half hours on its code, both versions included, and $33 and 1.7 hours on its network. Astra spent $11 to $18 and 35 to 40 minutes on each (at API prices). After that the frozen policy runs at the robot's own speed, with no LLM and under a second of CPU per episode. Live control pays on every episode: $2.72 and a median 8.4 minutes for Fable, $0.83 and 4.1 minutes for Astra, 18 to 40 times slower than the robot moves.
The break-even comes quickly (Figures 9 and 10). At 10 episodes, driving live is still the cheaper option for both models. By 17 to 19 episodes it costs more than writing the code, and at 100 episodes Fable's live control costs $272 and 14 hours of LLM time, against $45 and 2.6 hours for its code. Astra's costs less, $83 and 6.8 hours for 100 episodes, but still several times its written policies.
Figure 9. Fable: LLM cost and time to run 10 or 100 episodes
playground/LLMs for robot control
Fable's LLM cost, at API prices, and LLM time to run 10 or 100 episodes with each approach. A written policy's cost and time are paid once, both versions included, so its bars do not grow; live control pays per episode.
Figure 10. Astra: LLM cost and time to run 10 or 100 episodes
playground/LLMs for robot control
Astra's LLM cost and time, as in Figure 9. Astra's authoring cost is the middle of a $11 to $18 range per format.
Per successful episode the gap is wider. Over LIBERO's 500 held-out episodes, every written policy costs 7 to 10 cents per success with LIBERO's standard check (8 to 10 with the strict one), Fable's and Astra's alike. Driving live costs $5.44 per success with Fable and $2.59 with Astra, about 30 to 75 times more. Since the written policies cost about the same per success, choosing between them comes down to how often they fail: Fable's revised code fails about 1 episode in 8, Astra's best policy about 3 in 5.
Fable against Astra
Fable was far more accurate, and Astra faster and cheaper (Figure 11). Fable's written policies beat Astra's by 40 to 78 points (all p < 0.001). Live, Fable led 50% to 32%; with 50 episodes that gap is not statistically resolved (p = 0.12). Astra finished each first version about 6 times faster, for a quarter to a half of the cost, and drove live twice as fast at under a third of the cost per episode.
The two worked differently. Fable spent about an hour and a half on each first version, working out the arm's kinematics, its controller and the camera from source code before writing a policy. Astra spent about 15 minutes on each. Fable did best in code; Astra did as well or better in weights.
Figure 11. Fable against Astra, held-out success
playground/LLMs for robot control
Held-out success for Fable (blue) and Astra (orange) on each approach. Written policies use the strict check on the same 500 episodes; *live control uses LIBERO's usual check on the same 50. p-values compare the two models on those episodes.
The three approaches at a glance
Writing a policy was the more cost-effective choice for both models: past about 17 episodes it costs less than driving live, and per success it is about 30 to 75 times cheaper. It was also more reliable for Fable, whose code and network beat its own live driving by about 40 points on the same check, but not for Astra, whose written policies and live driving all succeeded about a third of the time. The bigger divide was the model: Fable's weakest approach, driving live, still scored above Astra's best (Figure 12).
Figure 12. The three approaches at a glance
Model
Approach
Success
LLM cost, 100 episodes
LLM time, 100 episodes
Fable
Code as policy†
88.2%
$45once
2.6 honce
Weight-space network
84.2%
$33once
1.7 honce
Live control*
50.0%
$272$2.72 each
14.0 h8.4 min each
Astra
Code as policy
33.2%
$15*once
35 minonce
Weight-space network
38.0%
$16*once
40 minonce
Live control*
32.0%
$83$0.83 each
6.8 h4.1 min each
playground/LLMs for robot control
Success for the revised written policies under the strict check (500 held-out episodes) and *live control under LIBERO's usual check (50). Cost, at API prices, and time are the LLM's, to run 100 episodes; a written policy's are paid once, both versions included. †Soup and sauce at its first version; 94.2% as submitted. Astra's authoring costs are the middle of a $11 to $18 range. Fable's code time includes a wait on a permission prompt.
Strict scoring
LIBERO counts an episode as a success the first moment its goal check passes, and ends it there. The check looks at where the objects are, not at whether the robot has let go. A policy can therefore score by carrying an object into the goal region and stopping there, or by passing through the region on its way to a failed placement. An adversarial review of our first round found policies doing exactly that. So our main results use a stricter check, set before round two: the goal must hold while the gripper touches nothing but the robot, for 20 control steps in a row (one second). The round-two authors were told about it, and episodes kept running past the first pass, so both checks come from the same runs.
The choice of check moves scores some, but not much (Figure 13). Six of the eight written policies change by a point or less. The exception is Fable's network, which scores 8 to 9 points higher under LIBERO's check, almost all of it on the wine rack: 40 of 50 under LIBERO's check and 1 under the strict one (Figure 14). One comparison depends on the check. After revision, Fable's code leads its network under the strict check (88.2% to 84.2%, p = 0.06) but trails it under LIBERO's (89.2% to 92.8%, p = 0.04). The other comparisons come out the same either way.
Live control ran before the strict check existed. Each episode ended the moment LIBERO's check passed, with the agent told to stop, and at that moment the gripper was still holding the object in 18 of Fable's 25 successes and 5 of Astra's 16. Live control therefore has no strict score of its own, and the main results report it under LIBERO's check. As an assisted diagnostic only, we replayed each success, continued it with actions we wrote, and applied the strict check. Holding still with the gripper as it was keeps 8 of Fable's successes and 10 of Astra's; opening the gripper and holding still keeps 18 and 10; opening it and lifting away keeps 23 and 13. For comparison, the strict check alone would have cost our first-round policies 0 to 18 points.
Figure 13. LIBERO's success check against the strict check
playground/LLMs for robot control
Held-out success with LIBERO's usual check (filled) and with the strict check (hollow), for every written policy (500 episodes each) and for live control (50 episodes each). Fable's revised code leaves soup and sauce at its first version. Live control's strict values come from the assisted open-and-lift replay described above, not from the agents themselves.
Figure 14. Fable's code and network on the wine rack
playground/LLMs for robot control
Held-out scene 0 of “put the wine bottle on the rack”, run by Fable's v2 code (left) and v2 network (right), at real speed. The code sets the bottle in the rack, lets go and backs away: LIBERO's check passes at step 123, and the strict check at step 150, one second after release. The network carries the bottle into the rack, and LIBERO's goal check passes at step 73 with the bottle still in the gripper, which LIBERO counts as a success. It then lets go before the bottle is seated, and the bottle falls. Across the 50 held-out scenes, the code scores 50 under LIBERO's check and 47 under the strict one; the network scores 40 and 1.
Standard scoring and π0.5
Most published LIBERO results use LIBERO's usual check, so Figure 15 lists ours that way too, with π0.5 alongside for context. Physical Intelligence reports per-suite scores for π0.5 fine-tuned on LIBERO's demonstrations; weighted to our mix of task suites, they come to about 97.2%. That is a reference point, not a matched comparison. It averages each suite's tasks rather than our ten, comes from their runs rather than ours, and π0.5 was trained on demonstrations of these tasks and uses a wrist camera, neither of which our authors had.
Figure 15. Our results with LIBERO's usual success check
Policy
LIBERO's check
Strict check
Fable, code as policy, first version
88.8%
87.8%
Fable, code as policy, revised†
89.2%
88.2%
Fable, weight-space network, first version
82.8%
74.8%
Fable, weight-space network, revised
92.8%
84.2%
Fable, live control
50.0%
no strict score of its own
Astra, code as policy, first version
10.0%
10.0%
Astra, code as policy, revised
33.2%
33.2%
Astra, weight-space network, first version
35.8%
35.2%
Astra, weight-space network, revised
38.2%
38.0%
Astra, live control
32.0%
no strict score of its own
π0.5, published, weighted to our task mix
≈97.2%
not measured
playground/LLMs for robot control
Held-out success with LIBERO's usual check, next to the strict check used in the rest of the post. Written policies: 500 episodes each; live control: 50. π0.5's figure is Physical Intelligence's published per-suite LIBERO score, weighted to our mix of three spatial, three object, two goal and two long-horizon tasks; we did not run π0.5 ourselves. †Soup and sauce at its first version; as submitted, 95.2% with LIBERO's check and 94.2% with the strict one.
Takeaways
We found that a frontier LLM can write a strong robot policy without ever running it or simulating it. Fable's zero-shot code solved 87.8% of held-out episodes on ten LIBERO tasks under a strict success check, working only from source code and assets.
We found that writing a policy is more cost-effective than driving the robot live for both models, past about 17 episodes. It was also more reliable for Fable; for Astra, live control matched its written policies on success but cost about 30 times more per success.
We found that code beat weights for Fable on the first try, by 13 points, and weights beat code for Astra; after one revision, neither gap was statistically resolved.
We found that one look at 20 recorded rollouts clearly helped two of the four written policies, by 9 and 23 points.
We found that Fable was far more accurate than Astra, and Astra faster and cheaper.
We found that LIBERO's success check can be passed without letting go, but that a stricter check moves most of our scores by a point or less.
This is a first glance. Each cell is one authoring run, so our confidence intervals cover variation across episodes, not across authoring attempts; a second run could land differently. We used 10 of LIBERO's 40 tasks, and live control was scored on 50 episodes per model rather than 500, with LIBERO's usual check. The π0.5 figure is published context, not a run on our episodes. The models ran in their own harnesses, Claude Code and Codex, so model and harness are confounded, as is what “high effort” means for each.
Addendum: GPT-6.2 Astra
Added November 17, 2026. OpenAI released GPT-6.2 Astra on November 17. We ran it through the same experiment, changing nothing but the model: the same rooms (byte-for-byte identical to GPT-6 Astra's), brief, materials, prompts, container, Codex version (0.167.1), reasoning effort (high), scoring harness and live-control runner. Astra 6.2 wrote a first version of each policy cold, saw the same 20 development rollouts and made one revision, and drove the robot live on the same 50 scenes. Before accepting each frozen policy we read every script it ran and every command in its sessions; all four passed, with nothing simulated, fitted or tested in a loop. Two operational changes are logged and change no score: one live episode crashed at startup, before the model produced any output, and was rerun from scratch; and we later ran fewer episodes at once and scored at lower CPU priority. As above, each policy is one authoring run.
Figure 16. GPT-6.2 Astra against Fable 5.2 and GPT-6 Astra
Policy
GPT-6.2 Astra
Fable 5.2
GPT-6 Astra
Code, first version
88.0% (84.9–90.6) [92.4%]
87.8% [88.8%]
10.0% [10.0%]
Code, revised
94.8% (92.5–96.4) [96.8%]
88.2%* [89.2%]
33.2% [33.2%]
Network, first version
47.8% (43.5–52.2) [57.0%]
74.8% [82.8%]
35.2% [35.8%]
Network, revised
78.0% (74.2–81.4) [79.0%]
84.2% [92.8%]
38.0% [38.2%]
Live control, LIBERO's check (50 episodes)
60% (46–72)
50%
32%
playground/LLMs for robot control
Held-out success under the strict check on LIBERO's 500 held-out episodes per written policy (95% Wilson interval for GPT-6.2 Astra), with LIBERO's usual check in brackets; live control is 50 episodes under LIBERO's usual check. *Fable's revised code as reported above (v2c).
Code. GPT-6.2 Astra is the strongest code author we have tested. Its first version was level with Fable's (on the same 500 episodes, 48 solved only by Astra 6.2 and 47 only by Fable, p = 1.0). Its revised code beat Fable's, 94.8% to 88.2% under the strict check (49 against 16 episodes, p = 0.00005; with LIBERO's check, 96.8% to 89.2%). The revision changed only the wine-rack task, which went from 14 of 50 to 48 of 50; the drawer bowl (41 of 50) and the two-object basket task (37 of 50) remain its weakest. With LIBERO's check its revised code sits about half a point below fine-tuned π0.5's published ≈97.2%, which, as in the main post, is a reference point rather than a matched comparison.
Networks. Fable still builds the better network: Astra 6.2's revised network scores 78.0% against Fable's 84.2% under the strict check (70 against 101 episodes, p = 0.02), and 79.0% against 92.8% with LIBERO's. It more than doubles GPT-6 Astra's network (215 against 15 episodes). It has 27 layers, and its weight matrices hold 2.5 billion entries, of which about 191,000 parameters are nonzero (183,960 weights and 7,180 biases).
Live control. GPT-6.2 Astra solved 30 of 50 scenes live, the best live result we have recorded. That is not significantly above Fable's 25 (16 against 11 scenes, p = 0.44) but is above GPT-6 Astra's 16 (18 against 4, p = 0.004). All 30 successes replay exactly; under the strict rule, 11, 16 or 24 of the 50 pass depending on whether the robot then holds still, opens and holds, or opens and retreats. On the same 50 scenes with LIBERO's check, both of its code versions beat its live control (17 and 18 scenes against 0), as did its revised network (13 against 3, p = 0.02); its first network tied it (9 against 11).
Figure 17. Three models' code on one held-out scene
playground/LLMs for robot control
Held-out scene 29 of “pick up the black bowl next to the plate and place it on the plate”, run by the revised code of GPT-6 Astra (left), Fable 5.2 (middle) and GPT-6.2 Astra (right), at 1.5× speed. GPT-6 Astra's and Fable's code run out of steps without placing the bowl; GPT-6.2 Astra's sets it on the plate, lets go and clears, passing the strict check at step 135. On this task GPT-6.2 Astra's code solved 49 of 50 scenes and Fable's 43; Fable solved no scene that Astra 6.2 missed. Each clip replays the scored episode exactly.
Cost and time. GPT-6.2 Astra's list price is $8 per million input tokens, $0.40 cached and $40 output: the same as Fable 5.2 for input and output, and 20% below GPT-6 Astra. Its code cost $8.34 and about 27 minutes of LLM time to write, both versions included, and its network $9.50 and about 42 minutes. Live, it cost $0.45 and a median 3.7 minutes per episode. Per success with LIBERO's check, that is 1.7 cents for its code, 2.4 cents for its network and 75 cents live: about a sixth of Fable's code (10.1 cents) and a third of Fable's network (7.1 cents). At the same list price, Astra 6.2 drove the robot live for 75 cents per success against Fable's $5.44, because it used far fewer tokens per episode (about 2,800 output tokens to Fable's 41,500) and succeeded more often. From GPT-6 Astra to GPT-6.2 Astra, live control became about 3.5 times cheaper per success ($2.58 to 75 cents) and written code 5 to 6 times cheaper, so the gap widened: driving live now costs about 43 times as much per success as Astra's code. (Recomputed the same way, every turn's tokens at list price, GPT-6 Astra's authoring cost is $17.86 for code and $18.36 for its network, the top of the $11 to $18 range given above.)
What changes. Fable is no longer the best written-policy author in every format: GPT-6.2 Astra writes the best code we have scored, while Fable still writes the best networks. Which format suits a model also depends on the model. Fable's revised network edged its code under LIBERO's check while its code led under the strict check; for Astra 6.2, code is the stronger format under both, by a wide margin. The central result holds for all three models: a policy the LLM writes once matches or beats the same LLM driving the robot live, at a small fraction of the cost per success. The caveats above apply: one authoring run per policy, ten of LIBERO's 40 tasks, 50 live episodes per model, model and harness confounded, and GPT-6.2 Astra run on its release day. This addendum's logs, prompts, frozen policies and per-episode results are in the replication bundle, in addendum_gpt62_astra/.
REPRODUCE THE STUDY
Data and code.
Every number on this page comes from these files. The bundle’s README lists the software versions and how to score a policy.
Tell us about your model and what you’d like to test. We’ll help scope a custom evaluation or work with you to add your policy to the Robot Intelligence Index.