Why don't you just ask Astra?

Noumenal · October 2026

"Just ask ChatGPT or Claude" has become the default answer for many questions lately. Write the email, fix the bug, summarise the paper. Most work done at a keyboard now looks like something an LLM can do. A few weeks ago the same question reached robotics. In a technical report, "GPT 6 Astra as an Embodied Policy" (Su et al., 2026), the authors put GPT 6 Astra in charge of simulated robot arms. They wanted to know two things. How well does Astra do when it drives a robot arm by itself, compared with a model built for robotics? And how well does it do in tandem with such a model?

The robotics model in the report is π0.5, Physical Intelligence's open vision-language-action (VLA) model. A VLA takes camera images and an instruction in plain text ("pick up the red ball") and outputs the arm's next movements, many times per second. It is trained on robot data only, so it is fast but narrow.

One of the report's benchmarks is RoboLab: ten pick-and-place tasks for a single Franka arm in simulation, with five attempts per task. π0.5 ran there zero-shot. It had never been trained on these tasks, only on DROID, a large collection of demonstrations recorded on real Franka arms, and nobody fine-tuned it for the benchmark. Astra alone drives the arm by looking at the camera images and writing out where the gripper should move next, a few steps at a time. In tandem, π0.5 proposes the next two seconds of motion and Astra reviews the proposal. It either lets the first part run unchanged or replaces it with a short correction of its own. Then π0.5 proposes again.

Astra alone wins clearly. On the report's second benchmark, RoboDojo, with two-armed tasks and a π0.5 fine-tuned on each task, the tandem comes out ahead (24 against 13 successes out of 50). Either way, Astra beats the robotics model. So people now ask the obvious question, even in robotics: why don't you just ask Astra?

There is a problem, which the authors of the report acknowledge: the latency between the robot and the LLM is still unsolved. Astra runs at its highest reasoning effort, and in our own runs a single call to an LLM at that effort easily takes 10 seconds or more. The robot has to stand still until the answer arrives. In tandem, the report calls Astra about 75 times per episode, and Astra alone about 155 times. In simulation you can pause the world while the model thinks, but in real robot control, latency matters.

In this post we explore the idea of calling the LLM only when it is needed. Between π0.5 and the LLM we put a harness: a small layer of code that watches π0.5, reads its own uncertainty, and decides at every step whether π0.5 can carry on, whether to step in with a simple move, or whether it is worth asking the LLM. The LLM leaves the control loop and becomes a supervisor that the robot calls when it needs help. We don't try to beat Astra. We try to make the VLA better, cheaply. On one task of the report's own RoboLab benchmark, π0.5 goes from 4 to 36 successes out of 40 with about 3.5 LLM calls per episode instead of a hundred, in under two minutes per episode. On LIBERO-Object the harness needs about 1.3 calls per episode. The post is for a technical audience and assumes some ML basics. All toy examples use made-up numbers. Numbers from our benchmark are labelled as such.

Summary

• An LLM in the control loop is slow and expensive. In our rebuild of the report's setup, the LLM made about 100 calls and took about 20 minutes per episode. The simulation waits for every call, which a real robot can't do.
• π0.5 fails for different reasons on different tasks. Sometimes it knows where the object is but won't commit to the grasp. Sometimes it heads for the wrong place.
• We put a small probabilistic harness between the VLA and the LLM. At every step it lets π0.5 continue, runs a cheap check, or asks the LLM.
• The harness steps in when π0.5 is uncertain: when its sampled plans spread out, when a cheap check disagrees with where it is heading, or when nothing happens for too long.
• On a RoboLab task, π0.5 goes from 4 to 36 successes out of 40 with 3.5 LLM calls per episode, in under two minutes and for about 4 cents per success.
• On LIBERO-Object the harness keeps nearly all of the LLM's success with about 1.3 calls per episode, 34 times faster and 70 times cheaper than letting the LLM drive.
• Asking has a price, and the LLM's answers are biased. A good harness asks rarely, and avoids asking the same question twice.
• Very important: wherever the VLA is already good, any intervention costs you.

Why not use Astra? Literally.

In the report, the tandem works like this. At every decision, π0.5 proposes a chunk of 50 steps of motion. Astra gets the camera images, the robot state, the history and the motion π0.5 proposes. It either runs the first 1 to 15 steps unchanged or replaces them with its own correction for 1 to 5 steps. On RoboDojo that comes to about 75 decisions and 12.5M tokens per episode, nearly all of them read from the prompt cache. The simulation waits for every answer. The authors describe this as treating robotics like "a pausable video game". (The report is a technical report, not peer-reviewed. We use it as a reference method, not as a leaderboard.)

To see what this costs, we rebuilt both of the report's setups on one of its own RoboLab tasks: "Place the butter on top of the raisin box". In the report, π0.5 solved it 0 times out of 5, while Astra alone and the tandem both solved it 5 out of 5. For the tandem we use the report's released code, with the same prompts and limits, and swap Astra for Claude Opus 5.5 at maximum effort. "LLM alone" is our version of Astra driving the arm directly: the LLM sees the camera images and moves the gripper itself, at most 5 cm per step. Later we repeat the comparison on a second benchmark, LIBERO-Object (see Results).

We ran each LLM setup twice, as a check that our rebuild works and to measure what it costs. The success rates are the report's.

Both LLM setups solved the task. In tandem, the LLM corrected π0.5's overshoot of the butter, then closed the gripper itself, exactly where π0.5 fails on its own. Alone, it worked out the butter's position from the images to within about 2 mm and placed it in 25 to 28 moves. But every episode took about 20 minutes, against under three for π0.5 alone, and almost 90% of that time went to waiting for the model, with the world frozen. On a real robot you can stop the arm, but you can't stop the world: a person walks in front of the camera, the wind moves an object, the ball rolls away.

Why is the LLM so slow? It reasons before it answers. At maximum effort it first works through a long chain of thought, often thousands of tokens, and it produces them one at a time, each one depending on the ones before. Even on fast hardware that takes seconds, often tens of seconds. We also tested low-effort mode, which surprisingly wasn't faster because it required many more calls to the LLM, though each call was executed faster. Our harness pauses too when it asks the LLM, and the arm holds still until the answer arrives. The big difference is how often.

None of this makes the LLM useless. It knows a lot about scenes, objects and instructions that π0.5 doesn't. The question is how to use it as a supervisor instead of putting it in the control loop.

Why a harness?

Foundation models for robots, VLAs like π0.5 and world-action models, have improved quickly, yet few are deployed in real industrial settings. The usual diagnosis is that more data, better data, or better architectures are needed. LLMs heard the same arguments, and scaling did pay off. But what turned capable LLMs into useful products was the harness: the code around a model that decides what it sees, when it is called, what it may do, and what is checked before an action runs. Coding agents like Claude Code and Codex are the clearest examples. The report's "review every chunk" is a harness too, just one that calls the model every time.

Harnesses earn their place for concrete reasons:

• They put checks between a model's output and the motors.
• They control cost and latency, which matters when one call takes 10 seconds or more.
• They record every decision, which serves as both an audit trail and training data.
• They let you swap the model, or replace it with a person, without retraining the robot.

Our harness adds three things. It reads π0.5's own uncertainty to decide when to step in. It treats every call to the LLM as having a price. And it keeps a few simple, general moves at hand for the moments when π0.5 hesitates. The rest of this post shows how.

Where π0.5 fails

Before building anything, we ran π0.5 alone on the butter task and watched it fail. It solved 4 of 40 episodes. In the failed episodes its fingertips still come within a few centimetres of the butter. It knows where the object is. It fails at committing to the grasp. The failures, most common first:

• it closes too briefly, and the next plan opens the gripper again
• it closes on the butter, then lets go before lifting
• it never closes
• it closes on nothing
• it drops the butter on the way

π0.5 alone: it closes on the butter, then lets go before lifting. Played at twice the speed.

On LIBERO-Object the failure is different. After we move the objects or change the instruction, π0.5 heads for where the target used to be, or for the wrong object. There the problem is where, not how. A good harness has to handle both.

How our harness works

We test on two benchmarks: the RoboLab task from the report and LIBERO-Object. The RoboLab task has a Franka arm, a butter box and a raisin box, and the instruction "Place the butter on top of the raisin box".

The RoboLab task at the start of an episode, over-shoulder camera (left) and wrist camera (right).

A typical LIBERO-Object task has a Franka arm, a basket and a handful of groceries, with an instruction such as "pick up the alphabet soup and place it in the basket" (the can in front).

A LIBERO object task as the scene camera sees it at the start of an episode. Rendered by us from the benchmark's starting state.

We split the job into two seats:

• π0.5 drives. It's fast and cheap, and good at what it was trained on. Outside of that it is often confidently wrong: wrong in ways its own confidence doesn't show.
• Experts (a cheap vision-language model or an LLM) answer questions. For example "where in this camera image is the alphabet soup?". In our benchmark the cheap model is NVIDIA Cosmos3-Edge, running locally, and the LLM is Claude Opus 5.5. The LLM's answers are good but slow, cost money, and (as we'll see) are biased. Nothing in the harness depends on who answers. Any model, or a person, that can answer questions about the scene fills this seat. In a real deployment the last resort is a human teleoperator. Our benchmark has none.

Between them sits our harness. One episode runs like this:

1. Ask once where things are. At the start the LLM marks the target and the goal in the camera image. With the depth image this gives positions in 3D. π0.5's input doesn't change.
2. Watch π0.5's plans. For every plan, the harness works out where the gripper will end up. If π0.5 heads twice in a row for a spot more than 7 cm from the target, the harness steps in.
3. Read π0.5's uncertainty. We ask π0.5 for five plans on the same camera image. If they disagree about closing the gripper next to the target, the harness closes it and holds. If they disagree elsewhere, it keeps the gripper as it is, so π0.5 can't drop what it holds while it hesitates.
4. Notice when nothing happens. If nothing is lifted three seconds after a grasp, the harness lets go, asks the LLM again where the target is, grasps from above itself, and asks the LLM whether the grasp holds.
5. Help with the carry. While carrying, the gripper stays closed. Over the goal, the harness sets the object down: up, across, down, and only then open.

In all other cases π0.5 simply drives. The harness asks the LLM at most nine times per episode. On average it asked 3.5 times.

Diagram to come: the 5 steps as a loop around π0.5.

None of these pieces knows about butter. At Noumenal we build harnesses with active inference. The harness carries a small generative model: a few variables for the task (where the target is, whether the gripper holds it) and a few numbers for how reliable each helper is. Every question has a price, and the harness asks when the information it expects to gain is worth that price. If you come from partially observable Markov decision processes (POMDPs), think of it as a small belief-state model with information gain and explicit prices. The next sections show the ideas behind it.

Reading π0.5's own uncertainty

π0.5 doesn't output one plan. It samples one, and a second sample on the same image can differ. So the harness asks π0.5 for several plans. If they spread out, π0.5 is unsure, and the harness steps in: it holds the gripper decision steady, and asks the LLM if that isn't enough. If they are very narrow, π0.5 is confident, but it can be confidently wrong, so a cheap check compares where it is heading with where the target is.

When is a question worth asking?

The harness asks the LLM only when the answer would teach it more than the question costs. For example, it asks the LLM whether the object is grasped only when nothing has been lifted after a grasp. Here is how it decides in general: first the idea, then the details.

The idea. The harness keeps a running estimate of where the target object is, with an error bar: "the mug is here, give or take 3 cm". Everything it does serves that estimate:

1. Keep a belief. A best guess for the target's position, plus an error bar for how unsure it is.
2. Let the error bar grow. Between answers the world moves a little, so the error bar slowly widens. If the gripper suddenly closes on nothing, the harness concludes the object slipped, and the error bar jumps.
3. Estimate what an answer would be worth. Once per π0.5 step, the harness asks itself: if I asked the LLM "where is the target?" right now, how much would the answer shrink my error bar? If I'm unsure, a lot. If I already know, hardly at all.
4. Compare with the price, and pick. Every question to the LLM has a fixed price, for the time and money it costs; letting π0.5 continue is free. If the expected gain is worth the price, the arm holds still, the LLM answers, and the answer is merged into the belief. Otherwise π0.5 keeps driving.

The harness's decision, once per π0.5 step

There is no hand-written rule like "ask at most 6 times" or "ask after a slip". Both fall out of the comparison. Once the harness knows where the target is, another answer isn't worth its price. After a slip it is unsure again, so a new answer is.

The math, compressed. Write the belief about the target's position along one axis as a Gaussian with standard deviation σ (the error bar), and an LLM answer as the true position plus noise with standard deviation σ_a. The expected information gain of one answer, in nats, is

$$\mathrm{IG}(\sigma) = \tfrac{1}{2} \ln\left(1 + \frac{\sigma^2}{\sigma_a^2}\right)$$

The harness asks if IG > c_ask, where c_ask is the price of one question to the LLM. We started with c_ask = ln 2, which is the gain of an answer that exactly halves σ. In words: ask only if the answer would at least halve the error bar.

A toy example. We strip the robot away: one block sits somewhere on a line (one axis of the table), and the LLM answers with the block's position plus a random error of about 12 mm. All numbers are made up. The harness starts knowing nothing (an error bar of 300 mm, the whole table), and at step 7 the block slips 45 mm to the side, which widens the harness's error bar. At every step, the harness asks only if the expected gain is worth the price.

The toy example, step by step

The harness asks the LLM once at the start, when it knows nothing, and then not again. With the error bar around 12–14 mm, another answer is worth about 0.4 nats, less than the price of 0.69. The slip at step 7 pushes the gain to 1.65 nats, so the harness asks the LLM once more and then goes back to letting π0.5 drive.

Whom to trust

An LLM's answer is evidence, not a command. The harness has two sources for where the target is: the LLM's answer, and where π0.5 is reaching. When they disagree, it has to decide which one to believe.

In LIBERO this happens with similar-looking objects. In the task below the instruction is "pick up the bbq sauce and place it in the basket", and the bbq sauce and the ketchup are two similar bottles, in one of our runs about 19 cm apart. If π0.5 reaches for one bottle and the LLM marks the other, the harness has to decide whom to believe.

The bbq-sauce task at the start of an episode. Rendered from the benchmark's own starting state.

A recorded LIBERO episode. The orange marks are the model's answers, which the harness folds in as evidence about where the object and the basket are; π0.5 then carries the object to the basket.

The idea. A disagreement has two possible explanations:

1. Same object, a bit of noise. π0.5 is going for the right object, and its reach and the LLM's mark are both slightly off. Then the two should be combined, and the estimate gets sharper.
2. Different objects. π0.5 is going for the wrong object. Then its reach says nothing about the target and should be ignored.

The harness keeps a probability for the first explanation, P(π0.5 right), and updates it after every reach. A small disagreement is easy to explain with noise, so P(right) barely moves. A large one is very unlikely under noise, so the harness concludes that π0.5 is heading for the wrong object. This is called explaining away: the wrong-object explanation absorbs the disagreement, so the target estimate isn't dragged toward π0.5's reach.

Two reaches, one LLM mark: what the harness concludes

A toy example. On a line again. The LLM has marked the target at 0, so the harness's estimate is 0 ± 22 mm. π0.5's reach scatters by about 15 mm, and the table is 600 mm wide. Now π0.5 reaches toward a point 0, 20, 50, 100 or 190 mm away from the mark. If π0.5 is right, its reach is averaged with the mark. If it is wrong, the reach is ignored. The harness weighs the two explanations with Bayes' rule, starting from P(right) = 0.95 (π0.5 picked the right object 39 times out of 40 in our LIBERO runs):

Up to 50 mm, π0.5 is believed: the reach counts as a second measurement of the same object, and the error bar shrinks from 22 to about 12.5 mm, so the next question to the LLM is worth less. At 190 mm, π0.5 is written off and the estimate stays exactly at the LLM's mark. At 100 mm the harness can't tell: the estimate is a blur of both explanations with an error bar of ±31 mm, and that is where asking the LLM again pays off most.

The LLM's errors are correlated

Both toy examples treat every LLM answer as the true position plus fresh, independent noise. Our measurements with Opus over 101 LIBERO episodes and 377 answers say this is wrong in two specific ways: the answers share a systematic offset (a bias), and wrong-object answers tend to repeat.

The bias. Most of the error is a fixed offset. Within an episode, the LLM's marks for the same object scatter by only about 10 mm, but they share an offset of about 20 mm. On one task Opus put its mark on the same pixel in every episode, 8.5 mm off each time. Asking again therefore adds almost nothing. With a shared offset in the model, a second answer about the same object is worth 0.05 nats, not the 0.35 nats that independent noise would suggest.

The second failure: the wrong object. This error isn't Gaussian at all. It is discrete: 8 of 72 first answers marked a different object than the one in the instruction. And it is correlated too. After an answer on the wrong object, the next answer was on the wrong object 3 times out of 9 (33%). After an answer on the right object, 3 times out of 80 (4%): a small sample, but a large difference. Two identical marks on the wrong bottle are therefore much weaker evidence than two independent marks would be. Our first version counted them as independent, ended up 23 mm sure about the wrong bottle, and asked for help it didn't need. The harness now carries a discrete variable for whether the answers refer to the intended object or another one, with exactly these switching rates.

What the harness does about it.

1. The bias is part of the state. The harness estimates the target position together with the shared bias of the answers. Repeated answers then pin down position plus bias, not the position, so the information gain of asking again collapses and the harness stops repeating the question.
2. It asks a question with a different failure mode. That is a check: draw a circle around the object π0.5 is reaching for and ask "is this the bbq sauce?". The answer is a yes/no judgement about a given object, not a new mark on the image, so it doesn't inherit the mark's offset, and it is a separate test of the identity of the object.

The RoboLab harness already acts on this. When a grasp fails, it doesn't ask for the position again from the same picture. It asks a different question, "is it grasped?", with the wrist camera.

When nothing happens

A common failure looks harmless: π0.5 closes the gripper on the object and then just waits. Nothing goes wrong, nothing goes right, and the clock runs out. The harness notices that nothing has been lifted three seconds after a grasp. It lets go, asks the LLM where the object is now, grasps from above itself, and asks the LLM with the wrist camera whether the grasp holds. Then π0.5 or the harness carries on.

With the harness: nothing is lifted after the grasp, so the harness asks the LLM where the butter is, grasps from above, checks the grasp, carries and places it. Played at twice the speed.

When π0.5 thinks it's done

Sometimes π0.5 picks up the wrong object, drops it in the basket and stops. From its own point of view the task is done. The harness compares how the episode unfolds with what successful episodes look like. When the two drift apart, its surprise grows, and past a threshold it asks the LLM what went wrong. The LLM sees the scene and the history so far and picks a fix, for example grasp again, or take the object out and start over.

First, do no harm

The first bar the harness has to clear is staying out of the way. An earlier version could steer the arm toward where the LLM said the target was. On LIBERO tasks that π0.5 solves on its own every time, it lost 4 of 18 episodes. The LLM had marked the right objects. The interventions themselves caused the failures.

So the harness now leaves π0.5 alone as long as it looks fine, and only steps in when π0.5 is clearly off course. The requirement is simple. Where π0.5 can do the task, match it and almost never ask. Where it can't, ask and recover what π0.5 loses.

Results

We report two benchmarks, both in simulation. First the RoboLab task from the report, then LIBERO-Object. Every arm uses the same π0.5 policy. The LLM baselines run Claude Opus 5.5 at maximum effort. The harness runs it at default effort.

RoboLab: butter on the raisin box

With the harness, π0.5 solves 36 of 40 episodes, with 3.5 LLM calls per episode, faster than π0.5 alone and for about 4 cents per success. The LLM setups solve the task too, but take about 20 minutes and $4 to$5 per episode.

LIBERO-Object

π0.5 knows the LIBERO-Object tasks well, so we change them in two ways it hasn't seen, 50 episodes each. Re-targeted: the scene is the usual one, but the instruction names a different object. Shuffled: every object is moved to another object's place.

With the harness, π0.5 goes from 22 to 50 completions on the re-targeted tasks and from 5 to 47 on the shuffled ones. Against the LLM alone, the harness gives up two shuffled completions (47 against 49) for episodes that are about 34 times faster and 70 times cheaper per success. It asks the LLM about 1.3 times per episode instead of 48 or 110. An episode with the harness takes about as long as one with π0.5 alone. Reviewing every chunk is the worst of both: π0.5 and the LLM slow each other down, and it costs more than the LLM alone.

All arms pause the simulation while the LLM answers, so the time comparison is fair between arms, but not to the real world. On a real robot the arm holds while the harness waits for an answer, and the world doesn't.

What's next

More tasks. The same harness next runs on a second RoboLab task. Its pieces are general, so we expect them to carry over.

Learning across episodes. The harness already learns within an episode how noisy the answers are and how π0.5's confidence relates to its accuracy. The next step is to carry that across episodes, so that calls per episode fall over time while success holds. We have built this, but it doesn't help yet.

Data from deployment. Every check and every question is logged with the harness's reason and the answer it got. That record is the domain- and task-specific data foundation models need to improve, collected while the robot does useful work under supervision. We expect this to be the harness's longer-term payoff. We haven't measured it yet.

The real robot. The same split, a fast policy for the reflexes and a slow model for the hard questions with a harness that prices the questions, is what we are building for our vegetation-management robot in the field.

Summary

• An LLM can rescue a VLA that's out of its depth, but in the loop at every step it is slow, expensive, and needs a world that waits.
• π0.5 can fail at where or at how. Asking the LLM once where things are fixes neither on its own.
• π0.5's own uncertainty is the best signal for when to step in: the spread of its sampled plans.
• With a few simple moves and a few questions, π0.5 goes from 4 to 36 out of 40 on the report's RoboLab task, at 3.5 calls and 4 cents per success. On LIBERO-Object the harness keeps nearly all of the LLM's success at about 1.3 calls per episode.
• Where π0.5 is good, the best thing the harness can do is nothing.

Every model is wrong somewhere. Nothing moves on its word.