1NVIDIA Research 2Georgia Institute of Technology 3University of Utah
A robot in an unmapped home has to decide what to find out and how to act, often before it has seen where things are. Vision-language agents bring the priors for this, but when they call tools directly nothing declares what makes an action correct. Belief-space task and motion planning declares it, but pays for explicit distributions over the whole house and a hand-written problem.
TAMPAgent lets the agent act as the belief-space policy while its output keeps the planner's contract: parameterized actions whose constraints tools must certify before anything moves. Beliefs stay in language until a constraint needs a value from a tool.
Method
TAMPAgent runs one loop. The agent writes the problem from its current belief, the planner grounds the actions, the agent plans with a hypothesis, the planner turns the plan into constraints, and the agent solves them with tools. Only then does the robot act, observe, and start again. Below, each step is shown on the run from the video.
The agent turns the instruction into a planning problem: a goal and the current value of every variable it needs. Values the robot has not observed are hypotheses written in words. Here the agent only knows the black table, so it guesses that both the apple and the bowl are there, and plans to find the apple, pick it, find the bowl, and place it.
![The agent writes the problem P0 with the goal In[apple, bowl] and language beliefs, then its first plan.](figures/method/step1.png)
The robot looks for the apple from several viewpoints. Segmentation finds nothing, then only a sliver with a single depth pixel, so the observation constraint of detect cannot be certified: the planner reports an epistemic gap and will not let the robot pick.
The agent uses its priors to rank where the object could be, and commits to the most likely explanation: the penguin plush hides it. Its plan moves the penguin out of the way before looking again.
The planner grounds the plan into a factor graph. The continuous choices of each action, such as the grasp, arm pose, path, and placement, become variables, and each action's preconditions become constraints over them. A variable shared by two actions couples them: the grasp chosen to pick the penguin must also let the robot set it down.
The agent composes a computation graph of tool calls, segmentation, grasp generation, inverse kinematics, and motion planning, and runs it until every constraint of the first action has a value a tool has certified.
The planner accepts only when every constraint of the first action is certified. The robot executes that one action and observes, and the agent updates its belief. Once the penguin is set aside, the apple is in view: what was a language hypothesis is now a certified RGB-D estimate. The loop starts again from the first step.
The same loop carries the task to the end. Two base motions toward the apple fail, so the agent picks it from where the robot stands. While carrying it, the robot searches for the bowl until perception certifies its location. No placement fits from the first stance, so the agent repositions the robot, and the apple goes into the bowl.
Sandbox
The agent is a coding agent, Codex or Claude Code, in a locked-down container. It has a shell, Python and a handbook, and it reaches the robot only through a gateway that forwards a fixed list of routes. Everything below is from simulation: how the sandbox is built, the handbook the agent reads, and one run of task T10, loop by loop.
Each step is a small Python file that states what the agent believes, what it wants to do and, after running, what it achieved. Every loop starts with ranked hypotheses in plain language, and every reply is saved.
It writes the problem and its inventory, proposes one complete plan that the planner compiles into a CSP, and binds tool receipts to the CSP's variables. These change the planner's belief, never the robot.
It cannot move the robot directly. It asks for a grant for the exact locked step, and the planner issues one only when every constraint of that step is certified. Then it dispatches once and completes the step.
The run is task T10, where a cracker box and a mustard bottle hide the apple; the T10 video shows the full rollout. The handbook is the simulation profile's; the real robot uses the same pages, with its own parts where the robots differ.
Evaluation
Ten simulated tasks ask the robot to place the apple on the plate on the dining table, and differ in where the apple starts and what hides or blocks it. Every method gets the same short goal prompt and no description of the scene. The robot, an RB-Y1, starts facing away from the apple every time. We compare against two baselines without a planner, over 20 episodes per task: Code as Policies (CaP), where the same agent writes code that calls the same tools, and Graph as Policy (GaP), where helper agents author and review graphs of tool calls and the robot moves only through a validated graph. The tasks fall into four groups: mobile manipulation (T1, T2), non-prehensile tasks where something must be moved first (T4 to T6, T8, T10), occlusion tasks where the apple is hidden (T3, T7), and T9, where the apple is hidden in the fridge behind a cup that must be moved. All methods run GPT-6-Astra at low reasoning effort. Each episode has 15 minutes of simulated time plus 5 minutes for each blocker, and 25 minutes for T3, T9, and T10. On T9 we measure learning from attempts instead of running 20 episodes. Three more tasks, T11 to T13, move to a larger home with other objects, where the robot must search across rooms; in T13 a chair also blocks the doorway. The charts are from the RB-Y1; the videos show rollouts on the RB-Y1 and on a Unitree G1.
Rollout on the G1.
T1Mobile manipulation
The apple is in the open on the dining table, next to the plate.
What needs to be done
Time budget: 15 min of simulated time
Close-up around the apple at the start.
T2Mobile manipulation
The apple is in the open on the kitchen counter, away from the dining table.
What needs to be done
Time budget: 15 min of simulated time
Rollout on the RB-Y1.
T3Occlusion
The apple is on a shelf inside the closed refrigerator, so no view of the room shows it.
What needs to be done
Time budget: 25 min of simulated time
Rollout on the RB-Y1 and G1.
T4Non-prehensile
The apple is on the counter, with a cup right in front of it that blocks every grasp.
What needs to be done
Time budget: 20 min of simulated time
Rollout on the RB-Y1.
T5Non-prehensile
The apple is on the counter, and an open drawer below it blocks the approach.
What needs to be done
Time budget: 20 min of simulated time
Close-up around the apple at the start.
T6Non-prehensile
The apple is on the counter behind a cup, and an open drawer blocks the approach.
What needs to be done
Time budget: 25 min of simulated time
Rollout on the RB-Y1.
T7Occlusion
A cracker box on the counter hides the apple from every view.
What needs to be done
Time budget: 20 min of simulated time
Rollout on the RB-Y1.
T8Non-prehensile
The apple and the plate are on the dining table, with a cracker box standing in front of the plate.
What needs to be done
Time budget: 20 min of simulated time
Close-up around the apple at the start. The refrigerator doors are drawn translucent here; in the simulation they are closed.
T9Occlusion + non-prehensile
The apple is inside the closed refrigerator, behind a cup on the shelf.
What needs to be done
Time budget: 25 min of simulated time
Rollout on the RB-Y1.
T10Non-prehensile
The apple is at the far end of the counter, hidden behind a mustard bottle, with other objects around it.
What needs to be done
Time budget: 25 min of simulated time
Rollout on the RB-Y1.
T11Mobile manipulation across rooms
Instruction: place bowl from living room to kitchen table. The robot starts in the kitchen. The bowl is out of sight on a round table in the living room, next to a small book.
What needs to be done
Rollout on the RB-Y1.
T12Mobile manipulation across rooms
Instruction: throw away the used sponge. The robot starts in the kitchen, where neither the sponge nor the trash bin is in view.
What needs to be done
Rollout on the RB-Y1.
T13Across rooms, non-prehensile
Instruction: place bowl from living room to kitchen table. The same instruction and home as T11, but a chair now stands in the opening between the kitchen and the living room.
What needs to be done
What each result shows, as the question it answers, is in the FAQ.
Gallery
Other real runs, on the RB-Y1 and on a Unitree G1.
Questions
All three methods solve every episode when the apple is in the open (T1, T2). When it is hidden or blocked (T3 to T8 and T10), TAMPAgent succeeds in 71% of episodes, against 45% for CaP and 50% for GaP. Over all tasks, it succeeds in 77%, against 57% and 61%. A plausible but infeasible plan, such as picking the apple through the cup, fails in the planner before the robot moves, and the failed constraint names what the agent must clear first.
An episode fails when an object falls or the scene is destroyed, or when its time runs out. Every motion TAMPAgent executes was certified first, so over the tasks it drops an object or destroys the scene in 13 episodes, against 26 for CaP and 14 for GaP, and it times out in 28, against 51 and 56.
The planner keeps a compact record of the problem and its progress, so the agent reads and writes less. Per minute of simulated time, TAMPAgent uses fewer uncached input and output tokens than CaP and GaP. GaP is more structured than CaP, but its agent has to come up with each computation graph by itself, while TAMPAgent's planner already supplies the actions and the constraints each graph must certify.
| Method | Uncached input / sim-min | Output / sim-min |
|---|---|---|
| TAMPAgent | 19.3k | 1.75k |
| CaP | 20.3k | 1.91k |
| GaP | 27.4k | 2.73k |
On T9, the apple sits inside the fridge behind a cup. Each method starts without how-to guidance, keeps notes in a memory across attempts, and retries until it succeeds. TAMPAgent and GaP succeed on their third attempt and CaP on its fourth. Over these attempts, TAMPAgent uses 42% and 31% fewer uncached input tokens than GaP and CaP, and 9% and 12% fewer output tokens.
On a real RB-Y1 in an office lab, TAMPAgent runs three categories of tasks. In simple pick and place, the robot puts an object such as a mustard bottle in a receptacle, finding both by exploring open surfaces. With an out-of-view receptacle, the robot carries the object while it searches, and plans the placement once the receptacle is certified. With a hidden object, as in the video above, the agent hypothesizes what hides the target and moves it first. Hidden objects cost the most, 973k uncached input and 127k output tokens per episode on average, against 376k and 39k for simple pick and place.
Every constraint of an executed action was certified by a tool under the robot's current perception. The agent can still choose a poor plan, and tools can return false positives, so a certified grasp can still fail.
Skills are hand designed, though learned policies can join as tools. Cloud-hosted agents are slow: the robot is idle for more than 90 percent of an episode while the agent reasons.
Citation
@misc{mishra2026tampagent,
title = {{TAMPAgent}: Agentic Belief-Space Planning for
Mobile Manipulation},
author = {Mishra, Utkarsh A. and Degirmenci, Alperen and Chen, Yongxin and
Xu, Danfei and Hermans, Tucker and Garrett, Caelan},
year = {2026},
note = {Under review}
}