A humanoid controller you can prompt

Robot control is now
an LLM tool call.

Chen Tessler*,1Yifeng Jiang*,1Alec Otto2Yi Shi†,1Umar Iqbal1Gal Chechik1Xue Bin Peng1

1 NVIDIA2 Dollhouse Robotics* Equal contribution† Work done while interning at NVIDIA

Language models have become good at reading a scene and deciding what a robot should do next. What they can’t do is make a humanoid move. Metatron closes that gap in three steps.

Train Metatron once. It learns from a large library of motions to turn simple goals into whole-body movement: where a hand should be, where a foot should land, how fast to walk. It is never retrained.

Let a language model write the skills. A skill is a short program that picks those goals for one behavior: sit down, steer through a room, get back up. The model writes it, tests it in simulation, and fixes what fails.

Call them at run time. The model calls a skill the way it calls any other tool. The skill feeds goals to Metatron and the robot performs the behavior, with no new training. The model itself never enters the control loop.

Real-robot recordings

You say what to do.

Metatron works out how to move.

01 / What you tell it

Set the goals.
Let the body respond.

Most humanoid controllers need a full-body pose at every step. Metatron just needs goals: where a hand should be, which way to face, how fast to walk.

What & where

Say what and where.

Roughly where a hand, a foot, or the torso should be, or how fast to travel and which way. Tighten it up only when the task needs precision.

When

Say when.

Arrival times can be rough too. A skill can line goals up in sequence and shift their timing as the task unfolds.

What stays free

Leave the rest free.

Pin down only what matters. Metatron fills in the rest of the body on its own, no full pose required.

The overlays show the goals we gave it. Everything else about the motion is Metatron’s.

G1 / MuJoCo simulation

Change the goals, not the controller.

Speed, direction, and timed foot targets steer the walk. When the robot gets knocked down, the skill switches to one goal: get upright.

SOMA / IsaacLab simulation · separate policy

Run, slide, climb. Just change the goals.

Speed and direction goals get it running. For the slide, ask for a low pelvis and one foot out front, and the body drops into it with a leg stretched ahead. For the climb, ask for the pelvis above the ledge, and Metatron works out where the hands and feet go.

02 / How skills get written

Refine the goals.
Not the policy.

Describe the behavior you want. A coding agent writes the skill, tries it in simulation, and fixes what fails. Metatron itself never changes.

YouDescribe the behavior
Coding agent + motion examplesWrite the skill
Simulation + frozen MetatronTest it, fix it, repeat

Two separate moments. Offline, a language model turns a behavior into a skill: it writes the goal-setting code, tests it in simulation, and fixes what fails. At run time, the skill is called like any other tool and feeds goals to Metatron at 50 Hz. The model never runs inside the control loop, and a human decides which skills get deployed.

Real robot / A skill written with Claude Code

A better walk,
through better goals.

With only speed and direction to go on, the robot’s steps came out hesitant. We asked Claude Code to fit a simple foot-swing rule to motion-capture clips. Its timed foot targets give the walk a confident rhythm, and the controller itself never changed.

Speed + direction+ timed foot targets

The demonstrated improvement in hardware gait quality and stability is assessed qualitatively.

What does a skill look like?

Two feet.
Half a cycle apart.

Faster walking means quicker, longer steps. Each foot swings back and forth on a cosine and lifts for half of every cycle, with the two feet half a cycle apart. The skill works out where each foot should be a few moments ahead and adds those foot goals to the speed-and-direction prompt.

For sideways travel the stance widens and the swing turns to follow the travel direction, with less side-to-side motion. Metatron does the rest of the body.

Moving-gait pseudocode based on the C++ deployment. Standing holds and turning corrections are omitted.

View simplified skill code
# speed in m/s; frequency in Hz; distances in m
frequency = min(2.0, 0.83 + 0.93 * speed)
amplitude = 0.13 + 0.065 * speed
phase += 2 * pi * frequency * dt

for time in future_times:
    for foot, shift in [(right, 0), (left, pi)]:
        angle = phase + 2 * pi * frequency * time + shift
        offset = stance_offset(foot, travel_angle)
        offset.xy += amplitude * cos(angle) * swing_axis
        offset.z += 0.08 * max(0, -sin(angle))
        goals.add(foot, to_world(offset, time), time)

# Foot goals + velocity + heading → frozen Metatron
03 / Agent-authored skills · G1 in MuJoCo

New tricks.
Same controller.

For these simulated tasks, GPT-6 Astra wrote and tuned the goal-setting code, the timing, and the motion references, with a human directing. The controller was never retrained.

Running → jump goals → running

Four hurdles. One continuous run.

Run at the bar, switch to a jump just before it, and go back to running the moment both feet clear it. Four times in a row, with no reset.

4 × 20 cm hurdles · 5 m apart · zero bar contacts in this run

Goals supplied Concept + pseudocode

Speed, direction, and a torso path guide the run. At 2.6 m from a bar, swap the speed goal for timed torso, hand, and foot targets taken from a recorded jump lined up over the bar. Once both feet clear, go back to running goals.

# Conceptual pseudocode — not a runnable API
for bar in course:  # four bars, 5 m apart
    run(speed=3.3, heading=forward, lane=center)
    when distance_to(bar) <= 2.6:
        track_jump_goals(aligned_over=bar)
    when both_feet_clear(bar, margin=0.12):
        resume_running(ramp_seconds=0.32)

Approaches average 3.07–3.17 m/s; the robot slows during each jump. The bar locations are supplied by task code, not perceived by the policy. Illustrative names; control details abbreviated.

Running → recovery → running

Fall. Get up. Carry on.

The robot runs into a 40 cm bar, falls, gets up, and resumes running.

3.83 m/s before impact · running again 3.28 s after the fall

Goals supplied Concept + pseudocode

Speed and torso goals guide the run; the hands and feet are free. If the torso stays low for 120 ms, the skill switches to one request: stand upright. Once it is, back to running. No obstacle avoidance, no reset, no teleport.

# Conceptual pseudocode — not a runnable API
# Height is measured above the lowest body.
height = torso_height / upright_height
if not recovering:
    recovering = below_for(height, 0.65, seconds=0.12)
elif height >= 0.90:
    recovering = False
if recovering:
    goals = timed_upright_pose(torso, hands, feet)
else:
    goals = track_running_goals(velocity, torso_path, heading)
policy.track(goals)

Illustrative names; control details abbreviated.

Motion reference + timed release

Send the ball down the lane.

A recorded bowling motion guides the swing. After the release it’s all physics: gravity, the ball, the pins.

9 of 10 pins knocked down

Goals supplied Concept + pseudocode

A recorded bowling motion supplies the torso, hand, and foot poses. The scene lets go of the ball at a set moment, and physics takes it from there.

# Conceptual pseudocode — not a runnable API
horizons = [0.1, 0.2, 0.4, 0.6, 0.8]
poses = bowling_reference.sample(future_motion_times)
goals = body_pose_goals(poses, torso + hands + feet,
                        horizons, frames="global + local")
policy.track(goals)
if time >= 2.3:
    scene.release_ball_grasp()  # Physics takes over.

Illustrative names; control details abbreviated.

Motion reference + physical racket

Meet the ball.

A recorded forehand swings the racket into a dropped ball and sends it over the net.

5.37 m/s outgoing ball speed

Goals supplied Concept + pseudocode

A recorded forehand supplies the body poses, and the hand targets carry the racket. The ball drops at a set moment: a timed drill, not vision.

# Conceptual pseudocode — not a runnable API
horizons = [0.1, 0.2, 0.4, 0.6, 0.8]
poses = forehand_reference.sample(future_motion_times)
policy.track(body_pose_goals(
    poses, torso + hands + feet, horizons,
    frames="global + local"))
if time >= 1.2:
    scene.drop_ball()  # Racket contact sets its flight.

Illustrative names; control details abbreviated.

Foot targets + contact physics

Follow through on the kick.

Timed foot goals swing the kicking foot into a football that just sits there. The ball flies because it was hit, not because we scripted it.

2 m approach · 4.82 m/s peak ball speed

Goals supplied Concept + pseudocode

Walk up with speed and torso goals. Near the ball, switch to timed foot targets: wind up, strike, follow through. The standing foot stays planted and the hands stay free.

# Conceptual pseudocode — not a runnable API
if distance_to_ball > 0.7:
    goals = approach_goals(speed=1.5, heading=toward_ball)
else:
    goals = timed_kick(
        support_foot=planted, torso=balanced_stance,
        kicking_foot=[(0.08, backswing), (0.18, strike),
                      (0.36, forward_and_up), (0.98, settle)])
policy.track(goals)  # No force or velocity is applied to the ball.

Illustrative names; control details abbreviated.

04 / The training recipe

Learn from complete motion.
Act on partial goals.

Complete motions teach the coordination. A second stage then hides most of each motion, so Metatron learns to work from whatever goals it is given.

01

Train the tracker

A tracker learns to imitate complete recorded motions, every joint of them.

Reinforcement learning
02

Distill Metatron

Metatron learns from that tracker with most of each motion hidden.

On-policy distillation
03

Deploy the policy.
Supply goals in code.

On-device · C++ skill codeskill(state) → goals
Frozen Metatronactions
No task-specific policy training

After stage 2, Metatron is done learning. Every behavior after that is a skill: code that sets goals, with no more training. On the robot, C++ skill code reads the current state and updates the goals; Metatron turns them into joint targets.

05 / Try it yourself · MuJoCo

Interactive
Web Demo

Steer the G1, change how it moves, or give it somewhere to go. The real controller and the physics run right in your browser.

Launch interactive demo

Desktop recommended
First load downloads and prepares the model. Later visits can reuse cached weights.

NVIDIA Metatron browser demo showing G1 getting up from the floor
06 / More G1 demonstrations

More ways to move.

Real recordings of the G1, all from the one controller. Pick a behavior, then a recording, and see which goals produced it.

G1 / Real robot

Direct the motion,
not every joint.

An Xbox controller sets how fast and which way. Timed foot goals give the walk its rhythm, and the controller handles the rest of the body.

Goals supplied
VelocityHeadingFoot targets
Policy weights unchanged

The skill sets goals. It never touches the joints directly.

All clips in this section show the real G1 at 1× speed; where a safety rig was used, it stays visible. Switching behaviors changes which goals are given, not the controller.

Metatron · The film 2:49

Inside the goal interface

Say only what matters.

Mix any of these goals in one prompt. Metatron fills in the rest of the motion.

Where is the target defined?

From motion data to skill code

Train the controller. Then build skills.

Two stages train the controller. The third writes skills, with the controller frozen.