RoboTTT: Context Scaling for Robot Policies

Yunfan Jiang^1,2, Yevgen Chebotar¹, Ruijie Zheng¹, Fengyuan Hu¹, Yunhao Ge¹, Jimmy Wu¹, Tianyuan Dai^1,3, Scott Reed¹, Li Fei-Fei^2,†, Yuke Zhu^1,3,†, Linxi “Jim” Fan^1,†

¹NVIDIA²Stanford University³The University of Texas at Austin^†Equal advising

Paper

Abstract

Recent robot foundation models operate with single-step or short-history visuomotor context. We introduce Test-Time-Training Robot Policies (RoboTTT), a robot model and training recipe that scale visuomotor context to 8K timesteps, three orders of magnitude beyond state-of-the-art policies, without growing inference latency. At this context length, we unlock new robot capabilities: one-shot in-context imitation from human video demonstrations, on-the-fly policy improvement, robustness to perturbations, and stronger performance on multi-stage, long-horizon tasks. We also observe, for the first time, steady gains in closed-loop performance as pretraining context length scales. At its core, RoboTTT integrates Test-Time Training into robot foundation models such as Vision-Language-Action policies, yielding a sequence model whose recurrent state consists of fast weights, parameters updated by gradient descent during both training and inference, compressing histories into weight space and retrieving contextual information for long-context conditioning. To scale training context length, the recipe combines sequence action forcing with truncated backpropagation through time. On challenging real-robot manipulation tasks, RoboTTT improves overall performance by 87% over the single-step context baseline and fully completes a five-minute, ten-stage assembly task, which no baseline ever does. RoboTTT trained with 8K-timestep context outperforms the same model pretrained with 1K timesteps by 62%, suggesting context length as a new scaling axis for robot foundation models.

Learned RoboTTT Policies

RoboTTT performs three long-horizon, dexterous bimanual assembly tasks on a YAM setup. Below are full rollouts of the learned policy on each.

Pup Go CarVehicle model assembly · 5 min

Gear BotRemote-controlled robot assembly · 5 min

CircuitUnseen configuration · 1-2 min

RoboTTT Model

What is Test-Time Training?

The TTT loss

The token $x_t$ is projected into a key $k$ and value $v$ , and the self-supervised loss $\ell(W; x_t)$ takes one of two forms:

TTT-KVB (Sun et al., 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden StatesarXiv:2407.04620; Zhang et al., 2025Test-Time Training Done RightarXiv:2505.23884) minimizes a key–value binding loss (e.g. MSE), $\ell(W; x_t)=\lVert f(k; W)-v\rVert^2$ .

TTT-E2E (Tandon et al., 2025End-to-End Test-Time Training for Long ContextarXiv:2512.23675; Behrouz et al., 2025Nested Learning: The Illusion of Deep Learning ArchitecturesarXiv:2512.24695) instead backpropagates end-to-end through the inner loop from the final task loss (e.g. cross-entropy in language modeling).

RoboTTT uses the key–value binding loss (TTT-KVB).

A TTT layer carries a small model inside it. The layer's hidden state is that model's fast weights $W$ ; its update rule is one gradient step on a self-supervised loss $\ell$ — for example, reconstructing the token it just received — and its output comes from the freshly updated model. Learning never stops at test time: the incoming stream is compressed into the weights.

Concretely, each incoming token $x_t$ drives two operations:

Update — one gradient step on the self-supervised loss $\ell$ writes the token into the fast weights,

W_t = W_{t-1} - \eta \, \nabla \ell(W_{t-1}; x_t),

(1)

Apply — the freshly updated model maps the token to the output,

o_t = f(x_t; W_t).

(2)

Because the fast weights are a fixed-size state carried forward in time, the layer conditions on arbitrarily long histories at a cost that stays constant as the context grows — the stream is compressed into weights, not cached.

TTT for Robot Actions

RoboTTT integrates TTT layers into a vision-language-action (VLA) policy while preserving compatibility with the underlying architecture; we instantiate it on GR00T N1.7GR00T N1: An Open Foundation Model for Generalist Humanoid RobotsarXiv:2503.14734. The result is a sequence model whose recurrent state consists of fast weights — updated by gradient descent during both training and inference — that compress the interaction history into weight space and retrieve it for long-context conditioning.

Gating for Preserving Pretrained Capabilities

RoboTTT is initialized from the pretrained VLA, so each inserted TTT layer is added through a learned tanh gate $\alpha \in \mathbb{R}^d$ (initialized near zero) to avoid overwriting what the base model already knows:

o = \tanh(\alpha) \odot o_{\mathrm{TTT}} + o_{\mathrm{attn}},

(3)

where $o_{\mathrm{TTT}}$ is the TTT-layer output (Eq. 2) and $o_{\mathrm{attn}}$ the attention output. At the start of training $\tanh(\alpha) \approx 0$ , so the TTT branch is silent and the pretrained model runs unchanged; the gate then opens only as far as test-time training helps.

Sequence Training

RoboTTT is trained on robot trajectory sequences, so the fast weights update within each training sequence rather than only at test time. This lets the model learn both a good fast-weight initialization $W_0$ and the update dynamics that fill it: the TTT projections $\theta_Q, \theta_K, \theta_V$ and $W_0$ are ordinary learnable parameters, with $W_0$ meta-learned through gradients of gradients (Finn et al., 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep NetworksarXiv:1703.03400; Tandon et al., 2025End-to-End Test-Time Training for Long ContextarXiv:2512.23675). For each sequence we run TTT in the inner loop, evaluate the per-step flow-matching loss $\ell_t$ at every timestep, and optimize the averaged sequence loss:

\mathcal{L}_{\mathrm{fm}}(\xi; W_0) = \frac{1}{T}\sum_{t=1}^{T} \ell_t(\xi_t;\, W_{t-1}),

(4)

where $\xi_t$ bundles the shared language instruction with the image, proprioception, and action chunk at step $t$ , and $W_{t-1}$ is the fast-weight state entering step $t$ — updated to $W_t$ inside the TTT layers (Eq. 1). Each optimization step updates both the regular model weights and the initialization $W_0$ .

Sequence Action Forcing

The action head $v_\theta$ is trained with flow matching. In sequence training we find it necessary to sample the noise level independently for each action chunk — a technique we call sequence action forcing. Sharing one noise level across the whole sequence makes entire sequences uniformly easy or hard to denoise and destabilizes training. With each chunk noised as $A_t^{\tau} = \tau A_t + (1-\tau)\,\epsilon$ (flow-matching level $\tau \in [0, 1]$ , $\epsilon \sim \mathcal{N}(0, I)$ ), the objective is

\mathcal{L}_{\mathrm{fm}}(\xi; W_0) = \frac{1}{T}\sum_{t=1}^{T} \mathbb{E}_{\tau_t,\,\epsilon}\big\lVert v_\theta(\Phi_t, A_t^{\tau_t}, q_t;\, W_{t-1}) - (A_t - \epsilon)\big\rVert^2,

(5)

with each $\tau_t$ drawn independently.

TBPTT

Backpropagating through a whole sequence stores activations for every timestep, so GPU memory grows with the sequence length. RoboTTT instead uses truncated backpropagation through time (TBPTT): the sequence is split into fixed-length segments, and gradients flow only within a segment. Crucially, the fast weights are carried across segment boundaries — so TTT still runs over the entire sequence — while their gradients are detached there. Memory is then set by the segment length rather than the total length, letting training contexts grow arbitrarily long under a fixed memory budget; the initialization $W_0$ still receives gradients through the first segment.

Effective Learning from Context

RoboTTT decouples fast-weight updates from slow-weight updates: masking the flow-matching loss on selected timesteps turns them into pure context — updating the fast weights without supervising an action. Because masking chooses what conditions the policy and what supervises it, two very different conditioning methods fall out of the same recipe.

Imitation from in-context video demonstrations prepends a human video sequence $\xi^{\mathrm{video}}$ — loss masked, fast weights only — to a robot trajectory $\xi^{\mathrm{robot}}$ that carries the action loss. Trained this way, a single human video of an unseen configuration conditions the policy for one-shot imitation at test time.

DAgger Distillation applies the same masking to a correction rollout, but swaps the roles: the robot's own suboptimal actions become the context while the human corrections become the targets. That asymmetry — failures as context, corrections as targets — is precisely how the human's failure-to-correction mapping is distilled into the fast weights: the model learns to produce a correction in response to a failure, rather than imitate corrections in isolation, making fuller use of the same data. We view it as an instantiation of algorithm distillation (Laskin et al., 2023In-context Reinforcement Learning with Algorithm DistillationarXiv:2210.14215) in robotics — the improvement process induced by human interventions becomes the policy's own fast-weight adaptation. At test time the model performs these corrections online, without human intervention; its own corrections then enter the history and are absorbed into the fast weights, exactly as the human corrections were during training.

Evaluation Results

Main Results

We compare RoboTTT against three baselines: GR00T N1.7 (single-step context), GR00T N1.7 Hist. (one extra history frame), and GDN — a Gated DeltaNet (Yang et al., 2024Gated Delta Networks: Improving Mamba2 with Delta RulearXiv:2412.06464) recurrent-memory model with a fixed-size state. RoboTTT consistently outperforms them on dexterous, long-horizon tasks.

Task Completion Score (%)

RoboTTT (Ours)GR00T N1.7GR00T N1.7 Hist.GDN

Figure 1: Task completion score (%) on each task and averaged across the three tasks. RoboTTT (green) scores highest on every task and on average.

Context Scaling

Scaling pretraining context length yields steady gains in closed-loop performance. Pretrained at context lengths from 128 timesteps to 8K, RoboTTT improves steadily with no sign of saturation. GDN shows no such trend.

Task Completion Score (%)

RoboTTT (Ours)GDNSingle-Step ContextShort Context

Figure 2: Task completion score (%) averaged across the three tasks as pretraining context length grows from 128 to 8K timesteps (log-scaled x-axis). RoboTTT improves steadily out to 8K; GDN and the two dashed baselines — best single-step context (GR00T N1.7) and best short context (GR00T N1.7 Hist.) — show no such trend.

New Capabilities

Scaling visuomotor context to 8K timesteps — about five minutes at 30 Hz control, three orders of magnitude beyond state-of-the-art policies — lets RoboTTT condition on far more of its own history. Beyond the stronger closed-loop performance shown above, this unlocks three capabilities that short-context policies lack: one-shot imitation from a single in-context human video, on-the-fly self-improvement, and robustness to physical perturbations.

One-Shot Imitation from In-Context Human Videos

Conditioning on a single in-context human video of an unseen configuration, RoboTTT reproduces the assembly from that one demonstration alone, succeeding where recurrent-memory baselines fail. Each rollout below uses the same prompt (“assemble circuit”); only the in-context video specifies which components go where.

On-the-Fly Improvement

Through DAgger Distillation, RoboTTT learns an implicit correction algorithm and recovers from its own mistakes online, without human intervention.

Perturbation Robustness

When a human removes an installed part mid-episode, RoboTTT conditions on its own rollout to return and reinstall it.

Conclusion

We present RoboTTT, a robot model and training recipe that scale the visuomotor context of robot policies to 8K timesteps. At this context length, RoboTTT unlocks new robot capabilities: one-shot imitation from in-context human video demonstrations, on-the-fly policy improvement, robustness to external perturbations, and stronger closed-loop performance on multi-stage, long-horizon tasks. At its core, RoboTTT integrates Test-Time Training into robot foundation models as the sequence modeling mechanism along the time dimension, and its training recipe, combining sequence action forcing with truncated backpropagation through time, makes training on long sequences tractable. We observe for the first time that scaling pretraining context length yields steady gains in closed-loop performance, suggesting context length as a new scaling axis for robot foundation models.

Acknowledgments

We thank Frederik Ebert, Letian Fu, Abhishek Gupta, Joel Jang, Hanjung Kim, Dantong Niu, Rutav Shah, You Liang Tan, Josiah Wong, Haoyu Xiong, Ruohan Zhang, Tianyuan Zhang, and Chuning Zhu for constructive discussions. We thank Zhe Zhang and Connor Pedersen for compute cluster support. We thank Amy Nguyen, Matin Furutan, Matin Nikoui, Mona Abbas, Lion Park, Ramanpreet Singh, Alaa Eltayeb, Dona Alhabibi, Marcelo Kulik, Nachiket Timmanagoudar, and Josiah Minor for real-robot operation, data collection, and evaluation. We thank Tri Cao and Yuqi Xie for help with the release. Last but not least, we thank the NVIDIA GEAR Team and the Stanford Vision and Learning Lab for their continuous support.