Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation
Grana is a compositional reward function that measures how much of a long-horizon robot manipulation task is complete. It tracks progress lost after mistakes while preserving credit for completed steps. On real robots, this progress-based reward model supports policy learning through training-time reinforcement learning (RL) and action selection through test-time model predictive control (MPC).
01 · Modeling
Estimating whole-task progress from subtasks.
A failed insertion can erase the progress of the current attempt while leaving an earlier assembly step intact. Grana models these separately: it estimates progress within the unfinished subtask and combines it with credit for completed steps.
Compositional model architecture
Task context
“Assemble an airplane model”
+ subtask descriptions
Grana Subtask Selector
Any subtask selector can be used.
Multi-view observations
VLM θ
+ LoRA
Progress head
fp
Shared across subtasks
Per-timestep features
Local subtask progress
Subtask composition
Global task progress
- Frozen
- Trainable
Task context (assemble an airplane model, with its nine subtask descriptions) and four camera views go into a frozen vision–language model with trainable LoRA. By default, Grana’s own VLM selects the unfinished subtask and passes its description to the model; any other subtask selector can be used. A trainable progress head, shared across subtasks, predicts that subtask’s local progress from 0 to 1. Composition adds one share of credit for every completed subtask: global progress equals k plus local progress, divided by K. The animation replays the outputs of Grana, adapted to airplane assembly, on one selected held-out episode at nine times speed by default, with light smoothing; this episode is not in the paper. Grana moves through all nine subtasks, its local progress restarts at each new subtask, and global progress climbs to 0.99 without losing the credit of completed subtasks.
Constructing progress supervision
We construct dense progress targets from sparse annotations of subtask boundaries, action-quality intervals, and subtask completion. Annotators mark suboptimal and regressed intervals; unmarked intervals are nominal. No frame-by-frame numerical progress scores are needed.
Progress supervision in this work comes from human annotations; our released codebase also supports GPT-assisted annotation for future use.
- Nominal
- Execution follows the task normally.Progress increases.
- Suboptimal
- Degraded execution without physical failure, such as a misaligned screw that remains held.Progress increases more slowly.
- Regressed
- Physical failure loses local progress, such as dropping the screw.Local progress resets to zero.
Then compose: whole-task target = (completed subtasks + local progress) / total subtasks. A local reset keeps credit for earlier steps that remain complete.
Reset local progress. Preserve completed work.
Airplane assembly · Insert the screw · Subtask 7 of 9
After the screw is lost, handling the drill does not restore insertion progress. The target stays at 6/9 until insertion resumes, then reaches 7/9 when the screw is inserted.
Nominal execution · Six completed subtasks retain their credit.
Target-construction details: Paper Sec. 3.2 and Appendix A.
02 · Reusable Knowledge
Learn reusable progress knowledge from shorter tasks.
Long-horizon tasks are harder to collect data for and difficult to anticipate in advance. We therefore learn reusable knowledge about task progress from diverse, mostly short manipulation tasks. This knowledge supports zero-shot prediction on unseen tasks and efficient adaptation to longer tasks with limited demonstrations.
We pretrain the reward model on 200 tasks with subtask descriptions and action-quality annotations, and evaluate zero-shot predictions on 50 held-out tasks. The long assembly tasks and both offline-RL tasks are excluded from pretraining.
03 · Evaluation
Reward model evaluation.
We test whether the progress knowledge learned during pretraining transfers to unseen tasks, longer procedures, and a different robot. We also examine the contribution of composition, the cost of inference, and how predictions respond when execution fails.
Reward prediction accuracy
Zero-shot on unseen tasks
Across the four reward sources shared by these evaluations, lower held-out RMSE corresponds to higher or tied policy success on Star insertion and Tube transfer. Paper Figs. 5–6.
Adapting to long tasks
With the architecture and adaptation data held fixed, reward-model pretraining reduces Grana’s RMSE by 15.8–28.3% across airplane and car assembly at both adaptation budgets. Paper Table 6.
Show the table
| Method | Airplane (9 subtasks) | Car (10 subtasks) | ||
|---|---|---|---|---|
| 15 traj. | 100 traj. | 15 traj. | 100 traj. | |
| RoboMeter | 0.493 | 0.487 | 0.658 | 0.549 |
| SARM | 0.747 | 0.485 | 1.124 | 0.539 |
| SARM2 | 0.498 | 0.461 | 0.629 | 0.472 |
| Grana, no pretraining | 0.382 | 0.311 | 0.502 | 0.445 |
| Grana | 0.308 | 0.262 | 0.394 | 0.319 |
Five long-horizon tasks
GPT6 refers to GPT6-Astra-high throughout this page by 2026/09/15.
Show the table
| Reward model | RMSE on 24–30 unseen test trajectories per task ↓ | ||||
|---|---|---|---|---|---|
| Circuit assembly (up to 3 subtasks) |
Firetruck assembly (up to 9 subtasks) |
Airplane assembly (up to 9 subtasks) |
Red car assembly (up to 10 subtasks) |
Pupgo car assembly (up to 18 subtasks) |
|
| GPT6: Composition | 0.2920 | 0.2914 | 0.4505 | 0.5240 | 1.8135 |
| SARM (D15) | 0.3612 | 0.5406 | 0.7474 | 1.1240 | 0.4042 |
| SARM2 (D15) | 0.2975 | 0.5166 | 0.4980 | 0.6290 | 0.3885 |
| RoboMeter (D15) | 0.1864 | 0.5202 | 0.4930 | 0.6580 | 0.3189 |
| Ours (D15) | 0.1563 | 0.2264 | 0.3080 | 0.3940 | 0.2561 |
Gains from compositional modeling
Show comparison table · Table 9
| Model | Direct prediction | + Composition |
|---|---|---|
| Base Qwen3.5-4B | 4.541 | 3.068 |
| GPT6 | 1.578 | 0.524 |
GPT6 refers to GPT6-Astra-high throughout this page by 2026/09/15.
Show comparison table · Table 8
| Configuration | D15 | D100 |
|---|---|---|
| Direct whole-task | 0.462 | 0.359 |
| Subtask composition (Grana) | 0.382 | 0.311 |
D15 and D100 use 15 and 100 adaptation trajectories, respectively. Within each budget, both predictors use the same base Qwen initialization and matched adaptation data, without reward-model pretraining. Composition uses automatic subtask selection; RMSE includes selection errors.
Transfer to new robots
Inference cost
Inference latency and API fees
Reward scoring is repeated across recorded trajectories and candidate futures, making latency and deployment cost important. Grana’s 4B model runs locally with ≤10 GB GPU memory and no API fee; a complete query takes 0.156 s on one H100, compared with 41.55 s for GPT6 composition through its API.
Grana0.156 s GPT6 composition41.55 s
2 calls per query (GPT6 latency over 8,000 calls)
Grana$0≤10 GB VRAM GPT6 composition≈ $38.08
API fee per trajectory (1 fps, 340 s trajectories)
GPU memory. The 4B model uses at most 10 GB of GPU VRAM in BF16 at batch size 1, with 8 timesteps × 4 camera views = 32 images per call. Using fewer camera views, fewer timesteps, or lower numerical precision reduces VRAM usage further.
Sequential timing. The 155.95 ms (≈ 0.156 s) latency on one NVIDIA H100, an average on HotWheels episodes, sums two model calls executed sequentially at batch size 1: subtask selection and local-progress prediction. Processing independent queries in parallel can improve throughput.
Image workload. A typical complete scoring query uses two model calls, each with 8 sampled timesteps × 4 camera views: 2 × 8 × 4 = 64 image inputs in total (32 per call). This input workload may contribute to GPT6’s longer API latency.
Show the table
| Scoring procedure / hardware | Latency per query | Speedup | GPU VRAM | API fee US$ / trajectory |
|---|---|---|---|---|
| Grana NVIDIA H100 | 155.95 ms | 266.4× | ≤10 GB | No API fee |
| Grana NVIDIA RTX 4090 | 451.06 ms | 92.1× | ≤10 GB | No API fee |
| Grana NVIDIA RTX 5090 | 334.19 ms | 124.3× | ≤10 GB | No API fee |
| Grana NVIDIA GB300 | 123.30 ms | 337.0× | ≤10 GB | No API fee |
| GPT6 composition Total | 29.00 + 12.55 =41.55 s | 1.0× | — | ≈ $38.08 |
| Subtask selection | 29.00 s | — | — | ≈ $20.31 |
| Local-progress prediction | 12.55 s | — | — | ≈ $17.77 |
Speedup is 41.55 s divided by the complete query latency. Dashes indicate unreported API-side memory or speedups not defined for component-only calls.
GPT6 latencies were measured over 8,000 calls; Grana's are averages on HotWheels episodes. For both Grana and GPT6 composition, the per-query latency sums sequential subtask selection and local-progress prediction. Grana was measured on one GPU in BF16 at batch size 1. API costs use 1 fps on trajectories averaging 340 s in the reported experiment; local GPU costs are not priced.
GPT6 refers to GPT6-Astra-high throughout this page by 2026/09/15.
How reward predictions respond to progress regression
A useful progress reward should reflect what a mistake actually undoes. These examples compare whether predictions drop when local progress is lost, retain credit for completed steps that remain intact, and rise again during recovery.
Progress is shown in accumulated subtask units; pink marks annotated failure, and boundary triangles mark predictions outside the plotted range.
04.1 · Application
Train-time RL.
Grana turns changes in predicted progress into learning signals for real robot policies. Its zero-shot rewards improve offline RL on Star insertion and Tube transfer; task-adapted rewards also improve long-horizon assembly through progress-weighted behavioral cloning.
Across four reward sources, lower held-out RMSE corresponds to higher or tied policy success on both offline-RL tasks.
Show reward comparisons and training iterations · Figs. 6–7
04.2 · Application · Test-time MPC
Improve execution without updating the policy.
A policy can get stuck or move on before finishing a step. Grana uses progress estimates to decide when planning is needed and to select among imagined futures. On tractor assembly, Grana-guided MPC improves success from 2/10 to 9/10 across ten initial layouts, without updating the policy.
Show action selection and tractor results · Appendix E
Grana decides when to switch MPC on and off. While MPC is on, it scores batches of four futures predicted by GR00T-2. When predicted progress trends downward for all four candidates, it rejects the batch and requests new futures.
From an accepted batch, the robot executes the candidate with the highest final predicted whole-task progress, composed from the selected subtask and its local progress. It then replans from the new observation.
In tractor assembly, GR00T-2 samples four action chunks per round, each with a predicted 1.6 s video. A Grana variant with a Qwen3.5-0.8B backbone, adapted to this task, scores the futures. With the same GR00T-2 policy run from 10 different initial layouts, success rises from 2/10 without MPC to 9/10 with MPC, without any policy update. Paper Appendix E and Fig. 9.
05 · Citation
Citation.
@misc{grana2026,
title = {{Grana}: Compositional Reward Modeling for Long-Horizon Robot Manipulation},
author = {Cao, Zi-ang and Zheng, Kaiyuan and Xu, Yinzhen and
Duan, Haonan and Gu, Jiasheng and Hu, Fengyuan and
Zhu, Nan and Lei, Kun and Lin, Kevin and Zheng, Ruijie and
Gao, Shenyuan and Zhu, Fangqi and Zhang, Zhe and
Poovendran, Radha and Fan, Linxi and Wang, Guanzhi and Zhu, Yuke},
year = {2026},
url = {https://research.nvidia.com/labs/gear/grana/}
}