Scroll

Grana: Compositional Reward Modeling for Long-Horizon Robot Manipulation

Grana is a compositional reward function that measures how much of a long-horizon robot manipulation task is complete. It tracks progress lost after mistakes while preserving credit for completed steps. On real robots, this progress-based reward model supports policy learning through training-time reinforcement learning (RL) and action selection through test-time model predictive control (MPC).

  1. Zi-ang Cao1,2,*
  2. Kaiyuan Zheng1,3,*
  3. Yinzhen Xu1
  4. Haonan Duan1
  5. Jiasheng Gu1
  6. Fengyuan Hu1
  7. Nan Zhu1
  8. Kun Lei1
  9. Kevin Lin1
  10. Ruijie Zheng1
  11. Shenyuan Gao1
  12. Fangqi Zhu1
  13. Zhe Zhang1
  14. Radha Poovendran3
  15. Linxi Fan1,†
  16. Guanzhi Wang1,†
  17. Yuke Zhu1,2,†
  • 1NVIDIA
  • 2The University of Texas at Austin
  • 3University of Washington

*Equal contribution†Equal advising

01 · Modeling

Estimating whole-task progress from subtasks.

A failed insertion can erase the progress of the current attempt while leaving an earlier assembly step intact. Grana models these separately: it estimates progress within the unfinished subtask and combines it with credit for completed steps.

Compositional model architecture

Task context

“Assemble an airplane model”

+ subtask descriptions

Grana Subtask Selector

Any subtask selector can be used.

Multi-view observations

Top
Bottom
Left wristLeft
Right wristRight

VLM θ

+ LoRA

Progress head

fp

Shared across subtasks

Per-timestep features

Local subtask progress

Subtask composition

Global task progress

  • Frozen
  • Trainable
Architecture overview: task and subtask descriptions plus multi-view observations feed a shared vision-language model; subtask selection and local progress are composed into whole-task progress.

Task context (assemble an airplane model, with its nine subtask descriptions) and four camera views go into a frozen vision–language model with trainable LoRA. By default, Grana’s own VLM selects the unfinished subtask and passes its description to the model; any other subtask selector can be used. A trainable progress head, shared across subtasks, predicts that subtask’s local progress from 0 to 1. Composition adds one share of credit for every completed subtask: global progress equals k plus local progress, divided by K. The animation replays the outputs of Grana, adapted to airplane assembly, on one selected held-out episode at nine times speed by default, with light smoothing; this episode is not in the paper. Grana moves through all nine subtasks, its local progress restarts at each new subtask, and global progress climbs to 0.99 without losing the credit of completed subtasks.

From local to global task progress. By default, Grana’s own vision–language model selects the unfinished subtask; any other subtask selector can be used. The shared model estimates progress for the selected description, and composition adds credit for completed subtasks. Playback: 9× real time. Grana is adapted to airplane assembly; this replay uses one held-out episode (K = 9) with lightly smoothed scores and is not included in the paper.

Constructing progress supervision

We construct dense progress targets from sparse annotations of subtask boundaries, action-quality intervals, and subtask completion. Annotators mark suboptimal and regressed intervals; unmarked intervals are nominal. No frame-by-frame numerical progress scores are needed.

Progress supervision in this work comes from human annotations; our released codebase also supports GPT-assisted annotation for future use.

Nominal
Execution follows the task normally.Progress increases.
Suboptimal
Degraded execution without physical failure, such as a misaligned screw that remains held.Progress increases more slowly.
Regressed
Physical failure loses local progress, such as dropping the screw.Local progress resets to zero.

Then compose: whole-task target = (completed subtasks + local progress) / total subtasks. A local reset keeps credit for earlier steps that remain complete.

Reset local progress. Preserve completed work.

Airplane assembly · Insert the screw · Subtask 7 of 9

After the screw is lost, handling the drill does not restore insertion progress. The target stays at 6/9 until insertion resumes, then reaches 7/9 when the screw is inserted.

Constructed target Completed credit Regressed
0:00.0 / 0:37.2 4× playback

Nominal execution · Six completed subtasks retain their credit.

Constructed supervision, not a model prediction. This example contains nominal and regressed intervals; suboptimal execution follows the slower-growth rule above. Paper Fig. 8.

Target-construction details: Paper Sec. 3.2 and Appendix A.

02 · Reusable Knowledge

Learn reusable progress knowledge from shorter tasks.

Long-horizon tasks are harder to collect data for and difficult to anticipate in advance. We therefore learn reusable knowledge about task progress from diverse, mostly short manipulation tasks. This knowledge supports zero-shot prediction on unseen tasks and efficient adaptation to longer tasks with limited demonstrations.

We pretrain the reward model on 200 tasks with subtask descriptions and action-quality annotations, and evaluate zero-shot predictions on 50 held-out tasks. The long assembly tasks and both offline-RL tasks are excluded from pretraining.

03 · Evaluation

Reward model evaluation.

We test whether the progress knowledge learned during pretraining transfers to unseen tasks, longer procedures, and a different robot. We also examine the contribution of composition, the cost of inference, and how predictions respond when execution fails.

Reward prediction accuracy

Zero-shot on unseen tasks

Across the four reward sources shared by these evaluations, lower held-out RMSE corresponds to higher or tied policy success on Star insertion and Tube transfer. Paper Figs. 5–6.

Adapting to long tasks

With the architecture and adaptation data held fixed, reward-model pretraining reduces Grana’s RMSE by 15.8–28.3% across airplane and car assembly at both adaptation budgets. Paper Table 6.

Show the table
Long-task adaptation · RMSE in subtask units ↓ · paper Table 4
MethodAirplane (9 subtasks)Car (10 subtasks)
15 traj.100 traj.15 traj.100 traj.
RoboMeter0.4930.4870.6580.549
SARM0.7470.4851.1240.539
SARM20.4980.4610.6290.472
Grana, no pretraining0.3820.3110.5020.445
Grana0.3080.2620.3940.319

Five long-horizon tasks

GPT6 refers to GPT6-Astra-high throughout this page by 2026/09/15.

Show the table
Reward model RMSE on 24–30 unseen test trajectories per task ↓
Circuit
assembly (up to 3 subtasks)
Firetruck
assembly (up to 9 subtasks)
Airplane
assembly (up to 9 subtasks)
Red car
assembly (up to 10 subtasks)
Pupgo car
assembly (up to 18 subtasks)
GPT6: Composition0.29200.29140.45050.52401.8135
SARM (D15)0.36120.54060.74741.12400.4042
SARM2 (D15)0.29750.51660.49800.62900.3885
RoboMeter (D15)0.18640.52020.49300.65800.3189
Ours (D15)0.15630.22640.30800.39400.2561
Ours (D15) Best in column In subtask units; lower is better.

Gains from compositional modeling

Show comparison table · Table 9
Zero-shot car assembly · RMSE in subtask units ↓ · selected rows from Table 9
ModelDirect prediction+ Composition
Base Qwen3.5-4B4.5413.068
GPT61.5780.524

GPT6 refers to GPT6-Astra-high throughout this page by 2026/09/15.

Show comparison table · Table 8
Airplane assembly · RMSE in subtask units ↓ · Table 8
ConfigurationD15D100
Direct whole-task0.4620.359
Subtask composition (Grana)0.3820.311

D15 and D100 use 15 and 100 adaptation trajectories, respectively. Within each budget, both predictors use the same base Qwen initialization and matched adaptation data, without reward-model pretraining. Composition uses automatic subtask selection; RMSE includes selection errors.

Transfer to new robots

Inference cost

Inference latency and API fees

Reward scoring is repeated across recorded trajectories and candidate futures, making latency and deployment cost important. Grana’s 4B model runs locally with ≤10 GB GPU memory and no API fee; a complete query takes 0.156 s on one H100, compared with 41.55 s for GPT6 composition through its API.

Grana0.156 s GPT6 composition41.55 s

2 calls per query (GPT6 latency over 8,000 calls)

Grana$0≤10 GB VRAM GPT6 composition≈ $38.08

API fee per trajectory (1 fps, 340 s trajectories)

GPU memory. The 4B model uses at most 10 GB of GPU VRAM in BF16 at batch size 1, with 8 timesteps × 4 camera views = 32 images per call. Using fewer camera views, fewer timesteps, or lower numerical precision reduces VRAM usage further.

Sequential timing. The 155.95 ms (≈ 0.156 s) latency on one NVIDIA H100, an average on HotWheels episodes, sums two model calls executed sequentially at batch size 1: subtask selection and local-progress prediction. Processing independent queries in parallel can improve throughput.

Image workload. A typical complete scoring query uses two model calls, each with 8 sampled timesteps × 4 camera views: 2 × 8 × 4 = 64 image inputs in total (32 per call). This input workload may contribute to GPT6’s longer API latency.

Show the table
Inference latency, speedup, GPU memory, and API fees
Scoring procedure / hardware Latency per query Speedup GPU VRAM API fee US$ / trajectory
Grana NVIDIA H100155.95 ms266.4×≤10 GBNo API fee
Grana NVIDIA RTX 4090451.06 ms92.1×≤10 GBNo API fee
Grana NVIDIA RTX 5090334.19 ms124.3×≤10 GBNo API fee
Grana NVIDIA GB300123.30 ms337.0×≤10 GBNo API fee
GPT6 composition Total29.00 + 12.55 =41.55 s1.0×—≈ $38.08
Subtask selection29.00 s——≈ $20.31
Local-progress prediction12.55 s——≈ $17.77

Speedup is 41.55 s divided by the complete query latency. Dashes indicate unreported API-side memory or speedups not defined for component-only calls.

GPT6 latencies were measured over 8,000 calls; Grana's are averages on HotWheels episodes. For both Grana and GPT6 composition, the per-query latency sums sequential subtask selection and local-progress prediction. Grana was measured on one GPU in BF16 at batch size 1. API costs use 1 fps on trajectories averaging 340 s in the reported experiment; local GPU costs are not priced.

GPT6 refers to GPT6-Astra-high throughout this page by 2026/09/15.

How reward predictions respond to progress regression

A useful progress reward should reflect what a mistake actually undoes. These examples compare whether predictions drop when local progress is lost, retain credit for completed steps that remain intact, and rise again during recovery.

Progress is shown in accumulated subtask units; pink marks annotated failure, and boundary triangles mark predictions outside the plotted range.

04.1 · Application

Train-time RL.

Grana turns changes in predicted progress into learning signals for real robot policies. Its zero-shot rewards improve offline RL on Star insertion and Tube transfer; task-adapted rewards also improve long-horizon assembly through progress-weighted behavioral cloning.

Across four reward sources, lower held-out RMSE corresponds to higher or tied policy success on both offline-RL tasks.

Show reward comparisons and training iterations · Figs. 6–7

04.2 · Application · Test-time MPC

Improve execution without updating the policy.

A policy can get stuck or move on before finishing a step. Grana uses progress estimates to decide when planning is needed and to select among imagined futures. On tractor assembly, Grana-guided MPC improves success from 2/10 to 9/10 across ten initial layouts, without updating the policy.

Show action selection and tractor results · Appendix E

Grana decides when to switch MPC on and off. While MPC is on, it scores batches of four futures predicted by GR00T-2. When predicted progress trends downward for all four candidates, it rejects the batch and requests new futures.

From an accepted batch, the robot executes the candidate with the highest final predicted whole-task progress, composed from the selected subtask and its local progress. It then replans from the new observation.

In tractor assembly, GR00T-2 samples four action chunks per round, each with a predicted 1.6 s video. A Grana variant with a Qwen3.5-0.8B backbone, adapted to this task, scores the futures. With the same GR00T-2 policy run from 10 different initial layouts, success rises from 2/10 without MPC to 9/10 with MPC, without any policy update. Paper Appendix E and Fig. 9.

05 · Citation

Citation.

@misc{grana2026,
  title     = {{Grana}: Compositional Reward Modeling for Long-Horizon Robot Manipulation},
  author    = {Cao, Zi-ang and Zheng, Kaiyuan and Xu, Yinzhen and
               Duan, Haonan and Gu, Jiasheng and Hu, Fengyuan and
               Zhu, Nan and Lei, Kun and Lin, Kevin and Zheng, Ruijie and
               Gao, Shenyuan and Zhu, Fangqi and Zhang, Zhe and
               Poovendran, Radha and Fan, Linxi and Wang, Guanzhi and Zhu, Yuke},
  year      = {2026},
  url       = {https://research.nvidia.com/labs/gear/grana/}
}