Technical report
CodeDecoding types
are rich.
Typical outputs span both text, such as thinking and sub-task planning, and visual primitives including boxes, points, depth, and 2D/3D trajectories for manipulation and driving.
VFlashEnabling System-1 VLMs
for Physical AI.
Perception Visual plans Action
A generalist decoding framework for fast, accurate visual planning. Decode complete primitives in parallel for faster robot and driving inference.
System-1 visual planning for Physical AI.
Embodied models do not necessarily think in text; they can also think in visual plans. Predicting where to grasp, how to move, and how a scene will evolve bridges perception and action for robots and autonomous vehicles. However, agents must continually update these plans, and autoregressive generation of coordinate tokens makes detailed visual plans increasingly expensive to generate.
We introduce VFlash, a generalist decoding framework for Physical AI that adapts VLM speculative decoding around a central insight: the known structure of visual primitives can be exploited to achieve spatial accuracy without reproducing the autoregressive target’s exact tokens. A lightweight drafter trained on annotated geometry can generate complete visual primitives in a parallel draft and achieve more accurate robot trajectories than its autoregressive target.
01 · Physical AI tasks
Decoding for Physical AI.
Applying speculative decoding with a fixed block size is not optimal for Physical AI. Generalist decoding must accommodate rich output types, known structures, and lengths specified on demand.
Decoding types
Decoding outputs
are structured.
Points, bounding boxes, and trajectories have known geometric structures. This structure lets blocks cover entire primitives while assigning coordinate roles to token positions.
Decoding requirements
are versatile.
A generalist decoder must support variable output lengths. NAVSIM requires 8 waypoints over 4 seconds, Waymo 20 over 5 seconds, and Alpamayo 64 over 6.4 seconds.
VFlash supports rich Physical AI outputs with systematic parallel decoding designs. A generic token window does not capture where one such primitive begins or ends.
Flash Primitive Decoding.
A tool call declares the output length, while the primitive type specifies the coordinate layout. Using frozen VLM context, a lightweight drafter predicts the entire primitive in one parallel pass. The target VLM then processes the plan to update its context and extract features used for downstream action.
→→→
INSTRUCTION
Plan the gripper’s next motion.
Target VLMfrozen
Visual + language contextREQUESTED LAYOUT10 points · 3D
32 positionsVFlashOne parallel drafter pass
SUB-TASK PLAN
Finding 1
Boundary alignment matters.
Training and decoding blocks should match complete primitive boundaries, allowing the drafter to learn and predict the full geometry together rather than fragments split by fixed token windows. VFlash introduces a structured tool call to declare the length of a variable-size primitive; its type determines the coordinate layout. Flash Primitive Decoding allocates a block for the complete geometry and fills the entire primitive in one parallel draft conditioned on frozen VLM features.
Finding 2
Position alignment matters.
Each output position needs a known role before parallel prediction begins. Different coordinate values can occupy different numbers of text tokens and shift later positions. We use dedicated coordinate tokens or fixed-width numeric fields to keep waypoint, axis, and delimiter positions consistent. This consistent layout supports parallel prediction with constraints applied to each position.
Finding 3
Learning from geometry.
The target VLM remains frozen while we train a lightweight drafter on annotated geometry. The drafter has five non-causal transformer layers and is trained with token cross-entropy on annotated primitives. It predicts complete visual primitives in parallel without requiring token-level agreement. Sub-task text is decoded speculatively and verified by the target VLM.
02 · Beyond the AR target
A drafter that surpasses
its autoregressive target.
VFlash reduces all six trajectory error metrics across ShareRobot-Bench and VABench-V relative to autoregressive (AR) decoding. Across three robot datasets and three target variants, mean 3D displacement error (ADE) decreases by 42.6% relative to the autoregressive target.
Trajectory evaluation
Training variant
Metric
AR targetVFlash
03 · Generalist decoding
Principled parallelism across visual tasks.
VFlash predicts complete 2D/3D trajectories in parallel. Matching training and decoding blocks to primitive boundaries lets the drafter learn and predict the full geometry together. Across all evaluated 2D/3D robot settings, trajectory errors decrease, including average 3D waypoint error.
With frozen MolmoMotion-AR, VFlash reduces DAVIS average displacement error by 27% and achieves comparable overall quality on dense 3D motion forecasting. Single-rollout generation is approximately 25× faster on H100 at batch size one. VFlash drafts each future frame of query points as one block.
On COCO and Dense200, VFlash delivers 2.6–2.8× more boxes per second, with mean F1 close to the autoregressive target. Flash Primitive Decoding predicts all coordinates of a box in a single parallel draft. Each block covers one box, and subsequent blocks handle additional objects.
The same drafter retains verified decoding for sub-task text. Text uses 16-position blocks: one accepted anchor and 15 proposals. The target verifies the block in one pass and keeps the longest matching prefix plus its own next token. Both methods produce an identical sub-task description for the example shown.
04 · Robot manipulation
Visual plans improve downstream control.
The downstream world action models and diffusion action heads differ in architecture, but share a common design: they cross-attend to VLM features encoding the generated visual plans. We obtain these features by passing VFlash’s predicted primitives through the target VLM. The action models are trained for robot manipulation or driving, while the VLM remains frozen.
Images + instruction→VLM + VFlash→Visual-plan features→Action policy
MolmoSpaces · world action model
The plan’s geometry matters.
62.6%
Baseline
61.0%
AR plans
63.7%
VFlash · w/o visual plan
69.3%
VFlash · 3D-only
70.2%
VFlash
The robot policy is a world action model (WAM) obtained by post-training Cosmos-Predict2.5 for control. It keeps its camera and robot-state inputs and receives the frozen VLM’s question, sub-task, and trajectory features through its existing cross-attention pathway. The WAM predicts 20-step action chunks and replans every three chunks. VFlash reaches 70.2% average success versus 62.6% for the baseline.
With the same WAM architecture, autoregressive 2D+3D plans yield 61.0% success versus VFlash’s 70.2%. Success falls to 63.7% when image features replace trajectory features while retaining the generated sub-task, demonstrating the value of visual geometry beyond language.
Explore MolmoSpaces latency ↓Four-suite results and comparison setup
| Configuration | VLM context | Decoder | Primitive | Avg. (%) ↑ | Success rate by suite (%) ↑ | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Image | Question | Sub-task | Trajectory | Pick | Pick & Place | Open | Close | ||||
| Original baseline | Original text encoder | — | — | 62.6 | 81.1 | 55.7 | 49.3 | 64.2 | |||
| AR plans | — | ✓ | ✓ | ✓ | AR | 2D + 3D | 61.0 | 81.2 | 50.4 | 29.8 | 82.4 |
| VFlash · w/o visual plan | ✓ | ✓ | ✓ | — | VFlash | — | 63.7 | 80.2 | 52.5 | 44.3 | 77.8 |
| 3D-only plans | — | ✓ | ✓ | ✓ | VFlash | 3D | 69.3 | 81.7 | 73.0 | 48.3 | 74.2 |
| VFlash | — | ✓ | ✓ | ✓ | VFlash | 2D + 3D | 70.2 | 83.6 | 67.3 | 52.8 | 77.0 |
Checkmarks identify which VLM states condition the WAM. All variants use the same WAM architecture and training budget. Success is averaged equally across the four suites.
LIBERO family · diffusion action head
Gains remain under perturbations.
A PIv3 action head reads frozen VLM features through cross-attention. The baseline encodes images and instructions using Qwen3-VL-4B. Our VLM supplies states through the predicted sub-task or full plan, predicting a sub-task and ten-waypoint 2D+3D trajectory from primary-camera and wrist-camera images before each eight-action chunk.
05 · Latency and deployment
Accelerating Visual Planning across Physical AI
Physical AI is latency-sensitive. VFlash pairs a target VLM with a compact drafter. The runtime manages drafting, verification where required, KV-cache updates, and downstream feature extraction. Measurements use customized TensorRT-Edge-LLM runtimes and TensorRT engines on Jetson Thor, DGX Spark, and H100.
Efficient visual planning for autonomous driving.
VFlash goes beyond robot manipulation. With a frozen planning VLM and a ReCogDrive-derived action head, it achieves competitive driving performance on NAVSIM and Waymo while substantially reducing decoding latency. The same variable-length decoder also supports the trajectories used by Alpamayo.
| Method | NAVSIM | Waymo | ||
|---|---|---|---|---|
| PDMS ↑ | ADE3s (m) ↓ | ADE5s (m) ↓ | RFS ↑ | |
| ReCogDrive-RL | 89.6 | - | - | - |
| ExploreVLA-RL | 90.4 | - | - | - |
| Fast-dDrive | - | 1.25 | 2.91 | 7.82 |
| MindVLA-U1-RL | - | 1.09 | 2.66 | 7.87 |
| Qwen-Drive-1.0-RL | 90.7 | 1.19 | 2.67 | 7.91 |
| VFlash-RL | 90.7 | 1.22 | 2.70 | 7.86 |
12×faster decoding, robot planning
19–42× driving (NAVSIM, Waymo)
12×faster decoding, robot planning
22–49× driving (NAVSIM, Waymo)
11×faster decoding, robot planning
11–24× driving (NAVSIM, Waymo)
Removing the bottleneck.
Autoregressive decoding accounts for 75–87% of measured model-stage latency on Thor. VFlash reduces this stage by 12–42×, yielding 5× faster robot inference and over 3× faster NAVSIM driving inference on Jetson Thor and DGX Spark.
Preprocess + visionPrefillDecodingAction model
AR3,552.7 ms
DFlash1,828.4 ms
VFlash703.3 ms
MODEL-PIPELINE SPEEDUP5.1×
vs. autoregressive decoding
2.6× vs. DFlashDecoding: 3,107.6 → 258.2 ms.
Measurements use batch size one and process one request at a time. CUDA events measure model-stage latency, including target feature preparation. Speedups compare AR and VFlash on the same task and hardware platform.
- TensorRT-Edge-LLM + TensorRT
- Per-GPU C++ runtimes use FP16 VLM engines and KV caches with no weight or KV quantization.
- CuTe DSL FMHA + XQA
- FMHA runs vision and prefill; XQA runs decoding and verification, supported by cache and position plugins.
- CUDA-optimized action models
- The driving head uses a BF16 TensorRT engine; WAM runs head-dimension-128 CuTe attention on Thor and Spark.
Implementation settings and latency measurement
Precision and hardware. Batch size one, one request at a time. Vision, target, and draft engines use FP16 with FP16 KV caches, without weight or KV quantization. Action-model precision is configured separately: DiT weights are BF16, with FP16 attention in the optimized robot WAM.
| Platform | Architecture | CUDA | TensorRT | Customized Edge-LLM base |
|---|---|---|---|---|
| Jetson AGX Thor | SM110 | 13.0 | 10.15.2.7 | 0.9.0 |
| DGX Spark | GB10 / SM121 | 13.0.88 | 10.14.1.48 | 0.10.1 |
| H100 80 GB | SM90 | 12.9.86 | 10.11.0.33 | 0.10.1 robotics; 0.9.0 + SM90 attention bridge for driving |
Primitive and feature passes. VFlash uses verified 16-position blocks for text, then drafts a whole visual primitive. One cached target prefill over the accepted primitive produces downstream features and is included in decoding time. AR, DFlash, and VFlash receive the same supplied tool-call JSON in the measured pipelines.
Action-model optimizations. On Thor and Spark, the WAM uses a custom head-dimension-128 CuTe DSL attention kernel; the H100 baseline uses native TensorRT attention. Driving latency is measured with ten DDIM steps to match the comparison planner. The paper evaluates driving quality with five DDIM steps.
Timing boundary. Each request begins with fresh vision computation and model state. CUDA events measure the instrumented stages, including host launch gaps. Model loading, engine building, graph capture, warmup, file loading, CPU preprocessing outside the measured modules, and output detokenization are excluded. Totals sum the stage means and report model-pipeline latency rather than full sensor-to-actuator latency.
Sampling. Thor robotics uses 24 records and ten passes after five warmups; Spark and H100 robotics use one pass over the same 24 records. Driving uses five warmups followed by ten passes over five records.
Team
1NVIDIA · 2Johns Hopkins University · 3Hong Kong Polytechnic University · 4Purdue University · 5The University of Hong Kong
* Equal contribution. † Work done during an internship at NVIDIA. ‡ Corresponding author.
BibTeX
@misc{chou2026vflash,
title = {{VFlash}: Fast, Accurate Visual Planning via Flash Primitive Decoding},
author = {Chou, Yu-Cheng and Wang, Shihao and Huang, Jin and Huang, Chi-Pin and Yang, Chiao-An and Fan, Xianzhe and Wang, Jing and Li, Zhiqi and Kuang, Yuanguo and Zheng, Ruijie and Bjorck, Johan and Liu, Sifei and Su, Hang and Badki, Abhishek and Zhang, Lei and Yuille, Alan and Fan, Linxi and Kautz, Jan and Yu, Zhiding},
year = {2026},
url = {https://research.nvidia.com/labs/lpr/vflash/}
}