Universal LiDAR tokenizer
A fully attention-based range-image tokenizer preserves sparse geometry and native temporal resolution in a shared latent space across scanning patterns.
Spatial Intelligence Lab
Into the LidarVerse
A Real-Time Unified LiDAR World Model Across Tasks and Sensors.
Overview
From new-world generation to reconstruction-grounded resimulation
LidarVerse brings generative and reconstruction-based LiDAR simulation into one conditional world model. The inference regime is determined by the available scene evidence, not by a separate architecture or prediction head.
Scene-level conditions guide new-world generation; an observed LiDAR prefix turns the same model into a point-cloud forecaster; and sparse observed sweeps form a persistent spatial cache that anchors generalizable resimulation without per-scene optimization. The distilled causal checkpoint runs at 10 FPS on a single H100, while the universal LiDAR tokenizer supports 32-, 64-, and 128-beam sensors.
System
One system, from compression to real-time interaction
Unified bidirectional teacher
Initialized from Cosmos-Transfer2.5-2B and adapted to LiDAR latents, a flow-matching DiT learns one interface for the LiDAR prefix, HD-Map, spatial cache, and text.
Four-step causal student
Teacher forcing initializes a block-autoregressive student, and self-forcing distribution matching enables four-step generation of each four-frame block.
Temporal and spatial memory
A rolling nine-frame K/V cache retains recent dynamics, while a world-aligned persistent spatial cache preserves confirmed geometry beyond that temporal window.
Interactive Action-Conditioned Generation
Choose a recorded scene and an ego action to generate a LiDAR rollout that responds to the requested action.
Full HD-Map and ego action
Log
Action
Loading video…
0:00 / 0:00
One causal checkpoint, three LiDAR simulation regimes
Changing only the available scene evidence moves the same real-time model from generation to forecasting to reconstruction-grounded resimulation.
With no observed scene geometry, scene-level conditions guide new-world generation while a rolling spatial cache is built from generated sweeps.
Input · 3s
Forecast · 3s
Loading video…
01 / 03
Controllable generation
Generate temporally consistent LiDAR from HD-Map and actor-layout conditions.
0:00 / 0:00
Across Sensors and Datasets
One HD-Map, three LiDAR configurations
After joint adaptation, one model generates the distinct scanning patterns of 32-beam nuScenes, 64-beam Waymo, and 128-beam NVIDIA PAI from the same Waymo HD-Map.
0:00 / 0:00
Quantitative Results
Quantitative evaluation across LiDAR simulation regimes
We evaluate the shared tokenizer, then the same causal checkpoint across autoregressive generation, point-cloud forecasting, and generalizable resimulation.
LiDAR video tokenizer comparison
MMD ×10⁵ · ↓ lower is better
Tokenizer architecture ablation
Autoregressive LiDAR generation
HD-Map-conditioned and interactive action-conditioned generation · MMD ×10⁵
Conditional point-cloud forecasting
30 observed frames at 10 Hz → 30 future frames at 10 Hz
Generalizable LiDAR resimulation
Novel-view interpolation and extrapolation without per-scene optimization
Tokenizer
Representative metric comparison
Each plot uses its own zero-based y-axis. Hover or focus a mark to inspect its values.