Foundation4D

Towards a unified perception and reconstruction model for autonomous vehicles

NVIDIA

White Paper Dataset

TL;DR

Foundation4D turns multi-camera driving video into geometry, 3D objects, semantic occupancy, and Gaussian reconstructions. A shared backbone combines information across cameras and time, while video processing refines camera trajectories and links objects into tracks. We also release the PAI–Foundation4D benchmark, with dense depth and occupancy annotations for 1,253 driving clips, and use predicted geometry and trajectories as priors for LiDAR-free NuRec.

Method

Per-chunk inference

Multi-camera image windows enter a shared encoder and factorized attention; task heads are illustrated with actual camera pose, geometry, Gaussian, detection and occupancy examples

Video processing

Three views of video processing: predict overlapping windows; align their shared F08 observation with Sim(3) window stitching; or use relative-pose constraints from the predicted windows for global SE(3) pose graph optimization. PGO is an alternative assembly path from raw window poses. Detections and the assembled ego trajectory are transformed into a shared world frame for offline object tracking.

Quantitative results

White Paper ↗

Leading results among the methods compared in the white paper, with models adapted to each benchmark.

Depth estimation · Waymo & nuScenes

MethodWaymonuScenes
AbsRel ↓δ₁.₂₅ ↑AbsRel ↓δ₁.₂₅ ↑
DVGT-10.2072.870.1878.97
DVGT-20.2670.930.2275.32
LiAuto-GeoX0.1975.720.2762.75
GeoUP0.1387.300.1980.05
Foundation4D0.0991.970.1586.44

48-frame windows; scale alignment per window. δ₁.₂₅ is reported in %. Training data and geometric inputs differ across methods. Table 2 ↗

Camera trajectories · Waymo & nuScenes

Full-video evaluation · White paper, Table 3 ↗
DatasetAssemblyMethodATE ↓ (cm)RTE ↓ (cm)RRE ↓ (°)
Waymo · Stitching
WaymoStitchingDVGT-162.333.540.06
Foundation4D32.703.200.06
Waymo · PGO+LM
WaymoPGO+LMDVGT-154.552.380.05
Foundation4D23.621.560.06
nuScenes · Stitching
nuScenesStitchingDVGT-131.023.980.07
Foundation4D22.132.150.05
nuScenes · PGO+LM
nuScenesPGO+LMDVGT-130.541.240.07
Foundation4D17.481.190.05

Both models use 48-frame windows with 5-frame overlap and the same video assembly system on official validation splits. Evaluation fits one Sim(3) alignment per complete clip; the depth table aligns each window separately.

ATE: absolute trajectory error; RTE/RRE: relative translation/rotation error. PGO+LM includes a linear-motion prior. Bold marks the lowest error within each dataset and assembly pair, including ties.

Precise Dense 3D Annotation Pipeline

Dataset ↗

PAI–Foundation4D extends NVIDIA’s Physical AI Autonomous Vehicles dataset with dense depth and semantic occupancy annotations, including per-voxel velocities, for 1,253 driving clips.

Actual PAI recording inputs: RGB images, measured LiDAR scans, ego poses and cuboid tracks. LiDAR aggregation feeds depth rendering and semantic voxelization, illustrated with Spectral metric depth and semantic occupancy annotations.

Application to LiDAR-free NuRec

Foundation4D supplies camera trajectories, dense geometry and actor tracks for NuRec. With multi-camera video and GPS/IMU trajectories, NuRec optimizes a dynamic Gaussian scene for novel-view rendering without LiDAR.

Bottom row: LiDAR-free NuRec reconstruction