fVDB: A Deep-Learning Framework for Sparse, Large-Scale, and High-Performance Spatial Intelligence

fVDB : Large-Scale and High Performance Spatial Intelligence

fVDB is a Python and C++ deep learning framework for building spatial intelligence applications on large-scale, and high resolution 3D data built on top of PyTorch.
download Install description Documentation code Code article Paper

fVDB Highlights

What is fVDB?

A unified deep learning framework for volumetric data. fVDB exposes a large suite of composable differentiable operators which allow users to build spatial intelligence applications such as generative AI, radiance fields, simulation, and 3D reconstruction. Under the hood, fVDB builds upon and extends NanoVDB, a best-in-class GPU spatial acceleration data structure. This allows fVDB to operate on large-scale, sparse, and high-resolution 3D data with maximum efficiency.
fVDB differentiable operators. Here's a summary of the operators that fVDB currently supports. These include operators to load a wide variety of 3D data in many formats (points, meshes, voxels, etc.), layers to build sparse 3D neural networks, primitives for volume rendering, ray tracing, and Gaussian splatting, just to name a few. This is only the beginning and we plan on adding many more!

Why fVDB?

Volumetric data is everywhere The amount of volumetric data collected is growing as industry continues to deploy robots, autonous vehicles, and industrial automation solutions. With this growth in data, there is a parallel growth in demand to operationalize this data with spatial intelligence, i.e. machine learning algorithms which can reason about the physical world. To satisfy this demand, deep learning on volumetric data has grown into a mainstream area of academic research.


Deep learning on volumetric data is hard. However, unlike deep learning on images and text, volumetric data is often too big to fit in ram, sparse, and non uniform (different inputs have different sizes and can't be stacked into a tensor easily). In practice, volumetric data collected in the real world is very large in spatial scale and resolution.

Lack of tooling for 3D Deep Learning: Current deep learning pipelines for volumetric data are implemented by duct taping together a suite of small academic libraries that have incompatible APIs, and sub-optimal performance. The result is that algorithms developed by researchers can't be deployed on real world data.

fVDB Use-Cases

Surface Reconstruction from Sparse Points: fVDB is used by NKSR to reconstruct 3D meshes from oriented point clouds.


3D Generative AI: fVDB is used by XCube to build a generative model for 3D shapes and scenes. The objects in the video below are all AI generated.

Generative 3D Scene Reconstruction: fVDB is used by SCcube to reconstruct millions of Gaussians with a range of 102.4m x 102.4m in 20 seconds from sparse views (only 3 images).

Unbounded driving scene generation: fVDB is used by InfiniCube to generate unbounded dynamic driving scenes from world models.

Accelerating city-scale radiance fields: fVDB is used by NeRF-XL to raytrace city-scale radiance fields accross multiple GPUs.