3D/4D Content Creation & Editing

Overview

We build systems for dynamic 3D content across reconstruction and generation, as well as editing and interaction. For reconstruction, we focus on feed‑forward and neural techniques to reconstruct simulation‑ready worlds from sensor data, improving fidelity, robustness, and scalability. For generation, we develop controllable methods to create high‑quality 3D/4D assets and dynamic scenes from text, images, and videos, leveraging camera‑aware video diffusion models, mesh‑aware LLMs, and efficient 3D priors. For interaction and editing, we develop techniques to enhance, relight and re‑condition captured and generated content — harmonizing renderings, synthesizing and removing weather, and correcting artifacts for photorealistic simulation.

Focus Areas

Text/Image/Video‑to‑3D/4D • Controllability • Scene Generation • Neural Reconstruction • Simulation‑Ready Assets • Interactive Editing • Real-Time Methods

Reconstruction

AI methods for reconstructing and/or generating simulation environments in which end-to-end policy models can be tested, evaluated, and trained in closed-loop.

Points of Contact: Riccardo de Lutio, Katarina Tothova

ArtiFixer: Enhancing and Extending 3D Reconstruction

Enhances and extends sparse 3D reconstructions with an auto‑regressive diffusion model that generates hundreds of consistent novel views in a single pass.

World from Motion: Dynamic Gaussian Reconstruction

Uses a controllable video generator as a prior to recover persistent, dynamic Gaussian reconstructions from monocular video.

Déjà View: Looping Transformers for 3D Reconstruction

Applies one looped transformer block recurrently, turning refinement steps into an inference‑time compute knob that matches far larger feed‑forward models at a fraction of the parameters.

VGG‑T3: Feed‑Forward 3D Reconstruction at Scale

Distills the varying‑length key‑value scene representation into a fixed‑size MLP, removing the quadratic cost that limits offline feed‑forward reconstruction to few input images.

Resources & Publications

Generation

Controllable methods to synthesize high‑quality 3D/4D assets and dynamic scenes.

Points of Contact: Riccardo de Lutio, Katarina Tothova, Shengyu Huang, Bryan Chu

Cosmos‑Drive‑Dreams: Synthetic Data Generation

Scalable synthetic driving data: controllable, multi‑view, temporally consistent video and LiDAR generated from world foundation models.

OmniDreams: Real‑Time Generative World Model

Autoregressively generates action‑conditioned video in real time, driving closed‑loop simulation on top of NVIDIA Cosmos.

GEN3C: 3D‑aware Video Generation

Guiding video generation with explicit 3D caches for precise camera control and 3D consistency.

Infinicube: Scene Generation

Generating unbounded dynamic driving scenes with world‑guided video models.

Resources & Publications

Interaction and Editing

AI methods for editing, enhancing and relighting 3D content across representations — from captured to generated — turning imperfect renderings into coherent, photorealistic environments for simulation, films and games.

Points of Contact: Riccardo de Lutio, Katarina Tothova

DiffusionHarmonizer: Online Diffusion Enhancer

Single‑step, temporally‑conditioned enhancer that harmonizes inserted objects and removes reconstruction artifacts online, on a single GPU.

WeatherWeaver: Weather Synthesis and Removal

Controllable addition and removal of weather and scene conditions in video, preserving physical and spatial coherence.