3D/4D Content Creation & Editing

AI methods for reconstructing, generating and editing dynamic 3D assets and scenes.

Overview

We build systems for dynamic 3D content across reconstruction and generation, as well as editing and interaction. For reconstruction, we focus on feed‑forward and neural techniques to reconstruct simulation‑ready worlds from sensor data, improving fidelity, robustness, and scalability. For generation, we develop controllable methods to create high‑quality 3D/4D assets and dynamic scenes from text, images, and videos, leveraging camera‑aware video diffusion models, mesh‑aware LLMs, and efficient 3D priors. For interaction and editing, we develop techniques to breathe life into static scenes, allowing dynamic simulation and interactive editing across 3D representations, from captured to generated.

Focus Areas

Text/Image/Video‑to‑3D/4D • Controllability • Scene Generation • Neural Reconstruction • Simulation‑Ready Assets • Interactive Editing • Real-Time Methods

Reconstruction

AI methods for reconstructing and/or generating simulation environments in which end-to-end policy models can be tested, evaluated, and trained in closed-loop.

Point of Contact: Zan Gojcic

Resources & Publications

Generation

Controllable methods to synthesize high‑quality 3D/4D assets and dynamic scenes.

Points of Contact: Zan Gojcic, Jun Gao, Or Litany, Jonathan Lorraine, Seung Wook Kim, Xuanchi Ren, Frank Shen

Resources & Publications

Interaction and Editing

AI methods for augmenting assets across representations - from generated to captured - with dynamic properties to enable physical interactive environments for films, games and robotics. Interactive editing with AI in the loop to empower workflows across different areas.

Point of Contact: Masha Shugrina

Resources & Publications