LucidLyra: Feed-Forward Generative 3D Scene Reconstruction and Joint Understanding via Multi-Task Distillation

Oliver Hahn1,2 Rodrigo Marcuzzi1 Sangeetha Grama Srinivasan1
Alessandro Burzio1,3 Sherwin Bahmani† Jiahui Huang†
Stefan Roth2,4,5 Shengyu Huang1 Qunjie Zhou1 Laura Leal-Taixé1
1NVIDIA 2TU Darmstadt 3Università degli Studi di Modena e Reggio Emilia 4ELIZA 5hessian.AI

†Work performed while at NVIDIA

Paper Code coming soon Citation

TL;DR: LucidLyra turns a single image into a queryable 3D Gaussian scene that jointly represents geometry, appearance, open-vocabulary semantics, and object instances. Our unified feed-forward decoder distills knowledge from a video generation backbone and vision foundation models using the backbone's generated videos. We further extend it to infer structured dynamic 3D scenes from an input video. LucidLyra's generated scenes enable physical AI simulation, 3D editing, and 3D asset mining.

Abstract

We present LucidLyra, a framework that makes generatively reconstructed 3D scenes directly queryable for semantics and instances in a feed-forward manner from video latents. While generative 3D reconstruction can synthesize virtual environments from a single image, downstream applications in robotics and physical AI require not only visual realism but also access to scene semantics and object identities. Existing 3D understanding approaches, however, require captured multi-view scenes, have limited generalization capabilities, or rely on expensive per-scene processing and optimization. To bridge this gap, our method distills complementary knowledge from a camera-controlled video diffusion model and frozen 2D vision foundation models into 3D Gaussians that jointly encode geometry, appearance, open-vocabulary semantics, and class-agnostic instance cues. The resulting scenes support novel-view rendering, open-vocabulary segmentation, and object grouping without per-scene optimization at test time, while training requires neither manual 3D annotations nor captured multi-view data. Furthermore, our latent formulation naturally extends to monocular video, reconstructing structured dynamic 3D scenes across time. Our method outperforms baselines combining state-of-the-art 3DGS segmentation approaches with generative 3D scene reconstruction on curated synthetic and real-image-conditioned benchmarks, without compromising reconstruction quality. By turning renderable but unstructured outputs into queryable 3D worlds, LucidLyra enables downstream applications including 3D scene editing, asset extraction, and robotic simulation.

Method

LucidLyra framework: a frozen camera-controlled video model and 2D teachers supervise a Gaussian decoder that predicts geometry, appearance, semantic features, and instance features.

Given a single image, LucidLyra uses a frozen camera-controlled video model to generate multi-view latents. A Gaussian decoder combines these latents with camera rays and C-RADIO input image features, then predicts geometry, appearance, semantics, and instance embeddings through separate reconstruction and understanding heads. During training, frozen SAM/SAM2 and projected SigLIP2 teachers extract object tracks and language-aligned features from the generated videos. Reconstruction, semantic, and instance losses jointly train our 3DGS decoder without captured multi-view data or manual 3D annotations. Given video latents a single decoder pass produces a scene supporting novel-view rendering, open-vocabulary semantic segmentation, and instance clustering without per-scene optimization. For dynamic scenes, time encodings and query-frame features enable decoding a 3D scene state at a requested time from a monocular input video.

Generative Image-to-3D Reconstruction and Understanding

3D scenes reconstructed from single images generated from text, showing RGB appearance, semantic and instance features (PCA), open-vocabulary semantic segmentation, and instance segmentation across novel viewpoints.

Real-World Image-to-3D Reconstruction and Understanding on Waymo

3D scenes reconstructed from single real-world images from the Waymo Open Dataset, showing RGB appearance, semantic and instance features (PCA), open-vocabulary semantic segmentation, and instance segmentation across novel viewpoints.

Generative Video-to-4D Reconstruction and Understanding

Dynamic 3D scenes reconstructed from videos produced by text-to-image and image-to-video generation, showing RGB appearance, semantic and instance features (PCA), open-vocabulary semantic segmentation, and instance segmentation across viewpoints and time.

Application: Robot Simulation

Robot navigation and manipulation in NVIDIA Isaac Sim using LucidLyra scenes. Extracted meshes provide collision geometry, semantic labels support task planning, and Gaussian rendering preserves scene appearance.

Application: 3D Scene Editing

LucidLyra’s semantic and instance fields enable object selection by text or clicks for translation, rotation, scaling, duplication, and deletion. OmniPaint and a subsequent LucidLyra pass complete exposed background regions.

Application: 3D Asset Mining

Six segmented views of each selected object from a LucidLyra scene guide Pixal3D to reconstruct textured meshes. PARTICULATE then estimates moving parts and articulation.

Baseline Comparison

Open-vocabulary semantic segmentation and instance segmentation compared with two-stage baselines combining Lyra’s generative pipeline with existing 3DGS segmentation methods. LucidLyra+ refines semantic labels while preserving geometry and appearance. Input images are generated or drawn from LucidLyra-ADE20K, and LucidLyra-SDG.

Citation

BibTeX Download .bib 
@misc{hahn2026lucidlyra,
  title  = {{LucidLyra}: Feed-Forward Generative {3D} Scene Reconstruction and Joint Understanding via Multi-Task Distillation},
  author = {Oliver Hahn and Rodrigo Marcuzzi and Sangeetha Grama Srinivasan
            and Alessandro Burzio and Sherwin Bahmani and Jiahui Huang
            and Stefan Roth and Shengyu Huang and Qunjie Zhou and Laura Leal-Taix{\'e}},
  year   = {2026},
  note   = {arXiv:2610.40495}
}