Grounded-Exo2Ego: Structured Semantic Grounding for Robust
Exocentric-to-Egocentric Video Generation

Shengze Wang, Michael Stengel, Tianye Li, Seonwook Park,

Amrita Mazumdar, Koki Nagano, Alex Trevithick, Shalini De Mello

NVIDIA

Paper (incl. supplementary) arXiv (coming soon) Code (in progress)

TLDR: Exo-to-ego is an important topic that can help solve robots' need for in-the-wild ego videos and enable new VR experiences. We propose a systematic solution at both the data and architecture level. We also go beyond the popular geometry-conditioned view synthesis approach, and show that semantic grounding helps further improve quality.

Exo input

Vista4D

EgoX

Ours

Exo input

Vista4D

EgoX

Ours

Exo input

Vista4D

EgoX

Ours

Exo input

Vista4D

EgoX

Ours

Auto-playing clips, same exo input fed through every pipeline. CPR and Cooking are New Activities; Omelet and Bike repair are Unseen. Full New Activities + Unseen comparisons appear in the "Qualitative comparisons" section below.

If the videos don't play, please click the video boxes or change to a different browser.

Abstract

Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard geometric conditioning becomes highly unreliable under extreme view changes and large unobservable regions.

We present Grounded-Exo2Ego, a principled framework that addresses these challenges at both the architectural and data levels. Architecturally, Grounded-Exo2Ego is a dual-branch video diffusion model that couples a geometric anchoring branch, which conditions the generation on the rendering of a 3D reconstruction, with a novel semantic grounding branch, which goes beyond the prevailing geometry-based approach and improves quality by synthesizing challenging regions based on object-level context.

Additionally, we found that the overlooked issue of camera-reconstruction misalignment severely undermines exo-to-ego learning. We thus introduce a camera re-localization algorithm that resolves this issue and substantially improves quality across all metrics. We further develop a fully automated synthetic data engine that generates and renders rigged 3D characters in procedurally generated environments. Evaluation on the challenging EgoExo4D dataset shows that our method outperforms recent state-of-the-art approaches by large margins across all metrics. Detailed ablations validate improvements from each of our contributions at both the data and architectural level.

Grounded-Exo2Ego. From a single exocentric video (top left), we lift the scene into a semantically grounded 3D reconstruction that anchors a dual-branch video diffusion model. The generated egocentric views (bottom) are substantially more accurate than those from existing methods.

Exo → Ego inherently harder than general view synthesis

(a) Exo input

(b) 3D at ego

(d) EgoX

(e) Ours

(f) GT

SOTA (e.g. Vista4D) often do monocular depth estimation from the input view (a) and render from the ego view (b) to generate the conditioning ego rendering video. However, the ego rendering (b) is often riddled with holes and corrupted geometry, leading to bad results (c). Recent exo-to-ego methods such as EgoX perform better (d) but fail to capture the exact viewpoint and content. Ours (e) produces ego videos much closer to the ground truth (f).

Method overview

Synthetic data engine

Real-world ego–exo data is still noisy and inaccurate, so we build a generative synthetic engine with exact camera poses, depth, and segmentation: 600+ interior scenes, 500+ rigged characters, 3,000 animations → 20K+ ego–exo training clips. The hundreds of characters are generated in a few hours.

Qualitative comparisons

If the videos don't play, please click the video boxes or change to a different browser.

Exo input

Vista4D

EgoX

Ours

Cooking — IIITH 43-1

Cooking — IIITH 71-6

Cooking — IIITH 96-2

CPR — NUS 08-1

Qualitative comparison on EgoExo4D (New Activities).

If the videos don't play, please click the video boxes or change to a different browser.

Exo input

Vista4D

EgoX

Ours

CPR — NUS 27-2

CPR — NUS 47-1

Basketball — SFU 05-8

COVID test — SFU 002-12

Qualitative comparison on EgoExo4D (New Activities).

If the videos don't play, please click the video boxes or change to a different browser.

Exo input

Vista4D

EgoX

Ours

Omelet — UTokyo 5-1001-2

Cooking — Minnesota 040-4

Bike repair — Indiana 01-1

Cooking — Minnesota 040-2

Qualitative comparison on EgoExo4D (Unseen).

In-the-wild examples

Ego video colors are re-harmonized to the exo input for visualization purposes only.

If the videos don't play, please click the video boxes or change to a different browser.

Exo input

Ours

Laofangu

Home

Chunk-autoregressive Exo2Ego (work in progress)

We also share our on-going experiments in chunk-autoregressive video generation, showing that a short finetuning stage lets the model continue from its own output and roll out much longer takes.

Quantitative results

Main results

Quantitative comparison on EgoExo4D. Best in bold, second-best underlined. Evaluated on the samples using EgoX's evaluation protocol. Unseen indicates brand new environments not seen during training. New Action (renamed from EgoX's Seen) indicate unseen actions but in environment seen during training. Grey rows show the change over EgoX. Our numbers are averaged over five independent generations. Loc Err and Contour are normalized by the output side length. Values in [brackets] are metrics the EgoX paper does not report, so we measured them ourselves.

Method	Image Metrics				Object Metrics			Video Metrics
Method	PSNR ↑	SSIM ↑	LPIPS ↓	CLIP-I ↑	Loc Err ↓	IoU ↑	Contour ↓	FVD ↓	T-LPIPS ↓
New Action (new actions, seen environments)
Exo2Ego-V	14.53	0.384	0.569	0.774	0.326	0.074	—	622.47	—
TrajectoryCrafter	13.05	0.375	0.606	0.780	0.210	0.128	—	546.09	—
Wan-Fun-Control	12.25	0.463	0.617	0.810	0.235	0.076	—	595.07	—
Wan-VACE	12.95	0.413	0.626	0.829	0.228	0.114	—	508.69	—
Vista4D	10.39	0.301	0.670	0.796	0.139	0.254	0.090	647.51	4.40
EgoX	16.05	0.556	0.498	0.896	0.129	0.363	[0.071]	184.47	[2.688]
Ours	18.64	0.560	0.346	0.910	0.031	0.576	0.027	119.9	1.454
Δ vs EgoX	+2.59 dB	+0.7%	−30.5%	+1.6%	−76.0%	+58.7%	−62.0%	−35.0%	−45.9%
Unseen (new actions, new environments)
Exo2Ego-V	12.70	0.439	0.597	0.679	0.447	0.003	—	1283.50	—
TrajectoryCrafter	12.24	0.297	0.619	0.778	0.400	0.039	—	821.71	—
Wan-Fun-Control	13.59	0.439	0.604	0.799	0.399	0.042	—	968.78	—
Wan-VACE	12.17	0.345	0.638	0.820	0.400	0.038	—	1045.45	—
Vista4D	10.68	0.255	0.668	0.811	0.146	0.229	0.087	794.09	2.91
EgoX	14.38	0.457	0.552	0.877	0.312	0.092	[0.083]	440.64	[2.375]
Ours	16.05	0.460	0.467	0.894	0.052	0.412	0.044	385.8	1.485
Δ vs EgoX	+1.67 dB	+0.7%	−15.4%	+1.9%	−83.3%	+347.8%	−47.0%	−12.4%	−37.5%

Ablation study

Ablation studies on EgoExo4D. Variants are named by what is removed relative to the full model Ours: obj tok are the per-object semantic tokens, obj mask the reprojected segmentation masks that ground them, and synth the synthetic training data. Results are averaged over five independent generations. Given ego attn mask measures Ours with attention masks taken from the ground-truth egocentric video instead of reprojected from the exo video, so it is an upper bound rather than a deployable configuration. Rows marked ^* are our own re-implementations of EgoX, ported to the LTX-2.3 backbone so that the comparison isolates one component at a time; EgoX's released code targets a different backbone, so these were rebuilt to the best of our ability from the paper and the reference implementation, and they may understate what the original authors would achieve. The unmarked EgoX row carries the numbers reported in their paper.

Variant	Image Metrics				Object Metrics			Video Metrics
Variant	PSNR ↑	SSIM ↑	LPIPS ↓	CLIP-I ↑	Loc Err ↓	IoU ↑	Contour ↓	FVD ↓	T-LPIPS ↓
New Action (new actions, seen environments)
EgoX	16.05	0.556	0.498	0.896	0.129	0.363	[0.071]	184.47	[2.688]
EgoX (LTX, data)^*	17.05	0.506	0.473	0.875	0.065	0.336	0.061	233.4	3.115
EgoX (LTX, data, reloc)^*	17.95	0.532	0.403	0.889	0.045	0.456	0.041	169.6	2.354
Ours (no obj tok, mask, synth)	17.69	0.529	0.400	0.894	0.040	0.501	0.036	163.9	1.593
Ours (no obj mask, synth)	18.06	0.541	0.377	0.900	0.035	0.540	0.031	142.8	1.540
Ours (no synth)	18.39	0.552	0.355	0.908	0.032	0.569	0.028	122.9	1.417
Ours	18.64	0.560	0.346	0.910	0.031	0.576	0.027	119.9	1.454
Given ego attn mask	18.94	0.568	0.330	0.913	0.026	0.605	0.022	113.8	1.437
Unseen (new actions, new environments)
EgoX	14.38	0.457	0.552	0.877	0.312	0.092	[0.083]	440.64	[2.375]
EgoX (LTX, data)^*	15.50	0.439	0.533	0.853	0.085	0.243	0.078	534.6	2.693
EgoX (LTX, data, reloc)^*	15.92	0.449	0.497	0.868	0.070	0.316	0.063	467.4	2.063
Ours (no obj tok, mask, synth)	15.67	0.445	0.491	0.882	0.062	0.374	0.054	417.9	1.598
Ours (no obj mask, synth)	15.76	0.451	0.481	0.883	0.058	0.385	0.050	399.9	1.596
Ours (no synth)	16.00	0.460	0.462	0.892	0.052	0.418	0.046	372.4	1.416
Ours	16.05	0.460	0.467	0.894	0.052	0.412	0.044	385.8	1.485
Given ego attn mask	16.30	0.467	0.453	0.897	0.046	0.451	0.038	373.6	1.462

Acknowledgements

We thank Maksim Eisenstein, Milos Hasan, Umar Iqbal, Christian Jacobsen, Pekka Janis, Christian Laforte, Jiefeng Li, Edward Liu, and Juho Marttila for valuable discussions and help.

Citation

arXiv link coming soon — please cite the project page for now.

@online{groundedexo2ego2026,
  title  = {Grounded-Exo2Ego: Structured Semantic Grounding for
            Robust Exocentric-to-Egocentric Video Generation},
  author = {Wang, Shengze and Stengel, Michael and Li, Tianye and
            Park, Seonwook and Mazumdar, Amrita and Nagano, Koki
            and Trevithick, Alex and De~Mello, Shalini},
  year   = {2026},
  url    = {https://research.nvidia.com/labs/amri/projects/grounded-exo2ego/},
}

Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

Abstract

Exo → Ego inherently harder than general view synthesis

Method overview

Synthetic data engine

Qualitative comparisons

In-the-wild examples

Chunk-autoregressive Exo2Ego (work in progress)

Quantitative results

Main results

Ablation study

Acknowledgements

Citation

Grounded-Exo2Ego: Structured Semantic Grounding for Robust
Exocentric-to-Egocentric Video Generation