NVIDIA AI-Mediated Reality and Interaction Research Group

Welcome to the homepage of NVIDIA Research’s AI-Mediated Reality and Interaction (AMRI) Group, led by Dr. Shalini De Mello. Our mission is to create interactive physical AIs capable of operating in dynamic 4D worlds. We span fundamental research on the algorithms, theory and application of human-AI interaction and 4D world modeling using new ideas in AI, neural rendering, graphics, generative modeling, large language models, human behavior understanding and digital human creation.

We have several positions open and are looking for motivated Research Interns for 2027. Interested graduate students should email, specifically with the subject titled “Seeking Research Internship 2027”, the team lead or specific team members directly.

News

August 2026 - 3 new papers accepted to ECCV 2026!

July 2026 - 1 paper accepted to SIGGRAPH 2026 and 1 paper accepted to HPG 2026!

June 2026 - We are hosting 4 workshops at CVPR 2026!

December 2025 - 2 papers accepted to NeurIPS 2025! We are also presenting 1 Emerging Technologies demo at SIGGRAPH Asia 2025.

Members

Avatar

Shalini De Mello

Team Leader

Avatar

Alex Trevithick

Computer Vision, World Models, Machine Learning

Avatar

Amrita Mazumdar

Streaming, Compression, 3D Representations

Avatar

Christian Jacobsen

Computer Vision, Generative Modeling, Human Robot Interaction

Avatar

Haithem Turki

3D Reconstruction, Generative Models, Simulation

Avatar

Jonghyun Kim

AI display research, AR/VR glasses design, Holographic near-eye displays

Avatar

Koki Nagano

Digital Humans, 3D Generative Models, Media Forensics

Avatar

Michael Stengel

Computer Graphics, Telepresence

Avatar

Seonwook Park

Computer Vision, Perceiving Humans

Avatar

Shengze Wang

Lifelike Robots, Digital Humans, Telepresence

Avatar

Tianye Li

Digital Humans, Computer Vision, Graphics

Publications

Quickly discover relevant content by filtering publications.

COSY: Compositional 3DGS Synthesis for Disentangled Human Head Editing

European Conference on Computer Vision (ECCV), 2026

Recent 3D Gaussian Splatting (3DGS) GANs for human heads synthesize and render photorealistic 3D models in real-time and offer a vast variety in identity and appearance. However, controlling specific semantic attributes such as hair color or glasses …
Project arXiv Github

DETRAM: End-to-end DEtection, Tracking and Recovery of HumAn Meshes

European Conference on Computer Vision (ECCV), 2026

In the task of human mesh recovery (HMR), multi-person scenes are particularly difficult to handle due to the many entities that appear and occlusions between them over time. In particular for video inputs, there is a need to track each entity …
arXiv

Instant Expressive Gaussian Head Avatars at Over 100 FPS

European Conference on Computer Vision (ECCV), 2026

Portrait animation has witnessed tremendous quality improvements thanks to recent advances in video diffusion models. However, these 2D methods often compromise 3D consistency and speed, limiting their applicability in real-world scenarios, such as …
Project arXiv Video

Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation

ArXiv Preprint, 2026

Generating egocentric video from a single exocentric video is an emerging and important topic for AR/VR and physical AI. Compared with conventional novel view synthesis, exo-to-ego generation is a significantly harder task because the standard …
Project arXiv PDF

Real-time 3D Visualization of Radiance Fields on Light Field Displays

High-Performance Graphics (HPG), 2026

Radiance fields, including their recent efficient forms such as 3D Gaussian Splatting and Sparse Voxels, have revolutionized photorealistic 3D scene visualization by enabling high-fidelity reconstruction of complex environments, making them a natural …
Project arXiv PDF Github

TSMC: Time-Varying Scene Mesh Compression

ACM SIGGRAPH, 2026

Time-varying scene meshes are a widely used representation for volumetric video, offering immersive six degrees of freedom (6DoF) interaction in virtual environments. However, their significantly larger size presents major challenges for efficient …
Project Github

World from Motion: Generative Dynamic Gaussian Reconstruction from Monocular Video

ArXiv Preprint, 2026

We present World from Motion, a method for generating freely renderable dynamic 3D Gaussian representations from monocular videos. Our approach conditions a video model on dense, pixel-aligned renderings that encode appearance, geometry, and 3D scene …
Project arXiv PDF Video

DyaPlex: Full-Duplex Speech-Motion Model for Dyadic Interaction

ArXiv Preprint, 2026

We present DyaPlex, a streaming, full-duplex speech-and-motion model designed for dyadic interaction. To capture the continuous and reciprocal nature of human communication, this full-duplex capability empowers the agent to simultaneously perceive …
Project arXiv PDF

VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents

ArXiv Preprint, 2026

Natural human conversation is full-duplex and audio-visual: people simultaneously speak and listen while continuously interpreting and producing nonverbal cues, such as nods, smiles, and gestures. To support successful human-agent interaction, agents …
Project arXiv PDF

Play4D: Accelerated and Interactive Free-viewpoint Video Streaming for Virtual Reality and Light Field Displays

ACM SIGGRAPH Asia 2025 Emerging Technologies, 2025

We present Play4D, a new method for accelerating and interactive free-viewpoint video streaming for virtual reality and light field displays. Our method is based on the observation that the face of a puppeteer is often visible in the background of …
Project PDF
See all publications