NVIDIA AI-Mediated Reality and Interaction Research Group

Welcome to the homepage of NVIDIA’s AI-Mediated Reality and Interaction Research Group, led by Dr. Shalini De Mello. Our mission is to re-imagine, using AI, communication between humans, and between humans and machines. Our research spans AI, vision and graphics. We are particularly interested in:

  • Digital Humans: generative models for digital humans
  • Human Understanding: embodied perception of human state and behavior
  • Core AI Technologies: neural rendering, generative AI, streaming and displays
  • New Experiences: enabled by AI (3D video conferencing, content creation, XR, LLMs, etc.)

We are looking for motivated full-time research scientists and interns! Interested graduate students please email the team lead or specific members directly.

News

December 2024 - 1 New paper published at SIGGRAPH Asia 2024 and 1 New paper published at NeurIPS 2024!

August 2024 - 1 New paper published in Nature, and 1 New paper accepted to ECCV 2024!

April 2024 - 4 papers accepted to CVPR 2024 and 1 paper accepted to ICLR 2024!

Members

Avatar

Shalini De Mello

Team Leader

Avatar

Amrita Mazumdar

Streaming, Compression, 3D Representations

Avatar

Jonghyun Kim

AI display research, AR/VR glasses design, Holographic near-eye displays

Avatar

Koki Nagano

Digital Humans, 3D Generative Models, Media Forensics

Avatar

Michael Stengel

Computer Graphics, Telepresence

Avatar

Omer Shapira

XR, Graphics, Human-Computer Interaction

Avatar

Seonwook Park

Computer Vision, Perceiving Humans

Avatar

Tianye Li

Digital Humans, Computer Vision, Graphics

Publications

Quickly discover relevant content by filtering publications.

Large Étendue 3D Holographic Display with Content-adpative Dynamic Fourier Modulation

SIGGRAPH ASIA (Conference Track), 2024

Emerging holographic display technology offers unique capabilities for next-generation virtual reality systems. Current holographic near-eye displays, however, only support a small étendue, which results in a direct tradeoff between achievable field …
Project PDF Video

QUEEN: QUantized Efficient ENcoding for Streaming Free-viewpoint Videos

Advances in Neural Information Processing Systems (NeurIPS), 2024

Online free-viewpoint video (FVV) streaming is a challenging problem, which is relatively under-explored. It requires incremental on-the-fly updates to a volumetric representation, fast training and rendering to satisfy real-time constraints and a …
Project arXiv PDF Blog

BLADE: Single-view Body Mesh Learning through Accurate Depth Estimation

ArXiv Preprint, 2024

Single-image human mesh recovery is a challenging task due to the ill-posed nature of simultaneous body shape, pose, and camera estimation. Existing estimators work well on images taken from afar, but they break down as the person moves close to the …
Project arXiv PDF Video

Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head Videos

European Conference on Computer Vision (ECCV), 2024

Modern generators render talking-head videos with impressive photorealism, ushering in new user experiences such as videoconferencing under constrained bandwidth budgets. Their safe adoption, however, requires a mechanism to verify if the rendered …
Project arXiv PDF Video

Dream-in-4D: A Unified Approach for Text- and Image-guided 4D Scene Generation

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images. However, the challenging problem of text-to-4D dynamic 3D scene generation with diffusion guidance remains …
Project arXiv Github

GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
🏆 Highlight top 2%

Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable …
Project arXiv

RegionGPT: Towards Region Understanding Vision Language Model

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the …
Project arXiv

What You See Is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries of scenes from collections of 2D images via neural volume rendering. Yet, the significant memory and …
Project arXiv PDF

3D Reconstruction with Generalizable Neural Fields using Scene Priors

International Conference on Learning Representations (ICLR), 2024

High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable …
Project arXiv

Coherent 3D Portrait Video Reconstruction via Triplane Fusion

ArXiv Preprint, 2024

Recent breakthroughs in single-image 3D portrait reconstruction have enabled telepresence systems to stream 3D portrait videos from a single camera in real-time, potentially democratizing telepresence. However, per-frame 3D reconstruction exhibits …
Project arXiv PDF
See all publications