NVIDIA AI-Mediated Reality and Interaction Research Group

Welcome to the homepage of NVIDIA’s AI-Mediated Reality and Interaction Research Group, led by Dr. Shalini De Mello. Our mission is to re-imagine, using AI, communication between humans, and between humans and machines. Our research spans AI, vision and graphics. We are particularly interested in:

  • Digital Humans: generative models for digital humans
  • Human Understanding: embodied perception of human state and behavior
  • Core AI Technologies: neural rendering, generative AI, streaming and displays
  • New Experiences: enabled by AI (3D video conferencing, content creation, XR, LLMs, etc.)

We are looking for motivated interns! Interested graduate students please email specific team members directly.

News

April 2024 - 4 papers accepted to CVPR 2024 and 1 paper accepted to ICLR 2024!

January 2024 - We presented 2 papers at NeuRIPS 2023!

Members

Avatar

Shalini De Mello

Team Leader

Avatar

Amrita Mazumdar

Streaming, Compression, 3D Representations

Avatar

Jonghyun Kim

Display Technology

Avatar

Koki Nagano

Digital Humans, 3D Generative Models, Media Forensics

Avatar

Matthew Chan

Generative Learning, 3D Representations

Avatar

Michael Stengel

Computer Graphics, Telepresence

Avatar

Omer Shapira

XR, Graphics, Human-Computer Interaction

Avatar

Tianye Li

Digital Humans, Computer Vision, Graphics

Publications

Quickly discover relevant content by filtering publications.

Dream-in-4D: A Unified Approach for Text- and Image-guided 4D Scene Generation

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images. However, the challenging problem of text-to-4D dynamic 3D scene generation with diffusion guidance remains …
Project arXiv Github

GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable …
Project arXiv

RegionGPT: Towards Region Understanding Vision Language Model

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the …
Project arXiv

What You See Is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries of scenes from collections of 2D images via neural volume rendering. Yet, the significant memory and …
Project arXiv PDF

3D Reconstruction with Generalizable Neural Fields using Scene Priors

International Conference on Learning Representations (ICLR), 2024

High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable …
Project arXiv

Convolutional State Space Models for Long-Range Spatiotemporal Modeling

Advances in Neural Information Processing Systems (NeurIPS), 2023

Effectively modeling long spatiotemporal sequences is challenging due to the need to model complex spatial correlations and long-range temporal dependencies simultaneously. ConvLSTMs attempt to address this by updating tensor-valued states with …
Project arXiv PDF Github

Generalizable One-shot 3D Neural Head Avatar

Advances in Neural Information Processing Systems (NeurIPS), 2023

We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image. Existing methods either involve time-consuming optimization for a specific person with multiple images, or they struggle to synthesize intricate …
Project arXiv PDF Github

Generative Novel View Synthesis with 3D-Aware Diffusion Models

IEEE International Conference on Computer Vision (ICCV), 2023
🏆 Oral, top 2%

We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image. Our model samples from the distribution of possible renderings consistent with the input and, even in the presence of ambiguity, is …
Project arXiv

AI-Mediated 3D Video Conferencing

ACM SIGGRAPH 2023 Emerging Technologies, 2023

We present an AI-mediated 3D video conferencing system that can reconstruct and autostereoscopically display a life-sized talking head using consumer-grade compute resources and minimal capture equipment. Our 3D capture uses a novel 3D lifting method …
Project doi PDF

Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis

ACM Transactions on Graphics (SIGGRAPH), 2023

We present a one-shot method to infer and render a photorealistic 3D representation from a single unposed image (e.g., face portrait) in real-time. Given a single RGB input, our image encoder directly predicts a canonical triplane representation of a …
Project doi arXiv
See all publications