Publications

Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head Videos

European Conference on Computer Vision (ECCV), 2024

Modern generators render talking-head videos with impressive photorealism, ushering in new user experiences such as videoconferencing under constrained bandwidth budgets. Their safe adoption, however, requires a mechanism to verify if the rendered …
Project arXiv PDF Video

What You See Is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

3D-aware Generative Adversarial Networks (GANs) have shown remarkable progress in learning to generate multi-view-consistent images and 3D geometries of scenes from collections of 2D images via neural volume rendering. Yet, the significant memory and …
Project arXiv PDF

RegionGPT: Towards Region Understanding Vision Language Model

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the …
Project arXiv

GAvatar: Animatable 3D Gaussian Avatars with Implicit Mesh Learning

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024
🏆 Highlight top 2%

Gaussian splatting has emerged as a powerful 3D representation that harnesses the advantages of both explicit (mesh) and implicit (NeRF) 3D representations. In this paper, we seek to leverage Gaussian splatting to generate realistic animatable …
Project arXiv

Dream-in-4D: A Unified Approach for Text- and Image-guided 4D Scene Generation

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Large-scale diffusion generative models are greatly simplifying image, video and 3D asset creation from user-provided text prompts and images. However, the challenging problem of text-to-4D dynamic 3D scene generation with diffusion guidance remains …
Project arXiv Github

Full-colour 3D holographic augmented-reality displays with metasurface waveguides

Nature, 2024

Emerging spatial computing systems seamlessly superimpose digital information on the physical environment observed by a user, enabling transformative experiences across various domains, such as entertainment, education, communication and training. …
Project Blog

3D Reconstruction with Generalizable Neural Fields using Scene Priors

International Conference on Learning Representations (ICLR), 2024

High-fidelity 3D scene reconstruction has been substantially advanced by recent progress in neural fields. However, most existing methods train a separate network from scratch for each individual scene. This is not scalable, inefficient, and unable …
Project arXiv

Generalizable One-shot 3D Neural Head Avatar

Advances in Neural Information Processing Systems (NeurIPS), 2023

We present a method that reconstructs and animates a 3D head avatar from a single-view portrait image. Existing methods either involve time-consuming optimization for a specific person with multiple images, or they struggle to synthesize intricate …
Project arXiv PDF Github

Convolutional State Space Models for Long-Range Spatiotemporal Modeling

Advances in Neural Information Processing Systems (NeurIPS), 2023

Effectively modeling long spatiotemporal sequences is challenging due to the need to model complex spatial correlations and long-range temporal dependencies simultaneously. ConvLSTMs attempt to address this by updating tensor-valued states with …
Project arXiv PDF Github

Generative Novel View Synthesis with 3D-Aware Diffusion Models

IEEE International Conference on Computer Vision (ICCV), 2023
🏆 Oral, top 2%

We present a diffusion-based model for 3D-aware generative novel view synthesis from as few as a single input image. Our model samples from the distribution of possible renderings consistent with the input and, even in the presence of ambiguity, is …
Project arXiv

AI-Mediated 3D Video Conferencing

ACM SIGGRAPH 2023 Emerging Technologies, 2023

We present an AI-mediated 3D video conferencing system that can reconstruct and autostereoscopically display a life-sized talking head using consumer-grade compute resources and minimal capture equipment. Our 3D capture uses a novel 3D lifting method …
Project doi PDF

Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis

ACM Transactions on Graphics (SIGGRAPH), 2023

We present a one-shot method to infer and render a photorealistic 3D representation from a single unposed image (e.g., face portrait) in real-time. Given a single RGB input, our image encoder directly predicts a canonical triplane representation of a …
Project doi arXiv

Zero-shot Pose Transfer for Unrigged Stylized 3D Characters

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Transferring the pose of a reference avatar to stylized 3D characters of various shapes is a fundamental task in computer graphics. Existing methods either require the stylized characters to be rigged, or they use the stylized character in the …
Project arXiv PDF Video

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023
🏆 Highlight top 2%

We present ODISE: Open-vocabulary DIffusion-based panoptic SEgmentation, which unifies pre-trained text-image diffusion and discriminative models to perform open-vocabulary panoptic segmentation. Text-to-image diffusion models have the remarkable …
Project arXiv PDF Video Github

GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

We propose GazeNeRF, a 3D-aware method for the task of gaze redirection. Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. Instead, we build on the intuition that the face region and eye balls are …
Project arXiv PDF Video Github

Affordance Diffusion: Synthesizing Hand-Object Interactions

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023

Recent successes in image synthesis are powered by large-scale diffusion models. However, most methods are currently limited to either text- or image-conditioned generation for synthesizing an entire image, texture transfer or inserting objects into …
Project arXiv PDF Github

GPViT: A High Resolution Non-Hierarchical Vision Transformer with Group Propagation

International Conference on Learning Representations (ICLR), 2023
🏆 Oral, Notable top 25%

We present the Group Propagation Vision Transformer (GPViT): a novel non-hierarchical (i.e. non-pyramidal) transformer model designed for general visual recognition with high-resolution features. High-resolution features (or tokens) are a natural fit …
Project arXiv PDF Github

QuadStream: A Quad-Based Scene Streaming Architecture for Novel Viewpoint Reconstruction

ACM Transactions on Graphics (SIGGRAPH), 2022

Streaming rendered 3D content over a network to a thin client device, such as a phone or a VR/AR headset, brings high-fidelity graphics to platforms where it would not normally possible due to thermal, power, or cost constraints. Streamed 3D content …
doi

Time-multiplexed neural holography: a flexible framework for holographic near-eye displays with fast heavily-quantized spatial light modulators

ACM SIGGRAPH 2022 Conference Proceedings, 2022

Holographic near-eye displays offer unprecedented capabilities for virtual and augmented reality systems, including perceptually important focus cues. Although artificial intelligence–driven algorithms for computer-generated holography (CGH) have …
Project doi

Holographic glasses for virtual reality

ACM SIGGRAPH 2022 Conference Proceedings, 2022

We present Holographic Glasses, a holographic near-eye display system with an eyeglasses-like form factor for virtual reality. Holographic Glasses are composed of a pupil-replicating waveguide, a spatial light modulator, and a geometric phase lens to …
Project doi

GroupViT: Semantic Segmentation Emerges from Text Supervision

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

Grouping and recognition are important components of visual scene understanding, e.g., for object detection and semantic segmentation. With end-to-end deep learning systems, grouping of image regions usually happens implicitly via top-down …
Project arXiv PDF Github

FreeSOLO: Learning to Segment Objects without Annotations

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

Instance segmentation is a fundamental vision task that aims to recognize and segment each object in an image. However, it requires costly annotations such as bounding boxes and segmentation masks for learning. In this work, we propose a fully …
Project arXiv PDF Github

Efficient Geometry-Aware 3D Generative Adversarial Networks

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022
🏆 Oral, top 4%

Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute-intensive or make approximations that are …
Project arXiv

CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022

Recent advances show that Generative Adversarial Networks (GANs) can synthesize images with smooth variations along semantically meaningful latent directions, such as pose, expression, layout, etc. While this indicates that GANs implicitly learn …
Project arXiv PDF Github

Learning Contrastive Representation for Semantic Correspondence

International Journal of Computer Vision (IJCV), 2022

Dense correspondence across semantically related images has been extensively studied, but still faces two challenges: 1) large variations in appearance, scale and pose exist even for objects from the same category, and 2) labeling pixel-level dense …
Project arXiv Github

Unfiltered holography: optimizing high diffraction orders without optical filtering for compact holographic displays

Optics letters, 2021

Computer-generated holography suffers from high diffraction orders (HDOs) created from pixelated spatial light modulators, which must be optically filtered using bulky optics. Here, we develop an algorithmic framework for optimizing HDOs without …
Project doi

Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrations

CoRR, 2021

Sharing autonomy between robots and human operators could facilitate data collection of robotic task demonstrations to continuously improve learned models. Yet, the means to communicate intent and reason about the future are disparate between humans …
arXiv

Speckle-free holography with partially coherent light sources and camera-in-the-loop calibration

Science advances, 2021

Computer-generated holography (CGH) holds transformative potential for a wide range of applications, including direct-view, virtual and augmented reality, and automotive display systems. While research on holographic displays has recently made …
Project doi

Self-Supervised Object Detection via Generative Image Synthesis

IEEE International Conference on Computer Vision (ICCV), 2021

We present SSOD ? the first end-to-end analysis-by-synthesis framework with controllable GANs for the task of self-supervised object detection. We use collections of real-world images without bounding box annotations to learn to synthesize and detect …
Project arXiv PDF

Neural 3D holography: learning accurate wave propagation models for 3D holographic virtual and augmented reality displays

ACM Transactions on Graphics (SIGGRAPH), 2021

Holographic near-eye displays promise unprecedented capabilities for virtual and augmented reality (VR/AR) systems. The image quality achieved by current holographic displays, however, is limited by the wave propagation models used to simulate the …
Project doi

A Distributed, Decoupled System for Losslessly Streaming Dynamic Light Probes to Thin Clients

Proceedings of the 12th ACM Multimedia Systems Conference, 2021

We present a networked, high-performance graphics system that combines dynamic, high-quality, ray traced global illumination computed on a server with direct illumination and primary visibility computed on a client. This approach provides many of the …
doi

Weakly-Supervised Physically Unconstrained Gaze Estimation

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
🏆 Oral

A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios. In contrast, videos of human interactions in unconstrained environments are abundantly available …
Project arXiv PDF

Learning to Track Instances without Video Annotations

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
🏆 Oral

Tracking segmentation masks of multiple instances has been intensively studied, but still faces two fundamental challenges: 1) the requirement of large-scale, frame-wise annotation, and 2) the complexity of two-stage approaches. To resolve these …
Project arXiv PDF

Learning Continuous Environment Fields via Implicit Functions

International Conference on Learning Representations (ICLR), 2021

We propose a novel scene representation that encodes reaching distance ? the distance between any position in the scene to a goal along a feasible trajectory. We demonstrate that this environment field representation can directly guide the dynamic …
Project arXiv PDF

Contrastive Syn-to-Real Generalization

International Conference on Learning Representations (ICLR), 2021

Training on synthetic data can be beneficial for label or data-scarce scenarios. However, synthetically trained models often suffer from poor generalization in real domains due to domain gaps. In this work, we make a key observation that the …
Project arXiv PDF Github

Optimizing image quality for holographic near-eye displays with michelson holography

Optica, 2021

We introduce Michelson holography (MH), a holographic display technology that optimizes image quality for emerging holographic near-eye displays. Using two spatial light modulators (SLMs), MH is capable of leveraging destructive interference to …
Project doi

Self-Learning Transformations for Improving Gaze and Head Redirection

Advances in Neural Information Processing Systems (NeurIPS), 2020

Many computer vision tasks rely on labeled data. Rapid progress in generative modeling has led to the ability to synthesize photorealistic images. However, controlling specific aspects of the generation process such that the data can be used for …
Project arXiv PDF Video Github

Online Adaptation for Consistent Mesh Reconstruction in the Wild

Advances in Neural Information Processing Systems (NeurIPS), 2020

This paper presents an algorithm to reconstruct temporally consistent 3D meshes of deformable object instances from videos in the wild. Without requiring annotations of 3D mesh, 2D keypoints, or camera pose for each video frame, we pose video-based …
Project arXiv PDF

Optical gaze tracking with spatially-sparse single-pixel detectors

2020 IEEE international symposium on mixed and augmented reality (ISMAR), 2020

Gaze tracking is an essential component of next generation displays for virtual reality and augmented reality applications. Traditional camera-based gaze trackers used in next generation displays are known to be lacking in one or multiple of the …
doi

Self-supervised Single-view 3D Reconstruction via Semantic Consistency

European Conference on Computer Vision (ECCV), 2020

We learn a self-supervised, single-view 3D reconstruction model that predicts the 3D mesh shape, texture and camera pose of a target object with a collection of 2D images and silhouettes. The proposed method does not necessitate 3D supervision, …
Project arXiv PDF Video

Neural holography

ACM SIGGRAPH 2020 Emerging Technologies, 2020

Holographic displays promise unprecedented capabilities for direct-view displays as well as virtual and augmented reality (VR/AR) applications. However, one of the biggest challenges for computer-generated holography (CGH) is the fundamental tradeoff …
Project doi

Self-Supervised Viewpoint Learning From Image Collections

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020

Training deep neural networks to estimate the viewpoint of objects requires large labeled training datasets. However, manually labeling viewpoints is notoriously hard, error-prone, and time-consuming. On the other hand, it is relatively easy to mine …
Project arXiv PDF Video Github

Prescription AR: a fully-customized prescription-embedded augmented reality display

Optics Express, 2020

In this paper, we present a fully-customized AR display design that considers the user’s prescription, interpupillary distance, and taste of fashion. A free-form image combiner embedded inside the prescription lens provides augmented images onto the …
doi arXiv

Toward Standardized Classification of Foveated Displays

IEEE transactions on visualization and computer graphics, 2020

Emergent in the field of head mounted display design is a desire to leverage the limitations of the human visual system to reduce the computation, communication, and display workload in power and form-factor constrained systems. Fundamental to this …
doi

Joint-task Self-supervised Learning for Temporal Correspondence

Advances in Neural Information Processing Systems (NeurIPS), 2019

This paper proposes to learn reliable dense correspondence from videos in a self-supervised manner. Our learning process integrates two highly related tasks: tracking large image regions and establishing fine-grained pixel-level associations between …
Project arXiv PDF Video Github

Latency of 30 ms benefits first person targeting tasks more than refresh rate above 60 Hz

SIGGRAPH Asia 2019 Technical Briefs, 2019

In competitive sports, human performance makes the difference between who wins and loses. In some competitive video games (esports), response time is an essential factor of human performance. When the athlete’s equipment (computer, input and output …
doi

Learning Propagation for Arbitrarily-structured Data

IEEE International Conference on Computer Vision (ICCV), 2019

Processing an input signal that contains arbitrary structures, e.g., superpixels and point clouds, remains a big challenge in computer vision. Linear diffusion, an effective model for image processing, has been recently integrated with deep learning …
Project arXiv PDF

Few-Shot Viewpoint Estimation

British Machine Vision Conference (BMVC), 2019

Viewpoint estimation for known categories of objects has been improved significantly thanks to deep networks and large datasets, but generalization to unknown categories is still very challenging. With an aim towards improving performance on unknown …
Project arXiv PDF

Few-Shot Adaptive Gaze Estimation

IEEE International Conference on Computer Vision (ICCV), 2019
🏆 Oral

Inter-personal anatomical differences limit the accuracy of person-independent gaze estimation networks. Yet there is a need to lower gaze errors further to enable applications requiring higher quality. Further gains can be achieved by personalizing …
Project arXiv PDF Github

Content-Consistent Generation of Realistic Eyes with Style

IEEE International Conference on Computer Vision Workshops (ICCVW), 2019
🏆 Winner (1st place) Synthetic Eye Generation Challenge

Accurately labeled real-world training data can be scarce, and hence recent works adapt, modify or generate images to boost target datasets. However, retaining relevant details from input data in the generated images is challenging and failure could …
Project arXiv PDF Github

Matching Prescription & Visual Acuity: Towards AR for Humans

ACM SIGGRAPH 2019 Emerging Technologies, 2019

An increasingly important part of usuable near-eye displays to allow use by users who use vision correction such as that provided by glasses and contact lenses. Recent research indicates that over 20% of world population is myopic, and this …
doi

Foveated AR: Dynamically-foveated Augmented Reality Display.

ACM Transactions on Graphics (SIGGRAPH), 2019

We present a near-eye augmented reality display with resolution and focal depth dynamically driven by gaze tracking. The display combines a traveling microdisplay relayed off a concave half-mirror magnifier for the high-resolution foveal region, with …
doi

Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data

2019 International Conference on Robotics and Automation (ICRA), 2019

We present structured domain randomization (SDR), a variant of domain randomization (DR) that takes into account the structure of the scene in order to add context to the generated data. In contrast to DR, which places objects and distractors …
doi

NVGaze: An Anatomically-Informed Dataset for Low-Latency, Near-Eye Gaze Estimation

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2019

Quality, diversity, and size of training dataset are critical factors for learning-based gaze estimators. We create two datasets satisfying these criteria for near-eye gaze estimation under infrared illumination: a synthetic dataset using …
Project doi

Near-eye display and tracking technologies for virtual and augmented reality

Computer Graphics Forum, 2019

Virtual and augmented reality (VR/AR) are expected to revolutionise entertainment, healthcare, communication and the manufacturing industries among many others. Near-eye displays are an enabling vessel for VR/AR applications, which have to tackle …
doi

Fluidic Elastomer Actuators for Haptic Interactions in Virtual Reality

IEEE Robotics and Automation Letters, 2019

Virtual reality experiences via immersive optics and sound are becoming ubiquitous; there are several consumer systems (e.g., Oculus Rift and HTC Vive) now available with these capabilities. Other sensory experiences, such as that of touch remain …
doi

RetroTracker: Upgrading Existing Virtual Reality Tracking Systems

2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 2019

Virtual reality systems often make use of spatially tracked handheld props in the form of controllers or specialized objects to add realism and interaction. Tracking these objects today relies on the use of expensive, bulky, and power-consuming …
doi

Switchable Temporal Propagation Network

European Conference on Computer Vision (ECCV), 2018

Videos contain highly redundant information between frames. Such redundancy has been studied extensively in video compression and encoding, but is less explored for more advanced video processing. In this paper, we propose a learnable unified …
Project arXiv PDF

Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection

ACM Transactions on Graphics (SIGGRAPH), 2018

Redirected walking techniques can enhance the immersion and visual-vestibular comfort of virtual reality (VR) navigation, but are often limited by the size, shape, and content of the physical environments.We propose a redirected walking technique …
doi

Light-weight Head Pose Invariant Gaze Tracking

IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018
🏆 Best Paper (runner up) Workshop on Analysis and Modeling of Faces and Gestures

Unconstrained remote gaze tracking using off-the-shelf cameras is a challenging problem. Recently, promising algorithms for appearance-based gaze estimation using convolutional neural networks (CNN) have been proposed. Improving their robustness to …
Project arXiv PDF

A variable shape and variable stiffness controller for haptic virtual interactions

2018 IEEE International Conference on Soft Robotics (RoboSoft), 2018

This paper presents an entirely compliant controller handle for use in virtual and augmented reality environments. The controller handle transitions between two static states: a semi-rigid, large diameter state when pneumatically pressurized and a …
doi

Learning Affinity via Spatial Propagation Networks

Advances in Neural Information Processing Systems (NIPS), 2017

In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity …
Project PDF

Retinal 3D: augmented reality near-eye display via pupil-tracked light field projection on retina

ACM Transactions on Graphics (SIGGRAPH), 2017

We introduce an augmented reality near-eye display dubbed “Retinal 3D.” Key features of the proposed display system are as follows: Focus cues are provided by generating the pupil-tracked light field that can be directly projected onto …
doi

Near-eye varifocal augmented reality display using see-through screens

ACM Transactions on Graphics (SIGGRAPH), 2017

We introduce an augmented reality near-eye display dubbed “Retinal 3D.” Key features of the proposed display system are as follows: Focus cues are provided by generating the pupil-tracked light field that can be directly projected onto …
doi

Varifocal virtuality: a novel optical layout for near-eye display

ACM SIGGRAPH 2017 Emerging Technologies, 2017

Augmented reality (AR) has recently gained momentum in the form of a variety of available optical see-through near-eye displays (NEDs) such as the Meta 2and the Microsoft Hololens. These devices are a big step forward towards Sutherland’s …
doi PDF

Stretchable Transducers for Kinesthetic Interactions in Virtual Reality

ACM SIGGRAPH 2017 Emerging Technologies, 2017

Commercial virtual reality devices such as the Oculus Rift and HTC Vive enable experience designers to emulate a number of human sensory inputs with computer simulations. Commercial experiences have demonstrated plausibly realistic audiovisual …
doi

Mixed-primary factorization for dual-frame computational displays.

ACM Transactions on Graphics (SIGGRAPH), 2017

Increasing resolution and dynamic range of digital color displays is challenging with designs confined by cost and power specifications. This necessitates modern displays to trade-off spatial and temporal resolution for color reproduction capability. …
doi PDF

Dynamic Facial Analysis: From Bayesian Filtering to Recurrent Neural Network

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

Facial analysis in videos, including head pose estimation and facial landmark localization, is key for many applications such as facial animation capture, human activity recognition, and human-computer interaction. In this paper, we propose to use a …
Project PDF