1. [Publications](/publications) 2. GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields # GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields ![](/sites/default/files/styles/wide/public/publications/tease_figure_2%20copy_2.png?itok=N4v18poG) We propose GazeNeRF, a 3D-aware method for the task of gaze redirection. Existing gaze redirection methods operate on 2D images and struggle to generate 3D consistent results. Instead, we build on the intuition that the face region and eye balls are separate 3D structures that move in a coordinated yet independent fashion. Our method leverages recent advancements in conditional image-based neural radiance fields and proposes a two-branch architecture that predicts volumetric features for the face and eye regions separately. Rigidly transforming the eye features via a 3D rotation matrix provides fine-grained control over the desired gaze angle. The final, redirected image is then attained via differentiable volume compositing. Our experiments show that this architecture outperforms naively conditioned NeRF baselines as well as previous state-of-the-art 2D gaze redirection methods in terms of redirection accuracy and identity preservation. Code and models will be released for research purposes. ## Authors Alessandro Ruzzi (ETH Zürich) Xiangwei Shi (Delft University of Technology) Xi Wang (ETH Zürich) Gengyan Li (ETH Zürich) [Shalini De Mello](/person/shalini-de-mello) Hyung Jin Chang (University of Birmingham) Xucong Zhang (Delft University of Technology) Otmar Hilliges (ETH Zürich) ## Publication Date Saturday, June 17, 2023 ## Published in [IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2023](https://cvpr2023.thecvf.com/) ## Research Area [Computer Vision](/research-area/computer-vision) [Human Computer Interaction](/research-area/human-computer-interaction) [VR, AR and Display Technology](/research-area/virtual-augmented-reality) ## External Links [Project Page](https://x-shi.github.io/GazeNeRF.github.io/) [Code](https://github.com/AlessandroRuzzi/GazeNeRF) [ArXiv](https://arxiv.org/abs/2212.04823) [Video](https://www.youtube.com/watch?v=JwqKbmUR3DE) ## Uploaded Files [Paper](https://d1qx31qr3h6wln.cloudfront.net/publications/2023_gazenerf_3d_aware_gaze_redirec-Camera-ready%20PDF.pdf "Open file in new window")1.31 MB [Supplementary](https://d1qx31qr3h6wln.cloudfront.net/publications/2023_gazenerf_3d_aware_gaze_redirec-Camera-ready%20Supplemental%20Material.pdf "Open file in new window")606.9 KB ## Copyright This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to .