1. [Publications](/index.php/publications) 2. Weakly-Supervised Physically Unconstrained Gaze Estimation # Weakly-Supervised Physically Unconstrained Gaze Estimation ![](/sites/default/files/styles/wide/public/publications/main_motiv.png?itok=9irEB6Xt) A major challenge for physically unconstrained gaze estimation is acquiring training data with 3D gaze annotations for in-the-wild and outdoor scenarios. In contrast, videos of human interactions in unconstrained environments are abundantly available and can be much more easily annotated with frame-level activity labels. In this work, we tackle the previously unexplored problem of weakly-supervised gaze estimation from videos of human interactions. We leverage the insight that strong gaze-related geometric constraints exist when people perform the activity of "looking at each other" (LAEO). To acquire viable 3D gaze supervision from LAEO labels, we propose a training algorithm along with several novel loss functions especially designed for the task. With weak supervision from two large scale CMU-Panoptic and AVA-LAEO activity datasets, we show significant improvements in (a) the accuracy of semi-supervised gaze estimation and (b) cross-domain generalization on the state-of-the-art physically unconstrained in-the-wild Gaze360 gaze estimation benchmark. We open source our code at . ## Authors Rakshit Kothari (Rochester Institute of Technology) [Shalini De Mello](/index.php/person/shalini-de-mello) [Umar Iqbal](/index.php/person/umar-iqbal) [Wonmin Byeon](/index.php/person/wonmin-byeon) Seonwook Park (Lunit Inc.) [Jan Kautz](/index.php/person/jan-kautz) ## Publication Date Saturday, June 19, 2021 ## Published in [IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2021](http://cvpr2021.thecvf.com/) ## Research Area [Artificial Intelligence and Machine Learning ](/index.php/research-area/machine-learning-artificial-intelligence) [Computer Vision](/index.php/research-area/computer-vision) [Esports](/index.php/research-area/esports) [Human Computer Interaction](/index.php/research-area/human-computer-interaction) ## External Links [Code](https://github.com/NVlabs/weakly-supervised-gaze) [ArXiv](https://arxiv.org/abs/2105.09803) ## Uploaded Files [Paper (PDF)](https://d1qx31qr3h6wln.cloudfront.net/publications/2021_CVPR_Weakly-Supervised%20Physically%20Unconstrained%20Gaze%20Estimation.pdf "Open file in new window")9.2 MB [Supplementary (PDF)](https://d1qx31qr3h6wln.cloudfront.net/publications/2021_CVPR_Weakly-Supervised%20Physically%20Unconstrained%20Gaze%20Estimation-Supp.pdf "Open file in new window")3.05 MB ## Award Oral ## Copyright This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to .