1. [Publications](/publications) 2. Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks # Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks ![](/sites/default/files/styles/wide/public/pubs/2016-09_Facial-Performance-Capture/laine2016tr1_thumb.png?itok=Lyrj5BAj) We present a real-time deep learning framework for video-based facial performance capture—the dense 3D tracking of an actor’s face given a monocular video. Our pipeline begins with accurately capturing a subject using a high-end production facial capture pipeline based on multi-view stereo tracking and artist-enhanced animations. With 5–10 minutes of captured footage, we train a convolutional neural network to produce high-quality output, including self-occluded regions, from a monocular video sequence of that subject. Since this 3D facial performance capture is fully automated, our system can drastically reduce the amount of labor involved in the development of modern narrative-driven video games or films involving realistic digital doubles of actors and potentially hours of animated dialogue per character. We compare our results with several state-of-the-art monocular real-time facial capture techniques and demonstrate compelling animation inference in challenging areas such as eyes and lips. ## Authors [Samuli Laine](/person/samuli-laine) [Tero Karras](/person/tero-karras) [Timo Aila](/person/timo-aila) Antti Herva (Remedy Entertainment) Shunsuke Saito (Pinscreen, University of Southern California) Ronald Yu (Pinscreen, University of Southern California) Hao Li (USC Institute for Creative Technologies, University of Southern California, Pinscreen) Jaakko Lehtinen (NVIDIA, Aalto University) ## Publication Date Thursday, July 20, 2017 ## Published in [Symposium on Computer Animation 2017](http://sca17.cs.columbia.edu/index.html) ## Research Area [Computer Graphics](/research-area/computer-graphics) [Computer Vision](/research-area/computer-vision) ## External Links [Video (YouTube)](https://youtu.be/VtttfrmfMZw) ## Uploaded Files [Paper (PDF)](https://research.nvidia.com/sites/default/files/publications/laine2017sca_paper_0.pdf "Open file in new window")8.88 MB ## Copyright Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from . SCA ’17, July 28-30, 2017, Los Angeles, CA, USA © 2017 Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery.