Deep Learned Face Swapping in Feature Film Production

Abstract

In visual effects for film, replacement of stunt performers’ facial likeness for their doubled actor counterparts using traditional computer graphics methods is a multi-stage, labor intensive task. Recently, deep learning techniques have made a compelling argument to train neural networks to learn how to take an image of a person’s face and convincingly infer a rendered image of a second person’s face with a previously unseen perspective, pose and lighting environment. A novel method is discussed for bringing deep neural network face swapping to feature film production which utilizes facial recognition for the discovery of training data. Our method further innovates in the area of utilizing traditional CG assets for informing some of the shortcomings of ML techniques. Connected with a technique for feature engineering during training dataset assembly, our Face Fabrication System enables Wētā FX to deliver final picture quality for use in production.