1. [Publications](/publications)
2. Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
 
 # Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

  ![Publication image](/sites/default/files/styles/wide/public/default_images/default.jpeg?itok=qUFsuJCP "Publication image")

 Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts. This is the goal of semi-supervised learning, which exploits more widely available unlabeled data to complement small labeled data sets. In this paper, we propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. Concretely, we learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images supplemented with only few labeled ones. We build our architecture on top of StyleGAN2, augmented with a label synthesis branch. Image labeling at test time is achieved by first embedding the target image into the joint latent space via an encoder network and test-time optimization, and then generating the label from the inferred embedding. We evaluate our approach in two important domains: medical image segmentation and part-based face segmentation. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization, such as transferring from CT to MRI in medical imaging, and photographs of real faces to paintings, sculptures, and even cartoons and animal faces. Project Page: <https://research.nvidia.com/labs/toronto-ai/semanticGAN/>



 ## Authors



Daiqing Li (NVIDIA)

Junlin Yang (NVIDIA, Yale University)

[Karsten Kreis](/person/karsten-kreis)

Antonio Torralba (MIT)

Sanja Fidler (NVIDIA, University of Toronto, Vector Institute)

 

 

 ## Publication Date



Saturday, June 19, 2021

 

 ## Published in



[IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021](https://arxiv.org/abs/2104.05833)

 

 ## Research Area



[Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence)

[Computer Vision](/research-area/computer-vision)

[Generative AI](/research-area/generative-ai)

 

 

 ## External Links



[Project Website](https://research.nvidia.com/labs/toronto-ai/semanticGAN/)