Multimodal Conditional Image Synthesis with Product-of-Experts GANs

NVIDIA Corporation
ECCV 2022
Research project behind the AI tool GauGAN2 and GauGAN360
TL;DR: PoE-GAN can synthesize images conditioned on an arbitrary combination of multiple modalities.

Abstract: Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator. Through our carefully designed training scheme, PoE-GAN learns to synthesize images with high quality and diversity. Besides advancing the state of the art in multimodal conditional image synthesis, PoE-GAN also outperforms the best existing unimodal conditional image synthesis approaches when tested in the unimodal setting.

Text Segmentation Sketch

Left: input modalities, right: the output image. Click each modality to choose whether it is used to synthesize the output image.