Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Xun Huang

Arun Mallya

Ting-Chun Wang

Ming-Yu Liu

NVIDIA Corporation

ECCV 2022

Research project behind the AI tool GauGAN2 and GauGAN360

Paper Demo (GauGAN2) Demo (GauGAN 360)

TL;DR: PoE-GAN can synthesize images conditioned on an arbitrary combination of multiple modalities.

Abstract: Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator. Through our carefully designed training scheme, PoE-GAN learns to synthesize images with high quality and diversity. Besides advancing the state of the art in multimodal conditional image synthesis, PoE-GAN also outperforms the best existing unimodal conditional image synthesis approaches when tested in the unimodal setting.

Text

Segmentation

Sketch

Left: input modalities, right: the output image. Click each modality to choose whether it is used to synthesize the output image.

Video (ECCV 2022)

Video (GauGAN2 Official)

Video (GauGAN360)

Product-of-Experts

Our goal is to train a single generative model that captures the image distribution conditioned on an arbitrary subset of modalities. In this paper, we consider four different modalities including text, semantic segmentation, sketch, and style reference. Intuitively, each input modality adds a constraint the synthesized image must satisfy. The set of images that satisfies all constraints is the intersection of the sets each of which satisfies one individual constraint. As illustrated in the figure below, we model this by assuming that the jointly conditioned probability distribution is proportional to the product of singly conditioned probability distributions. Under this setting, for the product distribution to have a high density in a region, each of the individual distributions needs to have a high density in that region, thereby satisfying each constraint. This has also been previously referred to as product-of-experts.

The product of distributions is analogous to the intersection of sets.

Generator Design

The figure below shows the architecture of our generator. We encode each modality into a feature vector which is then aggregated in Global PoE-Net using product-of-experts. The decoder generates the image using the output of Global PoE-Net and skip connections from the segmentation and sketch encoders.

Our generator architecture. Check our paper to see more details of the architecture.

Discriminator Design

We propose a multimodal projection discriminator that generalizes projection discriminator to handling multiple conditional inputs. Unlike the standard projection discriminator that computes a single inner product between the image embedding and the conditional embedding, we compute one inner product per input modality and add them together to obtain the final loss.

Comparison between the standard projection discriminator (left) and our multimodal projection discriminator (right).

Results

When tested using a single input modality, PoE-GAN outperforms previous state-of-the-art approaches specifically designed for that modality, such as the segmentation-to-image methods (SPADE, OASIS) and the text-to-image synthesis methods (DF-GAN, DM-GAN + CL).

Segmentation

SPADE

OASIS

PoE-GAN (ours)

Comparison with Segmentation-to-Image Synthesis Methods on MS-COCO.

Text

DF-GAN

DM-GAN + CL

PoE-GAN (ours)

A man riding a snowboard down a snow covered slope.

A red stop sign sitting on the side of a road.

A kitchen with a counter and a table with chairs.

A view of mountains from the window of a jet airplane.

A blueberry cake is on a plate and is topped with butter.

Three men each holding something and posing for a picture.

A plate holds a good size portion of a cooked, mixed dish that includes broccoli and pasta.

A red blue and yellow train and some people on a platform.

A desk set up as a workstation with a laptop.

Comparison with Text-to-Image Synthesis Methods on MS-COCO.

PoE-GAN can produce diverse output images when conditioned on an arbitrary subset of modalities. Below we show there random samples from PoE-GAN conditioned on two modalities (text + segmentation, text + sketch, and segmentation + sketch) on a dataset of landscape images.

Condition #1
(Text)

Condition #2 (Segmentation)

Samples

Trees near a lake in an autumn rainy day.

Huge ocean waves crash into rocks in a day with colorful clouds.

Condition #1
(Text)

Condition #2 (Sketch)

Samples

Waterfall and river between mountains.

A white sand beach near cyan ocean.

Condition #1 (Segmentation)

Condition #2 (Sketch)

Samples

Below we show there random samples from PoE-GAN conditioned on two modalities including style reference (segmentation + style, text + style, and sketch + style).

Condition #1 (Segmentation)

Condition #2 (Style)

Samples

Condition #1 (Sketch)

Condition #2 (Style)

Samples

Condition #1
(Text)

Condition #2 (Style)

Samples

A lake surrounded by mountains.

A lake surrounded by mountains.

PoE-GAN becomes an unconditional generative model when given no input modalities. Below are uncurated samples generated unconditionally by PoE-GAN.

PoE-GAN can also be used to generate 360° panorama images. Below are some examples.

Citation

@inproceedings{huang2022poegan,
    title={Multimodal Conditional Image Synthesis with Product-of-Experts {GANs}},
    author={Xun Huang and Arun Mallya and Ting-Chun Wang and Ming-Yu Liu},
    booktitle={ECCV},
    year={2022}
}