DiffCollage, a scalable probabilistic model that synthesizes large content in parallel, including long images, looped motions, and 360 images, with diffusion models only trained on pieces of the content
For business inquiries, please visit our website and submit the form: NVIDIA Research Licensing
DiffCollage construct a new diffusion to generate large content by leveraging diffusion models trained on generating pieces of the large content. To approximate the joint distribution of the large content, we use a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. The below figure show DiffCollage generate a long image where only diffusion models on small patches are trained.
Example : DiffCollage in generating long images
Given a diffusion model trained to generate 1024x1024 images, we compare various methods to synthesize infinite image, includeing standard generation (Baseline), autoregressive replacement, autoregressive reconstruction-based replacement. Our method not only generate higher quality, but also enjoys fast generation in the parallel approach.
Long images generated by various approaches that only use diffusion models trained on smaller square images
Using the pretrained Human Motion Diffusion Model, DiffCollage is capable of synthesizing human motion with durations much longer than those present in the training data, as well as composited and looped motions.
Thanks to the versatility of diffusion models, DiffCollage offers flexible image generation. The following examples depict connected images generated from either style or pixel images. DiffCollage can be easily integrated and used with existing training-free image translation tasks.
Connect pixels
Connect styles
Long image inpainting
@inproceedings{zhange2023diffcollage,
title={DiffCollage: Parallel Generation of Large Content with Diffusion Models},
author={Qinsheng Zhang and Jiaming Song and Xun Huang and Yongxin Chen and Ming-yu Liu},
booktitle={CVPR},
year={2023}
}