DiffCollage teaser

DiffCollage, a scalable probabilistic model that synthesizes large content in parallel, including long images, looped motions, and 360 images, with diffusion models only trained on pieces of the content.

Abstract

We present DiffCollage, a compositional diffusion model that can generate large content by leveraging diffusion models trained on generating pieces of the large content. Our approach is based on a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. We apply DiffCollage to various tasks, including infinite image generation, panorama image generation, and long-duration text-guided motion generation. Extensive experimental results with a comparison to strong autoregressive baselines verify the effectiveness of our approach.

Methodology

DiffCollage constructs a new diffusion model to generate large content by leveraging diffusion models trained on generating pieces of the large content. To approximate the joint distribution of the large content, we use a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. The figure below shows DiffCollage generating a long image where only diffusion models on small patches are trained.

Example: DiffCollage in generating long images

Infinite Image Generation

Given a diffusion model trained to generate 1024×1024 images, we compare various methods to synthesize infinite images, including standard generation (Baseline), autoregressive replacement, and autoregressive reconstruction-based replacement. Our method not only generates higher quality, but also enjoys fast generation via the parallel approach.

Long images generated by various approaches that only use diffusion models trained on smaller square images

Motion Generation

Using the pretrained Human Motion Diffusion Model, DiffCollage is capable of synthesizing human motion with durations much longer than those present in the training data, as well as composited and looped motions.

360° Image Generation

DiffCollage is capable of generating 360° panorama images using diffusion models trained solely on regular images. Several examples are presented below.

Interactive 360° images

Versatile Generation and Image Translation

Thanks to the versatility of diffusion models, DiffCollage offers flexible image generation. The following examples depict connected images generated from either style or pixel images. DiffCollage can be easily integrated with existing training-free image translation tasks.

Connect pixels

Connect styles

Long image inpainting

  1. Aspect ratio images can pose a challenge for reconstruction-based methods like Recon, which may need to chunk the original images and can result in noticeable boundary artifacts.

Summary Video

Citation

@inproceedings{zhange2023diffcollage,
  title={DiffCollage: Parallel Generation of Large Content with Diffusion Models},
  author={Zhang, Qinsheng and Song, Jiaming and Huang, Xun and Chen, Yongxin and Liu, Ming-Yu},
  booktitle={IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023}
}