Generative AI | Research

Generative AI

Associated Publications

2024

Breathing Life Into Sketches Using Text-to-Video Priors

Rinon Gal, Yael Vinker, Yuval Alaluf, Amit Bermano, Daniel Cohen-Or, Ariel Shamir, Gal Chechik

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

YuChen Hu, Chen Chen, Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, EnSiong Chng

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, Ensiong Chng, Huck Yang

LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, Xiaohui Zeng

Generating images of rare concepts using pre-trained diffusion models

Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, Gal Chechik

2023

Point-Cloud Completion with Pretrained Text-to-image Diffusion Models

Yoni Kasten, Ohad Rahamim, Gal Chechik

HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

Chen Chen, YuChen Hu, Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, Ensiong Chng

SceneScape: Text-Driven Consistent Scene Generation

Rafail Fridman, Amit Abecasis, Yoni Kasten, Tali Dekel

Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models

Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, Amit Bermano

SIGGRAPH Asia 2023

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Srijith Radhakrishnan, Huck Yang, Sumeer Khan, Rohit Kumar, Narsis Kiani, David Gomez-Cabrero, Jesper Tegnér

XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, Francis Williams

ChipNeMo: Domain-Adapted LLMs for Chip Design

Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Brucek Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman, Sujeet Omar, Sreedhar Pratty, Jonathan Raman, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Walker Turner, Kaizhe Xu, Haoxing (Mark) Ren

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, Kangxue Yin

IEEE/CVF International Conference on Computer Vision (ICCV) 2023 (Oral)

DreamTeacher: Pretraining Image Backbones with Deep Generative Models

Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler

IEEE/CVF International Conference on Computer Vision (ICCV) 2023

ATT3D: Amortized Text-To-3D Object Synthesis

Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, James Lucas

Generative Novel View Synthesis with 3D-Aware Diffusion Models

Eric R. Chan, Koki Nagano, Matthew Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein

International Conference on Computer Vision (ICCV) 2023

Oral

Syntactic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment

Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, Gal Chechik

Oral presentation

Norm-guided latent space exploration for text-to-image generation

Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, Gal Chechik

VerilogEval: Evaluating Large Language Models for Verilog Code Generation

Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing (Mark) Ren

2023 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

Differentially Private Diffusion Models

Tim Dockhorn, Tianshi Cao, Arash Vahdat, Karsten Kreis

Transactions on Machine Learning Research (TMLR) 2023

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization

Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, Jun Gao

ACM Transactions On Graphics (SIGGRAPH 2023)

A Hybrid Generator Architecture for Controllable Face Synthesis

Dann Mensah, Nam Hee Kim, Miika Aittala, Samuli Laine, Jaakko Lehtinen

IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Research

Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, Wen-mei Hwu

Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis

Alexander Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, Koki Nagano

ACM Transactions On Graphics (SIGGRAPH 2023)

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Rinon Gal, Moab Arar, Yuval Atzmon, Amit Bermano, Gal Chechik, Daniel Cohen-Or

AI-Mediated 3D Video Conferencing

Michael Stengel, Koki Nagano, Chao Liu, Matthew Chan, Alex Trevithick, Shalini De Mello, Jonghyun Kim, David Luebke, Amrita Mazumdar, Shengze Wang, Mayoore Jaiswal

ACM SIGGRAPH Emerging Technologies 2023

Efficient Transformer Inference with Statically Structured Sparse Attention

Steve Dai, Hasan Genc, Rangharajan Venkatesan, Brucek Khailany

2023 60th ACM/IEEE Design Automation Conference (DAC)

Physics-Informed Optical Kernel Regression Using Complex-valued Neural Fields

Guojin Chen, Zehua Pei, Haoyu Yang, Yuzhe Ma, Bei Yu, Martin Wong

60th ACM/IEEE Design Automation Conference

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton

Robotics: Science and Systems (RSS) 2023

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models

Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, Sanja Fidler

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

Jiawei Yang, Marco Pavone, Yue Wang

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Hightlight top 10%

Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers

Yixuan Huang, Adam Conkey, Tucker Hermans

IEEE International Conference on Robotics and Automation (ICRA)

Magic3D: High-Resolution Text-to-3D Content Creation

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin

CVPR 2023 (Highlight)

BITS: Bi-level Imitation for Traffic Simulation

Danfei Xu, Yuxiao Chen, Boris Ivanovic, Marco Pavone

IEEE International Conference on Robotics and Automation (ICRA) 2023

ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg

The International Conference on Robotics and Automation (ICRA)

Guided Conditional Diffusion for Controllable Traffic Simulation

Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Gerry Che, Baishakhi Ray, Marco Pavone

IEEE International Conference on Robotics and Automation (ICRA) 2023

Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning

Boris Ivanovic, James Harrison, Marco Pavone

IEEE International Conference on Robotics and Automation (ICRA) 2023

Subpixel Deblurring of Anti-Aliased Raster Clip Art

Jinfan Yang, Nicholas Vining, Shakiba Kheradmand, Nathan Carr, Leonid Sigal, Alla Sheffer

Computer Graphics Forum (Proc. Eurographics 2023)

CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, Xue Bin Peng

Robust and Controllable Object-Centric Learning through Energy-based Models

Ruixiang Zhang, Gerry Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, Liam Paull

International Conference on Learning Representations (ICLR) 2023

Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, Frank Wang

BufFormer: A Generative ML Framework for Scalable Buffering

Rongjian Liang, Siddhartha Nath, Anand Rajaram, Jiang Hu, Haoxing (Mark) Ren

28th Asia and South Pacific Design Automation Conference

2022

"This is my unicorn, Fluffy": Personalizing frozen vision-language representations

Niv Cohen, Rinon Gal, Eli Meirom, Gal Chechik, Yuval Atzmon

Elucidating the Design Space of Diffusion-Based Generative Models

Tero Karras, Miika Aittala, Timo Aila, Samuli Laine

NeurIPS 2022 (oral)

NeurIPS Outstanding Paper

LION: Latent Point Diffusion Models for 3D Shape Generation

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis

Neural Information Processing Systems (NeurIPS) 2022

GENIE: Higher-Order Denoising Diffusion Solvers

Tim Dockhorn, Arash Vahdat, Karsten Kreis

Neural Information Processing Systems (NeurIPS) 2022

CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang, Sifei Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Diffusion Models for Adversarial Purification

Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, Anima Anandkumar

International Conference on Machine Learning (ICML), 2022

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

Seung Wook Kim, Karsten Kreis, Daiqing Li, Antonio Torralba, Sanja Fidler

Conference on Computer Vision and Pattern Recognition (CVPR) 2022 (Oral)

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, Antonio Torralba

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Efficient Geometry-aware 3D Generative Adversarial Networks

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Oral

StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

Rinon Gal, Or Patashnik, Haggai Maron, Amir Bermano, Gal Chechik, Daniel Cohen-Or

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Zhisheng Xiao, Karsten Kreis, Arash Vahdat

International Conference on Learning Representations (ICLR) 2022 (Spotlight)

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Tim Dockhorn, Arash Vahdat, Karsten Kreis

International Conference on Learning Representations (ICLR) 2022 (Spotlight)

DeePattern: Layout Pattern Generation with Transforming Convolutional Auto-Encoder

Haoyu Yang, Shuhe Li, Wen Chen, Piyush Pathak, Frank Gennari, Ya-Chieh Lai, Bei Yu

IEEE Transactions on Semiconductor Manufacturing

Best Paper Award

2021

ATISS: Autoregressive Transformers for Indoor Scene Synthesis

Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, Sanja Fidler

Neural Information Processing Systems (NeurIPS) 2021

EditGAN: High-Precision Semantic Image Editing

Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler

Neural Information Processing Systems (NeurIPS) 2021

Alias-Free Generative Adversarial Networks

Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, Timo Aila

Don’t Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, Karsten Kreis

Neural Information Processing Systems (NeurIPS) 2021

Score-based Generative Modeling in Latent Space

Arash Vahdat, Karsten Kreis, Jan Kautz

Neural Information Processing Systems (NeurIPS) 2021

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, Sanja Fidler

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2021

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

Zhisheng Xiao, Karsten Kreis, Jan Kautz, Arash Vahdat

International Conference on Learning Representations (ICLR) 2021 (Spotlight)

2020

Variational Amodal Object Completion

Huan Ling, David Acuna, Karsten Kreis, Seung Wook Kim, Sanja Fidler

Neural Information Processing Systems (NeurIPS) 2020

Training Generative Adversarial Networks with Limited Data

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, Timo Aila

Neural FFTs for Universal Texture Image Synthesis

Morteza Mardani, Guilin Liu, Aysegul Dundar, Shiqiu Liu, Andrew Tao, Bryan Catanzaro

Semi-Supervised StyleGAN for Disentanglement Learning

Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit B. Patel, Anima Anandkumar

International Conference on Machine Learning (ICML) 2020

Analyzing and Improving the Image Quality of StyleGAN

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, Timo Aila

SymGAN: Orientation Estimation without Annotation for Symmetric Objects

Phil Ammirato, Jonathan Tremblay, Ming-Yu Liu, Alexander Berg, Dieter Fox

2019

A Style-Based Generator Architecture for Generative Adversarial Networks

Tero Karras, Samuli Laine, Timo Aila

2018

MoCoGAN: Decomposing Motion and Content for Video Generation

Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, Bryan Catanzaro

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen