Generative AI

Associated Publications

2024

Diffusion-Reward Adversarial Imitation Learning

Chun-Mao Lai, Hsiang-Chun Wang, Ping-Chun Hsieh, Frank Wang, Min-Hung Chen, Shao-Hua Sun

Neural Information Processing Systems (NeurIPS)

Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

Yuchen Hu, Chen Chen, Huck Yang, Chengwei Qin, Pin-Yu Chen, Eng Siong Chng, Chao Zhang

NeurIPS

MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpainting

Chen Tessler, Kelly Guo, Ofir Nabati, Gal Chechik, Jason Peng

SIGGRAPH Asia 2024

Fugatto 1 - Foundational Generative Audio Transformer Opus 1

Rafael Valle, Rohan Badlani, Zhifeng Kong, Sang-gil Lee, Arushi Goel, Sungwon Kim, Joao Felipe Santos, Shuqi Dai, Siddharth Gururani, Aya AIJa'fari, Alex Liu, Kevin Shih, Wei Ping, Huck Yang, Bryan Catanzaro

From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichment

Yusuke Hirota, Ryo Hachiuma, Huck Yang, Yuta Nakashima

EMNLP

Learning to Move Like Professional Counter-Strike Players

David Durst, F. Xie, V. Sarukkai, Brennan Shacklett, Iuri Frosio, Chen Tessler, Joohwan Kim, C. Taylor, G. Bernstein, S. Choudhury, P. Hanrahan,, Kayvon Fatahalian

The 23rd ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA 2024)

VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool

Chia-Tung (Mark) Ho, Haoxing (Mark) Ren, Brucek Khailany

arXiv

Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling

Jaideep Pathak, Yair Cohen, Piyush Garg, Peter Harrington, Noah Brenowitz, Dale Durran, Morteza Mardani, Arash Vahdat, Shaoming Xu, Karthik Kashinath, Mike Pritchard

GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translators

Yuchen Hu, Chen Chen, Huck Yang, Ruizhe Li, Zhehuai Chen, Eng Siong Chng

ACL 2024

TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models

Gilad Deutch, Rinon Gal, Daniel Garibi, Or Patashnik, Daniel Cohen-Or

SIGGRAPH Asia 2024

Align Your Steps: Optimizing Sampling Schedules in Diffusion Models

Amirmojtaba Sabour, Sanja Fidler, Karsten Kreis

International Conference on Machine Learning (ICML) 2024

DoRA: Weight-Decomposed Low-Rank Adaptation

Shih-Yang Liu, Chien-Yi Wang, Hongxu Danny Yin, Pavlo Molchanov, Frank Wang, Kwang-Ting Cheng, Min-Hung Chen

International Conference on Machine Learning (ICML) 2024

Breathing Life Into Sketches Using Text-to-Video Priors

Rinon Gal, Yael Vinker, Yuval Alaluf, Amit Bermano, Daniel Cohen-Or, Ariel Shamir, Gal Chechik

CVPR 2024

fVDB: A Deep-Learning Framework for Sparse, Large-Scale, and High-Performance Spatial Intelligence

Francis Williams, Jiahui Huang, Jonathan Swartz, Gergely Klar, Vijay Thakkar, Matthew Cong, Xuanchi Ren, Ruilong Li, Clement Fuji-Tsang, Sanja Fidler, Eftychios Sifakis, Ken Museth

SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillation

Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, Hongsheng Li

Paper

Large Language Model (LLM) for Standard Cell Layout Design Optimization

Chia-Tung (Mark) Ho, Haoxing (Mark) Ren

The First IEEE International Workshop on LLM-Aided Design (LAD'24)

Best Paper Award

Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer

Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, Tali Dekel

CVPR 2024

Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models

Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, Karsten Kreis

CVPR 2024 (Highlight)

Nemotron-4 340B

Flexible Motion In-betweening with Diffusion Models

Setareh Cohan, Guy Tevet, Daniele Reda, Xue Bin Peng, Michiel van de Panne

Paper

Large Language Models are Efficient Learners of Noise-Robust Speech Recognition

YuChen Hu, Chen Chen, Huck Yang, Ruizhe Li, Chao Zhang, Pin-Yu Chen, EnSiong Chng

ICLR 2024

WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space

Katja Schwarz, Seung Wook Kim, Jun Gao, Sanja Fidler, Andreas Geiger, Karsten Kreis

International Conference on Learning Representations (ICLR) 2024

It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition

Chen Chen, Ruizhe Li, Yuchen Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, Ensiong Chng, Huck Yang

ICLR 2024

LCM-Lookahead for Encoder-based Text-to-Image Personalization

Rinon Gal, Or Lichter, Elad Richardson, Or Patashnik, Amit H Bermano, Gal Chechik, Daniel Cohen-Or

ECCV 2024

LATTE3D: Large-scale Amortized Text-To-Enhanced3D Synthesis

Kevin Xie, Jonathan Lorraine, Tianshi Cao, Jun Gao, James Lucas, Antonio Torralba, Sanja Fidler, Xiaohui Zeng

ECCV

Consolidating Attention Features for Multi-view Image Editing

Or Patashnik, Rinon Gal, Daniel Cohen-Or, Jun-Yan Zhu, Fernando De la Torre

SIGGRAPH Asia 2024

ConsiStory: Training-Free Consistent Text-to-Image Generation

Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, Yuval Atzmon

Paper

Generating images of rare concepts using pre-trained diffusion models

Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, Gal Chechik

AAAI 2024

2023

Point-Cloud Completion with Pretrained Text-to-image Diffusion Models

Yoni Kasten, Ohad Rahamim, Gal Chechik

NeurIPS 2023

HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models

Chen Chen, YuChen Hu, Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Chen, Ensiong Chng

NeurIPS 2023

SceneScape: Text-Driven Consistent Scene Generation

Rafail Fridman, Amit Abecasis, Yoni Kasten, Tali Dekel

NeurIPS 2023

Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition

Srijith Radhakrishnan, Huck Yang, Sumeer Khan, Rohit Kumar, Narsis Kiani, David Gomez-Cabrero, Jesper Tegnér

EMNLP

XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, Francis Williams

CVPR

Domain-Agnostic Tuning-Encoder for Fast Personalization of Text-To-Image Models

Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, Amit Bermano

SIGGRAPH Asia 2023

ChipNeMo: Domain-Adapted LLMs for Chip Design

Mingjie Liu, Teo Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, Siddhanth Dhodhi, Sameer Halepete, Eric Hill, Jiashang Hu, Sumit Jain, Brucek Khailany, George Kokai, Kishor Kunal, Xiaowei Li, Charley Lind, Hao Liu, Stuart Oberman, Sujeet Omar, Sreedhar Pratty, Jonathan Raman, Ambar Sarkar, Zhengjiang Shao, Hanfei Sun, Pratik P Suthar, Varun Tej, Walker Turner, Kaizhe Xu, Haoxing (Mark) Ren

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, Kangxue Yin

IEEE/CVF International Conference on Computer Vision (ICCV) 2023 (Oral)

Generative Novel View Synthesis with 3D-Aware Diffusion Models

Eric R. Chan, Koki Nagano, Matthew Chan, Alexander W. Bergman, Jeong Joon Park, Axel Levy, Miika Aittala, Shalini De Mello, Tero Karras, Gordon Wetzstein

International Conference on Computer Vision (ICCV) 2023

Oral

DreamTeacher: Pretraining Image Backbones with Deep Generative Models

Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler

IEEE/CVF International Conference on Computer Vision (ICCV) 2023

ATT3D: Amortized Text-To-3D Object Synthesis

Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, James Lucas

ICCV

Syntactic Binding in Diffusion Models: Enhancing Attribute Correspondence through Attention Map Alignment

Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, Gal Chechik

NeurIPS 2023

Oral presentation

Norm-guided latent space exploration for text-to-image generation

Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, Gal Chechik

NeurIPS 2023

VerilogEval: Evaluating Large Language Models for Verilog Code Generation

Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing (Mark) Ren

2023 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)

Differentially Private Diffusion Models

Tim Dockhorn, Tianshi Cao, Arash Vahdat, Karsten Kreis

Transactions on Machine Learning Research (TMLR) 2023

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization

Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, Jun Gao

ACM Transactions On Graphics (SIGGRAPH 2023)

AI-Mediated 3D Video Conferencing

Michael Stengel, Koki Nagano, Chao Liu, Matthew Chan, Alex Trevithick, Shalini De Mello, Jonghyun Kim, David Luebke, Amrita Mazumdar, Shengze Wang, Mayoore Jaiswal

ACM SIGGRAPH Emerging Technologies 2023

A Hybrid Generator Architecture for Controllable Face Synthesis

Dann Mensah, Nam Hee Kim, Miika Aittala, Samuli Laine, Jaakko Lehtinen

SIGGRAPH 2023

IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Research

Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, Wen-mei Hwu

KDD'23

Live 3D Portrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis

Alexander Trevithick, Matthew Chan, Michael Stengel, Eric R. Chan, Chao Liu, Zhiding Yu, Sameh Khamis, Manmohan Chandraker, Ravi Ramamoorthi, Koki Nagano

ACM Transactions On Graphics (SIGGRAPH 2023)

Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models

Rinon Gal, Moab Arar, Yuval Atzmon, Amit Bermano, Gal Chechik, Daniel Cohen-Or

SIGGRAPH 2023

Efficient Transformer Inference with Statically Structured Sparse Attention

Steve Dai, Hasan Genc, Rangharajan Venkatesan, Brucek Khailany

2023 60th ACM/IEEE Design Automation Conference (DAC)

Physics-Informed Optical Kernel Regression Using Complex-valued Neural Fields

Guojin Chen, Zehua Pei, Haoyu Yang, Yuzhe Ma, Bei Yu, Martin Wong

60th ACM/IEEE Design Automation Conference

StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

Weiyu Liu, Yilun Du, Tucker Hermans, Sonia Chernova, Chris Paxton

Robotics: Science and Systems (RSS) 2023

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models

Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, Sanja Fidler

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization

Jiawei Yang, Marco Pavone, Yue Wang

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, Shalini De Mello

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2023

Hightlight top 10%

Planning for Multi-Object Manipulation with Graph Neural Network Relational Classifiers

Yixuan Huang, Adam Conkey, Tucker Hermans

IEEE International Conference on Robotics and Automation (ICRA)

Magic3D: High-Resolution Text-to-3D Content Creation

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin

CVPR 2023 (Highlight)

BITS: Bi-level Imitation for Traffic Simulation

Danfei Xu, Yuxiao Chen, Boris Ivanovic, Marco Pavone

IEEE International Conference on Robotics and Automation (ICRA) 2023

ProgPrompt: Generating Situated Robot Task Plans Using Large Language Models

Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, Animesh Garg

The International Conference on Robotics and Automation (ICRA)

Guided Conditional Diffusion for Controllable Traffic Simulation

Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Gerry Che, Baishakhi Ray, Marco Pavone

IEEE International Conference on Robotics and Automation (ICRA) 2023

Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning

Boris Ivanovic, James Harrison, Marco Pavone

IEEE International Conference on Robotics and Automation (ICRA) 2023

Subpixel Deblurring of Anti-Aliased Raster Clip Art

Jinfan Yang, Nicholas Vining, Shakiba Kheradmand, Nathan Carr, Leonid Sigal, Alla Sheffer

Computer Graphics Forum (Proc. Eurographics 2023)

CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, Xue Bin Peng

SIGGRAPH 2023

Robust and Controllable Object-Centric Learning through Energy-based Models

Ruixiang Zhang, Gerry Che, Boris Ivanovic, Renhao Wang, Marco Pavone, Yoshua Bengio, Liam Paull

International Conference on Learning Representations (ICLR) 2023

Frido: Feature Pyramid Diffusion for Complex Scene Image Synthesis

Wan-Cyuan Fan, Yen-Chun Chen, Dongdong Chen, Yu Cheng, Lu Yuan, Frank Wang

AAAI 2023

BufFormer: A Generative ML Framework for Scalable Buffering

Rongjian Liang, Siddhartha Nath, Anand Rajaram, Jiang Hu, Haoxing (Mark) Ren

28th Asia and South Pacific Design Automation Conference

2022

"This is my unicorn, Fluffy": Personalizing frozen vision-language representations

Niv Cohen, Rinon Gal, Eli Meirom, Gal Chechik, Yuval Atzmon

ECCV 2022

Elucidating the Design Space of Diffusion-Based Generative Models

Tero Karras, Miika Aittala, Timo Aila, Samuli Laine

NeurIPS 2022 (oral)

NeurIPS Outstanding Paper

LION: Latent Point Diffusion Models for 3D Shape Generation

Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis

Neural Information Processing Systems (NeurIPS) 2022

GENIE: Higher-Order Denoising Diffusion Solvers

Tim Dockhorn, Arash Vahdat, Karsten Kreis

Neural Information Processing Systems (NeurIPS) 2022

Diffusion Models for Adversarial Purification

Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, Anima Anandkumar

International Conference on Machine Learning (ICML), 2022

CoordGAN: Self-Supervised Dense Correspondences Emerge from GANs

Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang, Sifei Liu

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

Seung Wook Kim, Karsten Kreis, Daiqing Li, Antonio Torralba, Sanja Fidler

Conference on Computer Vision and Pattern Recognition (CVPR) 2022 (Oral)

Efficient Geometry-aware 3D Generative Adversarial Networks

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, Gordon Wetzstein

IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2022

Oral

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, Antonio Torralba

IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022

StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators

Rinon Gal, Or Patashnik, Haggai Maron, Amir Bermano, Gal Chechik, Daniel Cohen-Or

SIGGRAPH 2022

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Tim Dockhorn, Arash Vahdat, Karsten Kreis

International Conference on Learning Representations (ICLR) 2022 (Spotlight)

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs