NVIDIA Toronto AI Lab

NVIDIA Toronto AI Lab

Introduction

Welcome to the homepage of the NVIDIA Toronto Artificial Intelligence Lab led by Professor Sanja Fidler. Our research group was founded in 2018, and is primarily based in Toronto.

The research interests of our lab lie at the intersection of computer vision, machine learning and computer graphics. Our group members are also part of or closely collaborate with academic labs such as the University of Toronto and Vector Institute. Since April 2024, we are happy to join forces with Ken Museth and his physics research team.

We invite applications for the following positions:

  • full time research scientist
  • full time research engineer
  • research scientist intern
  • research engineer intern

Graduate and senior undergraduate students interested in doing an internship in the NVIDIA Toronto AI Lab can directly fill this form. See this link for open positions, or contact our members for more details.

News

April 2024 - We are happy to join forces with Ken Museth and his physics research team!

December 2023 - Adaptive Shells for Efficient Neural Radiance Field Rendering received the Best Paper Award at SIGGRAPH Asia 2023.

August 2023 - Interactive AI Material Generation and Editing in NVIDIA Omniverse wins Real-Time Live at SIGGRAPH 2023.

July 2023 - Learning Physically Simulated Tennis Skills From Broadcast Videos was awarded an Honorable Mention at SIGGRAPH 2023.

June 2022 - Karsten Kreis co-organized a workshop on diffusion-based generative modeling at CVPR 2022. Or Litani and Towaki Takikawa organized the neural fields tutorial at CVPR 2022.

April 2021 - Our work was presented at GTC 2021.

December 2020 - New version of the website.

May 2020 - 40 Years on, PAC-MAN Recreated with AI by NVIDIA Researchers

December 2019 - 2D or Not 2D: NVIDIA Researchers Bring Images to Life with AI

November 2019 - NVIDIA Makes 3D Deep Learning Research Easy with Kaolin PyTorch Library

November 2019 - NVIDIA Research at ICCV: Generating New City Road Layouts with AI

October 2019 - NVIDIA Research at ICCV: Meta-Sim: Learning to Generate Synthetic Datasets

June 2019 - NVIDIA Research Released at CVPR Helps Developers Create Better Visual Datasets

Members

Avatar

Sanja Fidler

Avatar

Hassan Abu Alhaija

Avatar

David Acuna

Avatar

Matan Atzmon

Avatar

Riccardo de Lutio

Avatar

Clement Fuji Tsang

Avatar

Jun Gao

Avatar

Zan Gojcic

Avatar

Jianfei Guo

Avatar

Anita Hu

Avatar

Jiahui Huang

Avatar

Amlan Kar

Avatar

Seung Wook Kim

Avatar

Marc Law

Avatar

Huan Ling

Avatar

Or Litany

Avatar

Jonathan Lorraine

Avatar

James Lucas

Avatar

Rafid Mahmood

Avatar

Janick Martinez Esturo

Avatar

Jason Peng

Avatar

Or Perel

Avatar

Jonah Philion

Avatar

Davis Rempe

Avatar

Nicholas Sharp

Avatar

Frank Shen

Avatar

Masha Shugrina

Avatar

Michal Tyszkiewicz

Avatar

Zian Wang

Avatar

Francis Williams

Avatar

Kevin Xie

Avatar

Kangxue Yin

Avatar

Xiaohui Zeng

Avatar

Haotian Zhang

Current interns

Avatar

Arthur Allshire

Avatar

Ruilong Li

Avatar

Ruofan Liang

Avatar

Selena Ling

Avatar

Ashkan Mirzaei

Avatar

Sara Oblak

Avatar

Karran Pandey

Avatar

Mathis Petrovich

Avatar

Xuanchi Ren

Avatar

Amirmojtaba Sabour

Avatar

Jonathan Tseng

Avatar

Chenfeng Xu

Avatar

Yilun Xu

Avatar

Hongwei Yi

Research Topics

Our research combines machine learning, computer vision and computer graphics for innovative applications such as video games, simulations, film industry, data servers, medical imaging and self-driving cars. A non-exhaustive list of research topics studied in our group include:

  • Machine Learning for Computer Graphics: Differentiable rendering, 3D deep learning, computational shape analysis, manipulation of color distributions

  • Generative modeling: Synthetic data generation, dynamic environment simulation, 3D models.

  • Representation Learning: Semantic segmentation, road layout modeling, complex continuous data structures to represent hierarchies and graphs, motion planning, planner-centric metrics, numerical Optimization, federated simulation

  • Machine Learning with limited supervision: active learning, self-supervised learning, few-shot learning, domain adaptation, learning with noisy labels

Publications of the physics research team can be found here.

Long-Term Projects

Quickly discover relevant content by filtering publications.

Omniverse AI Toybox

Interactive 3D Tools as Extensions in NVIDIA Omniverse

NVIDIA is using AI to improve the Content Creation process, giving the power to create virtual worlds to creators everywhere. AI Toybox …
Code
Omniverse AI Toybox

Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research

Pytorch Library

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp …
Code Project Source Document
Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research

Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Pytorch Library

Kaolin is a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable …
PDF Code Project Source Document
Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Publications

Quickly discover relevant content by filtering publications.

Publications of the physics research team can be found here.

*

Large-scale Amortized Text-To-Enhanced3D Synthesis

ArXiv 2024

Recent text-to-3D generation approaches produce impressive 3D results but require time-consuming optimization that can take up to an …
PDF Project
Large-scale Amortized Text-To-Enhanced3D Synthesis

Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

CVPR 2024

We present XCube, a novel generative model for high-resolution sparse 3D voxel grids with arbitrary attributes. Our model can generate …
PDF Project
Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

Adaptive Shells for Efficient Neural Radiance Field Rendering

SIGGRAPH Asia 2023 (Best Paper Award)

Neural radiance fields achieve unprecedented quality for novel view synthesis, but their volumetric formulation remains expensive, …
PDF Project Video
Adaptive Shells for Efficient Neural Radiance Field Rendering

Compact Neural Graphics Primitives with Learned Hash Probing

SIGGRAPH Asia 2023

Compact neural graphics primitives (Ours) have an inherently small size across a variety of use cases with automatically chosen …
PDF Project
Compact Neural Graphics Primitives with Learned Hash Probing

Language-Guided Traffic Simulation via Scene-Level Diffusion

CoRL 2023

Realistic and controllable traffic simulation is a core capability that is necessary to accelerate autonomous vehicle (AV) development. …
PDF
Language-Guided Traffic Simulation via Scene-Level Diffusion

TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

ICCV 2023 (Oral)

We present TexFusion(Texture Diffusion), a new method to synthesize textures for given 3D geometries, using only large-scale …
PDF
TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion Models

ATT3D: Amortized Text-to-3D Object synthesis

ICCV 2023

Text-to-3D modeling has seen exciting progress by combining generative text-to-image models with image-to-3D methods like Neural …
PDF Project Video
ATT3D: Amortized Text-to-3D Object synthesis

DreamTeacher: Pretraining Image Backbones with Deep Generative Models

ICCV 2023

In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks …
PDF Project
DreamTeacher: Pretraining Image Backbones with Deep Generative Models

Learning Human Dynamics in Autonomous Driving Scenarios

ICCV 2023

Simulation has emerged as an indispensable tool for scaling and accelerating the development of self-driving systems. A critical aspect …
PDF
Learning Human Dynamics in Autonomous Driving Scenarios

Neural LiDAR Fields for Novel View Synthesis

ICCV 2023

We present Neural Fields for LiDAR (NFL), a method to optimise a neural field scene representation from LiDAR measurements, with the …
PDF Project
Neural LiDAR Fields for Novel View Synthesis

SKED: Sketch-guided Text-based 3D Editing

ICCV 2023

Text-to-image diffusion models are gradually introduced into computer graphics, recently enabling the development of Text-to-3D …
PDF Project
SKED: Sketch-guided Text-based 3D Editing

Towards Viewpoint Robustness in Bird's Eye View Segmentation

ICCV 2023

Autonomous vehicles (AV) require that neural networks used for perception be robust to different viewpoints if they are to be deployed …
PDF Project
Towards Viewpoint Robustness in Bird's Eye View Segmentation

Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting

TMLR 2023

Sim2Real domain adaptation (DA) research focuses on the constrained setting of adapting from a labeled synthetic source domain to an …
PDF
Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and Reweighting

Differentially Private Diffusion Models

TMLR 2023

While modern machine learning models rely on increasingly large training datasets, data is often limited in privacy-sensitive domains. …
PDF Code
Differentially Private Diffusion Models

CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

SIGGRAPH 2023

In this work, we present Conditional Adversarial Latent Models (CALM), an approach for generating diverse and directable behaviors for …
PDF Code Project
CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

Data-Free Learning of Reduced-Order Kinematic

SIGGRAPH 2023

Physical systems ranging from elastic bodies to kinematic linkages are defined on high-dimensional configuration spaces, yet their …
PDF Code Project Video
Data-Free Learning of Reduced-Order Kinematic

Flexible Isosurface Extraction for Gradient-Based Mesh Optimization

SIGGRAPH 2023

This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the …
PDF Code Project Video
Flexible Isosurface Extraction for Gradient-Based Mesh Optimization

Learning Physically Simulated Tennis Skills from Broadcast Videos

SIGGRAPH 2023 (Honorable Mention)

We present a system that learns diverse, physically simulated tennis skills from large-scale demonstrations of tennis play harvested …
PDF Project
Learning Physically Simulated Tennis Skills from Broadcast Videos

Synthesizing Physical Character-Scene Interactions

SIGGRAPH 2023

Movement is how people interact with and affect their environment. For realistic character animation, it is necessary to synthesize …
PDF Project
Synthesizing Physical Character-Scene Interactions

Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

CVPR 2023

Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion …
PDF Project
Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids

CVPR 2023

Indoor scene reconstruction from monocular images has long been sought after by augmented reality and robotics developers. Recent …
PDF Project Video
Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense Grids

Magic3D: High-Resolution Text-to-3D Content Creation

CVPR 2023

DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields …
PDF Project
Magic3D: High-Resolution Text-to-3D Content Creation

Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes

CVPR 2023

Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and …
PDF Project Video
Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban Scenes

Neural Kernel Surface Reconstruction

CVPR 2023 (Highlight)

We present a novel method for reconstructing a 3D im- plicit surface from a large-scale, sparse, and noisy point cloud. Our approach …
PDF Project
Neural Kernel Surface Reconstruction

NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models

CVPR 2023

Automatically generating high-quality real world 3D scenes is of enormous interest for applications such as virtual reality and …
PDF Project
NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Models

Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion

CVPR 2023

We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet …
PDF Project
Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion

VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

CVPR 2023

Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and …
PDF Code
VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion

Spacetime Representation Learning

ICLR 2023

Much of the data we encounter in the real world can be represented as directed graphs. In this work, we introduce a general family of …
PDF
Spacetime Representation Learning

Mask3D: Mask Transformer for 3D Instance Segmentation

ICRA 2023

Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed …
PDF Code Project Source Document
Mask3D: Mask Transformer for 3D Instance Segmentation

Omniverse AI Toybox

Interactive 3D Tools as Extensions in NVIDIA Omniverse

NVIDIA is using AI to improve the Content Creation process, giving the power to create virtual worlds to creators everywhere. AI Toybox …
Code
Omniverse AI Toybox

Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools

SIGGRAPH Asia (TOG) 2022

Neural Brushstroke Engine (NeuBE) includes a GAN model that learns to mimic many drawing media styles by learning from unlabeled …
PDF Code Project
Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing Tools

GENIE: Higher-Order Denoising Diffusion Solvers

NeurIPS 2022

Denoising diffusion models (DDMs) have emerged as a powerful class of generative models. A forward diffusion process slowly perturbs …
PDF Code Project
GENIE: Higher-Order Denoising Diffusion Solvers

GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

NeurIPS 2022

As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in …
PDF Code Project
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images

LION: Latent Point Diffusion Models for 3D Shape Generation

NeurIPS 2022

Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for …
PDF Code Project
LION: Latent Point Diffusion Models for 3D Shape Generation

Optimizing data collection for machine learning

NeurIPS 2022

Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what …
PDF Project
Optimizing data collection for machine learning

PADL: Language-Directed Physics-Based Character Control

SIGGRAPH Asia 2022

Developing systems that can synthesize natural and life-like motions for simulated characters has long been a focus for computer …
PDF Project
PADL: Language-Directed Physics-Based Character Control

XDGAN: Multi-Modal 3D Shape Generation in 2D Space

BMVC 2022

Generative models for 2D images has recently seen tremendous progress in quality, resolution and speed as a result of the efficiency of …
PDF Project
XDGAN: Multi-Modal 3D Shape Generation in 2D Space

Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research

Pytorch Library

NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp …
Code Project Source Document
Kaolin Wisp: A PyTorch Library and Engine for Neural Fields Research

Language-Grounded Indoor 3D Semantic Segmentation in the Wild

ECCV 2022

Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on …
PDF Code Project
Language-Grounded Indoor 3D Semantic Segmentation in the Wild

MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation

ECCV 2022

We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. This is inspired by the …
PDF Project
MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D Segmentation

Learning Smooth Neural Functions via Lipschitz Regularization

SIGGRAPH 2022

Neural implicit fields have recently emerged as a useful representation for 3D shapes. These fields are commonly represented as neural …
PDF Project
Learning Smooth Neural Functions via Lipschitz Regularization

ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters

SIGGRAPH 2022

The incredible feats of athleticism demonstrated by humans are made possible in part by a vast repertoire of general-purpose motor …
PDF Project
ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated Characters

Variable Bitrate Neural Fields

SIGGRAPH 2022

Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, …
PDF Code Project Video
Variable Bitrate Neural Fields

Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

CVPR 2022 (Oral)

Modern image generative models show remarkable sample quality when trained on a single domain or class of objects. In this work, we …
PDF Project
Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph Maps

Extracting Triangular 3D Models, Materials, and Lighting From Images

CVPR 2022 (Oral)

We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike …
PDF Code Project
Extracting Triangular 3D Models, Materials, and Lighting From Images

AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

CVPR 2022

In this paper, we address the problem of texture representation for 3D shapes for the challenging and underexplored tasks of texture …
PDF Code Project
AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

CVPR 2022

Annotating images with pixel-wise labels is a time-consuming and costly process. Recently, DatasetGAN showcased a promising alternative …
PDF Code Project
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

Frame Averaging for Equivariant Shape Space Learning

CVPR 2022

The task of shape space learning involves mapping a train set of shapes to and from a latent representation space with good …
PDF Project
Frame Averaging for Equivariant Shape Space Learning

Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior

CVPR 2022

Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, …
PDF Project
Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic Prior

How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

CVPR 2022

Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test …
PDF Project
How Much More Data Do I Need? Estimating Requirements for Downstream Tasks

Neural Fields as Learnable Kernels for 3D Reconstruction

CVPR 2022

We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression. Our …
PDF Project Poster Video
Neural Fields as Learnable Kernels for 3D Reconstruction

Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

ICLR 2022 (Spotlight)

Score-based generative models (SGMs) have demonstrated remarkable synthesis quality. SGMs rely on a diffusion process that gradually …
PDF Code Project
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion

Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

ICLR 2022 (Spotlight)

A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously …
PDF Code Project
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Domain Adversarial Training: A Game Perspective

ICLR 2022

The dominant line of work in domain adaptation has focused on learning invariant representations using domain-adversarial training. In …
PDF
Domain Adversarial Training: A Game Perspective

Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach

ICLR 2022

Active learning is the process of training a model with limited labeled data by selecting a core subset of an unlabeled data pool to …
PDF
Low-Budget Active Learning via Wasserstein Distance: An Integer Programming Approach

Spectral Unions of Partial Deformable 3D Shapes

Eurographics 2022

Spectral geometric methods have brought revolutionary changes to the field of geometry processing – however, when the data to be …
PDF
Spectral Unions of Partial Deformable 3D Shapes

Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

WACV 2022

The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised …
PDF Code
Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels

ATISS: Autoregressive Transformers for Indoor Scene Synthesis

NeurIPS 2021

The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many …
PDF Code Project Poster Video
ATISS: Autoregressive Transformers for Indoor Scene Synthesis

Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

NeurIPS 2021

We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such …
PDF Project Video
Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

Score-based Generative Modeling in Latent Space

NeurIPS 2021

Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution …
PDF Code Project
Score-based Generative Modeling in Latent Space

Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation

NeurIPS 2021

Autonomous driving relies on a huge volume of real-world data to be labeled to high precision. Alternative solutions seek to exploit …
PDF Project
Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation

Ultrahyperbolic Neural Networks

NeurIPS 2021 (Spotlight)

Riemannian space forms, such as the Euclidean space, sphere and hyperbolic space, are popular and powerful representation spaces in …
PDF
Ultrahyperbolic Neural Networks

DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

NeurIPS 2021

We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable …
PDF Project
DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer

Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

NeurIPS 2021

Although machine learning models trained on massive data have led to breakthroughs in several areas, their deployment in …
PDF Project
Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn Divergence

EditGAN: High-Precision Semantic Image Editing

NeurIPS 2021

Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN based image editing methods …
PDF Project
EditGAN: High-Precision Semantic Image Editing

Mix3D: Out-of-Context Data Augmentation for 3D Scenes

3DV 2021 (Oral)

We present Mix3D, a data augmentation technique for segmenting large-scale 3D scenes. Since scene context helps reasoning about object …
PDF Code Project
Mix3D: Out-of-Context Data Augmentation for 3D Scenes

Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

ICCV 2021 (Oral)

In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single …
PDF Project
Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting

3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations

ICCV 2021 (Oral)

We propose a method to create plausible geometric and texture style variations of 3D objects in the quest to democratize 3D content …
PDF Project
3DStyleNet: Creating 3D Shapes with Geometric and Texture Style Variations

Vector Neurons: A General Framework for SO(3)-Equivariant Networks

ICCV 2021 (Oral)

Invariance and equivariance to the rotation group have been widely discussed in the 3D deep learning community for pointclouds. Yet …
PDF Code Project
Vector Neurons: A General Framework for SO(3)-Equivariant Networks

Physics-based Human Motion Estimation and Synthesis from Videos

ICCV 2021

Human motion synthesis is an important problem with applications in graphics, gaming and simulation environments for robotics. Existing …
PDF Project
Physics-based Human Motion Estimation and Synthesis from Videos

Self-Supervised Real-to-Sim Scene Generation

ICCV 2021

Synthetic data is emerging as a promising solution to the scalability issue of supervised deep learning, especially when real data are …
PDF Project
Self-Supervised Real-to-Sim Scene Generation

f-Domain-Adversarial Learning: Theory and Algorithms

ICML 2021

Unsupervised domain adaptation is used in many machine learning applications where, during training, a model has access to unlabeled …
PDF Code Project
f-Domain-Adversarial Learning: Theory and Algorithms

Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed Detection

ICML 2021

Training on datasets with long-tailed distributions has been challenging for major recognition tasks such as classification and …
PDF

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

CVPR 2021 (Oral)

We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring …
Project
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

DriveGAN: Towards a Controllable High-Quality Neural Simulation

CVPR 2021 (Oral)

Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are …
PDF Project
DriveGAN: Towards a Controllable High-Quality Neural Simulation

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces

CVPR 2021 (Oral)

Neural signed distance functions (SDFs) are emerging as an effective representation for 3D shapes. State-of-the-art methods typically …
PDF Code Project Video
Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Surfaces

Weakly Supervised Learning of Rigid 3D Scene Flow

CVPR 2021 (Oral)

We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a …
PDF Project
Weakly Supervised Learning of Rigid 3D Scene Flow

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

CVPR 2021

Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human …
PDF Project
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection

CVPR 2021

3D object detection is an important yet demanding task that heavily relies on difficult to obtain 3D annotations. To reduce the …
PDF Project
3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection

Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

ICLR 2021 (Oral)

Differentiable rendering has paved the way to training neural networks to perform ‘inverse graphics’ tasks such as …
PDF Project
Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering

VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

ICLR 2021 (Spotlight)

Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from …
PDF Code
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models

gradSim: Differentiable simulation for system identification and visuomotor control

ICLR 2021

We consider the problem of estimating an object’s physical properties such as mass, friction, and elasticity directly from video …
PDF Code Project Video
gradSim: Differentiable simulation for system identification and visuomotor control

ReLMoGen: Integrating Reinforcement Learning and Motion Generation for Interactive Navigation

ICRA 2021

Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous …
PDF Project
ReLMoGen: Integrating Reinforcement Learning and Motion Generation for Interactive Navigation

UniCon: Universal Neural Controller For Physics-based Character Motion

UniCon

The field of physics-based animation is gaining importance due to the increasing demand for realism in video games and films, and has …
PDF Project Video
UniCon: Universal Neural Controller For Physics-based Character Motion

Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Pytorch Library

Kaolin is a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable …
PDF Code Project Source Document
Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research

Variational Amodal Object Completion

NeurIPS 2020

PDF

Learning Deformable Tetrahedral Meshes for 3D Reconstruction

NeurIPS 2020

3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer …
PDF Code Project Video
Learning Deformable Tetrahedral Meshes for 3D Reconstruction

Ultrahyperbolic Representation Learning

NeurIPS 2020

In machine learning, data is usually represented in a (flat) Euclidean space where distances between points are along straight lines. …
PDF Code
Ultrahyperbolic Representation Learning

Federated Simulation or Medical Imaging

MICCAI 2020 (Nominated for the Young Scientist Award)

Labelling data is expensive and time consuming especially for domains such as medical imaging that contain volumetric imaging data and …
PDF Project
Federated Simulation or Medical Imaging

Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generation

ECCV 2020

Procedural models are being widely used to synthesize scenes for graphics, gaming, and to create (labeled) synthetic datasets for ML. …
PDF Project

Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D

ECCV 2020

The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these …
PDF Code Project Video

Learning to Simulate Dynamic Environments with GameGAN

CVPR 2020

Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the …
PDF Project Video
Learning to Simulate Dynamic Environments with GameGAN

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

NeurIPS 2019

Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting …
PDF Code Project Video
Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Neural Turtle Graphics for Modeling City Road Layouts

ICCV 2019 (Oral)

We propose Neural Turtle Graphics (NTG), a novel gen- erative model for spatial graphs, and demonstrate its ap- plications in modeling …
PDF Project Video
Neural Turtle Graphics for Modeling City Road Layouts

Meta Sim: Learning to Generate Synthetic Datasets

ICCV 2019 (Oral)

Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our …
PDF Code Project
Meta Sim: Learning to Generate Synthetic Datasets

Gated-SCNN Gated Shape CNNs for Semantic Segmentation

ICCV 2019

Current state-of-the-art methods for image segmentation form a dense image representation where the color, shape and texture …
PDF Code Project
Gated-SCNN Gated Shape CNNs for Semantic Segmentation

Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations

CVPR 2019 (Oral)

We tackle the problem of semantic boundary prediction, which aims to identify pixels that belong to object(class) boundaries. We notice …
PDF Code Project
Devil is in the Edges: Learning Semantic Boundaries from Noisy Annotations

Contact

Our lab is located in Downtown Toronto (20 minutes away from the St-George campus of the University of Toronto) and hosts many students for their co-op programs. Motivated candidates can contact our members to apply for internships and research positions.

Unauthorized visitors are not permitted in the Toronto office.

  • 431 King St W, 6th floor, Toronto, ON M5V 3M4
  • Monday-Friday 9:00 to 18:00