High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic. In this work, we generate 2048x1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures. Furthermore, we extend our framework to interactive visual manipulation with two additional features.

Learning Binary Residual Representations for Domain-specific Video Streaming

We study domain-specific video streaming. Specifically, we target a streaming setting where the videos to be streamed from a server to a client are all in the same domain and they have to be compressed to a small size for low-latency transmission. Several popular video streaming services, such as the video game streaming services of GeForce Now and Twitch, fall in this category.

Parallel Jaccard and Related Graph Clustering Techniques

In this paper we propose to generalize Jaccard and related measures, often used as similarity coefficients between two sets. We define Jaccard, Dice-Sorensen and Tversky edge weights on a graph and generalize them to account for vertex weights. We develop an efficient parallel algorithm for computing Jaccard edge and PageRank vertex weights. We highlight that the weights computation can obtain more than 10x speedup on the GPU versus CPU on large realistic data sets.

Learning Affinity via Spatial Propagation Networks

In this paper, we propose spatial propagation networks for learning the affinity matrix for vision tasks. We show that by constructing a row/column linear propagation model, the spatially varying transformation matrix exactly constitutes an affinity matrix that models dense, global pairwise relationships of an image.

Jan Issac

In Memoriam: Jan Issac

We are saddened to share the loss of our friend and colleague Jan Issac, who passed away on Sunday, April 15. 

Jan joined NVIDIA in 2017 as a researcher and software engineer on the Robotics team in Seattle. A talented engineer, he made many contributions to our state-of-the-art experimental platform for robotics manipulation. 

An intelligent and inquisitive man, he loved good food, was an enthusiastic photographer, and enjoyed playing the piano.

Jan will be remembered for his persistence, his amazing ability to learn diverse new technologies, and his dedication to helping others.

Jan will be deeply missed. Please keep him and his family in your thoughts.

Benjamin Eckart

Ben Eckart received his Ph.D. in Robotics at Carnegie Mellon University in 2017, as well as an M.S. in Electrical Engineering, B.S. in Computer Science, and B.S. in Computer Engineering. He was an NVIDIA Graduate Fellow in 2014, and upon graduation joined NVIDIA as a Post-Doctorate Researcher in 2017. His research explores methods to represent and operate on 3D point cloud data and its applications to robotics, computer vision, augmented reality, and autonomous driving.  

Main Field of Interest: 

Progressive Growing of GANs for Improved Quality, Stability, and Variation

We train generative adversarial networks in a progressive fashion, enabling us to generate high-resolution images with high quality.

Consistent Video Filtering for Camera Arrays

Visual formats have advanced beyond single-view images and videos: 3D movies are commonplace, researchers have developed multi-view navigation systems, and VR is helping to push light field cameras to mass market. However, editing tools for these media are still nascent, and even simple filtering operations like color correction or stylization are problematic: naively applying image filters per frame or per view rarely produces satisfying results due to time and space inconsistencies. Our method preserves and stabilizes filter effects while being agnostic to the inner working of the filter.

Learning to Super-Resolve Blurry Face and Text Images

We present an algorithm to directly restore a clear high-resolution image from a blurry low-resolution input. This problem is highly ill-posed and the basic assumptions for existing super-resolution methods (requiring clear input) and deblurring methods (requiring high-resolution input) no longer hold. We focus on face and text images and adopt a generative adversarial network (GAN) to learn a category-specific prior to solve this problem. However, the basic GAN formulation does not generate realistic high-resolution images.


Subscribe to Research RSS