Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset

Scene motion, multiple reflections, and sensor noise introduce artifacts in the depth reconstruction performed by time-of-flight (ToF) cameras.
We propose a two-stage, deep-learning approach to address all of these sources of artifacts simultaneously.
We also introduce FLAT, a synthetic dataset of 2000 ToF measurements that capture all of these nonidealities, and allows to simulate different camera hardware. Using the Kinect 2 camera as a baseline, we show improved reconstruction errors over state-of-the-art methods, on both simulated and real data.

Approximate svBRDF Estimation From Mobile Phone Video

We describe a new technique for obtaining a spatially varying BRDF (svBRDF) of a flat object using printed fiducial markers and a cell phone capable of continuous flash video. Our homography-based video frame alignment method does not require the fiducial markers to be visible in every frame, thereby enabling us to capture larger areas at a closer distance and higher resolution than in previous work.

Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation

Given two consecutive frames, video interpolation aims at generating intermediate frame(s) to form both spatially and temporally coherent video sequences. While most existing methods focus on single-frame interpolation, we propose an end-to-end convolutional neural network for variable-length multi-frame video interpolation, where the motion interpretation and occlusion reasoning are jointly modeled. We start by computing bi-directional optical flow between the input images using a U-Net architecture.

Charles Loop

Charles Loop is a principal research scientist in visual computing research group at NVIDIA Research. He has spent most of his career as a Research Scientist, working on computer graphics, for companies such as Apple and Microsoft. He is best known as the inventor of Loop Subdivision, an algorithm used for creating smooth shapes used in areas such as medical imaging, special effects, and video games. Charles has been programming Graphics Processing Units (GPUs) for many years. His work appears in many academic publications and applications for digital content creation, such as font and surface rendering; including Pixar's Open SubDiv library.  More recently, he initiated a project at Microsoft called Holoportation, that was well received in computer vision and graphics communities. The project demonstrated a two-way telepresence system, allowing users wearing Augmented Reality (AR) display devices such as Hololens to interact with each other’s photo-realistic 3D holograms in real-time, while physically separated by thousands of miles. Most recently, Charles was Chief Scientist in 8i, a startup whose mission is to deliver photo-realistic human holograms. Charles holds an M.S. in Mathematics from the University of Utah, and a Ph.D. in Computer Science from the University of Washington. Charles is located in Redmond.




HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration

Point cloud registration sits at the core of many important and challenging 3D perception problems including autonomous navigation, SLAM, object/scene recognition, and augmented reality. In this paper, we present a new registration algorithm that is able to achieve state-of-the-art speed and accuracy through its use of a hierarchical Gaussian Mixture Model (GMM) representation. Our method constructs a top-down multi-scale representation of point cloud data by recursively running many small-scale data likelihood segmentations in parallel on a GPU.


Subscribe to Research RSS