Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form

In this paper we focus on the linear algebra theory behind feedforward (FNN) and recurrent (RNN) neural networks. We review backward propagation, including backward propagation through time (BPTT). Also, we obtain a new exact expression for Hessian, which represents second order effects. We show that for t time steps the weight gradient can be expressed as a rank-t matrix, while the weight Hessian is as a sum of t2 Kronecker products of rank-1 and WTAW matrices, for some matrix A and weight matrix W.

Benjamin Klenk

Benjamin Klenk joined NVIDIA research in July 2017 after completing his PhD studies in Computer Engineering at the Ruprecht-Karls University in Heidelberg, Germany (defended with summa cum laude in January 2018). His thesis focused on communication models for direct inter-GPU communication in which the GPU orchestrates the network traffic itself without the intervention of the CPU. He also holds a B.Eng. in Electrial Engineering from the Cooperative State Universtiy Mosbach, Germany and a M.Sc. in Computer Engineering from the Ruprecht-Karls University in Heidelberg, Germany.

Benjamin is a member of the Networking Research Group and his research focuses on GPU-centric networking archtictures, communication models, and distributed and parallel applications with a focus on the training of deep neural nets.

Main Field of Interest: 

Varun Jampani

I am a research scientist at Nvidia Research in Westford, US. Prior to joining Nvidia, I completed my PhD at Perceiving Systems department, Max-Planck Institute (MPI) for Intelligent Systems in Tübingen, Germany. I did my bachelors and masters in Computer Science at IIIT-Hyderabad, India.

My work lies at the intersection of Computer Vision and Machine Learning. Specifically, I am working on leveraging machine learning techniques for better inference in computer vision models. The main research question is how to make use of learning techniques such as deep neural networks and random forests for inference in structured prediction frameworks.

See my personal webpage for more details: varunjampani.github.io

Main Field of Interest: 

Michael Stengel

Michael Stengel joined Nvidia as a Research Scientist in 2017.

His research is focused on perceptual aspects in Computer Graphics, in particular hardware and algorithms for gaze-contingent real-time rendering.

Michael Stengel received a Diploma in Computational Visualistics from University of Magdeburg, Germany (2011) and holds a Ph.D. degree in Computer Science from TU Braunschweig, Germany (2016).

In 2010 he joined the Virtual Reality Lab at Volkswagen AG, Wolfsburg, Germany where he developed haptics algorithms for immersive rendering.

As a postdoctoral research scientist he joined in 2016 TU Delft and VU Medical Center, Amsterdam in the Netherlands where he developed a Virtual Reality headset with gaze tracking for subject monitoring during awake brain surgeries.


Yaosheng Fu

Yaosheng Fu joined NVIDIA in September, 2017 as a member of the architecture research team. His current interests include computer architecture, memory systems and parallel computing. Yaosheng received his Ph.D. degree in Electrical Engineering at Princeton University, NJ in 2017 and B.S. in Electronic Engineering at Tsinghua University, China in 2010.

Near-eye Light Field Holographic Rendering with Spherical Waves for Wide Field of View Interactive 3D Computer Graphics

Holograms have high resolution and great depth of field allowing the eye to view a scene much like seeing through a virtual window. Unfortunately, computer generated holography (CGH) does not deliver the same promise due to hardware limitations under plane wave illumination and large computational cost. Light field displays have been popular due to their capability to provide continuous focus cue. However, light field displays suffer from the trade offs between spatial and angular resolution, and do not model diffraction.

Perceptually-Guided Foveation for Light Field Displays

A variety of applications such as virtual reality and immersive cinema require high image quality, low rendering latency, and consistent depth cues. 4D light field displays support focus accommodation, but are more costly to render than 2D images, resulting in higher latency. The human visual system can resolve higher spatial frequencies in the fovea than in the periphery. This property has been harnessed by recent 2D foveated rendering methods to reduce computation cost while maintaining perceptual quality.

SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks

Convolutional Neural Networks (CNNs) have emerged as a fundamental technology for machine learning. High performance and extreme energy efficiency are critical for deployments of CNNs, especially in mobile platforms such as autonomous vehicles, cameras, and electronic personal assistants. This paper introduces the Sparse CNN (SCNN) accelerator architecture, which improves performance and energy efficiency by exploiting the zero-valued weights that stem from network pruning during training and zero-valued activations that arise from the common ReLU operator.

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors. Our core idea is to simultaneously optimize for geometry encoded in a signed distance field (SDF), textures from automatically-selected keyframes, and their camera poses along with material and scene lighting. To this end, we propose a joint surface reconstruction approach that is based on shape-from-shading (SfS) techniques and utilizes the estimation of spatially-varying spherical harmonics (SVSH) from subvolumes of the reconstructed scene. Through extensive examples and evaluations, we demo


Subscribe to Research RSS