STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation

Sampling-based model-predictive control (MPC) is a promising tool for feedback control of robots with complex, non-smooth dynamics, and cost functions. However, the computationally demanding nature of sampling-based MPC algorithms has been a key bottleneck in their application to high-dimensional robotic manipulation problems in the real world. Previous methods have addressed this issue by running MPC in the task space while relying on a low-level operational space controller for joint control.

Correcting Robot Plans with Natural Language Feedback

When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints). Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction.

Spatiotemporal Blue Noise Masks

Blue noise error patterns are well suited to human perception, and when applied to stochastic rendering techniques, blue noise masks can minimize unwanted low-frequency noise in the final image. Current methods of applying different blue noise masks to each rendered frame result in either white noise frequency spectra temporally, and thus poor convergence and stability, or lower quality spatially. We propose novel blue noise masks that retain high quality blue noise spatially, yet when animated produce values at each pixel that are well distributed over time.

Mohamed Tarek Ibn Ziad

Mohamed joined NVIDIA in June 2022 and is a member of the Architecture Research Group (ARG). His research interests include systems security, microarchitecture design, and hardware support for security. Dr. Tarek Ibn Ziad received his PhD from the Computer Science Department at Columbia University in 2022. During his PhD, Mohamed worked on hardware-software co-design for practical memory safety. More information about his prior research work can be found on his external website.

Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training

Data clipping is crucial in reducing noise in quantization operations and improving the achievable accuracy of quantization-aware training (QAT). Current practices rely on heuristics to set clipping threshold scalars and cannot be shown to be optimal. We propose Optimally Clipped Tensors And Vectors (OCTAV), a recursive algorithm to determine MSE-optimal clipping scalars. Derived from the fast Newton-Raphson method, OCTAV finds optimal clipping scalars on the fly, for every tensor, at every iteration of the QAT routine.

Jack Snyder

Jack is a research scientist in the networking research group. He finished his Ph.D.in 2022 at Duke University where his advisor was Alvin R. Lebeck. His dissertation focused on congestion control mechanisms and protocols for lossless networks. At Duke, he won the outstanding teaching award. He received his B.S. in computer science and mathematics from Rhodes College where he worked with Brian Larkins on parallel programming models. His research interests include HPC networking and hardware/software codesign for distributed systems. At Nvidia, Jack works on congestion control.

Sana Damani

Sana Damani joined NVIDIA Research in 2022 as a member of the Architecture Research Group. Her primary focus areas include compiler optimizations and hardware-software co-design. Before joining NVIDIA, she earned her PhD from the Georgia Institute of Technology, where her dissertation focused on optimized scheduling and allocation techniques for parallel architectures. She is also a recipient of the 2021 NVIDIA Graduate Fellowship.

Melih Elibol

Melih Elibol is a Senior Research Scientist in Programming Systems and Applications research at NVIDIA. His research aims to improve the ease of expressing scalable high performance programs using modern programming tools, as well as considering how numerical optimization and machine learning may be applied toward addressing emerging challenges in this space. He completed his Ph.D. at the University of California, Berkeley and A.L.B. at Harvard University.