People

William Dally

Bill Dally joined NVIDIA in January 2009 as chief scientist, after spending 12 years at Stanford University, where he was chairman of the computer science department. Dally and his Stanford team developed the system architecture, network architecture, signaling, routing and synchronization technology that is found in most large parallel computers today. Dally was previously at the Massachusetts Institute of Technology from 1986 to 1997, where he and his team built the J-Machine and the M-Machine, experimental parallel computer systems that pioneered the separation of mechanism from programming models and demonstrated very low overhead synchronization and communication mechanisms. From 1983 to 1986, he was at California Institute of Technology (CalTech), where he designed the MOSSIM Simulation Engine and the Torus Routing chip, which pioneered “wormhole” routing and virtual-channel flow control. He is a member of the National Academy of Engineering, a Fellow of the American Academy of Arts & Sciences, a Fellow of the IEEE and the ACM, and has received the ACM Eckert-Mauchly Award, the IEEE Seymour Cray Award, and the ACM Maurice Wilkes award. He has published over 250 papers, holds over 120 issued patents, and is an author of four textbooks. Dally received a bachelor's degree in Electrical Engineering from Virginia Tech, a master’s in Electrical Engineering from Stanford University and a Ph.D. in Computer Science from CalTech.  He was a cofounder of Velio Communications and Stream Processors.

Research Area(s)
Circuits and VLSI Design
Computer Architecture
High Performance Computing
Artificial Intelligence and Machine Learning
Networking
Programming Languages, Systems and Tools
Google Scholar
https://scholar.google.com/citations?user=YZHj-Y4AAAAJ&hl=en&oi=sra

Publications

2022

Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training
Charbel Sakr, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, Brucek Khailany, William Dally
2022 International Conference on Machine Learning (ICML)
A 17–95.6 TOPS/W Deep Learning Inference Accelerator with Per-Vector Scaled 4-bit Quantization for Transformers in 5nm
Ben Keller, Rangharajan Venkatesan, Steve Dai, Stephen Tell, Brian Zimmer, William Dally, Tom Gray, Brucek Khailany
2022 Symposium on VLSI Technology & Circuits Digest of Technical Papers

2021

Evolution of the Graphics Processing Unit (GPU)
William Dally, Steve Keckler, David B. Kirk
IEEE Micro Special Issue of the 50th Anniversary of the Microprocessor
Simba: scaling deep-learning inference with chiplet-based architecture
Yakun Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Brucek Khailany, Steve Keckler
Communications of the ACM
VS-QUANT: Per-Vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
Steve Dai, Rangharajan Venkatesan, Mark Ren, Brian Zimmer, William Dally, Brucek Khailany
MLSys 2021

2020

Accelerating Chip Design with Machine Learning
Brucek Khailany, Mark Ren, Steve Dai, Saad Godil, Ben Keller, Robert Kirby, Alicia Klinefelter, Rangharajan Venkatesan, Yanqing Zhang, Bryan Catanzaro, William Dally
IEEE Micro
A 0.32–128 TOPS, Scalable Multi-Chip-Module-Based Deep Neural Network Inference Accelerator With Ground-Referenced Signaling in 16 nm
Brian Zimmer, Rangharajan Venkatesan, Yakun Sophia Shao, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany
IEEE Journal of Solid-State Circuits (JSSC)
JSSC 2020 Best Paper Award

2019

MAGNet: A Modular Accelerator Generator for Neural Networks
Rangharajan Venkatesan, Sophia Shao, Miaorong Wang, Jason Clemons, Steve Dai, Matt Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Yanqing Zhang, Brian Zimmer, William Dally, Joel Emer, Steve Keckler, Brucek Khailany
International Conference On Computer Aided Design
Simba: Scaling Deep-Learning Inference withMulti-Chip-Module-Based Architecture
Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Brucek Khailany, Steve Keckler
52nd IEEE/ACM International Symposium on Microarchitecture
MICRO52 Best Paper Award
A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator Designed with a High-Productivity VLSI Methodology
Rangharajan Venkatesan, Sophia Shao, Brian Zimmer, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany
Hot Chips: A Symposium on High Performance Chips
A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator with Ground-Reference Signaling in 16nm
Brian Zimmer, Rangharajan Venkatesan, Sophia Shao, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany
Symposium on VLSI Circuits Digest of Technical Papers
Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference
Angad S. Rekhi, Brian Zimmer, Nikola Nedovic, Nigxi Liu, Rangharajan Venkatesan, Miaorong Wang, Brucek Khailany, William Dally, Tom Gray
Design Automation Conference (DAC)
A Fine-Grained GALS SoC with Pausible Adaptive Clocking in 16 nm FinFET
Matt Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Stephen Tell, Brian Zimmer, Tezaswi Raja, Kevin Zhou, William Dally, Brucek Khailany
ASYNC 2019
ASYNC 2019 Best Paper Award
A 1.17-pJ/b, 25-Gb/s/pin Ground-Referenced Single-Ended Serial Link for Off- and On-Package Communication Using a Process- and Temperature-Adaptive Voltage Regulator
John Poulton, John Wilson, Walker Turner, Brian Zimmer, Xi Chen, Sudhir Kudva, Sanquan Song, Stephen Tell, Nikola Nedovic, Wenxu Zhao, Sunil Sudhakaran, Tom Gray, William Dally
IEEE JOURNAL OF SOLID-STATE CIRCUITS

2018

Hardware-Enabled Artificial Intelligence
William Dally, Tom Gray, John Poulton, Brucek Khailany, John Wilson, Larry Dennison
Symposia on VLSI Technology and Circuits
Ground-Referenced Signaling for Intra-Chip and Short-Reach Chip-to-Chip Interconnects
Walker Turner, John Poulton, John Wilson, Xi Chen, Stephen Tell, Matt Fojtik, Trey Greer, Brian Zimmer, Sanquan Song, Nikola Nedovic, Sudhir Kudva, Sunil Sudhakaran, Rizwan Bashirullah, Wenxu Zhao, William Dally, Tom Gray
Custom Integrated Circuits Conference
A 1.17pJ/b 25Gb/s/pin Ground-Referenced Single Ended Serial Link for Off- and On-Package Communication in 16nm CMOS Using a Process- and Temperature-Adaptive Voltage Regulator
John Wilson, Walker Turner, John Poulton, Brian Zimmer, Xi Chen, Sanquan Song, Stephen Tell, Nikola Nedovic, Wenxu Zhao, Sunil Sudhakaran, Tom Gray, William Dally
ISSCC

2017

Fine-Grained DRAM: Energy-Efficient DRAM for Extreme Bandwidth Systems
Mike O'Connor, Niladrish Chatterjee, Donghyuk Lee, John Wilson, Aditya Agrawal, Steve Keckler, William Dally
Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchite…
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Steve Keckler, William Dally
IEEE/ACM International Symposium on Computer Architecture (ISCA)
Architecting an Energy-Efficient DRAM System for GPUs
Niladrish Chatterjee, Mike O'Connor, Donghyuk Lee, Daniel Johnson, Minsoo Rhu, Steve Keckler, William Dally
Proceedings of the 23rd IEEE International Symposium on High Performance Comput…

2016

Current parking regulator for zero droop/overshoot load transient response
Sudhir Kudva, William Dally, Trey Greer, Tom Gray
Applied Power Electronics Conference and Exposition (APEC)
A 6.5-to-23.3fJ/b/mm Balanced Charge-Recycling Bus in 16nm FinFET CMOS at 1.7-to-2.6Gb/s/wire with Clock Forwarding and Low-Crosstalk Contraflow Wiring
John Wilson, Matt Fojtik, John Poulton, Xi Chen, Stephen Tell, Trey Greer, Tom Gray, William Dally
International Solid-State Circuits Conference (ISSCC 2016)
A 28nm 2Mbit 6T SRAM with Highly Configurable Write Assist Implementation and Capacitor Based Sense Amplifier Input Offset Compen
Mahmut Sinangil, John Poulton, Matt Fojtik, Trey Greer, Stephen Tell, Andy Gotterba, Jesse Wang, Jason Golbus, William Dally, Tom Gray
Journal of Solid State Circuits

2015

Network Endpoint Congestion Control for Fine-Grained Communication
Ted Jiang, Larry Dennison, William Dally
SC15

2014

Scaling the Power Wall: A Path to Exascale
Oreste Villa, Daniel Johnson, Mike O'Connor, Evgeny Bolotin, David Nellans, Justin Luitjens, Nikolai Sakharnykh, Peng Wang, Paulius Micikevicius, Anthony Scudiero, Steve Keckler, William Dally
Proceedings of the International Conference for High Performance Computing, Net…

2013

A 0.54 pJ/b 20 Gb/s Ground-Referenced Single-Ended Short-Reach Serial Link in 28 nm CMOS for Advanced Packaging Applications
John Poulton, William Dally, Xi Chen, John Eyles, Trey Greer, Stephen Tell, John Wilson, Tom Gray
Journal of Solid State Circuits
A 0.54pJ/b 20Gb/s Ground-Referenced Single-Ended Short-Haul Serial Link in 28nm CMOS for Advanced Packaging Applications
John Poulton, William Dally, Xi Chen, John Eyles, Trey Greer, Stephen Tell, Tom Gray
ISSCC

2012

Unifying Primary Cache, Scratch, and Register File Memories in a Throughput Processor
Mark Gebhart, Steve Keckler, Brucek Khailany, Ronny Krashinsky, William Dally
International Symposium on Microarchitecture (MICRO)
A Hierarchical Thread Scheduler and Register File for Energy-Efficient Throughput Processors
Mark Gebhart, Daniel R. Johnson, David Tarjan, Steve Keckler, William Dally, Erik Lindholm, Kevin Skadron
ACM Transactions on Computer Systems (TOCS)

2011

A Compile-Time Managed Multi-Level Register File Hierarchy
Mark Gebhart, Steve Keckler, William Dally
International Symposium on Microarchitecture (MICRO)
GPUs and the Future of Parallel Computing
Steve Keckler, William Dally, Brucek Khailany, Michael Garland, David Glasco
IEEE Micro
Energy-efficient Mechanisms for Managing Thread Context in Throughput Processors
Mark Gebhart, Daniel R. Johnson, David Tarjan, Steve Keckler, William Dally, Erik Lindholm, Kevin Skadron
Proceedings of the 38th International Symposium on Computer Architecture (ISCA)

2010

The Even/Odd Synchronizer: A Fast, All-Digital Periodic Synchronizer
William Dally, Stephen Tell
16th International Symposium on Asynchronous Circuits and Systems

2007

A 14-mW 6.25-Gb/s Transceiver in 90-nm CMOS
John Poulton, Robert Palmer, Andy Fuller, Trey Greer, John Eyles, William Dally, Mark Horowitz
IEEE Journal of Solid-State Circuits