People

William Dally

Bill Dally joined NVIDIA in January 2009 as chief scientist, after spending 12 years at Stanford University, where he was chairman of the computer science department. Dally and his Stanford team developed the system architecture, network architecture, signaling, routing and synchronization technology that is found in most large parallel computers today. Dally was previously at the Massachusetts Institute of Technology from 1986 to 1997, where he and his team built the J-Machine and the M-Machine, experimental parallel computer systems that pioneered the separation of mechanism from programming models and demonstrated very low overhead synchronization and communication mechanisms. From 1983 to 1986, he was at California Institute of Technology (CalTech), where he designed the MOSSIM Simulation Engine and the Torus Routing chip, which pioneered “wormhole” routing and virtual-channel flow control. He is a member of the National Academy of Engineering, a Fellow of the American Academy of Arts & Sciences, a Fellow of the IEEE and the ACM, and has received the ACM Eckert-Mauchly Award, the IEEE Seymour Cray Award, and the ACM Maurice Wilkes award. He has published over 250 papers, holds over 120 issued patents, and is an author of four textbooks. Dally received a bachelor's degree in Electrical Engineering from Virginia Tech, a master’s in Electrical Engineering from Stanford University and a Ph.D. in Computer Science from CalTech.  He was a cofounder of Velio Communications and Stream Processors.

Research Area(s)
Circuits and VLSI Design
Computer Architecture
High Performance Computing
Artificial Intelligence and Machine Learning
Networking
Programming Languages, Systems and Tools
Google Scholar
https://scholar.google.com/citations?user=YZHj-Y4AAAAJ&hl=en&oi=sra

Publications

2023

A 0.297-pJ/Bit 50.4-Gb/s/Wire Inverter-Based Short-Reach Simultaneous Bi-Directional Transceiver for Die-to-Die Interface in 5-nm CMOS
Yoshinori Nishi, John W. Poulton, Xi Chen, Sanquan Song, Brian Zimmer, Stephen Tell, Nikola Nedovic, John Wilson, William Dally, Tom Gray
IEEE Journal of Solid-State Circuits ( Volume: 58, Issue: 4, April 2023)
A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm
Ben Keller, Rangharajan Venkatesan, Steve Dai, Stephen Tell, Brian Zimmer, Charbel Sakr, William Dally, Tom Gray, Brucek Khailany
Journal of Solid-State Circuits

2022

LNS-Madam: Low-Precision Training in Logarithmic Number System Using Multiplicative Weight Update
Jiawei Zhao, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, Mustafa Ali, Ming-Yu Liu, Brucek Khailany, William Dally, Anima Anandkumar
IEEE Transactions on Computers (Volume: 71, Issue: 12, 01 December 2022)
Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training
Charbel Sakr, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, Brucek Khailany, William Dally
2022 International Conference on Machine Learning (ICML)
A 0.297-pJ/bit 50.4-Gb/s/wire Inverter-Based Short-Reach Simultaneous Bidirectional Transceiver for Die-to-Die Interface in 5nm CMOS
Yoshinori Nishi, John W. Poulton, Xi Chen, Sanquan Song, Brian Zimmer, Walker Turner, Stephen Tell, Nikola Nedovic, John Wilson, William Dally, Tom Gray
2022 IEEE SYMPOSIUM ON VLSI TECHNOLOGY & CIRCUITS
A 17–95.6 TOPS/W Deep Learning Inference Accelerator with Per-Vector Scaled 4-bit Quantization for Transformers in 5nm
Ben Keller, Rangharajan Venkatesan, Steve Dai, Stephen Tell, Brian Zimmer, William Dally, Tom Gray, Brucek Khailany
2022 Symposium on VLSI Technology & Circuits Digest of Technical Papers

2021

Evolution of the Graphics Processing Unit (GPU)
William Dally, Steve Keckler, David B. Kirk
IEEE Micro Special Issue of the 50th Anniversary of the Microprocessor
Simba: scaling deep-learning inference with chiplet-based architecture
Yakun Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Brucek Khailany, Steve Keckler
Communications of the ACM
ACM Research Highlight
VS-QUANT: Per-Vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
Steve Dai, Rangharajan Venkatesan, Mark Ren, Brian Zimmer, William Dally, Brucek Khailany
MLSys 2021

2020

Accelerating Chip Design with Machine Learning
Brucek Khailany, Mark Ren, Steve Dai, Saad Godil, Ben Keller, Robert Kirby, Alicia Klinefelter, Rangharajan Venkatesan, Yanqing Zhang, Bryan Catanzaro, William Dally
IEEE Micro
A 0.32–128 TOPS, Scalable Multi-Chip-Module-Based Deep Neural Network Inference Accelerator With Ground-Referenced Signaling in 16 nm
Brian Zimmer, Rangharajan Venkatesan, Yakun Sophia Shao, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany
IEEE Journal of Solid-State Circuits (JSSC)
JSSC 2020 Best Paper award

2019

MAGNet: A Modular Accelerator Generator for Neural Networks
Rangharajan Venkatesan, Sophia Shao, Miaorong Wang, Jason Clemons, Steve Dai, Matt Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Yanqing Zhang, Brian Zimmer, William Dally, Joel Emer, Steve Keckler, Brucek Khailany
International Conference On Computer Aided Design (ICCAD)
Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture
Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Brucek Khailany, Steve Keckler
International Symposium on Microarchitecture (MICRO)
Best Paper award, IEEE Micro Top Picks in Computer Architecture (Honorable Mention)
A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator Designed with a High-Productivity VLSI Methodology
Rangharajan Venkatesan, Sophia Shao, Brian Zimmer, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany
Hot Chips: A Symposium on High Performance Chips
A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator with Ground-Reference Signaling in 16nm
Brian Zimmer, Rangharajan Venkatesan, Sophia Shao, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany
Symposium on VLSI Circuits
Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference
Angad S. Rekhi, Brian Zimmer, Nikola Nedovic, Nigxi Liu, Rangharajan Venkatesan, Miaorong Wang, Brucek Khailany, William Dally, Tom Gray
Design Automation Conference (DAC)
A Fine-Grained GALS SoC with Pausible Adaptive Clocking in 16 nm FinFET
Matt Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Stephen Tell, Brian Zimmer, Tezaswi Raja, Kevin Zhou, William Dally, Brucek Khailany
ASYNC 2019
ASYNC 2019 Best Paper Award
A 1.17-pJ/b, 25-Gb/s/pin Ground-Referenced Single-Ended Serial Link for Off- and On-Package Communication Using a Process- and Temperature-Adaptive Voltage Regulator
John Poulton, John Wilson, Walker Turner, Brian Zimmer, Xi Chen, Sudhir Kudva, Sanquan Song, Stephen Tell, Nikola Nedovic, Wenxu Zhao, Sunil Sudhakaran, Tom Gray, William Dally
IEEE JOURNAL OF SOLID-STATE CIRCUITS

2018

Hardware-Enabled Artificial Intelligence
William Dally, Tom Gray, John Poulton, Brucek Khailany, John Wilson, Larry Dennison
Symposia on VLSI Technology and Circuits
Ground-Referenced Signaling for Intra-Chip and Short-Reach Chip-to-Chip Interconnects
Walker Turner, John Poulton, John Wilson, Xi Chen, Stephen Tell, Matt Fojtik, Trey Greer, Brian Zimmer, Sanquan Song, Nikola Nedovic, Sudhir Kudva, Sunil Sudhakaran, Rizwan Bashirullah, Wenxu Zhao, William Dally, Tom Gray
Custom Integrated Circuits Conference
A 1.17pJ/b 25Gb/s/pin Ground-Referenced Single Ended Serial Link for Off- and On-Package Communication in 16nm CMOS Using a Process- and Temperature-Adaptive Voltage Regulator
John Wilson, Walker Turner, John Poulton, Brian Zimmer, Xi Chen, Sanquan Song, Stephen Tell, Nikola Nedovic, Wenxu Zhao, Sunil Sudhakaran, Tom Gray, William Dally
ISSCC

2017

Fine-Grained DRAM: Energy-Efficient DRAM for Extreme Bandwidth Systems
Mike O'Connor, Niladrish Chatterjee, Donghyuk Lee, John Wilson, Aditya Agrawal, Steve Keckler, William Dally
International Symposium on Microarchitecture (MICRO)
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Steve Keckler, William Dally
International Symposium on Computer Architecture (ISCA)
SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks
Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Steve Keckler, William Dally
arXiv
Architecting an Energy-Efficient DRAM System for GPUs
Niladrish Chatterjee, Mike O'Connor, Donghyuk Lee, Daniel Johnson, Minsoo Rhu, Steve Keckler, William Dally
International Symposium on High Performance Computer Architecture (HPCA)

2016

EIE: Efficient Inference Engine on Compressed Deep Neural Network
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, William Dally
International Symposium on Computer Architecture (ISCA)
Current parking regulator for zero droop/overshoot load transient response
Sudhir Kudva, William Dally, Trey Greer, Tom Gray
Applied Power Electronics Conference and Exposition (APEC)
A 6.5-to-23.3fJ/b/mm Balanced Charge-Recycling Bus in 16nm FinFET CMOS at 1.7-to-2.6Gb/s/wire with Clock Forwarding and Low-Crosstalk Contraflow Wiring
John Wilson, Matt Fojtik, John Poulton, Xi Chen, Stephen Tell, Trey Greer, Tom Gray, William Dally
International Solid-State Circuits Conference (ISSCC 2016)
A 28nm 2Mbit 6T SRAM with Highly Configurable Write Assist Implementation and Capacitor Based Sense Amplifier Input Offset Compen
Mahmut Sinangil, John Poulton, Matt Fojtik, Trey Greer, Stephen Tell, Andy Gotterba, Jesse Wang, Jason Golbus, William Dally, Tom Gray
Journal of Solid State Circuits

2015

Network Endpoint Congestion Control for Fine-Grained Communication
Ted Jiang, Larry Dennison, William Dally
SC15

2014

Scaling the Power Wall: A Path to Exascale
Oreste Villa, Daniel Johnson, Mike O'Connor, Evgeny Bolotin, David Nellans, Justin Luitjens, Nikolai Sakharnykh, Peng Wang, Paulius Micikevicius, Anthony Scudiero, Steve Keckler, William Dally
SC '14

2013

A 0.54 pJ/b 20 Gb/s Ground-Referenced Single-Ended Short-Reach Serial Link in 28 nm CMOS for Advanced Packaging Applications
John Poulton, William Dally, Xi Chen, John Eyles, Trey Greer, Stephen Tell, John Wilson, Tom Gray
Journal of Solid State Circuits
21st Century Digital Design Tools
William Dally, Chris Malachosky, Steve Keckler
Design Automation Conference (DAC)
A 0.54pJ/b 20Gb/s Ground-Referenced Single-Ended Short-Haul Serial Link in 28nm CMOS for Advanced Packaging Applications
John Poulton, William Dally, Xi Chen, John Eyles, Trey Greer, Stephen Tell, Tom Gray
ISSCC

2012

Unifying Primary Cache, Scratch, and Register File Memories in a Throughput Processor
Mark Gebhart, Steve Keckler, Brucek Khailany, Ronny Krashinsky, William Dally
International Symposium on Microarchitecture (MICRO)
A Hierarchical Thread Scheduler and Register File for Energy-Efficient Throughput Processors
Mark Gebhart, Daniel R. Johnson, David Tarjan, Steve Keckler, William Dally, Erik Lindholm, Kevin Skadron
ACM Transactions on Computer Systems (TOCS)

2011

A Compile-Time Managed Multi-Level Register File Hierarchy
Mark Gebhart, Steve Keckler, William Dally
International Symposium on Microarchitecture (MICRO)
GPUs and the Future of Parallel Computing
Steve Keckler, William Dally, Brucek Khailany, Michael Garland, David Glasco
IEEE Micro
Energy-efficient Mechanisms for Managing Thread Context in Throughput Processors
Mark Gebhart, Daniel R. Johnson, David Tarjan, Steve Keckler, William Dally, Erik Lindholm, Kevin Skadron
International Symposium on Computer Architecture (ISCA)

2010

The Even/Odd Synchronizer: A Fast, All-Digital Periodic Synchronizer
William Dally, Stephen Tell
16th International Symposium on Asynchronous Circuits and Systems

2007

A 14-mW 6.25-Gb/s Transceiver in 90-nm CMOS
John Poulton, Robert Palmer, Andy Fuller, Trey Greer, John Eyles, William Dally, Mark Horowitz
IEEE Journal of Solid-State Circuits