William Dally | Research

William Dally

Bill Dally joined NVIDIA in January 2009 as chief scientist, after spending 12 years at Stanford University, where he was chairman of the computer science department. Dally and his Stanford team developed the system architecture, network architecture, signaling, routing and synchronization technology that is found in most large parallel computers today. Dally was previously at the Massachusetts Institute of Technology from 1986 to 1997, where he and his team built the J-Machine and the M-Machine, experimental parallel computer systems that pioneered the separation of mechanism from programming models and demonstrated very low overhead synchronization and communication mechanisms. From 1983 to 1986, he was at California Institute of Technology (CalTech), where he designed the MOSSIM Simulation Engine and the Torus Routing chip, which pioneered “wormhole” routing and virtual-channel flow control. He is a member of the National Academy of Engineering, a Fellow of the American Academy of Arts & Sciences, a Fellow of the IEEE and the ACM, and has received the ACM Eckert-Mauchly Award, the IEEE Seymour Cray Award, and the ACM Maurice Wilkes award. He has published over 250 papers, holds over 120 issued patents, and is an author of four textbooks. Dally received a bachelor's degree in Electrical Engineering from Virginia Tech, a master’s in Electrical Engineering from Stanford University and a Ph.D. in Computer Science from CalTech. He was a cofounder of Velio Communications and Stream Processors.

Research Area(s)

Circuits and VLSI Design

Computer Architecture

High Performance Computing

Artificial Intelligence and Machine Learning

Programming Languages, Systems and Tools

Google Scholar

https://scholar.google.com/citations?user=YZHj-Y4AAAAJ&hl=en&oi=sra

Publications

2024

A 0.190-pJ/bit 25.2-Gb/s/wire Inverter-Based AC-Coupled Transceiver for Short-Reach Die-to-Die Interfaces in 5-nm CMOS

Yoshinori Nishi, John W. Poulton, Xi Chen, Sanquan Song, Brian Zimmer, Walker Turner, Stephen Tell, Nikola Nedovic, John Wilson, William Dally, Tom Gray

IEEE Journal of Solid-State Circuits (JSSC) (Volume: 59, Issue: 4, April 2024)

2023

A 0.190-pJ/bit 25.2-Gb/s/wire Inverter-Based AC-Coupled Transceiver for Short-Reach Die-to-Die Interfaces in 5-nm CMOS

Yoshinori Nishi, John W. Poulton, Xi Chen, Sanquan Song, Brian Zimmer, Walker Turner, Stephen Tell, Nikola Nedovic, John Wilson, William Dally, Tom Gray

2023 IEEE SYMPOSIUM ON VLSI TECHNOLOGY & CIRCUITS

A 0.297-pJ/Bit 50.4-Gb/s/Wire Inverter-Based Short-Reach Simultaneous Bi-Directional Transceiver for Die-to-Die Interface in 5-nm CMOS

Yoshinori Nishi, John W. Poulton, Walker Turner, Xi Chen, Sanquan Song, Brian Zimmer, Stephen Tell, Nikola Nedovic, John Wilson, William Dally, Tom Gray

IEEE Journal of Solid-State Circuits ( Volume: 58, Issue: 4, April 2023)

A 95.6-TOPS/W Deep Learning Inference Accelerator With Per-Vector Scaled 4-bit Quantization in 5 nm

Ben Keller, Rangharajan Venkatesan, Steve Dai, Stephen Tell, Brian Zimmer, Charbel Sakr, William Dally, Tom Gray, Brucek Khailany

Journal of Solid-State Circuits

2022

LNS-Madam: Low-Precision Training in Logarithmic Number System Using Multiplicative Weight Update

Jiawei Zhao, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, Mustafa Ali, Ming-Yu Liu, Brucek Khailany, William Dally, Anima Anandkumar

IEEE Transactions on Computers (Volume: 71, Issue: 12, 01 December 2022)

Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training

Charbel Sakr, Steve Dai, Rangharajan Venkatesan, Brian Zimmer, Brucek Khailany, William Dally

2022 International Conference on Machine Learning (ICML)

A 0.297-pJ/bit 50.4-Gb/s/wire Inverter-Based Short-Reach Simultaneous Bidirectional Transceiver for Die-to-Die Interface in 5nm CMOS

Yoshinori Nishi, John W. Poulton, Xi Chen, Sanquan Song, Brian Zimmer, Walker Turner, Stephen Tell, Nikola Nedovic, John Wilson, William Dally, Tom Gray

2022 IEEE SYMPOSIUM ON VLSI TECHNOLOGY & CIRCUITS

A 17–95.6 TOPS/W Deep Learning Inference Accelerator with Per-Vector Scaled 4-bit Quantization for Transformers in 5nm

Ben Keller, Rangharajan Venkatesan, Steve Dai, Stephen Tell, Brian Zimmer, William Dally, Tom Gray, Brucek Khailany

2022 Symposium on VLSI Technology & Circuits Digest of Technical Papers

2021

Evolution of the Graphics Processing Unit (GPU)

William Dally, Steve Keckler, David B. Kirk

IEEE Micro Special Issue of the 50th Anniversary of the Microprocessor

Simba: scaling deep-learning inference with chiplet-based architecture

Yakun Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Brucek Khailany, Steve Keckler

Communications of the ACM

ACM Research Highlight

VS-QUANT: Per-Vector Scaled Quantization for Accurate Low-Precision Neural Network Inference

Steve Dai, Rangharajan Venkatesan, Haoxing (Mark) Ren, Brian Zimmer, William Dally, Brucek Khailany

2020

Accelerating Chip Design with Machine Learning

Brucek Khailany, Haoxing (Mark) Ren, Steve Dai, Saad Godil, Ben Keller, Robert Kirby, Alicia Klinefelter, Rangharajan Venkatesan, Yanqing Zhang, Bryan Catanzaro, William Dally

A 0.32–128 TOPS, Scalable Multi-Chip-Module-Based Deep Neural Network Inference Accelerator With Ground-Referenced Signaling in 16 nm

Brian Zimmer, Rangharajan Venkatesan, Yakun Sophia Shao, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany

IEEE Journal of Solid-State Circuits (JSSC)

JSSC 2020 Best Paper award

2019

MAGNet: A Modular Accelerator Generator for Neural Networks

Rangharajan Venkatesan, Sophia Shao, Miaorong Wang, Jason Clemons, Steve Dai, Matt Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Yanqing Zhang, Brian Zimmer, William Dally, Joel Emer, Steve Keckler, Brucek Khailany

International Conference On Computer Aided Design (ICCAD)

Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture

Sophia Shao, Jason Clemons, Rangharajan Venkatesan, Brian Zimmer, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Brucek Khailany, Steve Keckler

International Symposium on Microarchitecture (MICRO)

Best Paper award, IEEE Micro Top Picks in Computer Architecture (Honorable Mention)

A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator Designed with a High-Productivity VLSI Methodology

Rangharajan Venkatesan, Sophia Shao, Brian Zimmer, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany

Hot Chips: A Symposium on High Performance Chips

A 0.11 pJ/Op, 0.32-128 TOPS, Scalable Multi-Chip-Module-based Deep Neural Network Accelerator with Ground-Reference Signaling in 16nm

Brian Zimmer, Rangharajan Venkatesan, Sophia Shao, Jason Clemons, Matt Fojtik, Ted Jiang, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Priyanka Raina, Stephen Tell, Yanqing Zhang, William Dally, Joel Emer, Tom Gray, Steve Keckler, Brucek Khailany

Symposium on VLSI Circuits

Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference

Angad S. Rekhi, Brian Zimmer, Nikola Nedovic, Nigxi Liu, Rangharajan Venkatesan, Miaorong Wang, Brucek Khailany, William Dally, Tom Gray

Design Automation Conference (DAC)

A Fine-Grained GALS SoC with Pausible Adaptive Clocking in 16 nm FinFET

Matt Fojtik, Ben Keller, Alicia Klinefelter, Nathaniel Pinckney, Stephen Tell, Brian Zimmer, Tezaswi Raja, Kevin Zhou, William Dally, Brucek Khailany

ASYNC 2019 Best Paper Award

A 1.17-pJ/b, 25-Gb/s/pin Ground-Referenced Single-Ended Serial Link for Off- and On-Package Communication Using a Process- and Temperature-Adaptive Voltage Regulator

John Poulton, John Wilson, Walker Turner, Brian Zimmer, Xi Chen, Sudhir Kudva, Sanquan Song, Stephen Tell, Nikola Nedovic, Wenxu Zhao, Sunil Sudhakaran, Tom Gray, William Dally

IEEE JOURNAL OF SOLID-STATE CIRCUITS

2018

Hardware-Enabled Artificial Intelligence

William Dally, Tom Gray, John Poulton, Brucek Khailany, John Wilson, Larry Dennison

Symposia on VLSI Technology and Circuits

Ground-Referenced Signaling for Intra-Chip and Short-Reach Chip-to-Chip Interconnects

Walker Turner, John Poulton, John Wilson, Xi Chen, Stephen Tell, Matt Fojtik, Trey Greer, Brian Zimmer, Sanquan Song, Nikola Nedovic, Sudhir Kudva, Sunil Sudhakaran, Rizwan Bashirullah, Wenxu Zhao, William Dally, Tom Gray

Custom Integrated Circuits Conference

A 1.17pJ/b 25Gb/s/pin Ground-Referenced Single Ended Serial Link for Off- and On-Package Communication in 16nm CMOS Using a Process- and Temperature-Adaptive Voltage Regulator

John Wilson, Walker Turner, John Poulton, Brian Zimmer, Xi Chen, Sanquan Song, Stephen Tell, Nikola Nedovic, Wenxu Zhao, Sunil Sudhakaran, Tom Gray, William Dally

ISSCC

2017

Fine-Grained DRAM: Energy-Efficient DRAM for Extreme Bandwidth Systems

Mike O'Connor, Niladrish Chatterjee, Donghyuk Lee, John Wilson, Aditya Agrawal, Steve Keckler, William Dally

International Symposium on Microarchitecture (MICRO)

SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks

Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Steve Keckler, William Dally

International Symposium on Computer Architecture (ISCA)

SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks

Angshuman Parashar, Minsoo Rhu, Anurag Mukkara, Antonio Puglielli, Rangharajan Venkatesan, Brucek Khailany, Joel Emer, Steve Keckler, William Dally

Architecting an Energy-Efficient DRAM System for GPUs

Niladrish Chatterjee, Mike O'Connor, Donghyuk Lee, Daniel Johnson, Minsoo Rhu, Steve Keckler, William Dally

International Symposium on High Performance Computer Architecture (HPCA)

2016

EIE: Efficient Inference Engine on Compressed Deep Neural Network

Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, William Dally

International Symposium on Computer Architecture (ISCA)

Current parking regulator for zero droop/overshoot load transient response

Sudhir Kudva, William Dally, Trey Greer, Tom Gray

Applied Power Electronics Conference and Exposition (APEC)

A 6.5-to-23.3fJ/b/mm Balanced Charge-Recycling Bus in 16nm FinFET CMOS at 1.7-to-2.6Gb/s/wire with Clock Forwarding and Low-Crosstalk Contraflow Wiring

John Wilson, Matt Fojtik, John Poulton, Xi Chen, Stephen Tell, Trey Greer, Tom Gray, William Dally

International Solid-State Circuits Conference (ISSCC 2016)

A 28nm 2Mbit 6T SRAM with Highly Configurable Write Assist Implementation and Capacitor Based Sense Amplifier Input Offset Compen

Mahmut Sinangil, John Poulton, Matt Fojtik, Trey Greer, Stephen Tell, Andy Gotterba, Jesse Wang, Jason Golbus, William Dally, Tom Gray

Journal of Solid State Circuits

2015

Network Endpoint Congestion Control for Fine-Grained Communication

Ted Jiang, Larry Dennison, William Dally

2014

Scaling the Power Wall: A Path to Exascale

Oreste Villa, Daniel Johnson, Mike O'Connor, Evgeny Bolotin, David Nellans, Justin Luitjens, Nikolai Sakharnykh, Peng Wang, Paulius Micikevicius, Anthony Scudiero, Steve Keckler, William Dally

2013

A 0.54 pJ/b 20 Gb/s Ground-Referenced Single-Ended Short-Reach Serial Link in 28 nm CMOS for Advanced Packaging Applications

John Poulton, William Dally, Xi Chen, John Eyles, Trey Greer, Stephen Tell, John Wilson, Tom Gray

Journal of Solid State Circuits

21st Century Digital Design Tools

William Dally, Chris Malachosky, Steve Keckler

Design Automation Conference (DAC)

A 0.54pJ/b 20Gb/s Ground-Referenced Single-Ended Short-Haul Serial Link in 28nm CMOS for Advanced Packaging Applications

John Poulton, William Dally, Xi Chen, John Eyles, Trey Greer, Stephen Tell, Tom Gray

2012

Unifying Primary Cache, Scratch, and Register File Memories in a Throughput Processor

Mark Gebhart, Steve Keckler, Brucek Khailany, Ronny Krashinsky, William Dally

International Symposium on Microarchitecture (MICRO)

A Hierarchical Thread Scheduler and Register File for Energy-Efficient Throughput Processors

Mark Gebhart, Daniel R. Johnson, David Tarjan, Steve Keckler, William Dally, Erik Lindholm, Kevin Skadron

ACM Transactions on Computer Systems (TOCS)

2011

A Compile-Time Managed Multi-Level Register File Hierarchy

Mark Gebhart, Steve Keckler, William Dally

International Symposium on Microarchitecture (MICRO)

GPUs and the Future of Parallel Computing

Steve Keckler, William Dally, Brucek Khailany, Michael Garland, David Glasco

Energy-efficient Mechanisms for Managing Thread Context in Throughput Processors

Mark Gebhart, Daniel R. Johnson, David Tarjan, Steve Keckler, William Dally, Erik Lindholm, Kevin Skadron

International Symposium on Computer Architecture (ISCA)

2010

The Even/Odd Synchronizer: A Fast, All-Digital Periodic Synchronizer

William Dally, Stephen Tell

16th International Symposium on Asynchronous Circuits and Systems

2007

A 14-mW 6.25-Gb/s Transceiver in 90-nm CMOS

John Poulton, Robert Palmer, Andy Fuller, Trey Greer, John Eyles, William Dally, Mark Horowitz

IEEE Journal of Solid-State Circuits