Scaling Implicit Parallelism via Dynamic Control Replication

We present dynamic control replication, a run-time program analysis that enables scalable execution of implicitly parallel programs on large machines through a distributed and efficient dynamic dependence analysis. Dynamic control replication distributes dependence analysis by executing multiple copies of an implicitly parallel program while ensuring that they still collectively behave as a single execution. By distributing and parallelizing the dependence analysis, dynamic control replication supports efficient, on-the-fly computation of dependences for programs with arbitrary control flow at scale. We describe an asymptotically scalable algorithm for implementing dynamic control replication that maintains the sequential semantics of implicitly parallel programs.

An implementation of dynamic control replication in the Legion runtime delivers the same programmer productivity as writing in other implicitly parallel programming models, such as Dask or TensorFlow, while providing better performance (11.4X and 14.9X respectively in our experiments), and scalability to hundreds of nodes. We also show that dynamic control replication provides good absolute performance and scaling for HPC applications, competitive in many cases with explicitly parallel programming systems.

Authors: 
Wonchan Lee (NVIDIA)
Elliott Slaughter (SLAC National Accelerator Laboratory)
Zhihao Jia (Carnegie Mellon University)
Mario Di Renzo (Sapienza University of Rome)
Manolis Papadakis (NVIDIA)
Galen Shipman (Los Alamos National Laboratory)
Patrick McCormick (Los Alamos National Laboratory)
Alex Aiken (Stanford University)
Publication Date: 
Saturday, February 27, 2021
Uploaded Files: