1. [Publications](/publications) 2. Monte Carlo Gradient Quantization # Monte Carlo Gradient Quantization ![Publication image](/sites/default/files/styles/wide/public/default_images/default.jpeg?itok=qUFsuJCP "Publication image") We propose Monte Carlo methods to leverage both sparsity and quantization to compress gradients of neural networks throughout training. On top of reducing the communication exchanged between multiple workers in a distributed setting, we also improve the computational efficiency of each worker. Our method, called Monte Carlo Gradient Quantization (MCGQ), shows faster convergence and higher performance than existing quantization methods on image classification and language modeling. Using both low-bit-width-quantization and high sparsity levels, our method more than doubles the rates of existing compression methods from 200x to 520x and 462x to more than 1200x on different language modeling tasks. Accepted to CVPR2020 workshop ## Authors Goncalo Mordido (Hasso Plattner Institute) [Matthijs Van keirsbilck](/person/matthijs-van-keirsbilck) [Alex Keller](/person/alex-keller) ## Publication Date Sunday, June 14, 2020 ## Published in [CVPR2020 Workshop](https://openaccess.thecvf.com/content_CVPRW_2020/papers/w40/Mordido_Monte_Carlo_Gradient_Quantization_CVPRW_2020_paper.pdf) ## Research Area [Algorithms and Numerical Methods](/research-area/algorithms) [Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence) ## External Links [IEEE link](https://ieeexplore.ieee.org/document/9150640) ## Copyright This material is posted here with permission of the IEEE. Internal or personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by writing to .