1. [Publications](/publications) 2. LNS-Madam: Low-Precision Training in Logarithmic Number System Using Multiplicative Weight Update # LNS-Madam: Low-Precision Training in Logarithmic Number System Using Multiplicative Weight Update ![](/sites/default/files/styles/wide/public/publications/LNS-Madam.JPG?itok=aYPCWOaT) Representing deep neural networks (DNNs) in low-precision is a promising approach to enable efficient acceleration and memory reduction. Previous methods that train DNNs in low-precision typically keep a copy of weights in high-precision during the weight updates. Directly training with low-precision weights leads to accuracy degradation due to complex interactions between the low-precision number systems and the learning algorithms. To address this issue, we develop a co-designed low-precision training framework, termed LNS-Madam, in which we jointly design a logarithmic number system (LNS) and a multiplicative weight update algorithm (Madam). We prove that LNS-Madam results in low quantization error during weight updates, leading to stable performance even if the precision is limited. We further propose a hardware design of LNS-Madam that resolves practical challenges in implementing an efficient datapath for LNS computations. Our implementation effectively reduces energy overhead incurred by LNS-to-integer conversion and partial sum accumulation. Experimental results show that LNS-Madam achieves comparable accuracy to full-precision counterparts with only 8 bits on popular computer vision and natural language tasks. Compared to FP32 and FP8, LNS-Madam reduces the energy consumption by over 90% and 55%, respectively. ## Authors Jiawei Zhao (Caltech) [Steve Dai](/person/steve-dai) [Rangharajan Venkatesan](/person/rangharajan-venkatesan) [Brian Zimmer](/person/brian-zimmer) Mustafa Ali (Purdue University) [Ming-Yu Liu](/person/ming-yu-liu) [Brucek Khailany](/person/brucek-khailany) [William Dally](/person/william-dally) Anima Anandkumar (NVIDIA) ## Publication Date Thursday, December 1, 2022 ## Published in [IEEE Transactions on Computers (Volume: 71, Issue: 12, 01 December 2022)](https://www.computer.org/csdl/journal/tc) ## Research Area [Artificial Intelligence and Machine Learning ](/research-area/machine-learning-artificial-intelligence) [Circuits and VLSI Design](/research-area/circuits) [Computer Architecture](/research-area/computer-architecture) ## External Links [IEEE Xplore](https://ieeexplore.ieee.org/abstract/document/9900267) ## Copyright This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. For more information, see