DNN gradient lossless compression: Can GenNorm be the answer?

Zhong Jing Chen, Eduin E. Hernandez, Yu-Chih Huang, Stefano Rini

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

2 Scopus citations

Abstract

In this paper, the problem of optimal gradient lossless compression in Deep Neural Network (DNN) training is considered. Gradient compression is relevant in many distributed DNN training scenarios, including the recently popular federated learning (FL) scenario in which each remote users are connected to the parameter server (PS) through a noiseless but rate limited channel. In distributed DNN training, if the underlying gradient distribution is available, classical lossless compression approaches can be used to reduce the number of bits required for communicating the gradient entries. Mean field analysis has suggested that gradient updates can be considered as independent random variables, while Laplace approximation can be used to argue that gradient has a distribution approximating the normal (Norm) distribution in some regimes. In this paper we argue that, for some networks of practical interest, the gradient entries can be well modelled as having a generalized normal (GenNorm) distribution. We provide numerical evaluations to validate that the hypothesis GenNorm modelling provides a more accurate prediction of the DNN gradient tail distribution. Additionally, this modeling choice provides concrete improvement in terms of lossless compression of the gradients when applying classical fix-to-variable lossless coding algorithms, such as Huffman coding, to the quantized gradient updates. This latter results indeed provides an effective compression strategy with low memory and computational complexity that has great practical relevance in distributed DNN training scenarios.

Original languageEnglish
Title of host publicationICC 2022 - IEEE International Conference on Communications
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages407-412
Number of pages6
ISBN (Electronic)9781538683477
DOIs
StatePublished - 2022
Event2022 IEEE International Conference on Communications, ICC 2022 - Seoul, Korea, Republic of
Duration: 16 May 202220 May 2022

Publication series

NameIEEE International Conference on Communications
Volume2022-May
ISSN (Print)1550-3607

Conference

Conference2022 IEEE International Conference on Communications, ICC 2022
Country/TerritoryKorea, Republic of
CitySeoul
Period16/05/2220/05/22

Keywords

  • Distributed optimization
  • DNN training
  • Generalized normal distribution
  • Lossless compression

Fingerprint

Dive into the research topics of 'DNN gradient lossless compression: Can GenNorm be the answer?'. Together they form a unique fingerprint.

Cite this