Sharp asymptotics on the compression of two-layer neural networks

Mohammad Hossein Amani, Simone Bombari, Marco Mondelli, Rattana Pukdee, Stefano Rini

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M <N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimize the population L2 loss between the outputs of the target and of the compressed network, under the assumption of Gaussian inputs. By using tools from high-dimensional probability, we show that this non-convex problem can be simplified when the target network is sufficiently over-parameterized, and provide the error rate of this approximation as a function of the input dimension and N. In this mean-field limit, the simplified objective, as well as the optimal weights of the compressed network, does not depend on the realization of the target network, but only on expected scaling factors. Furthermore, for networks with ReLU activation, we conjecture that the optimum of the simplified optimization problem is achieved by taking weights on the Equiangular Tight Frame (ETF), while the scaling of the weights and the orientation of the ETF depend on the parameters of the target network. Numerical evidence is provided to support this conjecture.

Original languageEnglish
Title of host publication2022 IEEE Information Theory Workshop, ITW 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages588-593
Number of pages6
ISBN (Electronic)9781665483414
DOIs
StatePublished - 2022
Event2022 IEEE Information Theory Workshop, ITW 2022 - Mumbai, India
Duration: 1 Nov 20229 Nov 2022

Publication series

Name2022 IEEE Information Theory Workshop, ITW 2022

Conference

Conference2022 IEEE Information Theory Workshop, ITW 2022
Country/TerritoryIndia
CityMumbai
Period1/11/229/11/22

Fingerprint

Dive into the research topics of 'Sharp asymptotics on the compression of two-layer neural networks'. Together they form a unique fingerprint.

Cite this