DEEP RELU NETWORKS PRESERVE EXPECTED LENGTH

Boris Hanin; Ryan Jeong; David Rolnick

DEEP RELU NETWORKS PRESERVE EXPECTED LENGTH

Boris Hanin, Ryan Jeong, David Rolnick

Operations Research & Financial Engineering

Princeton University

Research output: Contribution to conference › Paper › peer-review

Abstract

Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the network distorts length - if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs? It has been widely believed that this length grows exponentially in network depth. We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization. We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes. These theoretical results are corroborated by our experiments.

Original language	American English
State	Published - 2022
Event	10th International Conference on Learning Representations, ICLR 2022 - Virtual, Online Duration: Apr 25 2022 → Apr 29 2022

Conference

Conference	10th International Conference on Learning Representations, ICLR 2022
City	Virtual, Online
Period	4/25/22 → 4/29/22

ASJC Scopus subject areas

Language and Linguistics
Computer Science Applications
Education
Linguistics and Language

Cite this

@conference{6b8d8a6aa64c4c84b30d0f9cecedc60d,

title = "DEEP RELU NETWORKS PRESERVE EXPECTED LENGTH",

abstract = "Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the network distorts length - if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs? It has been widely believed that this length grows exponentially in network depth. We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization. We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes. These theoretical results are corroborated by our experiments.",

author = "Boris Hanin and Ryan Jeong and David Rolnick",

note = "Publisher Copyright: {\textcopyright} 2022 ICLR 2022 - 10th International Conference on Learning Representationss. All rights reserved.; 10th International Conference on Learning Representations, ICLR 2022 ; Conference date: 25-04-2022 Through 29-04-2022",

year = "2022",

language = "American English",

}

TY - CONF

T1 - DEEP RELU NETWORKS PRESERVE EXPECTED LENGTH

AU - Hanin, Boris

AU - Jeong, Ryan

AU - Rolnick, David

PY - 2022

Y1 - 2022

N2 - Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the network distorts length - if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs? It has been widely believed that this length grows exponentially in network depth. We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization. We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes. These theoretical results are corroborated by our experiments.

AB - Assessing the complexity of functions computed by a neural network helps us understand how the network will learn and generalize. One natural measure of complexity is how the network distorts length - if the network takes a unit-length curve as input, what is the length of the resulting curve of outputs? It has been widely believed that this length grows exponentially in network depth. We prove that in fact this is not the case: the expected length distortion does not grow with depth, and indeed shrinks slightly, for ReLU networks with standard random initialization. We also generalize this result by proving upper bounds both for higher moments of the length distortion and for the distortion of higher-dimensional volumes. These theoretical results are corroborated by our experiments.

UR - http://www.scopus.com/inward/record.url?scp=85132309710&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85132309710&partnerID=8YFLogxK

M3 - Paper

T2 - 10th International Conference on Learning Representations, ICLR 2022

Y2 - 25 April 2022 through 29 April 2022

ER -

DEEP RELU NETWORKS PRESERVE EXPECTED LENGTH

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this