TY - JOUR
T1 - Approximation Spaces of Deep Neural Networks
AU - Gribonval, Rémi
AU - Kutyniok, Gitta
AU - Nielsen, Morten
AU - Voigtlaender, Felix
PY - 2022
Y1 - 2022
N2 - We study the expressivity of deep neural networks. Measuring a network’s complexity by its number of connections or by its number of neurons, we consider the class of functions for which the error of best approximation with networks of a given complexity decays at a certain rate when increasing the complexity budget. Using results from classical approximation theory, we show that this class can be endowed with a (quasi)-norm that makes it a linear function space, called approximation space. We establish that allowing the networks to have certain types of “skip connections” does not change the resulting approximation spaces. We also discuss the role of the network’s nonlinearity (also known as activation function) on the resulting spaces, as well as the role of depth. For the popular ReLU nonlinearity and its powers, we relate the newly constructed spaces to classical Besov spaces. The established embeddings highlight that some functions of very low Besov smoothness can nevertheless be well approximated by neural networks, if these networks are sufficiently deep.
AB - We study the expressivity of deep neural networks. Measuring a network’s complexity by its number of connections or by its number of neurons, we consider the class of functions for which the error of best approximation with networks of a given complexity decays at a certain rate when increasing the complexity budget. Using results from classical approximation theory, we show that this class can be endowed with a (quasi)-norm that makes it a linear function space, called approximation space. We establish that allowing the networks to have certain types of “skip connections” does not change the resulting approximation spaces. We also discuss the role of the network’s nonlinearity (also known as activation function) on the resulting spaces, as well as the role of depth. For the popular ReLU nonlinearity and its powers, we relate the newly constructed spaces to classical Besov spaces. The established embeddings highlight that some functions of very low Besov smoothness can nevertheless be well approximated by neural networks, if these networks are sufficiently deep.
KW - Approximation spaces
KW - Besov spaces
KW - Deep neural networks
KW - Direct estimates
KW - Inverse estimates
KW - Piecewise polynomials
KW - ReLU activation function
KW - Sparsely connected networks
UR - http://www.scopus.com/inward/record.url?scp=85105443182&partnerID=8YFLogxK
U2 - 10.1007/s00365-021-09543-4
DO - 10.1007/s00365-021-09543-4
M3 - Journal article
SN - 0176-4276
VL - 55
SP - 259
EP - 367
JO - Constructive Approximation
JF - Constructive Approximation
IS - 1
ER -