J Stat Phys
DOI 10.1007/s10955-017-1836-5
Why Does Deep and Cheap Learning Work So Well?
Henry W. Lin1·Max Tegmark2·David Rolnick3
Received: 3 December 2016 / Accepted: 27 June 2017
© Springer Science+Business Media, LLC 2017
Abstract We show how the success of deep learning could depend not only on mathematics
but also on physics: although well-known mathematical theorems guarantee that neural net-
works can approximate arbitrary functions well, the class of functions of practical interest
can frequently be approximated through “cheap learning” with exponentially fewer param-
eters than generic ones. We explore how properties frequently encountered in physics such
as symmetry, locality, compositionality, and polynomial log-probability translate into excep-
tionally simple neural networks. We further argue that when the statistical process generating
the data is of a certain hierarchical form prevalent in physics and machine learning, a deep
neural network can be more efﬁcient than a shallow one. We formalize these claims using
information theory and discuss the relation to the renormalization group. We prove various
“no-ﬂattening theorems” showing when efﬁcient linear deep networks cannot be accurately
approximated by shallow ones without efﬁciency loss; for example, we show that nvariables
cannot be multiplied using fewer than 2nneurons in a single hidden layer.
Keywords Artiﬁcial neural networks ·Deep learning ·Statistical physics
1 Introduction
Deep learning works remarkably well, and has helped dramatically improve the state-of-
the-art in areas ranging from speech recognition, translation and visual object recognition
to drug discovery, genomics and automatic game playing [ 1,2]. However, it is still not fully
understood why deep learning works so well. In contrast to GOFAI (“good old-fashioned
AI”) algorithms that are hand-crafted and fully understood analytically, many algorithms
B Henry W. Lin
henrylin@college.harvard.edu
1Department of Physics, Harvard University, Cambridge, MA 02138, USA
2Department of Physics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
3Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
123
H. W. Lin et al.
using artiﬁcial neural networks are understood only at a heuristic level, where we empiri-
cally know that certain training protocols employing large data sets will result in excellentperformance. This is reminiscent of the situation with human brains: we know that if we traina child according to a certain curriculum, she will learn certain skills—but we lack a deepunderstanding of how her brain accomplishes this.
This makes it timely and interesting to develop new analytic insights on deep learning
and its successes, which is the goal of the present paper. Such improved understanding is notonly interesting in its own right, and for potentially providing new clues about how brainswork, but it may also have practical applications. Better understanding the shortcomings ofdeep learning may suggest ways of improving it, both to make it more capable and to makeit more robust [ 3].
1.1 The Swindle: Why Does “Cheap Learning” Work?
Throughout this paper, we will adopt a physics perspective on the problem, to prevent
application-speciﬁc details from obscuring simple general results related to dynamics, sym-metries, renormalization, etc, and to exploit useful similarities between deep learning and
statistical mechanics.
The task of approximating functions of many variables is central to most applications
of machine learning, including unsupervised learning, classiﬁcation and prediction, as illus-trated in Fig. 1. For example, if we are interested in classifying faces, then we may want our
neural network to implement a function where we feed in an image represented by a milliongreyscale pixels and get as output the probability distribution over a set of people that theimage might represent.
When investigating the quality of a neural net, there are several important factors to
consider:
Unsupervised
learning
Prediction Classificationp(x,y)
p(y|x) p(x|y)
Fig. 1 In this paper, we follow the machine learning convention that xrefers to data (e.g., an image) and yrefers
to underlying information about that data (such as a label for the image). Neural networks can be used to estimate(or sample from) probability distributions with respect to xandy, given many samples. Classiﬁcation involves
approximating the probability distribution of ygiven x, in the case that yis discrete-valued. This problem
may also be called prediction ,e . g .w h e n xis earlier data in a time series. Generation involves approximating
the probability distribution for xgiven y, or drawing samples from this distribution. Unsupervised learning
attempts to approximate or model the probability distribution of x, without any knowledge of y
123
Why Does Deep and Cheap…
•Expressibility : What class of functions can the neural network express?
•Efﬁciency : How many resources (neurons, parameters, etc) does the neural network
require to approximate a given function?
•Learnability : How rapidly can the neural network learn good parameters for approximat-
ing a function?
This paper is focused on expressibility and efﬁciency, and more speciﬁcally on the fol-
lowing well-known [ 4–6] problem: How can neural networks approximate functions well
in practice, when the set of possible functions is exponentially larger than the set of practi-cally possible networks? For example, suppose that we wish to classify megapixel greyscale
images into two categories, e.g., cats or dogs. If each pixel can take one of 256 values, then
there are 256
1000,000possible images, and for each one, we wish to compute the probability
that it depicts a cat. This means that an arbitrary function is deﬁned by a list of 2561000,000
probabilities, i.e., way more numbers than there are atoms in our universe (about 1078).
Yet neural networks with merely thousands or millions of parameters somehow manage
to perform such classiﬁcation tasks quite well. How can deep learning be so “cheap”, in thesense of requiring so few parameters?
We will see in below that neural networks perform a combinatorial swindle, replacing
exponentiation by multiplication: if there are say n=10
6inputs taking v=256 values
each, this swindle cuts the number of parameters from vntov×ntimes some constant
factor. We will show that this success of this swindle depends fundamentally on physics:although neural networks only work well for an exponentially tiny fraction of all possibleinputs, the laws of physics are such that the data sets we care about for machine learning(natural images, sounds, drawings, text, etc) are also drawn from an exponentially tiny fraction
of all imaginable data sets. Moreover, we will see that these two tiny subsets are remarkablysimilar, enabling deep learning to work well in practice.
The rest of this paper is organized as follows. In Sect. 2, we present results for shallow
neural networks with merely a handful of layers, focusing on simpliﬁcations due to locality,symmetry and polynomials. In Sect. 3, we study how increasing the depth of a neural network
can provide polynomial or exponential efﬁciency gains even though it adds nothing in termsof expressivity, and we discuss the connections to renormalization, compositionality andcomplexity. We summarize our conclusions in Sect. 4.
2 Expressibility and Efﬁciency of Shallow Neural Networks
Let us now explore what classes of probability distributions pare the focus of physics and
machine learning, and how accurately and efﬁciently neural networks can approximate them.We will be interested in probability distributions p(x|y),w h e r e xranges over some sample
space and ywill be interpreted either as another variable being conditioned on or as a model
parameter. For a machine learning example, we might interpret yas an element of some set
of animals {cat,dog,rabbit,...}andxas the vector of pixels in an image depicting such
an animal, so that p(x|y)fory=catgives the probability distribution of images of cats
with different coloring, size, posture, viewing angle, lighting condition, electronic cameranoise, etcFor a physics example, we might interpret yas an element of some set of metals
{iron,aluminum ,copper ,...}andxas the vector of magnetization values for different parts
of a metal bar. The prediction problem is then to evaluate p(x|y), whereas the classiﬁcation
problem is to evaluate p(y|x).
Because of the above-mentioned “swindle”, accurate approximations are only possible
for a tiny subclass of all probability distributions. Fortunately, as we will explore below,
123
H. W. Lin et al.
the function p(x|y)often has many simplifying features enabling accurate approximation,
because it follows from some simple physical law or some generative model with relativelyfew free parameters: for example, its dependence on xmay exhibit symmetry, locality and/or
be of a simple form such as the exponential of a low-order polynomial. In contrast, thedependence of p(y|x)onytends to be more complicated; it makes no sense to speak of
symmetries or polynomials involving a variable y=cat.
Let us therefore start by tackling the more complicated case of modeling p(y|x).T h i s
probability distribution p(y|x)is determined by the hopefully simpler function p(x|y)via
Bayes’ theorem:
p(y|x)=p(x|y)p(y)
/summationtext
y/primep(x|y/prime)(y/prime), (1)
where p(y)is the probability distribution over y(animals or metals, say) ap r i o r i ,b e f o r e
examining the data vector x.
2.1 Probabilities and Hamiltonians
It is useful to introduce the negative logarithms of two of these probabilities:
Hy(x)≡− lnp(x|y),
μy≡− lnp(y). (2)
Table 1is a brief dictionary translating between physics and machine-learning terminology.
Statisticians refer to −lnpas “self-information” or “surprisal”, and statistical physicists
refer to Hy(x)as the Hamiltonian , quantifying the energy of x(up to an arbitrary and irrel-
evant additive constant) given the parameter y. These deﬁnitions transform Eq. ( 1) into the
Boltzmann form
p(y|x)=1
N(x)e−[Hy(x)+μx], (3)
where
N(x)≡/summationdisplay
ye−[Hy(x)+μy]. (4)
Table 1 Physics-ML dictionary
Physics Machine learning
Hamiltonian Surprisal −lnp
Simple H Cheap learning
Quadratic H Gaussian p
Locality SparsityTranslationally symmetric H Convnet
Computing pfrom H Softmaxing
Spin BitFree energy difference KL-divergenceEffective theory Nearly lossless data distillationIrrelevant operator Noise
Relevant operator Feature
123
Why Does Deep and Cheap…
This recasting of Eq. ( 1) is useful because the Hamiltonian tends to have properties making
it simple to evaluate. We will see in Sect. 3that it also helps understand the relation between
deep learning and renormalization [ 7].
2.2 Bayes’ Theorem as a Softmax
Since the variable ytakes one of a discrete set of values, we will often write it as an index
instead of as an argument, as py(x)≡p(y|x). Moreover, we will often ﬁnd it convenient to
view all values indexed by yas elements of a vector, written in boldface, thus viewing py,
Hyandμyas elements of the vectors p,Handμ, respectively. Equation ( 3) thus simpliﬁes
to
p(x)=1
N(x)e−[H(x)+μ], (5)
using the standard convention that a function (in this case exp) applied to a vector acts on its
elements.
We wish to investigate how well this vector-valued function p(x)can be approximated
by a neural net. A standard n-layer feedforward neural network maps vectors to vectors
by applying a series of linear and nonlinear transformations in succession. Speciﬁcally, itimplements vector-valued functions of the form [ 1]
f(x)=σ
nAn···σ2A2σ1A1x, (6)
where the σiare relatively simple nonlinear operators on vectors and the Aiare afﬁne
transformations of the form Aix=Wix+bifor matrices Wiand so-called bias vectors bi.
Popular choices for these nonlinear operators σiinclude
•Local function apply some nonlinear function σto each vector element,
•Max-pooling compute the maximum of all vector elements,
•Softmax exponentiate all vector elements and normalize them to so sum to unity
˜σ(x)≡ex
/summationtext
ieyi. (7)
(We use ˜σto indicate the softmax function and σto indicate an arbitrary non-linearity,
optionally with certain regularity requirements).
This allows us to rewrite Eq. ( 5)a s
p(x)=˜σ[−H(x)−μ]. (8)
This means that if we can compute the Hamiltonian vector H(x)with some n-layer neural
net, we can evaluate the desired classiﬁcation probability vector p(x)by simply adding a
softmax layer. The μ-vector simply becomes the bias term in this ﬁnal layer.
2.3 What Hamiltonians can be Approximated by Feasible Neural Networks?
It has long been known that neural networks are universal1approximators [ 8,9], in the sense
that networks with virtually all popular nonlinear activation functions σ(x)can approximate
any smooth function to any desired accuracy—even using merely a single hidden layer.
1Neurons are universal analog computing modules in much the same way that NAND gates are universal
digital computing modules: any computable function can be accurately evaluated by a sufﬁciently large networkof them. Just as NAND gates are not unique (NOR gates are also universal), nor is any particular neuronimplementation—indeed, anygeneric smooth nonlinear activation function is universal [ 8,9].
123
H. W. Lin et al.
However, these theorems do not guarantee that this can be accomplished with a network of
feasible size, and the following simple example explains why they cannot: There are 22n
different Boolean functions of nvariables, so a network implementing a generic function
in this class requires at least 2nbits to describe, i.e., more bits than there are atoms in our
universe if n>260.
The fact that neural networks of feasible size are nonetheless so useful therefore implies
that the class of functions we care about approximating is dramatically smaller. We will seebelow in Sect. 2.4that both physics and machine learning tend to favor Hamiltonians that are
polynomials
2—indeed, often ones that are sparse, symmetric and low-order. Let us therefore
focus our initial investigation on Hamiltonians that can be expanded as a power series:
Hy(x)=h+/summationdisplay
ihixi+/summationdisplay
i≤jhijxixj+/summationdisplay
i≤j≤khijkxixjxk+··· . (9)
If the vector xhasncomponents ( i=1,..., n), then there are (n+d)!/(n!d!)terms of
degree up to d.
2.3.1 Continuous Input Variables
If we can accurately approximate multiplication using a small number of neurons, then
we can construct a network efﬁciently approximating any polynomial Hy(x)by repeated
multiplication and addition. We will now see that we can, using any smooth but otherwisearbitrary non-linearity σthat is applied element-wise. The popular logistic sigmoid activation
function σ(x)=1/(1+e
−x)will do the trick.
Theorem 1 Let fbe a neural network of the form f=A2σA1,w h e r e σacts elementwise by
applying some smooth non-linear function σto each element. Let the input layer, hidden layer
and output layer have sizes 2, 4 and 1, respectively. Then fcan approximate a multiplication
gate arbitrarily well.
To see this, let us ﬁrst Taylor-expand the function σaround the origin:
σ(u)=σ0+σ1u+σ2u2
2+O(u3). (10)
Without loss of generality, we can assume that σ2/negationslash=0: since σis non-linear, it must have a
non-zero second derivative at some point, so we can use the biases in A1to shift the origin
to this point to ensure σ2/negationslash=0. Equation ( 10) now implies that
m(u,v)≡σ(u+v)+σ(−u−v)−σ(u−v)−σ(−u+v)
4σ2
=uv/bracketleftbig
1+O/parenleftbig
u2+v2/parenrightbig/bracketrightbig
, (11)
where we will term m(u,v)themultiplication approximator . Taylor’s theorem guarantees
that m(u,v)is an arbitrarily good approximation of uvfor arbitrarily small |u|and|v|.
However, we can always make |u|and|v|arbitrarily small by scaling A1→λA1and
2The class of functions that can be exactly expressed by a neural network must be invariant under composition,
since adding more layers corresponds to using the output of one function as the input to another. Importantsuch classes include linear functions, afﬁne functions, piecewise linear functions (generated by the popularRectiﬁed Linear unit “ReLU” activation function σ(x)=max[0,x]), polynomials, continuous functions and
smooth functions whose n
thderivatives are continuous. According to the Stone-Weierstrass theorem, both
polynomials and piecewise linear functions can approximate continuous functions arbitrarily well.
123
Why Does Deep and Cheap…
uvuvContinuous multiplication gate:
w1 uvuvwBinary multiplication gate:
1-2
Fig. 2 Multiplication can be efﬁciently implemented by simple neural nets, becoming arbitrarily accurate as
λ→0(left)a n dβ→∞ (right ). Squares apply the function σ, circles perform summation, and lines multiply
by the constants labeling them. The “1” input implements the bias term. The left gate requires σ/prime/prime(0)/negationslash=0,
which can always be arranged by biasing the input to σ.T h e right gate requires the sigmoidal behavior
σ(x)→0a n dσ(x)→1a s x→− ∞ andx→∞ , respectively
then compensating by scaling A2→λ−2A2. In the limit that λ→∞ , this approximation
becomes exact.3In other words, arbitrarily accurate multiplication can always be achieved
using merely 4 neurons. Figure 2illustrates such a multiplication approximator. (Of course, a
practical algorithm like stochastic gradient descent cannot achieve arbitrarily large weights,though a reasonably good approximation can be achieved already for λ
−1∼10.)
Corollary 1 For any given multivariate polynomial and any tolerance /epsilon1> 0, there exists
a neural network of ﬁxed ﬁnite size N (independent of /epsilon1) that approximates the polynomial
to accuracy better than /epsilon1. Furthermore, N is bounded by the complexity of the polynomial,
scaling as the number of multiplications required times a factor that is typically slightlylarger than 4.
4
This is a stronger statement than the classic universal universal approximation theorems
for neural networks [ 8,9], which guarantee that for every /epsilon1there exists some N(/epsilon1), but allows
for the possibility that N(/epsilon1)→∞ as/epsilon1→0. An approximation theorem in [ 10] provides
an/epsilon1-independent bound on the size of the neural network, but at the price of choosing a
pathological function σ.
2.3.2 Discrete Input Variables
For the simple but important case where xis a vector of bits, so that xi=0o r xi=1, the
fact that x2
i=ximakes things even simpler. This means that only terms where all variables
3The limit where λ→∞ but|A1|2|A2|is held constant is very similar in spirit to the ’t Hooft limit in large
Nquantum ﬁeld theories where g2Nis held ﬁxed but N→∞ . The extra terms in the Taylor series which are
suppressed at large λare analogous to the suppression of certain Feynman diagrams at large N. The authors
thank Daniel Roberts for pointing this out.
4In addition to the four neurons required for each multiplication, additional neurons may be deployed to copy
variables to higher layers bypassing the nonlinearity in σ. Such linear “copy gates” implementing the function
u→uare of course trivial to implement using a simpler version of the above procedure: using A1to shift and
scale down the input to fall in a tiny range where σ/prime(u)/negationslash=0, and then scaling it up and shifting accordingly
with A2.
123
H. W. Lin et al.
are different need be included, which simpliﬁes Eq. ( 9)t o
Hy(x)=h+/summationdisplay
ihixi+/summationdisplay
i Dln 10≈23. In summary, when xis a bit string, an arbitrary function
py(x)can be evaluated by a simple 3-layer neural network: the middle layer uses sigmoid
functions to compute the products from Eq. ( 12), and the top layer performs the sums from
Eq. ( 12) and the softmax from Eq. ( 8).
2.4 What Hamiltonians Do We Want to Approximate?
We have seen that polynomials can be accurately approximated by neural networks using a
number of neurons scaling either as the number of multiplications required (for the continuouscase) or as the number of terms (for the binary case). But polynomials per se are no panacea:
with binary input, all functions are polynomials, and with continuous input, there are (n+
d)!/(n!d!)coefﬁcients in a generic polynomial of degree dinnvariables, which easily
becomes unmanageably large. We will now discuss situations in which exceptionally simplepolynomials that are sparse, symmetric and/or low-order play a special role in physics andmachine learning.
2.4.1 Low Polynomial Order
The Hamiltonians that show up in physics are not random functions, but tend to be poly-nomials of very low order, typically of degree ranging from 2 to 4. The simplest exampleis of course the harmonic oscillator, which is described by a Hamiltonian that is quadraticin both position and momentum. There are many reasons why low order polynomials showup in physics. Two of the most important ones are that sometimes a phenomenon can bestudied perturbatively, in which case, Taylor’s theorem suggests that we can get away witha low order polynomial approximation. A second reason is renormalization: higher order
123
Why Does Deep and Cheap…
terms in the Hamiltonian of a statistical ﬁeld theory tend to be negligible if we only observe
macroscopic variables.
At a fundamental level, the Hamiltonian of the standard model of particle physics has
d=4. There are many approximations of this quartic Hamiltonian that are accurate in speciﬁc
regimes, for example the Maxwell equations governing electromagnetism, the Navier-Stokesequations governing ﬂuid dynamics, the Alvén equations governing magnetohydrodynam-ics and various Ising models governing magnetization—all of these approximations haveHamiltonians that are polynomials in the ﬁeld variables, of degree dranging from 2 to 4.
This means that the number of polynomial coefﬁcients in many examples is not inﬁnite
as in Eq. ( 9) or exponential in nas in Eq. ( 12), merely of order O(n
4).
There are additional reasons why we might expect low order polynomials. Thanks to the
Central Limit Theorem [ 11], many probability distributions in machine learning and statistics
can be accurately approximated by multivariate Gaussians, i.e., of the form
p(x)=eh+/summationtext
ihjxi−/summationtext
ijhijxixj, (14)
which means that the Hamiltonian H=− lnpis a quadratic polynomial. More generally,
the maximum-entropy probability distribution subject to constraints on some of the lowestmoments, say expectation values of the form /angbracketleftxα1
1xα2
2···xαnn/angbracketrightfor some integers αi≥0 would
lead to a Hamiltonian of degree no greater than d≡/summationtext
iαi[12].
Image classiﬁcation tasks often exploit invariance under translation, rotation, and various
nonlinear deformations of the image plane that move pixels to new locations. All such spatialtransformations are linear functions ( d=1 polynomials) of the pixel vector x. Functions
implementing convolutions and Fourier transforms are also d=1 polynomials.
Of course, such arguments do not imply that we should expect to see low-order polynomials
in every application. If we consider some data set generated by a very simple Hamiltonian(say the Ising Hamiltonian), but then discard some of the random variables, the resultingmarginalized distribution can become quite complicated and of high order. Similarly, if we donot observe the random variables directly, but observe some generic functions of the randomvariables, the result will generally be a mess. These arguments, however, might indicate thatthe probability of encountering a Hamiltonian described by a low-order polynomial in someapplication might be signiﬁcantly higher than what one might expect from some naive prior.For example, a uniform prior on the space of all polynomials of degree Nwould suggest that
a randomly chosen polynomial would almost always have degree N, but this might be a bad
prior for real-world applications.
We should also note that even if a Hamiltonian is described exactly by a low-order poly-
nomial, we would not expect the corresponding neural network to reproduce a low-orderpolynomial Hamiltonian exactly in any practical scenario for a host of possible reasonsincluding limited data, the requirement of inﬁnite weights for inﬁnite accuracy, and the fail-ure of practical algorithms such as stochastic gradient descent to ﬁnd the global minimumof a cost function in many scenarios. So looking at the weights of a neural network trainedon actual data may not be a good indicator of whether or not the underlying Hamiltonian isa polynomial of low degree or not.
2.4.2 Locality
One of the deepest principles of physics is locality : that things directly affect only what is in
their immediate vicinity. When physical systems are simulated on a computer by discretizingspace onto a rectangular lattice, locality manifests itself by allowing only nearest-neighbor
123
H. W. Lin et al.
interaction. In other words, almost all coefﬁcients in Eq. ( 9) are forced to vanish, and the total
number of non-zero coefﬁcients grows only linearly with n. For the binary case of Eq. ( 9),
which applies to magnetizations (spins) that can take one of two values, locality also limitsthe degree dto be no greater than the number of neighbors that a given spin is coupled to
(since all variables in a polynomial term must be different).
Again, the applicability of these considerations to particular machine learning applica-
tions must be determined on a case by case basis. Certainly, an arbitrary transformation ofa collection of local random variables will result in a non-local collection. (This might ruinlocality in certain ensembles of images, for example). But there are certainly cases in physicswhere locality is still approximately preserved, for example in the simple block-spin renor-malization group, spins are grouped into blocks, which are then treated as random variables.To a high degree of accuracy, these blocks are only coupled to their nearest neighbors. Suchlocality is famously exploited by both biological and artiﬁcial visual systems, whose ﬁrstneuronal layer performs merely fairly local operations.
2.4.3 Symmetry
Whenever the Hamiltonian obeys some symmetry (is invariant under some transformation),the number of independent parameters required to describe it is further reduced. For instance,many probability distributions in both physics and machine learning are invariant undertranslation and rotation. As an example, consider a vector xof air pressures y
imeasured
by a microphone at times i=1,..., n. Assuming that the Hamiltonian describing it has
d=2 reduces the number of parameters Nfrom∞to(n+1)(n+2)/2. Further assuming
locality (nearest-neighbor couplings only) reduces this to N=2n, after which requiring
translational symmetry reduces the parameter count to N=3. Taken together, the constraints
on locality, symmetry and polynomial order reduce the number of continuous parameters inthe Hamiltonian of the standard model of physics to merely 32 [ 13]. Naturally, this does
not mean that modeling a real physical system requires merely 32 parameters - the objectsinvolved must be modeled also; there too, however, symmetry allows us to abstract awayfrom the information contained in individual particles to that summarizing components ofthe system.
Symmetry can reduce not merely the parameter count, but also the computational com-
plexity. For example, if a linear vector-valued function f(x)mapping a set of nvariables onto
itself happens to satisfy translational symmetry, then it is a convolution (implementable bya convolutional neural net; “convnet”), which means that it can be computed with nlog
2n
rather than n2multiplications using Fast Fourier transform.
3 Why Deep?
Above we investigated how probability distributions from physics and computer science
applications lent themselves to “cheap learning”, being accurately and efﬁciently approxi-mated by neural networks with merely a handful of layers. Let us now turn to the separatequestion of depth, i.e., the success of deep learning: what properties of real-world probability
distributions cause efﬁciency to further improve when networks are made deeper? This ques-tion has been extensively studied from a mathematical point of view [ 14–16], but mathematics
alone cannot fully answer it, because part of the answer involves physics. We will argue thatthe answer involves the hierarchical/compositional structure of generative processes togetherwith inability to efﬁciently “ﬂatten” neural networks reﬂecting this structure.
123
Why Does Deep and Cheap…
3.1 Hierarchical Processess
One of the most striking features of the physical world is its hierarchical structure. Spatially,
it is an object hierarchy: elementary particles form atoms which in turn form molecules, cells,organisms, planets, solar systems, galaxies, etcCausally, complex structures are frequently
created through a distinct sequence of simpler steps.
Figure 3gives two examples of such causal hierarchies generating data vectors y
0/mapsto→
y1/mapsto→.../mapsto→ynthat are relevant to physics and image classiﬁcation, respectively. Both
examples involve a Markov chain5where the probability distribution p(yi)at the ithlevel of
the hierarchy is determined from its causal predecessor alone:
pi=Mipi−1, (15)
where the probability vector pispeciﬁes the probability distribution of p(yi)according to
(pi)y≡p(yi)and the Markov matrix Mispeciﬁes the transition probabilities between two
neighboring levels, p(yi|yi−1). Iterating Eq. ( 15)g i v e s
pn=MnMn−1···M1p0, (16)
so we can write the combined effect of the the entire generative process as a matrix product.
In our physics example (Fig. 3, left), a set of cosmological parameters y0(the density of
dark matter, etc) determines the power spectrum y1of density ﬂuctuations in our universe,
which in turn determines the pattern of cosmic microwave background radiation y2reaching
us from our early universe, which gets combined with foreground radio noise from ourGalaxy to produce the frequency-dependent sky maps ( y
3) that are recorded by a satellite-
based telescope that measures linear combinations of different sky signals and adds electronicreceiver noise. For the recent example of the Planck Satellite [ 17], these datasets y
i,y2,...
contained about 101,1 04,1 08,1 09and 1012numbers, respectively.
More generally, if a given data set is generated by a (classical) statistical physics process,
it must be described by an equation in the form of Eq. ( 16), since dynamics in classical
physics is fundamentally Markovian: classical equations of motion are always ﬁrst orderdifferential equations in the Hamiltonian formalism. This technically covers essentially alldata of interest in the machine learning community, although the fundamental Markoviannature of the generative process of the data may be an in-efﬁcient description.
Our toy image classiﬁcation example (Fig. 3, right) is deliberately contrived and over-
simpliﬁed for pedagogy: y
0is a single bit signifying “cat or dog”, which determines a set of
parameters determining the animal’s coloration, body shape, posture, etcusing approxiate
probability distributions, which determine a 2D image via ray-tracing, which is scaled andtranslated by random amounts before a randomly generated background is added.
In both examples, the goal is to reverse this generative hierarchy to learn about the input
y≡y
0from the output yn≡x, speciﬁcally to provide the best possibile estimate of the
probability distribution p(y|y)=p(y0|yn)—i.e., to determine the probability distribution
for the cosmological parameters and to determine the probability that the image is a cat,respectively.
5If the next step in the generative hierarchy requires knowledge of not merely of the present state but also
information of the past, the present state can be redeﬁned to include also this information, thus ensuring thatthe generative process is a Markov process.
123
H. W. Lin et al.
y0=y
y1
y2
y3
>y3=T3(x)>y2=T2(x)>y0=T0(x)
y4M4M1
M3M2
>y1=T1(x)
x=y4add
foregroundssimulate
sky map
n, n , Q, T/S T, h b
Pixel 1 Pixel 2 T
6422347 6443428 -454.841
3141592 2718281 141.421
8454543 9345593 654.766
1004356 8345388 -305.567 ... ... ...TELESCOPE
DATACMB SKY
MAP
TRANSFORMEDOBJECTRAY-TRACED
OBJECTCATEGORY
LABEL
FINAL IMAGESOLIDWORKS
PARAMTERSPOWER
SPECTRUMCOSMO-
LOGICALPARAMTERScat or dog?take linear
combinations,
add noiseray trace select background select color,
shape & posture f3f2f1f0generate
fluctuationsscale & translate
FREQ UENC Y
MAPSparam 1 param 2 param 3
6422347 6443428 -454.841
3141592 2718281 141.421
8454543 9345593 654.7661004356 8345388 -305.567
 ... ... ...y0=y
y1
y2
y3
y4M4M1
M3M2
Fig. 3 Causal hierarchy examples relevant to physics ( left) and image classiﬁcation ( right ). As information
ﬂows down the hierarchy y0→y1→...→yn=y, some of it is destroyed by random Markov processes.
However, no further information is lost as information ﬂows optimally back up the hierarchy as /hatwideyn−1→...→
/hatwidey0. The right example is deliberately contrived and over-simpliﬁed for pedagogy; for example, translation and
scaling are more naturally performed before ray tracing, which in turn breaks down into multiple steps.
3.2 Resolving the Swindle
This decomposition of the generative process into a hierarchy of simpler steps helps resolve
the“swindle” paradox from the introduction: although the number of parameters requiredto describe an arbitrary function of the input data yis beyond astronomical, the generative
process can be speciﬁed by a more modest number of parameters, because each of its stepscan. Whereas specifying an arbitrary probability distribution over multi-megapixel images
xrequires far more bits than there are atoms in our universe, the information specifying
how to compute the probability distribution p(x|y)for a microwave background map ﬁts
into a handful of published journal articles or software packages [ 18–24]. For a megapixel
image of a galaxy, its entire probability distribution is deﬁned by the standard model ofparticle physics with its 32 parameters [ 13], which together specify the process transforming
primordial hydrogen gas into galaxies.
123
Why Does Deep and Cheap…
The same parameter-counting argument can also be applied to all artiﬁcial images of inter-
est to machine learning: for example, giving the simple low-information-content instruction“draw a cute kitten” to a random sample of artists will produce a wide variety of images y
with a complicated probability distribution over colors, postures, etc, as each artist makes
random choices at a series of steps. Even the pre-stored information about cat probabilitiesin these artists’ brains is modest in size.
Note that a random resulting image typically contains much more information than the
generative process creating it; for example, the simple instruction “generate a random string
of10
9bits” contains much fewer than 109bits. Not only are the typical steps in the generative
hierarchy speciﬁed by a non-astronomical number of parameters, but as discussed in Sect.2.4, it is plausible that neural networks can implement each of the steps efﬁciently.
6
A deep neural network stacking these simpler networks on top of one another would then
implement the entire generative process efﬁciently. In summary, the data sets and functionswe care about form a minuscule minority, and it is plausible that they can also be efﬁcientlyimplemented by neural networks reﬂecting their generative process. So what is the remainder?Which are the data sets and functions that we do notcare about?
Almost all images are indistinguishable from random noise, and almost all data sets and
functions are indistinguishable from completely random ones. This follows from Borel’stheorem on normal numbers [ 26], which states that almost all real numbers have a string of
decimals that would pass any randomness test, i.e., are indistinguishable from random noise.
Simple parameter counting shows that deep learning (and our human brains, for that matter)would fail to implement almost all such functions, and training would fail to ﬁnd any usefulpatterns. To thwart pattern-ﬁnding efforts. cryptography therefore aims to produces random-looking patterns. Although we might expect the Hamiltonians describing human-generateddata sets such as drawings, text and music to be more complex than those describing simplephysical systems, we should nonetheless expect them to resemble the natural data sets thatinspired their creation much more than they resemble random functions.
3.3 Sufﬁcient Statistics and Hierarchies
The goal of deep learning classiﬁers is to reverse the hierarchical generative process as wellas possible, to make inferences about the input yfrom the output x. Let us now treat this
hierarchical problem more rigorously using information theory.
Given P(y|x),asufﬁcient statistic T (x)is deﬁned by the equation P(y|x)=P(y|T(x))
and has played an important role in statistics for almost a century [ 27]. All the information
about ycontained in xis contained in the sufﬁcient statistic. A minimal sufﬁcient statistic [27]
is some sufﬁcient statistic T
∗which is a sufﬁcient statistic for all other sufﬁcient statistics.
This means that if T(y)is sufﬁcient, then there exists some function fsuch that T∗(y)=
f(T(y)). As illustrated in Fig. 3,T∗can be thought of as a an information distiller, optimally
compressing the data so as to retain all information relevant to determining yand discarding
all irrelevant information.
The sufﬁcient statistic formalism enables us to state some simple but important results
that apply to any hierarchical generative process cast in the Markov chain form of Eq. ( 16).
Theorem 2 Given a Markov chain described by our notation above, let T ibe a minimal
sufﬁcient statistic of P (yi|yn). Then there exists some functions f isuch that T i=fi◦Ti+1.
6Although our discussion is focused on describing probability distributions, which are not random, stochastic
neural networks can generate random variables as well. In biology, spiking neurons provide a good randomnumber generator, and in machine learning, stochastic architectures such as restricted Boltzmann machines[25] do the same.
123
H. W. Lin et al.
More casually speaking, the generative hierarchy of Fig. 3can be optimally reversed one
step at a time: there are functions f ithat optimally undo each of the steps, distilling out all
information about the level above that was not destroyed by the Markov process. Here isthe proof. Note that for any k ≥1, the “backwards” Markov property P (y
i|yi+1,yi+k)=
P(yi|yi+1)follows from the Markov property via Bayes’ theorem:
P(yi|yi+k,yi+1)=P(yi+k|yi,yi+1)P(yi|yi+1)
P(yi+k|yi+1)
=P(yi+k|yi+1)P(yi|yi+1)
P(yi+k|yi+1)
=P(yi|yi+1). (17)
Using this fact, we see that
P(yi|yn)=/summationdisplay
yi+1P(yi|yi+1yn)P(yi+1|yn)
=/summationdisplay
yi+1P(yi|yi+1)P(yi+1|Ti+1(yn)). (18)
Since the above equation depends on y nonly through T i+1(yn), this means that T i+1is a
sufﬁcient statistic for P (yi|yn). But since T iis the minimal sufﬁcient statistic, there exists a
function f isuch that T i=fi◦Ti+1.
Corollary 2 With the same assumptions and notation as theorem 2, deﬁne the function
f0(T0)=P(y0|T0)and f n=Tn−1.T h e n
P(y0|yn)=(f0◦f1◦···◦ fn)(yn). (19)
The proof is easy. By induction,
T0=f1◦f2◦···◦ Tn−1,which implies the corollary . (20)
Roughly speaking, Corollary 2 states that the structure of the inference problem reﬂects the
structure of the generative process. In this case, we see that the neural network trying toapproximate P (y|x)must approximate a compositional function. We will argue below in
Sect. 3.6that in many cases, this can only be accomplished efﬁciently if the neural network
has/greaterorsimilarn hidden layers.
In neuroscience parlance, the functions f
icompress the data into forms with ever more
invariance [28], containing features invariant under irrelevant transformations (for example
background substitution, scaling and translation).
Let us denote the distilled vectors /hatwideyi≡fi(/hatwideyi+1),w h e r e /hatwideyn≡y. As summarized by Fig.
3, as information ﬂows down the hierarchy y =y0→y1→...exn=x, some of it is
destroyed by random processes. However, no further information is lost as information ﬂowsoptimally back up the hierarchy as y →/hatwidey
n−1→···→ /hatwidey0.
3.4 Approximate Information Distillation
Although minimal sufﬁcient statistics are often difﬁcult to calculate in practice, it is frequently
possible to come up with statistics which are nearly sufﬁcient in a certain sense which wenow explain.
An equivalent characterization of a sufﬁcient statistic is provided by information theory
[29,30]. The data processing inequality [30] states that for any function fand any random
123
Why Does Deep and Cheap…
variables x,y,
I(x,y)≥I(x,f(y)), (21)
where Iis the mutual information :
I(x,y)=/summationdisplay
x,yp(x,y)logp(x,y)
p(x)p(y). (22)
A sufﬁcient statistic T(x)is a function f(x)for which “ ≥” gets replaced by “ =”i nE q .( 21),
i.e., a function retaining all the information about y.
Even information distillation functions fthat are not strictly sufﬁcient can be very useful
as long as they distill out most of the relevant information and are computationally efﬁcient.
For example, it may be possible to trade some loss of mutual information with a dramaticreduction in the complexity of the Hamiltonian; e.g., H
y(f(x))may be considerably easier
to implement in a neural network than Hy(x). Precisely this situation applies to the physical
example described in Fig. 3, where a hierarchy of efﬁcient near-perfect information distillers
fihave been found, the numerical cost of f3[23,24],f2[21,22],f1[19,20]a n d f0[17] scal-
ing with the number of inputs parameters nasO(n),O(n3/2),O(n2)and O(n3), respectively.
More abstractly, the procedure of renormalization, ubiquitous in statistical physics, can beviewed as a special case of approximate information distillation, as we will now describe.
3.5 Distillation and Renormalization
The systematic framework for distilling out desired information from unwanted “noise” inphysical theories is known as Effective Field Theory [ 31]. Typically, the desired information
involves relatively large-scale features that can be experimentally measured, whereas thenoise involves unobserved microscopic scales. A key part of this framework is known astherenormalization group (RG) transformation [ 31,32]. Although the connection between
RG and machine learning has been studied or alluded to repeatedly [ 7,33–36], there are
signiﬁcant misconceptions in the literature concerning the connection which we will nowattempt to clear up.
Let us ﬁrst review a standard working deﬁnition of what renormalization is in the context
of statistical physics, involving three ingredients: a vector yof random variables, a course-
graining operation Rand a requirement that this operation leaves the Hamiltonian invariant
except for parameter changes. We think of yas the microscopic degrees of freedom—typically
physical quantities deﬁned at a lattice of points (pixels or voxels) in space. Its probabilitydistribution is speciﬁed by a Hamiltonian H
y(x), with some parameter vector y.W ei n t e r p r e t
the map R:y→yas implementing a coarse-graining7of the system. The random variable
R(y)also has a Hamiltonian, denoted H/prime(R(y)), which we require to have the same functional
form as the original Hamiltonian Hy, although the parameters ymay change. In other words,
H/prime(R(x))=Hr(y)(R(x))for some function r. Since the domain and the range of Rcoincide,
this map Rcan be iterated ntimes Rn=R◦R◦··· R, giving a Hamiltonian Hrn(y)(Rn(x))
for the repeatedly renormalized data. Similar to the case of sufﬁcient statistics, P(y|Rn(x))
will then be a compositional function.
7A typical renormalization scheme for a lattice system involves replacing many spins (bits) with a single
spin according to some rule. In this case, it might seem that the map Rcould not possibly map its domain
onto itself, since there are fewer degrees of freedom after the coarse-graining. On the other hand, if we let thedomain and range of Rdiffer, we cannot easily talk about the Hamiltonian as having the same functional form,
since the renormalized Hamiltonian would have a different domain than the original Hamiltonian. Physicistsget around this by taking the limit where the lattice is inﬁnitely large, so that Rmaps an inﬁnite lattice to an
inﬁnite lattice.
123
H. W. Lin et al.
Contrary to some claims in the literature, effective ﬁeld theory and the renormaliza-
tion group have little to do with the idea of unsupervised learning and pattern-ﬁnding.Instead, the standard renormalization procedures in statistical physics are essentially afeature extractor for supervised learning, where the features typically correspond to long-
wavelength/macroscopic degrees of freedom. In other words, effective ﬁeld theory onlymakes sense if we specify what features we are interested in. For example, if we are givendata xabout the position and momenta of particles inside a mole of some liquid and is tasked
with predicting from this data whether or not Alice will burn her ﬁnger when touching theliquid, a (nearly) sufﬁcient statistic is simply the temperature of the object, which can inturn be obtained from some very coarse-grained degrees of freedom (for example, one coulduse the ﬂuid approximation instead of working directly from the positions and momenta of∼10
23particles). But without specifying that we wish to predict (long-wavelength physics),
there is nothing natural about an effective ﬁeld theory approximation.
To be more explicit about the link between renormalization and deep-learning, consider
a toy model for natural images. Each image is described by an intensity ﬁeld φ(r),w h e r e
ris a 2-dimensional vector. We assume that an ensemble of images can be described by a
quadratic Hamiltonian of the form
Hy(φ)=/integraldisplay/bracketleftBig
y0φ2+y1(∇φ)2+y2/parenleftbig
∇2φ/parenrightbig2+···/bracketrightBig
d2r. (23)
Each parameter vector ydeﬁnes an ensemble of images; we could imagine that the ﬁctitious
classes of images that we are trying to distinguish are all generated by Hamiltonians Hywith
the same above form but different parameter vectors y. We further assume that the function
φ(r)is speciﬁed on pixels that are sufﬁciently close that derivatives can be well-approximated
by differences. Derivatives are linear operations, so they can be implemented in the ﬁrst layerof a neural network. The translational symmetry of Eq. ( 23) allows it to be implemented with
a convnet. If can be shown [ 31] that for any course-graining operation that replaces each
block of b×bpixels by its average and divides the result by b
2, the Hamiltonian retains the
form of Eq. ( 23) but with the parameters yireplaced by
y/prime
i=b2−2iyi. (24)
This means that all parameters yiwith i≥2 decay exponentially with bas we repeatedly
renormalize and bkeeps increasing, so that for modest b, one can neglect all but the ﬁrst few
yi’s. What would have taken an arbitrarily large neural network can now be computed on a
neural network of ﬁnite and bounded size, assuming that we are only interested in classifyingthe data based only on the coarse-grained variables. These insufﬁcient statistics will stillhave discriminatory power if we are only interested in discriminating Hamiltonians whichall differ in their ﬁrst few C
k. In this example, the parameters y0andy1correspond to “relevant
operators” by physicists and “signal” by machine-learners, whereas the remaining parameterscorrespond to “irrelevant operators” by physicists and “noise” by machine-learners.
The ﬁxed point structure of the transformation in this example is very simple, but one can
imagine that in more complicated problems the ﬁxed point structure of various transforma-tions might be highly non-trivial. This is certainly the case in statistical mechanics problemswhere renormalization methods are used to classify various phases of matters; the point hereis that the renormalization group ﬂow can be thought of as solving the pattern-recognitionproblem of classifying the long-range behavior of various statistical systems.
123
Why Does Deep and Cheap…
In summary, renormalization can be thought of as a type of supervised learning,8where
the large scale properties of the system are considered the features. If the desired featuresare not large-scale properties (as in most machine learning cases), one might still expectthe a generalized formalism of renormalization to provide some intuition to the problem byreplacing a scale transformation with some other transformation. But calling some proce-dure renormalization or not is ultimately a matter of semantics; what remains to be seen iswhether or not semantics has teeth, namely, whether the intuition about ﬁxed points of therenormalization group ﬂow can provide concrete insight into machine learning algorithms.In many numerical methods, the purpose of the renormalization group is to efﬁciently andaccurately evaluate the free energy of the system as a function of macroscopic variables ofinterest such as temperature and pressure. Thus we can only sensibly talk about the accuracyof an RG-scheme once we have speciﬁed what macroscopic variables we are interested in.
3.6 No-Flattening Theorems
Above we discussed how Markovian generative models cause p(x|y)to be a composition of
a number of simpler functions fi. Suppose that we can approximate each function fiwith
an efﬁcient neural network for the reasons given in Sect. 2. Then we can simply stack these
networks on top of each other, to obtain an deep neural network efﬁciently approximating
p(x|y).
But is this the most efﬁcient way to represent p(x|y)? Since we know that there are
shallower networks that accurately approximate it, are any of these shallow networks asefﬁcient as the deep one, or does ﬂattening necessarily come at an efﬁciency cost?
To be precise, for a neural network fdeﬁned by Eq. ( 6), we will say that the neural network
f
/lscript
/epsilon1is the ﬂattened version of fif its number /lscriptof hidden layers is smaller and f/lscript
/epsilon1approximates
fwithin some error /epsilon1(as measured by some reasonable norm). We say that f/lscript
/epsilon1is aneuron-
efﬁcient ﬂattening if the sum of the dimensions of its hidden layers (sometimes referred to as
the number of neurons Nn)i sl e s st h a nf o r f. We say that f/lscript
/epsilon1is asynapse-efﬁcient ﬂattening if
the number Nsof non-zero entries (sometimes called synapses) in its weight matrices is less
than for f.T h i sl e t su sd e ﬁ n et h e ﬂattening cost of a network fas the two functions
Cn(f,/lscript ,/epsilon1) ≡min
f/lscript
/epsilon1Nn(f/lscript
/epsilon1)
Nn(f), (25)
Cs(f,/lscript ,/epsilon1) ≡min
f/lscript
/epsilon1Ns(f/lscript
/epsilon1)
Ns(f), (26)
specifying the factor by which optimal ﬂattening increases the neuron count and the synapse
count, respectively. We refer to results where Cn>1o r Cs>1 for some class of functions
fas“no-ﬂattening theorems” , since they imply that ﬂattening comes at a cost and efﬁcient
ﬂattening is impossible. A complete list of no-ﬂattening theorems would show exactly whendeep networks are more efﬁcient than shallow networks.
8A subtlety regarding the above statements is presented by the Multi-scale Entanglement Renormalization
Ansatz (MERA) [ 37]. MERA can be viewed as a variational class of wave functions whose parameters can
be tuned to to match a given wave function as closely as possible. From this perspective, MERA is as an
unsupervised machine learning algorithm, where classical probability distributions over many variables are
replaced with quantum wavefunctions. Due to the special tensor network structure found in MERA, theresulting variational approximation of a given wavefunction has an interpretation as generating an RG ﬂow.Hence this is an example of an unsupervised learning problem whose solution gives rise to an RG ﬂow. Thisis only possible due to the extra mathematical structure in the problem (the speciﬁc tensor network found inMERA); a generic variational Ansatz does not give rise to any RG interpretation and vice versa.
123
H. W. Lin et al.
There has already been very interesting progress in this spirit, but crucial questions remain.
On one hand, it has been shown that deep is not always better, at least empirically for someimage classiﬁcation tasks [ 38]. On the other hand, many functions fhave been found for which
the ﬂattening cost is signiﬁcant. Certain deep Boolean circuit networks are exponentiallycostly to ﬂatten [ 39]. Two families of multivariate polynomials with an exponential ﬂattening
cost C
nare constructed in [ 14]. Poggio et al. [ 6], Mhaskar et al. [ 15], Mhaskar and Poggio
[16] focus on functions that have tree-like hierarchical compositional form, concluding that
the ﬂattening cost Cnis exponential for almost all functions in Sobolev space. For the ReLU
activation function, [ 40] ﬁnds a class of functions that exhibit exponential ﬂattening costs;
[41] study a tailored complexity measure of deep versus shallow ReLU networks. Eldan
and Shamir [ 42] shows that given weak conditions on the activation function, there always
exists at least one function that can be implemented in a 3-layer network which has anexponential ﬂattening cost. Finally, [ 43,44] study the differential geometry of shallow versus
deep networks, and ﬁnd that ﬂattening is exponentially neuron-inefﬁcient. Further workelucidating the cost of ﬂattening various classes of functions will clearly be highly valuable.
3.7 Linear No-Flattening Theorems
In the mean time, we will now see that interesting no-ﬂattening results can be obtained evenin the simpler-to-model context of linear neural networks [ 45], where the σoperators are
replaced with the identity and all biases are set to zero such that A
iare simply linear operators
(matrices). Every map is speciﬁed by a matrix of real (or complex) numbers, and compositionis implemented by matrix multiplication.
One might suspect that such a network is so simple that the questions concerning ﬂat-
tening become entirely trivial: after all, successive multiplication with ndifferent matrices
is equivalent to multiplying by a single matrix (their product). While the effect of ﬂatteningis indeed trivial for expressibility (fcan express any linear function, independently of how
many layers there are), this is not the case for the learnability , which involves non-linear and
complex dynamics despite the linearity of the network [ 45]. We will show that the efﬁciency
of such linear networks is also a very rich question.
Neuronal efﬁciency is trivially attainable for linear networks, since all hidden-layer neu-
rons can be eliminated without accuracy loss by simply multiplying all the weight matricestogether. We will instead consider the case of synaptic efﬁciency and set /lscript=/epsilon1=0.
Many divide-and-conquer algorithms in numerical linear algebra exploit some factoriza-
tion of a particular matrix Ain order to yield signiﬁcant reduction in complexity. For example,
when Arepresents the discrete Fourier transform (DFT), the fast Fourier transform (FFT)
algorithm makes use of a sparse factorization of Awhich only contains
O(nlogn)non-zero
matrix elements instead of the naive single-layer implementation, which contains n2non-
zero matrix elements. As ﬁrst pointed out in [ 46], this is an example where depth helps and,
in our terminology, of a linear no-ﬂattening theorem: fully ﬂattening a network that performsan FFT of nvariables increases the synapse count N
sfrom O(nlogn)toO(n2),i.e., incurs a
ﬂattening cost Cs=O(n/logn)∼O(n). This argument applies also to many variants and
generalizations of the FFT such as the Fast Wavelet Transform and the Fast Walsh-HadamardTransform.
Another important example illustrating the subtlety of linear networks is matrix multipli-
cation. More speciﬁcally, take the input of a neural network to be the entries of a matrix M
and the output to be NM, where both MandNhave size n×n. Since matrix multiplication
is linear, this can be exactly implemented by a 1-layer linear neural network. Amazingly,the naive algorithm for matrix multiplication, which requires n
3multiplications, is not opti-
123
Why Does Deep and Cheap…
mal: the Strassen algorithm [ 47] requires only O(nω)multiplications (synapses), where
ω=log27≈2.81, and recent work has cut this scaling exponent down to ω≈2.3728639
[48]. This means that fully optimized matrix multiplication on a deep neural network has a
ﬂattening cost of at least Cs=O(n0.6271361).
Low-rank matrix multiplication gives a more elementary no-ﬂattening theorem. If Ais a
rank- kmatrix, we can factor it as A=BCwhere Bis ak×nmatrix and Cis an n×kmatrix.
Hence the number of synapses is n2for an /lscript=0 network and 2 nkfor an /lscript=1-network,
giving a ﬂattening cost Cs=n/2k>1 as long as the rank k 0, there exists a neural
network that can implement the function/producttextn
i=1xiusing a single hidden layer of 2nneurons.
123
Why Does Deep and Cheap…
Furthermore, this is the smallest possible number of neurons in any such network with only
a single hidden layer.
This result may be compared to problems in Boolean circuit complexity, notably the
question of whether TC0=TC1[50]. Here circuit depth is analogous to number of layers,
and the number of gates is analogous to the number of neurons. In both the Boolean circuitmodel and the neural network model, one is allowed to use neurons/gates which have anunlimited number of inputs. The constraint in the deﬁnition of TC
ithat each of the gate
elements be from a standard universal library ( AND,OR,NOT,Majority ) is analogous
to our constraint to use a particular nonlinear function. Note, however, that our theorem isweaker by applying only to depth 1, while TC
0includes all circuits of depth O(1).
A.1 Proof that 2nNeurons are Sufﬁcient
A neural network with a single hidden layer of mneurons that approximates a product gate
forninputs can be formally written as a choice of constants aijandwjsatisfying
m/summationdisplay
j=1wjσ/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBigg
≈n/productdisplay
i=1xi. (A1)
Here, we use ≈to denote that the two sides of ( A1) have identical Taylor expansions up
to terms of degree n; as we discussed earlier in our construction of a product gate for two
inputs, this exables us to achieve arbitrary accuracy /epsilon1by ﬁrst scaling down the factors xi,
then approximately multiplying them and ﬁnally scaling up the result.
We may expand ( A1) using the deﬁnition σ(x)=/summationtext∞
k=0σkxkand drop terms of the Taylor
expansion with degree greater than n, since they do not affect the approximation. Thus, we
wish to ﬁnd the minimal msuch that there exist constants aijandwjsatisfying
σnm/summationdisplay
j=1wj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggn
=n/productdisplay
i=1xi, (A2)
σkm/summationdisplay
j=1wj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggk
=0, (A3)
for all 0 ≤k≤n−1. Let us set m=2n, and enumerate the subsets of {1,..., n}as
S1,..., Smin some order. Deﬁne a network of mneurons in a single hidden layer by setting
aijequal to the function si(Sj)which is −1i fi∈Sjand+1 otherwise, setting
wj≡1
2nn!σnn/productdisplay
i=1aij=(−1)|Sj|
2nn!σn. (A4)
In other words, up to an overall normalization constant, all coefﬁcients aijandwjequal±1,
and each weight wjis simply the product of the corresponding aij.
We must prove that this network indeed satisﬁes Eqs. ( A2)a n d( A3). The essence of our
proof will be to expand the left hand side of Eq. ( A1) and show that all monomial terms except
x1···xncome in pairs that cancel. To show this, consider a single monomial p(x)=xr1
1···xrnn
where r1+...+rn=r≤n.
Ifp(x)/negationslash=/producttextn
i=1xi, then we must show that the coefﬁcient of p(x)inσr/summationtextm
j=1wj/parenleftbig/summationtextn
i=1aijxi/parenrightbigris 0. Since p(x)/negationslash=/producttextn
i=1xi, there must be some i0such that ri0=0. In
other words, p(x)does not depend on the variable xi0. Since the sum in Eq. ( A1)i so v e ra l l
123
H. W. Lin et al.
combinations of ±signs for all variables, every term will be canceled by another term where
the (non-present) xi0has the opposite sign and the weight wjhas the opposite sign:
σrm/summationdisplay
j=1wj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggr
=σr/summationdisplay
Sj(−1)|Sj|
2nn!σr/parenleftBiggn/summationdisplay
i=1si(Sj)xi/parenrightBiggr
=σr/summationdisplay
Sj/negationslash/owneri0/bracketleftbigg(−1)|Sj|
2nn!σr/parenleftBiggn/summationdisplay
i=1si(Sj)xi/parenrightBiggr
+(−1)|Sj∪{i0}|
2nn!σr/parenleftBiggn/summationdisplay
i=1si(Sj∪{i0})xi/parenrightBiggr/bracketrightbigg
=/summationdisplay
Sj/negationslash/owneri0(−1)|Sj|
2nn!/bracketleftbigg/parenleftBiggn/summationdisplay
i=1si(Sj)xi/parenrightBiggr
−/parenleftBiggn/summationdisplay
i=1si(Sj∪{i0})xi/parenrightBiggr/bracketrightbigg
Observe that the coefﬁcient of p(x)is equal in/parenleftbig/summationtextn
i=1si(Sj)xi/parenrightbigrand/parenleftbig/summationtextn
i=1si(Sj∪{i0})xi/parenrightbigr,s i n c e ri0=0. Therefore, the overall coefﬁcient of p(x)in the
above expression must vanish, which implies that ( A3) is satisﬁed.
If instead p(x)=/producttextn
i=1xi, then all terms have the coefﬁcient of p(x)in/parenleftbig/summationtextn
i=1aijxi/parenrightbign
isn!/producttextn
i=1aij=(−1)|Sj|n!, because all n!terms are identical and there is no cancelation.
Hence, the coefﬁcient of p(x)on the left-hand side of ( A2)i s
σnm/summationdisplay
j=1(−1)|Sj|
2nn!σn(−1)|Sj|n!=1,
completing our proof that this network indeed approximates the desired product gate.
From the standpoint of group theory, our construction involves a representation of the
group G= Zn
2, acting upon the space of polynomials in the variables x1,x2,..., xn.T h e
group Gis generated by elements gisuch that giﬂips the sign of xiwherever it occurs. Then,
our construction corresponds to the computation
f(x1,..., xn)=(1−g1)(1−g2)···(1−gn)σ(x1+x2+...+xn).
Every monomial of degree at most n, with the exception of the product x1···xn,i ss e n tt o
0b y(1−gi)for at least one choice of i. Therefore, f(x1,..., xn)approximates a product
gate (up to a normalizing constant).
123
Why Does Deep and Cheap…
A.2 Proof that 2nNeurons are Necessary
Suppose that Sis a subset of {1,..., n}and consider taking the partial derivatives of ( A2)a n d
(A3), respectively, with respect to all the variables {xh}h∈S. Then, we obtain the equalities
n!σn
(n−|S|)!m/summationdisplay
j=1wj/productdisplay
h∈Sahj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggn−|S|
=/productdisplay
h/∈Sxh, (A5)
k!σk
(k−|S|)!m/summationdisplay
j=1wj/productdisplay
h∈Sahj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggk−|S|
=0, (A6)
for all 0 ≤k≤n−1. Let Adenote the 2n×mmatrix with elements
ASj≡/productdisplay
h∈Sahj. (A7)
We will show that Ahas full row rank. Suppose, towards contradiction, that ctA=0for
some non-zero vector c. Speciﬁcally, suppose that there is a linear dependence between rows
ofAgiven by
r/summationdisplay
/lscript=1c/lscriptAS/lscript,j=0, (A8)
where the S/lscriptare distinct and c/lscript/negationslash=0f o re v e r y /lscript.L e t sbe the maximal cardinality of any S/lscript.
Deﬁning the vector dwhose components are
dj≡wj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggn−s
, (A9)
taking the dot product of Eq. ( A8) with dgives
0=ctAd=r/summationdisplay
/lscript=1c/lscriptm/summationdisplay
j=1wj/productdisplay
h∈S/lscriptahj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggn−s
=/summationdisplay
/lscript|(|S/lscript|=s)c/lscriptm/summationdisplay
j=1wj/productdisplay
h∈S/lscriptahj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBiggn−|S/lscript|
(A10)
+/summationdisplay
/lscript|(|S/lscript|<s)c/lscriptm/summationdisplay
j=1wj/productdisplay
h∈S/lscriptahj/parenleftBiggn/summationdisplay
i=1aijxi/parenrightBigg(n+|S/lscript|−s)−|S/lscript|
.
Applying Eq. ( A6) (with k=n+|S/lscript|−s) shows that the second term vanishes. Substituting
Eq. ( A5) now simpliﬁes Eq. ( A10)t o
0=/summationdisplay
/lscript|(|S/lscript|=s)c/lscript(n−|S/lscript|)!
n!σn/productdisplay
h/∈S/lscriptxh, (A11)
i.e., to a statement that a set of monomials are linearly dependent. Since all distinct monomials
are in fact linearly independent, this is a contradiction of our assumption that the S/lscriptare distinct
andc/lscriptare nonzero. We conclude that Ahas full row rank, and therefore that m≥2n,w h i c h
concludes the proof.
123
H. W. Lin et al.
References
1. LeCun, Y ., Bengio, Y ., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)
2. Bengio, Y .: Learning deep architectures for AI, foundations and trends®. Mach. Learn. 2, 1–127 (2009)
3. Russell, S., Dewey, D., Tegmark, M.: Research priorities for robust and beneﬁcial artiﬁcial intelligence.
AI Mag. 36, 105–114 (2015)
4. Herbrich, R., Williamson, R.C.: Algorithmic luckiness. J. Mach. Learn. Res. 3, 175–212 (2002)
5. Shawe-Taylor, J., Bartlett, P.L., Williamson, R.C., Anthony, M.: Structural risk minimization over data-
dependent hierarchies. IEEE Trans. Inf. Theory 44, 1926–1940 (1998)
6. Poggio, T., Anselmi, F., Rosasco, L.: I-theory on depth vs width: hierarchical function composition. Center
Brains Minds Mach. (2015). Technical Reports
7. Mehta, P., Schwab, D.J.: An exact mapping between the variational renormalization group and deep
learning. arXiv:1410.3831 (2014)
8. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators.
Neural Netw. 2, 359–366 (1989)
9. Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2,
303–314 (1989)
10. Pinkus, A.: Approximation theory of the MLP model in neural networks. Acta Numer. 8, 143–195 (1999)
11. Gnedenko, B., Kolmogorov, A., Gnedenko, B., Kolmogorov, A.: Limit distributions for sums of indepen-
dent. Am. J. Math. 105, 28–35 (1954)
12. Jaynes, E.T.: Information theory and statistical mechanics. Phys. Rev. 106, 620 (1957)
13. Tegmark, M., Aguirre, A., Rees, M.J., Wilczek, F.: Dimensionless constants, cosmology, and other dark
matters. Phys. Rev. D 73, 023505 (2006)
14. Delalleau, O., Bengio, Y .: Shallow vs. deep sum-product networks. In: Advances in Neural Information
Processing Systems, pp. 666–674 (2011)
15. Mhaskar, H., Liao, Q., Poggio, T.: Learning functions: when is deep better than shallow. arXiv:1603.00988
(2016)
16. Mhaskar, H., Poggio, T.: Deep vs. shallow networks: an approximation theory perspective.
arXiv:1608.03287 (2016)
17. Adam, R., Ade, P., Aghanim, N., Akrami, Y ., Alves, M., Arnaud, M., Arroja, F., Aumont, J., Baccigalupi,
C., Ballardini, M., et al.: arXiv:1502.01582 (2015)
18. Seljak, U., Zaldarriaga, M.: A line of sight approach to cosmic microwave background anisotropies.
arXiv:astro-ph/9603033 (1996)
19. Tegmark, M.: How to measure CMB power spectra without losing information. Phys. Rev. D 55, 5895
(1997)
20. Bond, J., Jaffe, A.H., Knox, L.: Estimating the power spectrum of the cosmic microwave background.
P h y s .R e v .D 57, 2117 (1998)
21. Tegmark, M., de Oliveira-Costa, A., Hamilton, A.J.: High resolution foreground cleaned CMB map from
WMAP. Phys. Rev. D 68, 123523 (2003)
22. Ade, P., Aghanim, N., Armitage-Caplan, C., Arnaud, M., Ashdown, M., Atrio-Barandela, F., Aumont, J.,
Baccigalupi, C., Banday, A.J., Barreiro, R., et al.: Planck 2013 results. XII. Diffuse component separation.Astron. Astrophys. 571, A12 (2014)
23. Tegmark, M.: How to make maps from cosmic microwave background data without losing information.
Astrophys. J. Lett. 480, L87 (1997)
24. Hinshaw, G., Barnes, C., Bennett, C., Greason, M., Halpern, M., Hill, R., Jarosik, N., Kogut, A., Limon,
M., Meyer, S., et al.: First-year Wilkinson microwave anisotropy probe (WMAP) WMAP is the resultof a partnership between Princeton University and the NASA Goddard Space Flight Center. Scientiﬁc
guidance is provided by the WMAP Science Team. Observations: data processing methods and systematic
error limits. Astrophys. J. Suppl. Ser. 148, 63 (2003)
25. Hinton, G.: A practical guide to training restricted Boltzmann machines. Momentum 9, 926 (2010)
26. Émile Borel, M.: Les probabilités dénombrables et leurs applications arithmétiques. Rendiconti del Cir-
colo Matematico di Palermo (1884–1940) 27, 247–271 (1909)
27. Fisher, R.A.: On the mathematical foundations of theoretical statistics. Philos. Trans. R. Soc. Lond. Ser.
A Contain. Pap. Math. Phys. Charact. 222, 309–368 (1922)
28. Riesenhuber, M., Poggio, T.: Models of object recognition. Nat. Neurosci. 3, 1199–1204 (2000)
29. Kullback, S., Leibler, R.A.: On information and sufﬁciency. Ann. Math. Stat.
22, 79–86 (1951). doi: 10.
1214/aoms/1177729694
30. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (2012)31. Kardar, M.: Statistical Physics of Fields. Cambridge University Press, Cambridge (2007)
123
Why Does Deep and Cheap…
32. Cardy, J.: Scaling and Renormalization in Statistical Physics, vol. 5. Cambridge University Press, Cam-
bridge (1996)
33. Johnson, J.K., Malioutov, D.M., Willsky, A.S.: Lagrangian relaxation for MAP estimation in graphical
models. arXiv:0710.0013 (2007)
34. Bény, C.: Deep learning and the renormalization group. arXiv:1301.3124 (2013)
35. Saremi, S., Sejnowski, T.J.: Hierarchical model of natural images and the origin of scale invariance. Proc.
Natl. Acad. Sci. 110, 3071–3076 (2013). http://www.pnas.org/content/110/8/3071.full.pdf ,http://www.
pnas.org/content/110/8/3071.abstract
36. Miles Stoudenmire, E., Schwab, D.J.: Supervised learning with quantum-inspired tensor networks.
arXiv:1605.05775 (2016)
37. Vidal, G.: Class of quantum many-body states that can be efﬁciently simulated. Phys. Rev. Lett. 101,
110501 (2008). arXiv:quant-ph/0610099
38. Ba, J., Caruana, R.: Do deep nets really need to be deep? In: Advances in Neural Information Processing
Systems, pp. 2654–2662 (2014)
39. Hastad, J.: Almost optimal lower bounds for small depth circuits. In: Proceedings of the Eighteenth
Annual ACM Symposium on Theory of Computing, pp. 6–20. Organization ACM (1986)
40. Telgarsky, M.: Representation beneﬁts of deep feedforward networks. arXiv:1509.08101 (2015)
41. Montufar, G.F., Pascanu, R., Cho, K., Bengio, Y .: On the number of linear regions of deep neural networks.
In: Advances in Neural Information Processing Systems, pp. 2924–2932 (2014)
42. Eldan, R., Shamir, O.: The power of depth for feedforward neural networks. arXiv:1512.03965 (2015)
43. Poole, B., Lahiri, S., Raghu, M., Sohl-Dickstein, J., Ganguli, S.: Exponential expressivity in deep neural
networks through transient chaos. arXiv:1606.05340 (2016)
44. Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., Sohl-Dickstein, J.: On the expressive power of deep
neural networks. arXiv:1606.05336 (2016)
45. Saxe, A.M., McClelland, J.L., Ganguli, S.: Exact solutions to the nonlinear dynamics of learning in deep
linear neural networks. arXiv:1312.6120 (2013)
46. Bengio, Y ., LeCun, Y ., et al.: Scaling learning algorithms towards AI. Large Scale Kernel Mach. 34, 1–41
(2007)
47. Strassen, V .: Gaussian elimination is not optimal. Numer. Math. 13, 354–356 (1969)
48. Le Gall, F.: In: Proceedings of the 39th international symposium on symbolic and algebraic computation.
Organization ACM, pp. 296–303 (2014)
49. Carleo, G., Troyer, M.: Solving the quantum many-body problem with artiﬁcial neural networks.
arXiv:1606.02318 (2016)
50. V ollmer, H.: Introduction to Circuit Complexity: A Uniform Approach. Springer, Berlin (2013)
123