Internet Mathematics Vol. 1, No. 2: 226-251
A Brief History of
Generative Models for
Power Law and Lognormal
Distributions
Michael Mitzenmacher
Abstract. Recently, I became interested in a current debate over whether ﬁle size
distributions are best modelled by a power law distribution or a lognormal distribution.
In trying to learn enough about these distributions to settle the question, I found a rich
and long history, spanning many ﬁelds. Indeed, several recently proposed models from
the computer science community have antecedents in work from decades ago. Here,
Ib r i e ﬂy survey some of this history, focusing on underlying generative models that
lead to these distributions. One ﬁnding is that lognormal and power law distributions
connect quite naturally, and hence, it is not surprising that lognormal distributionshave arisen as a possible alternative to power law distributions across many ﬁelds.
1. Introduction
Power law distributions (also often referred to as heavy-tail distributions, Pareto
distributions, Zip ﬁan distributions, etc.) are now pervasive in computer science;
See, for example, [Broder et al. 00, Crovella and Bestavros 97, Faloutsos et al.99]. Numerous other examples can be found in the extensive bibliography of thispaper.
This paper was speci ﬁcally motivated by a recent paper by Downey [Downey
01] challenging the now conventional wisdom that ﬁle sizes are governed by a
© A K Peters, Ltd.
1542-7951/04 $0.50 per page 226
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 227
power law distribution. The argument was substantiated both by collected data
and by the development of an underlying generative model which suggested thatﬁle sizes were better modeled by a lognormal distribution.
1In my attempts
to learn more about this question, I was drawn to the history of lognormal
and power law distributions. As part of this process, I delved into past andpresent literature, and came across some interesting facts that appear not tobe well known in the computer science community. This paper presents what Ihave found, focusing speci ﬁcally on the models of processes that generate these
distributions.
Perhaps the most interesting discovery is that much of what we in the computer
science community have begun to understand and utilize about power law andlognormal distributions has long been known in other ﬁelds, such as economics
and biology. For example, models of a dynamically growing Web graph that
result in a power law distribution for in- and outdegrees have become the focus
of a great deal of recent study. In fact, as I describe below, very similar modelsdate back to at least the 1950s, and arguably back to the 1920s.
A second discovery is the argument over whether a lognormal or power law
distribution is a better ﬁt for some empirically observed distribution has been
repeated across many ﬁelds over many years. For example, the question of
whether income distribution follows a lognormal or power law distribution alsodates back to at least the 1950s. The issue arises for other ﬁnancial models, as
detailed in [Mandelbrot 97]. Similar issues continue to arise in biology [Jain and
Ramakumar 99], chemistry [Nakajima and Higurashi 98], ecology [Allen et al.
01, Sole et al. 00], astronomy [Wheatland and Sturrock 96], and informationtheory [Li 96, Perline 96]. These cases serve as a reminder that the problems weface as computer scientists are not necessarily new, and we should look to othersciences both for tools and understanding.
A third discovery from examining previous work is that power law and log-
normal distributions are intrinsically connected. Very similar basic generative
models can lead to either power law or lognormal distributions, depending onseemingly trivial variations. There is, t herefore, a reason why this argument as
to whether power law or lognormal distributions are more accurate has arisen
and repeated itself across a variety of ﬁelds.
The purpose of this paper is to explain some of the basic generative models
that lead to power law and lognormal distributions, and speci ﬁcally, to cover
how small variations in the underlying model can change the result from one tothe other. A second purpose is to provide along the way (incomplete) pointersto some of the recent and historically relevant scienti ﬁc literature.
1I elaborate on this speci ﬁc model in another paper [Mitzenmacher 02].
228 Internet Mathematics
This survey is intended to be accessible to a general audience; that is, it is
intended for computer science theorists , computer scientists who are not theo-
rists, and hopefully also people outside of computer science. Therefore, while
mathematical arguments and some probability will be used, the aim is for the
mathematics to be intuitive, clean, and comprehensible rather than rigorous andtechnical. In some cases, details may be suppressed for readability; interested
readers are referred to the original papers.
2. The Distributions: Basic Definitions and Properties
We begin by reviewing basic facts about power law and lognormal distributions.
For our purposes, a nonnegative random variable Xis said to have a power
lawdistribution if
Pr[X≥x]∼cx−α
for constants c>0a n dα>0. Here, f(x)∼g(x)r e p r e s e n t st h a tt h el i m i to ft h e
ratios goes to 1 as xgrows large. Roughly speaking, in a power law distribution
asymptotically the tails fall according to the power α. Such a distribution leads to
much heavier tails than other common models, such as exponential distributions.One speci ﬁc commonly used power law distribution is the Pareto distribution ,
which satis ﬁes
Pr[X≥x]=px
kQ−α
for some α>0a n d k> 0. The Pareto distribution requires X≥k.T h e
density function for the Pareto distribution is f(x)=αkαx−α−1.F o r a p o w e r
law distribution, usually αfalls in the range 0 <α≤2, in which case, Xhas
inﬁnite variance. If α≤1, then Xalso has in ﬁnite mean.
IfXhas a power law distribution, then in a log-log plot of Pr[ X≥x], also
known as the complementary cumulative distribution function , asymptotically
the behavior will be a straight line. This provides a simple empirical test for
whether a random variable has a power law given an appropriate sample. Forthe speci ﬁc case of a Pareto distribution, the behavior is exactly linear, as
ln(Pr[ X≥x]) =−α(lnx−lnk).
Similarly, on a log-log plot, the density function for the Pareto distribution is
also a straight line:
lnf(x)=(−α−1) lnx+αlnk+l nα.
A random variable Xhas a lognormal distribution if the random variable
Y=l n Xhas a normal (i.e., Gaussian) distribution. Recall that the normal
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 229
distribution Yis given by the density function
f(y)=1√
2πσe−(y−µ)2/2σ2
where µis the mean, σis the standard deviation ( σ2is the variance), and the
range is −∞ <y<∞. The density function for a lognormal distribution
therefore satis ﬁes
f(x)=1√
2πσxe−(lnx−µ)2/2σ2.
Note that the change of variables introduces an additional 1 /xterm outside of the
exponential term. The corresponding complementary cumulative distribution
function for a lognormal distribution is given by
Pr[X≥x]=8∞
z=x1√
2πσze−(lnz−µ)2/2σ2dz.
We will say that Xhas parameters µandσ2when the associated normal distrib-
ution Yhas mean µand variance σ2, where the meaning is clear. The lognormal
distribution is skewed, with mean eµ+1
2σ2,m e d i a neµ, and mode eµ−σ2.A l o g -
normal distribution has ﬁnite mean and variance, in contrast to the power law
distribution under natural parameters.
Despite its ﬁnite moments, the lognormal distribution is extremely similar in
shape to power law distributions, in the following sense: If Xhas a lognormal
distribution, then in a log-log plot of the complementary cumulative distributionfunction or the density function, the behavior will appear to be nearly a straightline for a large portion of the body of the distribution. Indeed, if the varianceof the corresponding normal distribution is large, the distribution may appearlinear on a log-log plot for several orders of magnitude.
To see this, let us look at the logarithm of the density function, which is
easier to work with than the complementary cumulative distribution function(although the same idea holds). We have
lnf(x)=−lnx−ln√
2πσ−(lnx−µ)2
2σ2(2.1)
=−(lnx)2
2σ2+pµ
σ2−1Q
lnx−ln√
2πσ−µ2
2σ2. (2.2)
Ifσis suﬃciently large, then the quadratic term of equation (2.2) will be small
for a large range of xvalues, and hence, the logarithm of the density function
will appear almost linear for a large range of values.
Finally, recall that normal distributions have the property that the sum of
two independent normal random variables Y1andY2with means µ1andµ2and
230 Internet Mathematics
variances σ2
1andσ2
2,respectively, is a normal random variable with mean µ1+µ2
and variance σ2
1+σ2
2. It follows that the product of two lognormally distributed
random variables also has a lognormal distribution.
3. Power Laws via Preferential Attachment
We now move from mathematical de ﬁnitions and properties to generative models.
For the power law distribution, we begin by considering the World Wide Web.The World Wide Web can naturally be thought of as a graph, with pages corre-sponding to vertices and hyperlinks corre sponding to directed edges. Empirical
work has shown that indegrees and outdegrees of vertices in this graph obey
power law distributions. There has subseq uently been a great deal of theoretical
work on designing random graph models that yield Web-like graphs [Barab´ asi
et al. 99, Broder et al. 00, Cooper and Frieze 01, Drinea et al. 00, Kleinberg etal. 99, Krapivsky and Redner 01, Kumar et al. 99a, Kumar et al 00]. An im-portant criterion for an appropriate random graph model is that it yields powerlaw distributions for the indegrees and outdegrees.
Most models are variations of the following theme. Let us start with a single
page, with a link to itself. At each time step, a new page appears, with outdegree1. With probability α<1, the link for the new page points to a page chosen
uniformly at random. With probability 1 −α, the new page points to page chosen
proportionally to the indegree of the page. This model exempli ﬁes what is often
called preferential attachment ; new objects tend to attach to popular objects. In
the case of the Web graph, new links tend to go to pages that already have links.
A simple, if slightly nonrigorous, argument for the above model goes as follows
[Barab´ asi et al. 99, Drinea et al. 00, Krapivsky and Redner 01, Kumar et al 00].
LetX
j(t)( o rj u s t Xjwhere the meaning is clear) be the number of pages with
indegree jwhen there are tpages in the system. Then, for j≥1,the probability
that Xjincreases is just
αXj−1/t+( 1−α)(j−1)Xj−1/t;
theﬁrst term is the probability a new link is chosen at random and chooses a
page with indegree j−1, and the second term is the probability that a new link
is chosen proportionally to the indegrees and chooses a page with indegree j−1.
Similarly, the probability that Xjdecreases is
αXj/t+( 1−α)jXj/t.
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 231
Hence, for j≥1, the growth of Xjis roughly given by
dXj
dt=α(Xj−1−Xj)+( 1−α)((j−1)Xj−1−jXj)
t.
Some mathematical purists may object to utilizing a continuous di ﬀerential
equation to describe what is clearly a discrete process. This intuitively appealingapproach can be justi ﬁed more formally using martingales [Kumar et al 00] and
in particular, the theoretical frameworks of Kurtz and Wormald [Drinea et al.
0 0 ,K u r t z8 1 ,W o r m a l d9 5 ] .
T h ec a s eo f X
0must be treated specially, since each new page introduces a
vertex of indegree 0.
dX0
dt=1−αX0
t.
Suppose in the steady state limit that Xj(t)=cj·t; that is, pages of indegree
jconstitute a fraction cjof the total pages. Then we can successively solve for
thecj. For example,
dX0
dt=c0=1−αX0
t=1−αc0,
from which we ﬁndc0=1
1+α. More generally, using the equation for dXj/dt,
weﬁnd that for j≥1,
cj(1 +α+j(1−α)) = cj−1(α+(j−1)(1−α)).
This recurrence can be used to determine the cjexactly. Focusing on the as-
ymptotics, we ﬁnd that for large j
cj
cj−1=1−2−α
1+α+j(1−α)∼1−w2−α
1−αWw1
jW
.
Asymptotically, for the above to hold, we have cj∼cj−2−α
1−αfor some constant
c, giving a power law. To see this, note that cj∼cj−2−α
1−αimplies
cj
cj−1∼wj−1
jW2−α
1−α
∼1−w2−α
1−αWw1
jW
.
Strictly speaking, to show it is a power law, we should consider c∗
k=
j≥kcj,
since we desire the behavior of the tail of the distribution. However, we have
c∗
k∼3
j≥kcj−2−α
1−α∼8∞
j=kcj−2−α
1−αdj∼cIk−1
1−α
232 Internet Mathematics
for some constant cI. More generally, if the fraction of items with weight jfalls
roughly proportionally to j−α, the fraction of items with weight greater than or
equal to jfalls roughly proportionally j1−α, a fact we make use of throughout.
Although the above argument was described in terms of degree on the Web
graph, this type of argument is clearly very general and applies to any sort ofpreferential attachment. In fact, the ﬁrst similar argument dates back to at
least 1925. It was introduced by Yule [Yule 25] to explain the distribution ofspecies among genera of plants, which had been shown empirically by Willis tosatisfy a power law distribution. While the mathematical treatment from 1925 isdiﬀerent than modern versions, the outline of the general argument is remarkably
similar. Mutations cause new species to d e v e l o pw i t h i ng e n e r a ,a n dm o r er a r e l y
mutations lead to entirely new genera. Mutations within a genus are more likelyto occur in a genus with more species, leading to the preferential attachment.
A clearer and more general development of how preferential attachment leads
to a power law was given by Simon [Simon 55] in 1955. Again, although Simonwas not interested in developing a model for the Web, he lists ﬁve applications
of this type of model in his introduction: distributions of word frequencies in
documents, distributions of numbers of papers published by scientists, distribu-
tion of cities by population, distribution of incomes, and distribution of speciesamong genera. Simon was aware of Yule’s previous work, and suggests his workis a generalization. Simon’s argument, except for notation and the scaling ofvariables, is painfully similar to the outline above.
As one might expect from Simon’s list of applications, power laws had been
observed in a variety of ﬁelds for some time; Simon was attempting to give a
mathematical argument explaining these observations. The earliest apparent ref-erence is to the work by Pareto [Pareto 1896] in 1897, who introduced the Paretodistribution to describe income distribution. The ﬁrst known attribution of the
power law distribution of word frequencies appears to be due to Estoup in 1916[Estoup 1916], although generally the idea (and its elucidation) are attributedto Zipf [Zipf 32, Zipf 35, Zipf 39]. Similarly, Zipf is often credited with notingthat city sizes appear to match a power law, although this idea can be traced
back further to 1913 and Auerbach [Auerbach 1913]. Lotka (circa 1926) found
in examining the number of articles produced by chemists that the distributionfollowed a power law [Lotka 26]; indeed, power laws of various forms appear inmany places in informetrics [Bookstein 90].
Although we now associate the argument above with the Web graph, even be-
fore the Web graph became popular, more formal developments of the argumentabove had been developed as part of the study of random trees. Speci ﬁcally,
consider the following recursive tree structure. Begin with a root node. At eachstep, a new node is added; its parent is chosen from the current vertices with
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 233
probability proportional to one plus the number of children of the node. This is
just another example of preferential attachment; indeed, it is essentially equiv-alent to the simple Web graph model described above with the probability α
of choosing a random node equal to 1/2. That the degree distribution of such
graphs obey a power law (in expectation) was proven in 1993 in works by Mah-moud, Smythe, and Szyma´ nski [Mahmound et al. 93]. See also the related [Lu
and Feng 98, Szyma´ nski 87, Pittel 94, Smythe and Mahmound 95].
Of course, in recognizing the relationship between the recent work on Web
graph models and this previous work, it would be remiss to not point out thatmodern developments have led to many new insights. Perhaps most impor-tant is the development of a connection between Simon’s model, which appearsamenable only to limiting analysis based on di ﬀerential equations, and purely
combinatorial models based on random graphs [B. Bollob´ as et al. 01, Mahmound
et al. 93, Smythe and Mahmound 95]. Such a connection is important for furtherrigorous analysis of these structures. Al so, current versions of Simon’s arguments
based on martingales provide a much more rigorous foundation [B. Bollob´ as et
al. 01, Cooper and Frieze 01, Kumar et al 00, Lu and Feng 98]. More recentwork has focused on greater understanding of the structure of graphs that arisefrom these kinds of preferential attachment model. It has been shown that inthe Web graph model described above, where new pages copy existing links, the
graphs have community substructures [Kumar et al 00], a property not found in
random graphs, but amply found in the actual Web [Gibson et al. 98, Kumar etal. 99b]. The diameter of these random Web graphs have also been the subjectof recent study [Albert et al. 99, Bollob´ as and Riordan to appear]. Still, it is
important to note how much was already known about the power law phenom-enon in various ﬁelds well before the modern e ﬀort to understand power laws on
the Web, and how much computer scientists had to reinvent.
4. Power Laws via Optimization
Mandelbrot had developed other argume nts for deriving power law distribu-
tions based on information theoretic considerations somewhat earlier than Si-mon [Mandelbrot 53]. His argument is very similar in spirit to other recentoptimization-based arguments for heavy tailed distributions [Carlson and Doyle99, Fabrikant et al. 02, Zhu et al. 01].
We sketch Mandelbrot’s framework, which demonstrates a power law in the
rank-frequency distribution of words. That is, the frequency p
jof the jth most-
used word, given as a fraction of the time that word appears, follows a powerlaw in j,s op
j∼cj−α. T h i si sas l i g h t l yd i ﬀerent ﬂavor than the type power law
234 Internet Mathematics
than we considered previously; Simon’s model considers the fraction of words
that appear jtimes. But, of course, the two are related. We clarify this by
following an argument of Bookstein [Bookstein 90].
Suppose we have a text where the number of words qkthat appear ktimes is
given by qk=Qk−αforα>1. Further, suppose for convenience we have one
most frequent word that appears kmtimes, so that we may write qk=(k/km)−α.
The number of words that appear kor more times is then approximately
8km
kwx
kmW−α
dx,
and hence, the rank jof a word that appears ktimes is approximately
j=jm
α−1^wjm
kWα−1
−1
.
Now solving for kin terms of j,w e ﬁnd that the jth most-used word appears
approximately
k=jm}(α−1)j
jm+1]−1/(α−1)
times, yielding a power law for the frequency pjas a function of j.
We now begin Mandelbrot’s argument. Co nsider some langua ge consisting of
nwords. The cost of using the jth word of the language in a transmission is Cj.
F o re x a m p l e ,i fw et h i n ko fE n g l i s ht e x t ,t h ec o s to faw o r dm i g h tb et h o u g h to fas the number of letters plus the additional cost of a space. Hence, a natural costfunction has C
j∼logdjfor some alphabet size d. Suppose that we wish to design
the language to optimize the average amount of information per unit transmission
cost. Here, we take the average amount of information to be the entropy. We
think of each word in our transmission as being selected randomly, and theprobability that a word in the transmission is the jth word of the language is p
j.
Then, the average information per word is the entropy H=−n
j=1pjlog2pj,
and the average cost per word is C=n
j=1pjCj. The question is how would
thepjbe chosen to minimize A=C/H . Taking derivatives, we ﬁnd
dA
dpj=CjH+Clog2(epj)
H2.
Hence, all the derivatives are 0 (and Ais, in fact, minimized) when pj=
2−HCj/C/e. Using Cj∼logdj,w eo b t a i nap o w e rl a wf o rt h e pj.2Mandel-
2The eagle-eyed reader might note that technically the result above does not quite match
ap o w e rl a wa sw eh a v ed e ﬁned it; just because Cj∼logdjdoes not strictly give us pj∼j−α.
In this case, this is a minor point; really Cjis within an additive constant of logdj,a n dw e
therefore ﬁnd that pjis within a constant multiplicative factor of a power law. We ignore this
distinction henceforth.
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 235
brot argues that a variation of this model matches empirical results for English
quite well.
Carlson and Doyle suggest a similar framework for analyzing ﬁle sizes and
forest ﬁles [Carlson and Doyle 99]. Fabrikant , Koutsoupias, and Papadimitriou
introduce combinatorial models for the Internet graph (which should not beconfused with the Web graph; the Internet graph consists of the servers and linksbetween them as opposed to Web pages) and ﬁle sizes based on local optimization
that also yield power laws [Fabrikant et al. 02].
As an aside, I found when reviewing the literature that Mandelbrot strongly
argued against Simon’s alternative assumptions and derivations based on pref-erential attachment when his article came out. This led to what is in retrospectan amusing but apparently at the time quite heated exchange between Simonand Mandelbrot in the journal Information and Control [Mandelbrot 59, Simon
60, Mandelbrot 61a, Simon 61a, Mandelbrot 61b, Simon 61b].
3
It is worth noting that economists appear to have given the nod to Simon and
the preferential attachment model. Indeed, a recent popular economics text byKrugman [Krugman 96] o ﬀers a derivation of the power law similar to Simon’s
argument.
4A more formal treatment is given by Gabaix [Gabaix 99].
5. Multiplicative Processes
Lognormal distributions are generated by processes that follow what the econo-mist Gibrat called the law of proportionate e ﬀect [Gibrat 30, Gibrat 31]. We
use the term multiplicative process to describe the underlying model. In biology,
such processes are used to described the growth of an organism. Suppose westart with an organism of size X
0.A t e a c h s t e p j, the organism may grow or
shrink, according to a random variable Fj,s ot h a t
Xj=FjXj−1.
3A few excerpts from the exchange are worth citing to demonstrate the disagreement. The
abstract of Mandelbrot’s initial note begins, “This note is a discussion of H. A. Simon’s model(1955) concerning the class of frequency distributions generally associated with the name of G.K. Zipf. The main purpose is to show that Simon’s model is analytically circular in the case ofthe linguistic laws of Estouf-Zipf and Willis-Yule.” [Mandelbrot 59] The abstract of Simon’s
rebuttal begins, “This note takes issue with a recent criticism by Dr. B. Mandelbrot of a certain
stochastic model to explain word-frequency data. Dr. Mandelbrot’s principal empirical andmathematical objections to the model are shown to be unfounded.” The subsequent abstractsare similar in spirit.
4As an interesting example of the pervasiveness of power-law behavior, one review of Krug-
man’s book, written by an urban geographer, accuses the author of excessive hubris for notnoting the signi ﬁcant contributions made by urban geographers with regard to Simon’s model
[Berry 99].
236 Internet Mathematics
The idea is that the random growth of an organism is expressed as a percentage
of its current weight, and is independent of its current actual size. If the Fk,1≤
k≤j,are all governed by independent lognormal distributions, then so is each
Fj, inductively, since the product of lognormal distributions is again lognormal.
More generally, approximately lognorma l distributions may be obtained even
if the Fjare not themselves lognormal. Speci ﬁcally, consider
lnXj=l nX0+j3
k=1lnFk.
Assuming the random variables ln Fksatisfy appropriate conditions, the Central
Limit Theorem says thatj
k=1lnFkconverges to a normal distribution, and
hence, for su ﬃciently large j,Xjis well approximated by a lognormal distri-
bution. In particular, if the ln Fkare independent and identically distributed
variables with ﬁnite mean and variance, then asymptotically Xjwill approach a
lognormal distribution.
Multiplicative processes are used in biology and ecology to describe the growth
of organisms or the population of a species. In economics, perhaps the most well-known use of the lognormal distribution derives from the Black-Scholes option
pricing model [Black and Scholes 73], which is a speci ﬁc application of Ito’s
lemma (see, e.g., [Hull 97, Ito 51]). In a simpli ﬁed version of this setting [Cox
et al. 79, Hull 97], the price of a security moves in discrete time steps, and theprice X
jchanges according to Xj=FjXj−1,w h e r e Fjis lognormally distrib-
uted. Using this model, Black and Scholes demonstrate how to use options toguarantee a risk-free return equivalent to the prevailing interest rate in a perfectmarket. Other applications in, for example, geology and atmospheric chemistryare given in [Crow and Shimizu 88]. More recently, as described below, Adamicand Huberman suggest that multiplicative processes may describe the growth ofsites on the Web as well as the growth of user tra ﬃco nw e bs i t e s[ H u b e r m a n
and Adamic 99, Huberman and Adamic 00]. Lognormal distributions have alsobeen suggested for ﬁle sizes [Barford et al. 99, Barford and Crovella 98, Downey
01].
The connection between multiplicative processes and the lognormal distribu-
tion can be traced back to Gibrat around 1930 [Gibrat 30, Gibrat 31], althoughKapteyn described, in other terms, an equivalent process in 1903 [Kapteyn1903], and McAlister described the lognormal distribution around 1879 [McAlis-ter 1879]. Aitchison and Brown suggest that the lognormal distribution may beab e t t e r ﬁt for income distribution than a power law distribution, representing
perhaps the ﬁrst time the question of whether a power law distribution or a log-
normal distribution gives a better ﬁt was fully developed [Aitchison and Brown
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 237
54, Aitchison and Brown 57]. It is interesting that when examining income dis-
tribution data, Aitchison and Brown observe that for lower incomes a lognormaldistribution appears a better ﬁt, while for higher incomes a power law distrib-
ution appears better; this is echoed in later work by Montroll and Schlesinger
[Montroll and Schlesinger 82, Montroll and Schlesinger 83], who o ﬀer a possi-
ble mathematical justi ﬁcation discussed below. Similar observations have been
given for ﬁle sizes [Barford et al. 99, Barford and Crovella 98].
5.1. Multiplicative Models and Power Law Distributions
Although the multiplicative model is used to generate lognormal or approxi-
mately distributions, only a small change from the lognormal generative processyields a generative process with a power l aw distribution. To provide a concrete
example, we consider the interesting history of work on income distributions.
Recall that Pareto introduced the Pareto distribution in order to explain in-
come distribution at the tail end of the nineteenth century. Champernowne
[Champernowne 53], in a work slightly predating Simon (and acknowledged bySimon, who suggested his work generalized and extended Champernowne), of-fered an explanation for this behavior. Suppose that we break income into dis-crete ranges in the following manner. We assume there is some minimum incomem.F o r t h e ﬁrst range, we take incomes between mandγm,f o rs o m e γ>1;
for the second range, we take incomes between γmandγ
2m. We therefore say
t h a tap e r s o ni si nc l a s s jforj≥1 if their income is between mγj−1andmγj.
Champernowne assumes that over each time step, the probability of an individ-ual moving from class ito class j, which we denote by p
ij, depends only on the
value of j−i. He then considers the equilibrium distribution of people among
classes. Under this assumption, Pa reto distributions can be obtained.
Let us examine a speci ﬁc case, where γ=2 , pij=2/3i fj=i−1, and
pij=1/3i fj=i+ 1. Of course, the case i= 1 is a special case; in this case,
p11=2/3. In this example, outside of class 1, the expected change in income
o v e ra n ys t e pi s0 . I ti sa l s oe a s yt oc h e c kt h a t ,i nt h i sc a s e ,t h ee q u i l i b r i u mprobability of being in class kis just 1 /2
k, and hence, the probability of being in
class greater than or equal to kis 1/2k−1. Hence the probability that a person’s
income Xis larger than 2k−1min equilibrium is given by
Pr[X≥2k−1m]=1/2k−1,
or
Pr[X≥x]=m/x
forx=2k−1m. This is a power law distribution.
238 Internet Mathematics
Note, however, the speci ﬁc model above looks remarkably like a multiplicative
model. Moving from one class to another can be thought of as either doublingor halving your income over one time step. That is, if X
tis your income after t
time steps, then
Xt=FtXt−1,
where Ftis 1/2w i t hp r o b a b i l i t y2 /3 and 2 with probability 1 /3. Again, E[Xt]=
E[Xt−1]. Our previous discussion therefore suggests that Xtshould converge to
a lognormal distribution for large t.
What is the di ﬀerence between the Champernowne model and the multiplica-
tive model? In the multiplicative model, income can become arbitrarily closeto zero through successive decreases; in the Champernowne model, there is aminimum income corresponding to the lowest class below which one cannot fall.This small change allows one model to produce a power law distribution whilethe other produces a lognormal. As long as there is a bounded minimum that
acts as a lower re ﬂective barrier to the multiplicative model, it will yield a power
law instead of a lognormal distribution. The theory of this phenomenon is more
fully developed in [Gabaix 99, Kesten 73].
6. Monkeys Typing Randomly
We return now to Mandelbrot’s optimization argument for the power law be-havior of word frequency in written language. A potentially serious objection toMandelbrot’s argument was developed by the psychologist Miller [Miller 57], whodemonstrated that the power law behavior of word frequency arises even with-out an underlying optimization problem. This result, explained below, shouldperhaps serve as warning: Just because one ﬁnds a compelling mechanism to
explain a power law does not mean that there are not other, perhaps simpler,
explanations.
Miller suggests the following experiment. A monkey types randomly on a
keyboard with ncharacters and a space bar. A space is hit with probability q;
all other characters are hit with equal probability (1 −q)/n.As p a c ei su s e dt o
separate words. We consider the f requency distribution of words.
It is clear that, as the monkey types, each word with c(nonspace) characters
occurs with probability
q
c=w1−q
nWc
q,
and there are ncwords of length c. (We allow the empty word of length 0 for
convenience.) The words of longer length are less likely and hence, occur lower in
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 239
the rank order of word frequency. In part icular, the word with frequency ranks
1+(nj−1)/(n−1) to ( nj+1−1)/(n−1) have jletters. Hence, the word with
frequency rank rj=njoccurs with probability
qj=qw1−q
nWlognrj
=q(rj)logn(1−q)−1,
and the power law behavior is apparent. Hence, the power law associated with
word frequency requires neither preferen tial attachment nor optimization; mon-
keys typing randomly would produce it.
Bell, Cleary, and Witten observe empirically that when the probabilities of
each letter are not equal, a smoother match to the power law develops [Bell etal. 90]. I am currently unaware of a proof similar to the one above demonstrat-ing that power law behavior occurs when the probabilities for each of the lettersare arbitrary. Indeed, to confuse the issue, one paper on the subject claims thatif the letter frequencies are not equal, a lognormal distribution occurs [Perline96] (see also [Gong et al. 01], where this claim is repeated). It is worth ex-amining this argument more carefully, since it demonstrates the confusion that
can arise in trying to distinguish models that generate power law and lognormal
distributions.
Perline notes that in the experiment with monkeys typing randomly, if we con-
sider words only of some ﬁxed length m,f o rmsuﬃciently large their frequency-
rank distribution will approximate a lognormal distribution, following the par-adigm of multiplicative processes. To see this, let the probabilities for our n
characters be p
1,p2,...,p n. Consider the generation a random m-letter word.
LetXitake on the value pjif the ith letter is j.T h e n Ym=X1X2...X mis
a random variable whose value corresponds to the probability that a word cho-sen uniformly at random from all m-letter words appears as the monkeys type.
We have that ln Y
m=m
k=1lnXi;s i n c et h e Xiare independent and identically
distributed, ln Ymconverges to a normal distribution by the Central Limit Theo-
rem, and hence, Ymconverges to a lognormal distribution. Notice that this holds
true even if all letter frequencies are equal, although in this case, the resulting
distribution is trivial.
Perline then argues that if we consider all words of length up to m, we still
obtain asymptotic convergence to a lognormal distribution. This follows from ageneralization of the Central Limit Theorem due to Anscombe. Intuitively, thisis because most words have length close to m, so the words with small length
are just noise in the distribution. This result requires that some two letters havediﬀerent probabilities of being hit.
From this, it might be tempting to conclude that the distribution, if the word
length is unrestricted, is also lognormal when letters do not all have the same
240 Internet Mathematics
probabilities. However, this does not follow. The problem is that for each value
ofmwe obtain a slightly di ﬀerent lognormal distribution. Hence, it is not
necessarily true that in the limit as mincreases we are getting closer and closer
to some ﬁnal lognormal distribution. Rathe r, we have a sequence of lognormal
distributions that is converging to some distribution. To justify that the result
need not be lognormal, I present an amusing example of my own devising.
Consider an alphabet with two letters: “a” occurs with probability q,“ b ”
occurs with probability q2, and a space occurs with probability 1 −q−q2.T h e
value qmust be chosen so that 1 −q−q2>0. In this case, every valid word the
monkey can type occurs with probability qj(1−q−q2)f o rs o m ei n t e g e r j.L e t
us say a word has pseudorank jif it occurs with probability qj(1−q−q2). There
is 1 word with pseudorank 0 (the empty word), 1 with pseudorank 1 (“a”), 2with pseudorank 2 (“aa” and “b”), and so on. A simple induction yields thatthe number of words with pseudorank kis in fact the ( k+1)st Fibonacci n um ber
F
k+1(where here we start with F0=0a n d F1= 1). This follows obviously from
the fact that to obtain the words with pseudorank kwe append an “a” to a word
with pseudorank k−1, or a “b” to a word with pseudorank k−2.
Recall that Fk≈Φk/√
5f o rl a r g e k,w h e r eΦ=( 1+√
5)/2. Alsok
i=1Fk=
Fk+2−1. Now the argument is entirely similar to the case where all items have
the same probability. If we ask for the frequency of the rj=Fjth most frequent
item, it has pseudorank j−2, and hence its frequency is
qj−2(1−q−q2)≈qlogΦ√
5rj−2(1−q−q2)=rlogΦq
j qlogΦ√
5−2(1−q−q2),
and again we have power law behavior.
There is nothing special about having two characters for this example; one
could easily expand it to include more complex generalized Fibonacci sequences.
A suitable generalization appears feasible for any probabilities p1,p2, ..., p n
associated with the ncharacters, although a formal proof is beyond the scope of
this survey.5Roughly, let p1be the largest of the pi,a n dl e t pj=pγj
1forj≥1.
Then the number of words with frequency greater than or equal to pk
1grows
approximately proportionally to (1 /c)k,w h e r e cis the unique real root between
0a n d1o fn
j=1xγj= 1. This is all we need for the monkeys to produce a power
law distribution, following the arguments above.
5Brian Conrad and I have constructed a formal treatment of this argument, using some
nontrivial analytic number theory. This work has been submitted for publication.
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 241
7. Double Pareto Distributions
Interestingly, there is another variation on the multiplicative generative model
that yields power law behavior. Recall that in the multiplicative model, if web e g i nw i t hv a l u e X
0and every step yields an independent and identically distrib-
uted multiplier from a lognormal distribution F, then any resulting distribution
Xtafter tsteps is lognormal. Suppose, however, that instead of examining Xt
for a speci ﬁc value of t,w ee x a m i n et h er a n d o mv a r i a b l e XTwhere Titself is
a random variable. As an example, when considering income distribution, inseeing the data we may not know how long each person has lived. If di ﬀerent
age groups are intermixed, the number of multiplicative steps each person maybe thought to have undergone may be thought of as a random variable.
This eﬀect was noticed as early as 1982 by Montroll and Schlesinger [Montroll
and Schlesinger 82, Montroll and Schlesinger 83]. They show that a mixture of
lognormal distributions based on a geometric distribution would have essentiallya lognormal body but a power law distribution in the tail. Huberman andAdamic suggest a pleasantly simple variation of the above result; in the casewhere the time Tis an exponential random variable, and we may think of the
number of multiplicative steps as being continuous, the resulting distributionofX
Thas a power law distribution [Huberman and Adamic 99, Huberman and
Adamic 00]. Huberman and Adamic go on to suggest that this result can explainthe power law distribution observed for the number of pages per site. As theWeb is growing exponentially, the age of a site can roughly be thought of asdistributed like an exponential random variable. If the growth of the number ofpages on a Web site follows a multiplicative process, the above result suggests apower law distribution.
In more recent independent work, Reed provides the correct full distribution
for the above model, which yields what he calls a double Pareto distribution
[Reed 00]. Speci ﬁcally, the resulting distribution has one Pareto tail distribution
for small values (below some point) and another Pareto tail distribution for largevalues (above the same point).
6
For example, consider for simplicity the case where if we stop a process at time
tthe result is a lognormal random variable with mean 0 and variance t. Then,
if we stop the process at an exponentially distributed time with mean 1 /λ,t h e
density function of the result is
f(x)=8∞
t=0λe−λt1√
2πtxe−(lnx)2/2tdt.
6For completeness, we note that Huberman and Adamic concentrate only on the tail of the
density function, and correctly determine the power law behavior. However, they miss thetwo-sided nature of the distribution. Reed gives the complete correct form, as we do below.
242 Internet Mathematics
Using the substitution t=u2gives
f(x)=2λ√
2πx8∞
u=0e−λu2−(lnx)2/2u2du.
An integral table gives us the identity
8∞
z=0e−az2−b/z2=1
25π
ae−2√
ab,
which allows us to solve for the resulting form. Note that in the exponent√
2abof the identity we have b=( l n x)2/2. Because of this, there are two
diﬀerent behaviors, depending on whether x≥1o r x≤1. For x≥1,f(x)=p0
λ/2Q
x−1−√
2λ, so the result is a power law distribution. For x≤1,f(x)=p0
λ/2Q
x−1+√
2λ.
The double Pareto distribution falls ni cely between the lognormal distribution
and the Pareto distribution. Like the Pareto distribution, it is a power lawdistribution. But while the log-log plot of the density of the Pareto distribution
is a single straight line, for the double Pareto distribution the log-log plot of the
density consists of two straight line segments that meet at a transition point.This is similar to the lognormal distribution, which has a transition point aroundits median e
µdue to the quadratic term, as shown in Equation (2.1). Hence, an
appropriate double Pareto distribution can closely match the body of a lognormaldistribution and the tail of a Pareto distribution. For example, Figure 1 showsthe complementary cumulative distribution function for a lognormal and a doublePareto distribution. (These graphs have only been minimally tuned to givea reasonable match.) The plots match quite well with a standard scale forprobabilities, as shown on the left. On the log-log scale, however, one can seethe diﬀerence in the tail behavior. The double Pareto distribution follows a
power law; the lognormal distribution has a clear curvature.
Reed also suggests a generalization of the above called a double Pareto-lognormal
distribution with similar properties [Reed 01]. The double Pareto-lognormal dis-
tribution has more parameters, but might allow closer matches with empirical
distributions.
It seems reasonable that in many processes the time an object has lived should
be considered a random variable as well, and hence, this model may prove moreaccurate for many situations. For examp le, that the double Pareto tail phenom-
enon could explain why income distributions and ﬁle size distributions appear
better modeled by a distribution with a lognormal body and a Pareto tail [Aitchi-son and Brown 54, Barford et al. 99, Barford and Crovella 98, Montroll andSchlesinger 82, Montroll and Schlesinger 83]. Reed presents empirical evidence
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 243
00.10.20.30.40.50.60.70.80.91
-6 -4 -2 0 2 4 6Probability
Log of Value (Base 2)ccdf: Lognormal and Double Pareto
Lognormal
Double Pareto
-10-8-6-4-20
-6 -4 -2 0 2 4 6Log of Probability (Base 2)
Log of Value (Base 2)ccdf: Lognormal and Double Pareto
Lognormal
Double Pareto
Figure 1 . Shapes of lognormal and double Pareto distributions.
in favor of using the double Pareto and double-Pareto lognormal distributions
for incomes and other applications [Reed 00, Reed 01].
To give an idea of why it might be natural for the time parameter to be
(roughly) exponentially distributed, I brie ﬂy describe a model that I introduced
in [Mitzenmacher 02]. This model combines ideas from the theory of recursivetrees, preferential attachment, and the double Pareto framework. Consider a
graph process that works as follows: At each step, with probability γ,an e w
node is introduced that becomes the root of a new tree. Each new node hasan associated size chosen independently and uniformly at random from a dis-tribution D
1. With probability 1 −γ, an existing node is chosen uniformly at
random, and it generates a child. The size of a child is equal to the size of itsparent, multiplied by some multiplicative factor chosen by a distribution D
2.I t
is easy to show that the distribution of the depths of the nodes generated in thismanner converges to a geometric distribution. Along each branch of the tree,the size of the nodes follows a multiplicative process. If D
1andD2are lognormal
distributions, then the size of a randomly chosen node is a geometric mixture oflognormally distributed random variables, which closely matches the exponentialmixture required for a double Pareto distribution. In fact, the tail behaviors arethe same. I use this model to explain ﬁle size distributions in [Mitzenmacher
02]; [Reed and Hughes 02] analyzes other similar models.
This line of thought also ties back into the discussion of monkeys typing ran-
domly. In the case of unrestricted word lengths and unequal letter probabilities,the word length is geometrically distributed, and the probability of a word of any(large) ﬁxed length is approximately lognormal, with the appropriate mean and
variance being proportional to the length of the word. Hence, the underlyingdistribution of word lengths is a geometric mixture of approximately lognormal
244 Internet Mathematics
random variables as in the framework above, and hence, the resulting power law
is unsurprising.
8. Conclusions
Power law distributions and lognormal distributions are quite natural modelsand can be generated from simple and intuitive generative processes. Because ofthis, they have appeared in many areas of science. This example should remindus of the importance of seeking out and recognizing work in other disciplines,even if it lies outside our normal purview. Since computer scientists invented
search engines, we really have little excuse. On a personal note, I was astounded
at how the Web and search engines have transformed the possibilities for miningprevious research; many of the decades-old articles (including the 1925 articleby Yule!) cited here are, in fact, available on the Web.
It is not clear that the above discussion settles one way or another whether
lognormal or power law distributions are better models for things like ﬁle size
distributions. Given the close relationship between the two models, it is not clearthat a de ﬁnitive answer is possible; it may be that in seemingly similar situations
slightly di ﬀerent assumptions prevail. The fact that power law distributions
arise for multiplicative models once the observation time is random or a lowerboundary is put into e ﬀect, however, may suggest that power laws are more
robust models. Indeed, following the work of Reed [Reed 00, Reed 01], werecommend the double Pareto distribution and its variants as worthy of furtherconsideration in the future.
From a more pragmatic point of view, it might be reasonable to use whichever
distribution makes it easier to obtain results. This runs the risk of being inac-curate; perhaps in some cases, the fact t hat power law distributions can have
inﬁnite mean and variance are salient features, and therefore substituting a log-
normal distribution loses this important characteristic. Also, if one is attemptingto predict future behavior based on current data, misrepresenting the tail of the
distribution could have severe consequences. For example, large ﬁles above a
certain size might be rare currently, and hence, both lognormal and power lawdistibutions based on current data might capture these rare events adequately.As computer systems with more memory proliferate, and even larger ﬁles become
more frequent, the prediction from two m odels may vary more substantially. The
recent work [Gong et al. 01] argues that for at least some network applicationsthe diﬀerence in tails is not important. We believe that formalizing this idea
is an important open question. Speci ﬁcally, it would be useful to know in a
more formal sense in what situations the small di ﬀerences between power laws
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 245
and lognormal distributions man ifest themselves in vastly di ﬀerent qualitative
behavior, and in what cases a power law distribution can be suitably approxi-mated by lognormal distributions.
For further reading, I strongly recommend the article by Xavier Gabaix [Gabaix
99], which provides both underlying mathematics and an economic perspectiveand history. Similarly, Mandelbrot provides both history about and his ownperspective on lognormal and power law distributions in a recent book [Mandel-brot 97]. Wentian Li has a web page devoted to Zipf’s law which is an excellentreference [Li 99]. For lognormal distributions, useful sources include the textby Aitchison and Brown [Aitchison and Brown 57] or the modern compendiumedited by Crow and Shimizu [Crow and Shimizu 88].
Acknowledgments. I would like to thank John Byers, Mark Crovella, Allen Downey, Alan
Frieze, Jon Kleinberg, and Eli Upfal for suggestions on how to improve this work. This
paper is supported in part by an Alfred P. Sloan Research Fellowship and NSF grant
CCR-9983832.
References
[Aiello et al. 00] W. Aiello, F. Chung, and L. Lu. “A Random Graph Model for Mas-
sive Graphs.” In Proceedings of the 32nd Annual ACM Symposium on Theory of
Computing , pp. 171—180. New York: ACM Press, 2000.
[Aitchison and Brown 54] J. Aitchison and J. A. C. Brown. “On Criteria for Descrip-
tions of Income Distribution.” Metroeconomica 6 (1954), 88—98.
[Aitchison and Brown 57] J. Aitchison and J. A. C. Brown. The Lognormal Distribu-
tion. Cambridge, UK: Cambridge University Press, 1957.
[Allen et al. 01] A. P. Allen, B. Li, and E. L. Charnov. “Population Fluctuations,
Power Laws and Mixtures of Lognormal Distributions.” Ecology Letters 4 (2001),
1—3.
[Albert et al. 99] R. Albert, H. Jeong, and A.-L. Barab´ asi. “Diameter of the World
Wide Web.” Nature 401 (1999), 130—131.
[Auerbach 1913] F. Auerbach. “Das Gesetz der Bevolkerungskonzentration.” Peter-
manns Geographische Mitteilungen LIX (1913), 73—76.
[Barab´ asi et al. 99] A.-L. Barab´ asi, R. Albert, and H. Jeong. “Mean-Field Theory for
Scale-Free Random Networks.” Physica A 272 (1999), 173—189.
[Barford et al. 99] P. Barford, A. Bestavros, A. Bradley, and M. Crovella. “Changes in
Web Client Access Patterns: Characteristics and Caching Implications.” World
Wide Web 2 (1999), 15—28.
[Barford and Crovella 98] P. Barford and M. Crovella. “Generating Representative
Web Workloads for Network and Server Performance Evaluation.” In Proceedings
of ACM SIGMETRICS , pp. 151—160. New York: ACM Press, 1998.
246 Internet Mathematics
[Bell et al. 90] T. C. Bell, J. G. Cleary, and I. H. Witten. Text Compression . Englewood
Cliﬀs, NJ: Prentice-Hall, 1990.
[Berry 99] B. Berry. “D´ ej`av u ,M r .K r u g m a n . ” Urban Geography 20 (1999) 1—2.
[Black and Scholes 73] F. Black and M. Scholes. “The Pricing of Options and Corpo-
rate Liabilities.” Journal of Political Economics 81 (1973), 637—654.
[Bollob´ as and Riordan to appear] B. Bollob´ as and O. Riordan. “The Diameter of a
Scale-Free Random Graph.” To appear.
[B. Bollob´ as et al. 01] B. Bollob´ a s ,O .R i o r d a n ,J .S p e n c e r ,a n dG .T u s n ´ ady. “The
Degree Sequence of a Scale-Free Random Process.” Random Structures and Algo-
rithms 18:3 (2001), 279—290.
[Bookstein 90] A. Bookstein. “Informetric Distributions, Part I: Uni ﬁed Overview.”
Journal of the American Society for Information Science 41:5 (1990), 368—375.
[Broder et al. 00] A. Broder, R. Kumar, F. Maghoul, P. Raghavan, S. Rajagopalan, R.
Stata, A. Tomkins, and J. Wiener. “Graph Structure in the Web: Experiments
and Models.” In Proc. of the 9th World Wide Web Conference , pp. 309—320.
Amsterdam: Elsevier Science, 2000.
[Carlson and Doyle 99] J. M. Carlson and J. Doyle. “Highly Optimized Tolerance: A
M e c h a n i s mf o rP o w e rL a w si nD e s i g n e dS y s t e m s . ” Physics Review E 60:2 (1999),
1412—1427.
[Champernowne 53] D. Champernowne. “A Model of Income Distribution.” Economic
Journal 63 (1953), 318—351.
[Cooper and Frieze 01] C. Cooper and A. Frieze. “On a General Model of Undirected
Web Graphs.” In Proceedings of the 9th Annual European Symposium on Algo-
rithms , pp. 500—511. Berlin—Heidelberg: Springer—Verlag, 2001.
[Cox et al. 79] J. Cox, S. Ross, and M. Rubinstien. “Option Pricing: A Simpli ﬁed
Approach.” Journal of Financial Economics 7 (1979), 229—265.
[Crovella and Bestavros 97] M. Crovella and A. Bestavros. “Self-Similarity in World
Wide Web Tra ﬃc: Evidence and Possible Causes.” IEEE/ACM Transactions on
Networking 5:6 (1997), 835—846.
[Crovella et al. 98] M. Crovella, M. S. Taqqu, and A. Bestavros. “Heavy-Tailed Prob-
ability Distributions in the World Wide Web.” In A Practical Guide to Heavy
Tails ,e d i t e db yR .J .A d l e r ,R .E .F e l d m a n ,a n dM .S .T a q q u ,p p .3 — 2 6 .L o n d o n :
Chapman and Hall, 1998.
[Crow and Shimizu 88] E. L. Crow and K. Shimizu (editors). Lognormal Distributions:
Theory and Applications .N e wY o r k :M a r k e lD e k k e r ,I n c . ,1 9 8 8 .
[Drinea et al. 00] E. Drinea, M. Enachescu, and M. Mitzenmacher. “Variations on
Random Graph Models of the Web.” Harvard Computer Science Technical Report
TR-06-01 , 2000.
[Downey 01] A. B. Downey. “The Structural Causes of File Size Distribu-
tions.” To appear in MASCOTS 2001 . Available from World Wide Web
(http://rocky.wellesley.edu/downey/ ﬁlesize/), 2001.
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 247
[Estoup 1916] J. B. Estoup. Gammes Stenographiques . Paris: Institut Stenographique
de France, 1916.
[Fabrikant et al. 02] A. Fabrikant, E. Koutsoupias, and C. H. Papadimitriou. “Heuris-
tically Optimized Tradeo ﬀs: A New Paradigm for Power Laws in the Internet.”
InProceedings of the 29th International Colloquium on Automata, Languages,
and Programming, Lecture Notes in Computer Science 2380, pp. 110—122. Berlin—
Heidelberg: Springer—Verlag, 2002.
[Faloutsos et al. 99] M. Faloutsos, P. Faloutsos, and C. Faloutsos. “On Power-Law
Relationships of the Internet Topology.” In Proceedings of the ACM SIGCOMM
1999 Conference , pp. 251—261. New York: ACM Press, 1999.
[Gabaix 99] X. Gabaix. “Zipf’s Law for Cities: An Explanation.” Quarterly Journal of
Economics 114 (1999), 739-767.
[Gibrat 30] R. Gibrat. “Une loi des r´ eparations ´ economiques: l’e ﬀet proportionnel.”
Bull. Statist. g´ en Fr. 19 (1930), 469.
[Gibrat 31] R. Gibrat. L e si n e g a l i t e se c o n o m i q u e s . Paris: Libraire du Recueil Sirey,
1931.
[Gibson et al. 98] D. Gibson, J. Kleinberg, and P. Raghavan. “Inferring Web Com-
munities from Link Topology.” In Proceedings of the 9th ACM Conference on
Hypertext and Hypermedia , pp. 225-234, New York: ACM Press, 1998.
[Gong et al. 01] W. Gong, Y.Liu, V. Misra, and D. Towsley. “On the Tails of Web File-
size Distributions.” In Proceedings of the Thirty-Ninth Annual Allerton Conference
on Communication, Control, and Computing , pp. 192—201. Urbana—Champaign,
IL: Coordinated Science Laboratory, University of Illinois, 2001.
[Huberman and Adamic 99] B. A. Huberman and L. A. Adamic. “Evolutionary Dy-
namics of the World Wide Web.” Technical Report, Xerox Palo Alto Research
Center, 1999. Appears as a brief communication in Nature 401 (1999), 131.
[Huberman and Adamic 00] B. A. Huberman and L. A. Adamic. “The Nature of Mar-
kets in the World Wide Web.” Quarterly Journal of Economic Commerce 1 (2000),
5—12.
[Hull 97] J. C. Hull. Introduction to Futures and Options Markets, Third edition. En-
gelwood Cli ﬀs, NJ: Prentice-Hall, Inc., 1997.
[Ito 51] K. Itˆ o.Stochastic Di ﬀerential Equations, Memoirs of the American Mathemat-
ical Society 4. Providence, RI: American Mathematical Society, 1951.
[Jain and Ramakumar 99] R. Jain and S. Ramakumar. “Stochastic Dynamics Mod-
eling of the Protein Sequence Length Distribution in Genomes: Implications for
Microbial Evolution.” Physica A 273 (1999), 476—485.
[Kapteyn 1903] J. C. Kapteyn. Skew Frequency Curves in Biology and Statistics, As-
tronomical Laboratory. Groningen: Noordho ﬀ, 1903.
[Kesten 73] H. Kesten. “Random Di ﬀerence Equations and Renewal Theory for Prod-
ucts of Random Matrices.” Acta Mathematica CXXXI (1973), 207—248.
248 Internet Mathematics
[Kleinberg et al. 99] J. Kleinberg, R. Kumar, P. Raghavan, S. Rajagopalan, and A.
Tomkins. “The Web as a Graph: Measurements, Models, and Methods.” InProceedings of the International Conference on Combinatorics and Computing,
Lecture Notes in Computer Science 1627, pp. 1—18. Berlin—Heidelberg: Springer—
Verlag, 1999.
[Krapivsky and Redner 01] P. L. Krapivsky and S. Redner. “Organization of Growing
Random Networks.” Physical Review E 63 (2001), 066123-1—066123-14.
[Krugman 96] P. Krugman. The Self-Organizing Economy . Cambridge, MA: Blackwell,
1996.
[Kumar et al. 99a] R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. “Ex-
tracting Large Scale Knowledge Bases from the Web.” In Proceedings of the 25th
VLDB Conference , pp. 639—650. San Francisco: Morgan Kaufmann, 1999.
[Kumar et al. 99b] R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. “Trawl-
ing the Web for Emerging Cyber-Communities.” In Proceedings of the 8th Inter-
national World Wide Web Conference , pp. 403-415. Amsterdam: Elsevier Science,
1999.
[Kumar et al 00] R. Kumar, P. Raghavan, S. Rajagopalan, D. Sivakumar, A. Tomkins,
and E. Upfal. “Stochastic Models for the Web Graph.” In Proceedings of the 41st
Annual Symposium on Foundations of Computer Science , pp. 57—65. New York:
ACM Press, 2000.
[ K u r t z 8 1 ]T .G .K u r t z , Approximation of Population Processes. Philadelphia, PA:
SIAM, 1981.
[Leland et al. 94] W. E. Leland, M. S. Taqqu, W. Willinger, and D. V. Wilson. “On the
Self-Similar Nature of Ethernet Tra ﬃc.”IEEE/ACM Transactions on Networking
(1994), 1—15.
[Li 96] W. Li. “Random Texts Exhibit Zipf’s-Law-Like Word Frequency Distribution.”
IEEE Transactions on Information Theory 38:6 (1992), 1842-1845.
[Li 99] W. Li. “References on Zipf’s Law.” Available from World Wide Web
(http://linkage.rockerfeller.edu/wli/zipf/), 1999.
[Mihail and Papadimitriou 02] M. Mihail and C. H. Papadimitriou. “On the Eigen-
value Power Law.” In Proceedings of RANDOM 2002 , pp. 254—262. Berlin—
Heidelberg: Springer—Verlag, 2002.
[Lotka 26] A. J. Lotka. “The Frequency Distribution of Scienti ﬁc Productivity.” Jour-
nal of the Washington Academy of Sciences 16 (1926), 317—323.
[Lu and Feng 98] J. Lu and Q. Feng. “Strong Consistency of the Number of the Vertices
of Given Degrees in Nonuniform Random Recursive Trees.” Yokohama Math. J.
45 (1998), 61—69.
[Mahmound et al. 93] H. Mahmound, R. Smythe, and J. Szyma´ nski. “On the Structure
of Plane-Oriented Recursive Trees and their Branches. Random Structures and
Algorithms 3 (1993), 255—266.
[Mandelbrot 53] B. Mandelbrot. “An Informational Theory of the Statistical Structure
of Languages.” In Communication Theory , edited by W. Jackson, pp. 486—502.
Woburn, MA: Butterworth, 1953.
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 249
[Mandelbrot 59] B. Mandelbrot. “A Note on a Class of Skew Distribution Function:
Analysis and Critique of a Paper by H.A. Simon.” Information and Control 2
(1959), 90—99.
[Mandelbrot 61a] B. Mandelbrot. “Final Note on a Class of Skew Distribution Func-
tions: Analysis and Critique of a Model due to H.A. Simon.” Information and
Control 4 (1961), 198—216.
[Mandelbrot 61b] B. Mandelbrot. “Post Scriptum to ‘Final Note’.” Information and
Control 4 (1961), 300—304.
[Mandelbrot 97] B. Mandelbrot. Fractals and Scaling in Finance .N e wY o r k : S p r i n g e r -
Verlag, 1997.
[McAlister 1879] D. McAlister. “The Law of the Geometric Mean.” Proceedings of the
Royal Society 29 (1879), 367.
[Medina et al. 00] A. Medina, I. Matta, and J. Byers. “On the Origin of Power Laws
in Internet Topologies.” Computer Communication Review 30:2 (2000), 18—28.
[Miller 57] G. A Miller. “Some E ﬀects of Intermittent Silence.” American Journal of
Psychology 70 (1957), 311—314.
[Mitzenmacher 02] M. Mitzenmacher. “Dynamic Models for File Size Distributions
and Double Pareto Distributions.” Preprint, available from World Wide Web
(http://www.eecs.harvard.edu/ ∼michaelm/NEWWORK/papers/), 2002.
[Mitzenmacher 01] M. Mitzenmacher. “A Bri ef History of Generative Models for Power
Law and Lognormal Distributions. In Proceedings of the Thirty-Ninth Annual
Allerton Conference on Communication, Control, and Computing , pp. 182—191,
Urbana—Champaign, IL: Coordinated Science Laboratory, University of Illinois,
2001.
[Montroll and Schlesinger 82] E. W. Montroll and M. F. Shlesinger. “On 1 /fNoise
and Other Distributions with Long Tails.” Proceedings of the National Academy
of Sciences 79 (1982), 3380—3383.
[Montroll and Schlesinger 83] E. W. Montroll and M. F. Shlesinger. “Maximum En-
tropy Formalism, Fractals, Scaling Phenomena, and 1 /fNoise: A Tale of Tails.”
Journal of Statistical Physics 32 (1983), 209—230.
[Nakajima and Higurashi 98] T. Nakajima and A. Higurashi. “A Use of Two-Channel
Radiances for an Aeresol Characterization from Space.” Geophysical Research Let-
ters25 (1998), 3815—3818.
[Pareto 1896] V. Pareto. Cours d’Economie Politique . Geneva: Droz, 1896.
[ P a x s o na n dF l o y d9 5 ] V .P a x s o na n dS .F l o y d .“ W i d e - A r e aT r a ﬃc: The Failure of
Poisson Modeling.” IEEE/ACM Transactions on Networking 3 (1995), 226—244.
[Perline 96] R. Perline. “Zipf’s Law, the Central Limit Theorem, and the Random
Division of the Unit Interval.” Physical Review E 54:1 (1996), 220—223.
[Pittel 94] B. Pittel. “Note on the Height of Recursive Trees and m-ary Search Trees.”
Random Structures and Algorithms 5 (1994), 337—347.
[Reed 00] W. J. Reed. “The Pareto Law of Incomes—An Explanation
and an Extension.” Submitted. Available from World Wide Web
(http://www.math.uvic.ca/faculty/reed/index.html), 2000.
250 Internet Mathematics
[ R e e d 0 1 ]W .J .R e e d .“ T h eD o u b l eP a r e t o - L o g n o r m a lD i s t r i b u t i o n — AN e wP a r a -
metric Model for Size Distribution.” Available from World Wide Web(http://www.math.uvic.ca/faculty/reed/index.html), 2001.
[Reed and Hughes 02] W. J. Reed and B. D. Hughes. “From Gene Fami-
lies and Genera to Incomes and Internet File Sizes: Why Power-
Laws Are So Common in Nature.” Available from World Wide Web
(http://www.math.uvic.ca/faculty/reed/index.html), 2002.
[Simon 55] H. A. Simon. “On a Class of Skew Distribution Functions.” Biometrika
42:3/4 (1955), 425—440.
[Simon 60] H. A. Simon. “Some Further Notes on a Class of Skew Distribution Func-
tions.” Information and Control 3 (1960), 80—88.
[Simon 61a] H. A. Simon. “Reply to ‘Final Note’.” Information and Control 4 (1961),
217—223.
[Simon 61b] H. A. Simon. “Reply to Dr. Mandelbrot’s Post Scriptum.” Information
and Control 4 (1961), 305—308.
[Smythe and Mahmound 95] R. Smythe and H. Mahmound. “A Survey of Recursive
Trees.” Theoretical Probability and Mathematical Statistics 51 (1995) 1—27.
[Sole et al. 00] R. V. Sole, D. Alonso, and A. McKane. “Connectivity and Scaling in
S-species Model Ecosystems.” Physica A 286 (2000), 337—344.
[Szyma´ nski 87] J. Szyma´ nski. “On a Nonuniform Random Recursive Tree.” Annals of
Discrete Mathematics 333 (1987), 297—306.
[Wheatland and Sturrock 96] M. S. Wheatland and P. A. Sturrock. “Avalanche Models
of Solar Flares and the Distribution of Active Regions.” The Astrophysical Journal
471 (1996), 1044—1048.
[Wormald 95] N. C. Wormald. “Di ﬀerential Equations for Random Processes and Ran-
dom Graphs.” Annals of Appl. Prob. 5 (1995), 1217—1235.
[Yule 25] G. Yule. “A Mathematical Theory of Evolution Based on the Conclusions of
Dr. J.C. Willis.” F.R.S. Philosophical Transactions of the Royal Society of London
(Series B) 213 (1925), 21—87.
[Zhu et al. 01] X. Zhu, J. Yu, and J. Doyle. “Heavy Tails, Generalized Coding, and
Optimal Web Layout.” In Proceedings of IEEE INFOCOM , pp. 1617—1626. Los
Alamitos, CA: IEEE Press, 2001.
[Zipf 32] G. Zipf. Selective Studies and the Principle of Relative Frequency in Language.
Cambridge, MA: Harvard University Press, 1932.
[Zipf 35] G. Zipf. The Psycho-Biology of Language: An Introduction to Dynamic Philol-
ogy. Boston, MA; Houghton Mi ﬄin Company, 1935.
[Zipf 39] G. Zipf. Human Behavior and the Principle of Least E ﬀort. Cambridge, MA:
Addison-Wesley, 1949.
Mitzenmacher: Generative Models for Power Law and Lognormal Distributions 251
Michael Mitzenmacher, Harvard University, Division of Engineering and Applied
Science, 33 Oxford St., Cambridge, MA 02138 (michaelm@eecs.harvard.edu)
Received November 27, 2002; accepted January 6, 2003.