Open Category Detection with PAC Guarantees.

Open Category Detection with PAC Guarantees
Si Liu* 1Risheek Garrepalli* 2Thomas G. Dietterich2Alan Fern2Dan Hendrycks3
Abstract
Open category detection is the problem of detect-
ing “alien” test instances that belong to categories
or classes that were not present in the training
data. In many applications, reliably detecting
such aliens is central to ensuring the safety and
accuracy of test set predictions. Unfortunately,
there are no algorithms that provide theoretical
guarantees on their ability to detect aliens under
general assumptions. Further, while there are al-
gorithms for open category detection, there are
few empirical results that directly report alien de-
tection rates. Thus, there are signiﬁcant theoret-
ical and empirical gaps in our understanding of
open category detection. In this paper, we take
a step toward addressing this gap by studying a
simple, but practically-relevant variant of open
category detection. In our setting, we are pro-
vided with a “clean” training set that contains
only the target categories of interest and an un-
labeled “contaminated” training set that contains
a fractionof alien examples. Under the as-
sumption that we know an upper bound on , we
develop an algorithm with PAC-style guarantees
on the alien detection rate, while aiming to mini-
mize false alarms. Empirical results on synthetic
and standard benchmark datasets demonstrate the
regimes in which the algorithm can be effective
and provide a baseline for further advancements.
1. Introduction
Most machine learning systems implicitly or explicitly as-
sume that their training experience is representative of their
test experience. This assumption is rarely true in real-world
deployments of machine learning, where “unknown un-
knowns”, or “alien” data, can arise without warning. Ig-
*Equal contribution1Department of Statistics, Oregon State
University, Oregon, USA2School of EECS, Oregon State Univer-
sity, Oregon, USA3University of California, Berkeley, California,
USA. Correspondence to: Si Liu <lius2@oregonstate.edu >.
Proceedings of the 35thInternational Conference on Machine
Learning , Stockholm, Sweden, PMLR 80, 2018. Copyright 2018
by the author(s).noring the potential for such aliens can lead to serious safety
concerns in many applications and signiﬁcantly degrade
the accuracy of test set predictions in others. For exam-
ple, consider a scientiﬁc application where a classiﬁer is
trained to recognize speciﬁc categories of insects in fresh-
water samples in order to detect important environmental
changes (Lytle et al., 2010). Test samples will typically
contain some fraction of specimens belonging to species
not represented in the training data. A classiﬁer that is un-
aware of these new species will misclassify the specimens
as belonging to existing species. This will produce incorrect
scientiﬁc conclusions.
The problem of open category detection is to detect such
alien examples at test time. An ideal algorithm for this prob-
lem would guarantee a user-speciﬁed alien-detection rate
(e.g., 95%), while attempting to minimize the false alarm
rate. Unfortunately, no existing algorithm provides such
guarantees under general conditions. In addition, empir-
ical evaluations of existing algorithms for open category
detection typically do not directly evaluate alien detection
rates, which are perhaps the most relevant for safety-critical
applications. Overall, our current theoretical and practical
understanding of open category detection is lacking from a
safety and accuracy perspective.
Is it possible to achieve open category detection with guar-
antees? In this paper, we take a step toward answering
this question by studying a simpliﬁed, but practically rel-
evant, problem setting. To motivate our setting, consider
the above insect identiﬁcation problem. At training time it
is reasonable to expect that a clean training set is available
that contains only the insect categories of interest. At test
time, a new sample will include insects from the training
categories along with some percentage of insects from new
alien categories. Further, scientists may have reasonable
estimates for this percentage based on their scientiﬁc knowl-
edge and practical experience. We would like to guarantee
that the system is able to raise an alarm for, say, 95% of the
insects from alien classes, with each alarm being examined
by a scientist. At the same time, we would like to avoid as
many “false alarms” as possible, since each alarm requires
scientist effort.
To formalize the example, our setting assumes two training
sets: a clean training dataset involving a ﬁnite set of cate-
Open Category Detection with PAC Guarantees
gories and a contaminated dataset that contains a fraction
of aliens. Our ﬁrst contribution is to show that, in this setting,
theoretical guarantees are possible given knowledge of an
upper bound on . In particular, we give an algorithm that
uses this knowledge to provide Probably Approximately
Correct (PAC) guarantees for achieving a user-speciﬁed
alien detection rate. While knowledge of a non-trivial upper
bound onmay not always be possible, in many situations
it will be possible to select a reasonable value based on
domain knowledge, prior data, or by inspecting a sample of
the test data.
The key idea behind our algorithm is to leverage modern
anomaly detectors, which are trained on the clean data. Our
algorithm combines the anomaly-score distributions over
the clean and contaminated training data in order to derive
an alarm threshold that achieves the desired guarantee on
the alien detection rate on new test queries. In theory the
detection rate guarantee will be met regardless of the quality
of the anomaly detector. The quality of the detector, how-
ever, has a signiﬁcant impact on the false alarm rate, with
better detectors leading to fewer false alarms.
We carry out experiments1on synthetic and benchmark
datasets using a state-of-the-art anomaly detector, the Iso-
lation Forest (Liu et al., 2008). We vary the amount of
training data, the fraction of alien data points, along with
the accuracy of the upper bound on provided to our algo-
rithm. The results indicate that our algorithm can achieve
the guaranteed performance when enough data is available,
as predicted by the theory. The results also show that for
the considered benchmarks, the Isolation Forest anomaly
detector is able to support non-trivial false positive rates
given enough data. The results also illustrate the inherent
difﬁculty of the problem for small datasets and/or small
values of. Overall, our results provide a useful baseline
for driving future work on open category detection with
guarantees.
2. Related Work
Open category detection is related to the problem of one-
class classiﬁcation, which aims to detect outliers relative
to a single training class. One-class SVMs (OCSVMs)
(Sch ¨olkopf et al., 2001) are popular for this problem. How-
ever, they have been found to perform poorly for open
category detection due to poor generalization (Zhou &
Huang, 2003), which has been partly addressed by later
work (Manevitz & Yousef, 2002; Wu & Ye, 2009; Jin et al.,
2004; Cevikalp & Triggs, 2012). OCSVMs have been em-
ployed in a multi-class setting similar to open category de-
tection (Heﬂin et al., 2012; Pritsos & Stamatatos, 2013).
1Code for reproducing our experiments can be found at
https://github.com/liusi2019/ocd.However, there are no direct mechanisms to control the alien
detection rate of these methods, which is a key requirement
for our problem setting.
Work on classiﬁcation with rejection/abstaining options
(Chow, 1970; Wegkamp, 2007; Tax & Duin, 2008;
Pietraszek, 2005; Geifman & El-Yaniv, 2017) allows classi-
ﬁers to abstain from making predictions when they are not
conﬁdent. While loosely related to open category detection,
these approaches do not directly consider the possibility of
novel categories, but rather focus on assessing conﬁdence
with respect to the known categories. Due to their closed-
world discriminative nature, it is easy to construct scenarios
where such methods are incorrectly conﬁdent about the class
of an alien and do not abstain.
A variety of prior work has addressed variants of open cat-
egory detection. This includes work on formalizing the
concept of “open space” to characterize the region of the fea-
ture space outside of the support of the training set (Scheirer
et al., 2013). Variants of SVMs have also been developed,
such as the One-vs-Set Machine (Scheirer et al., 2013) and
the Weibull-calibrated SVM (Scheirer et al., 2014). Ad-
ditional work has addressed open category detection by
tuning the decision boundary based on unlabeled data which
contains data from novel categories (Da et al., 2014). Ap-
proaches based on nearest neighbor methods have also been
proposed (Mendes J ´unior et al., 2017). None of these meth-
ods, however, allow for the direct control of alien detection
rates, nor do they provide theoretical guarantees.
There is also recent interest in open category detection for
deep neural networks applied to vision and text classiﬁcation
(Bendale & Boult, 2016; Shu et al., 2017). These methods
usually train a neural network in a standard closed-world
setting, but then analyze various activations in the network
in order to detect aliens. Another related line of work is
detection of out-of-distribution instances, which is similar
to open category detection but assumes that the test data
come from a completely different distribution compared to
the training distribution (Hendrycks & Gimpel, 2017; Liang
et al., 2018). All of this work is quite specialized to deep
neural networks and does not provide direct control of alien
detection rates or theoretical guarantees.
3. Problem Setup
We consider open category detection where there is an un-
known nominal data distribution D0over labeled examples
from a known set of category labels. We receive as input
a “clean” nominal training set S0containingki.i.d. draws
fromD0. In practice, S0will correspond to some curated
labeled data that contains only known categories of interest.
We also receive as input an unlabeled “mixture” dataset
Smthat contains npoints drawn i.i.d. from a mixture dis-
Open Category Detection with PAC Guarantees
tributionDm. Speciﬁcally, the mixture distribution Dmis
a combination of the nominal distribution D0and an un-
known alien distribution Da, which is a distribution over
novel categories (alien data points). We assume that Dais
stationary, so that all alien points that appear as future test
queries will also be drawn from Da.
At training time, we assume that Dmis a mixture distribu-
tion, with probability of generating an alien data point
fromDaand probability of 1of generating a nominal
point. Our results hold even if the test queries come from a
mixture with a different value of as long as the alien test
points are drawn from Da.
Given these datasets, our problem is to label test instances
fromDmas either “alien” or “nominal”. In particular, we
wish to achieve a speciﬁed alien detection rate , which is
the fraction of alien data points in Dmthat are classiﬁed as
“alien” (e.g., 95%). At the same time we would like the false
positive rate to be small, which is the fraction of nominal
data points incorrectly classiﬁed as aliens.
Our approach to this problem assumes the availability of an
anomaly detector that is trained on S0and assigns anomaly
scores to all data points in both S0andSm. Intuitively, the
anomaly scores order the test examples according to how
anomalous they appear relative to the nominal data (higher
scores being more anomalous). An ideal detector would
rank all alien data points higher than all nominals, though
in practice, the ordering will not be so clean. Our approach
labels data in Smby selecting a threshold on the anomaly
scores and labeling all data points with scores above the
threshold as aliens and the remaining points as nominals.
Our key challenge is to select a threshold that provides a
guarantee on the alien detection rate.
4. Algorithms for Open Category Detection
In order to obtain theoretical guarantees, our algorithm as-
sumes knowledge of the alien mixture probability that
generates the mixture data Sm. Later, we will show that
knowing an upper bound on is sufﬁcient to obtain a guar-
antee.
Our approach is based on considering the cumulative dis-
tribution functions (CDFs) over anomaly scores of a ﬁxed
anomaly detector. Let F0;Fa, andFmbe the CDFs of
anomaly scores for the nominal data distribution D0, alien
distribution Da, and mixture distribution Dmrespectively.
SinceDmis a simple mixture of D0andDa, we can write
Fmas
Fm(x) = (1)F0(x) +Fa(x):
From this we can derive the CDF for Fain terms ofFmand
F0:
Fa(x) =Fm(x)(1)F0(x)
:Given the ability to derive Fa, it is straightforward to achieve
an alien detection rate of 1q(e.g. 95%) by selecting an
anomaly score threshold qthat is theqquantile ofFaand
raising an alarm on all test queries whose anomaly score is
greater than q.
In reality, we do not have access to FmorF0and hence
cannot exactly determine Fa. Rather, we have samples
SmandS0. Thus, our algorithm works with the empirical
CDFs ^F0and ^Fm, which are simple step-wise constant
approximations, and estimates an empirical CDF over aliens:
^Fa(x) =^Fm(x)(1)^F0(x)
: (1)
Our algorithm computes the above estimate of ^Faand uses
it to select a threshold ^qto be the largest threshold such
that^Fa(^q)q, where 1qis the target alien detection
rate. This choice will minimize the number of false alarms.
The steps of this algorithm are as follows.
Algorithm 1
1:Get anomaly scores for all points in S0andSm, denoted
x1;x2;:::;xkandy1;y2;:::;ynrespectively.
2:Compute empirical CDFs ^F0and^Fm.
3:Calculate ^Fausing equation 1.
4:Output detection threshold
^q= maxfu2S:^Fa(u)qg;
whereS=fx1;x2;:::;xk;y1;y2;:::;yng.
Although ^Fmand^F0are both legal CDFs, the estimate
for^Fafrom step 3 may not be a legal CDF, because it is
the difference of two noisy estimates—it may not increase
monotonically and it may even be negative. A good tech-
nique for dealing with this problem is to employ isotoniza-
tion (Barlow & Brunk, 1972) and clipping. Isotonization
ﬁnds the monotonically increasing function ^F
aclosest to
^Fain squared error. To convert ^Fainto a legal CDF, deﬁne
Fa= minfmaxf^F
a;0g;1g, where the min and max opera-
tors are applied pointwise to their arguments. We performed
experiments (shown in the supplementary materials) to test
whether using Fain Step 4 would improve the performance
of the overall algorithm. We found that it did not.
5. Finite Sample Guarantee
In the limit of inﬁnite data (both nominal and mixture) and
perfect knowledge of ,^Fawill converge to the true alien
CDF, and our algorithm will achieve the desired alien de-
tection rate. In this section, we consider the ﬁnite data case
wherejS0j=jSmj=n. We derive a value for the sample
sizenthat guarantees with high probability over random
Open Category Detection with PAC Guarantees
draws ofS0andSm, that fraction 1qof the alien
test points will be detected, where is an additional error
incurred because of the ﬁnite sample size n.
Our key theoretical tool is a ﬁnite sample result on the
uniform convergence of empirical CDF functions (Massart,
1990). To use this result, we make the reasonable technical
assumption that the nominal and alien CDFs, F0andFa,
are continuous. In the following, let be the target alien
detection rate, qbe the input to Algorithm 1, ^qbe the
estimatedq-quantile of the alien CDF (step 4 of Alg. 1),
andbe an error parameter. The following theorem gives
the sample complexity for guaranteeing that 1of the
alien examples will be detected using threshold ^q.
Theorem 1. LetS0andSmbe nominal and mixture
datasets containing ni.i.d. samples from the nominal and
mixture data distributions respectively. For any 2(0;1q)
and2(0;1), if
n>1
2ln2
1p
11
22
2
;
then with probability at least 1, Algorithm 1 will return
a threshold ^qthat achieves an alien detection rate of at
least 1, where=q+.
The proof is in the Appendix. Note that ngrows as
O(1
22log1
). Hence, this guarantee is polynomial in all
relevant parameters, which we believe is the ﬁrst such guar-
antee for open category detection. The result can be gener-
alized to the case where n0< nm; in practice, the larger
the mixture sample Smis, the easier it is to estimate q,
because this provides more alien points for estimating the
q-th quantile of Fa.
The theorem gives us ﬂexibility in setting andq(the algo-
rithm input) to achieve a guarantee of 1. Theparameter
controls a trade-off between sample size and false alarm rate.
To minimize the false alarm rate, we want to make qlarge
(to obtain a larger threshold), so we want to set qclose to.
But, asq!,!0, andn!1 . To minimize the sample
sizen, we want to make qas small as possible, because that
allowsto be larger and hence nbecomes smaller. The
optimal setting of depends on how the false alarm rate
grows withq, which in turn depends on the relative shape
ofF0andFa. In a real safety application, we can estimate
these fromS0andSmand choose an appropriate qvalue.
What if we don’t know the exact value of ? If our algorithm
uses an upper bound 0on the trueto compute ^Fa, we
can still provide a guarantee. In this case, in addition to the
assumptions in Theorem 1, we need a concept of an anomaly
detector being admissible . We say that an anomaly detector
isadmissible for a problem, if the anomaly score CDFs
satisfyF0(x)Fm(x)for allx2R. Most reasonable
anomaly detectors will be admissible in this sense, sincethe alien CDF will typically concentrate more mass toward
larger anomaly score values compared to F0. Indeed, if this
is not the case, there is little hope since there is effectively
no signal to distinguish between aliens and nominals.
Corollary 1. Consider running Algorithm 1 using an upper
bound0on the true. Under the same assumptions as
Theorem 1, if the anomaly detector is admissible and
n>1
2ln2
1p
11
220
02
;
then with probability at least 1, Algorithm 1 will return
a threshold ^qthat achieves an alien detection rate of at
least 1, where=q+.
The proof is in the Appendix. While we can achieve a guar-
antee using an upper bound on 0, the returned threshold
will be more conservative (smaller) than if we had used the
true. This will result in higher false alarm rates, since
more nominal points will be above the threshold. Thus it is
desirable to use a value of 0that is as close to as possible.
6. Experiments
We performed experiments to answer four questions. Ques-
tion Q1: how accurate is our estimate of ^qas a function of
nand? Question Q2: how loose are the bounds from The-
orem 1? Question Q3: what are typical values of the false
alarm rates for various settings of nandon real datasets?
Question Q4: how do these observed values change if we
employ an overestimate 0>?
All of our experiments employ the Isolation Forest anomaly
detector (Liu et al., 2008), which has been demonstrated
to be a state-of-the-art detector in recent empirical studies
(Emmott et al., 2013). In the Supplementary Materials
we show similar results with the LODA anomaly detector
(Pevn ´y, 2015).
To address Q1 and Q2, we run controlled experiments
on synthetic data. The data points are generated from 9-
dimensional normal distributions. The dimensions of the
nominal distribution D0are independently distributed as
N(0;1). The alien distribution is similar, but with probabil-
ity 0.4, 3 of the 9 dimensions (chosen uniformly at random)
are distributed as N(3;1)and with probability 0.6, 4 of the
9 dimensions (chosen uniformly at random) follow N(3;1).
This ensures that the anomalies are not highly similar to
each other and models the situation in which there are many
different kinds of alien objects, not just a single alien class
forming a tight cluster.
In each experiment, the nominal dataset and the mixture
dataset are of the same size n, and the mixture dataset
contains a proportion of anomaly points. We ﬁxed
the target quantile to be q= 0:05. The experiments are
Open Category Detection with PAC Guarantees
0.650.70.750.80.850.90.951Recall
5!=0.01100⋯10000100⋯10000!=0.05100⋯10000!=0.10100⋯10000!=0.20!=0.50)=100⋯10000
Figure 1. Comparison of recall achieved by ^qcompared to oracle
recall of 0.95. Error bars are 95% conﬁdence intervals. Settings
ofnandincrease from left to right starting with = 0:01and
n2f100;500;1K;5K;10Kgup to= 0:5andn= 10 K.
carried out for n2 f100;500;1K;5K;10Kgand2
f0:01;0:05;0:10;0:20;0:50g. For testing, we create two
large datasets G0andGa, withG0being a pure nominal
dataset,Gabeing a pure alien dataset, and jG0j=jGaj=
20K. The Isolation Forest algorithm computes 1000 full
depth isolation trees on the nominal data. Each tree is grown
on a randomly-selected 20% subsample of the clean data
points. We compute anomaly scores for the nominal points
via out-of-bag estimates and anomaly scores for the mix-
ture points,G0, andGausing the full isolation forest. For
each combination of nand, we repeat the experiment
100times. We measure the fraction of aliens detected (the
“recall”) and the fraction of nominal points declared to be
alien (the “false positive rate”) by applying the ^qestimate
to threshold the anomaly scores in G0andGa.
To assess the accuracy of our ^qestimates (Q1), we could
compare them to the true values. However, this comparison
is hard to interpret, because is expressed on the scale
of anomaly scores, which are somewhat arbitrary. Instead,
Figure 1 plots the recall achieved by ^q. If^qhad been
estimated perfectly, the recall would always be 1q= 0:95.
However, we see that the recall is often less than 0.95, which
indicates that ^qis over-estimated, especially when nand
are small. This behavior is predicted by our theory, where
we see that the sample size requirements grow inversely
with2. For largerandn, the recall guarantee is generally
achieved. Figure 2 compares the false positive rate of the
true oracleqto the false positive rate of the estimate ^q. For
each combination of andn, we have 100 replications of
the experiment and therefore 100 estimates ^aand 100 FPR
rates. For each of these, the true FPR is computed using G0.
2!=0.01100⋯10000100⋯10000!=0.05100⋯10000!=0.10100⋯10000!=0.20!=0.50)=100⋯1000000.020.040.060.080.10.12FPRFigure 2. Comparison of oracle FPR to the FPR achieved by ^q.
Error bars span from the 25th to 75th percentile with the blue dot
marking the median of the 100 trials. Orange markers indicate the
oracle FPR. Settings of nandincrease from left to right starting
with= 0:01andn2f100;500;1K;5K;10Kgup to= 0:5
andn= 10 K.
The error bars summarize the resulting 100 FPR values by
the median and inter-quartile range. We see that for small n
and, the FPR can be quite different from the oracle rate,
but for larger nand, the estimates are very good.
To assess the looseness of the bounds (Q2), for each combi-
nation ofnand, we ﬁx= 0:05and compute the value
ofsuch that 95 of the 100 runs achieved a recall of at least
1(thusempirially achieves the 1guarantee). We
then compute =qand the corresponding required
sample size naccording to Theorem 1. Figure 3 shows a
Figure 3. The log sample size nrequired by Theorem 1 in order
to guarantee the actual observed recall versus the log actual sample
sizen.
Open Category Detection with PAC Guarantees
plot ofnversus the actual n. The distance of these points
from then=ndiagonal line show that the theory is fairly
loose, although it becomes tighter as ngets large.
00.050.10.150.20.250.30.35
00.10.20.30.40.5FPR
αLandsatpagebOCRLetter recogShuttleCovertype
Figure 4. False positive rates on six UCI datasets as a function of
(q= 0:05,= 0:05).
Benchmark Data Experiments. To address our third and
fourth questions, we performed experiments on six UCI
multiclass datasets: Landsat, Opt.digits, pageb, Shuttle,
Covertype and MNIST. In addition to these, we provide
results for the Tiny ImageNet dataset. In each multiclass
dataset, we split the classes into two groups: nominal and
alien. For Tiny ImageNet, we train a deep neural network
classiﬁer on 200 nominal classes and treat the remaining
800 as aliens. The nominal classes for UCI datasets are
MNIST(1,3,7), Landsat(1,7), OCR(1,3,4,5,7), pageb(1,5),
Letter recognition(1,3), and Shuttle(1,4). We generated
0.860.880.90.920.940.960.98
00.10.20.30.40.5Recall!LandsatpagebOCRLetter recogShuttleCovertype
Figure 5. Recall rates on six UCI datasets as a function of (q=
0:05,= 0:05)
0.30.350.40.450.50.550.60.650.7
0.050.150.250.350.45FPR!MNISTTiny ImagenetFigure 6. False positive rates on two image datasets as a function
of(q= 0:05;= 0:05).
nominal and mixture datasets for various values of . The
value ofnfor each dataset is 1532 for Landsat,788 for Letter
recognition, 568 for OCR, 4912 for pageb, 5000 for Shuttle,
13,624 for Covertype, 11,154 for MNIST, and 10,000 for
Tiny ImageNet. Because we cannot create datasets with
largen, we cannot measure the true value of q.
After computing the anomaly scores for both nominal and
mixture datasets, we applied Algorithm 1 within a 10-fold
cross validation. We divide the mixture data points at ran-
dom into 10 groups. For each fold, we estimate ^Faand^a
from 9 of the 10 groups and then score the mixture points in
the held-out fold according to ^a. In all other respects, the
experimental protocol is the same as for the synthetic data.
For Tiny ImageNet, the anomaly scores are obtained by
applying a baseline method (Hendrycks & Gimpel, 2017).
To answer Q3, Figures 4 and 6 plot the false positive rate as
a function of for the UCI and vision datasets, respectively.
We see that the FPR ranges from 3.6% to 26.9% on UCI
depending on the dataset and the level of . The vision
datasets have higher FPR, especially MNIST, which has a
large number of alien classes that are not distinguished well
by the anomaly detector. The FPR depends primarily on
the domain, because the key issue is how well the anomaly
detector distinguishes between nominal and alien examples.
The false alarm rate generally improves as increases. In
some applications, it may be possible to enrich Smso that
is larger on the training set to take advantage of this
phenomenon. It is interesting to note that once ^ahas been
computed, it can be applied to test datasets having different
(or unknown) values of .
Figures 5 and 7 plot the recall rate as a function of for
the UCI and vision datasets. We set q= 0:05in these
experiments. Theorem 1 only guarantees a recall of 1q,
Open Category Detection with PAC Guarantees
0.880.890.90.910.920.930.940.950.960.970.98
0.050.150.250.350.45Recall
!MNISTTiny Imagenet
Figure 7. Recall rates on two image datasets as a function of
(q= 0:05;= 0:05).
00.050.10.150.20.250.30.350.4
00.0020.0040.0060.0080.01α'	-αrecall'	-recallfpr'	-fpr
Figure 8. Change in recall and false positive rate as a function of
0for six UCI datasets; 2f0:1;0:2;0:4g
wheredepends on n. Hence, it is nice to see that for
three of the domains (Shuttle, Covertype, and Landsat) in
UCI and for both vision datasets, the recall is very close to
1q= 0:95. These are the domains with the largest values
ofn. The value of has a bigger impact on recall than it
does on FPR. This is because the effective number of alien
training examples is n, which can be very small for some
datasets when = 0:1. This shows that in applications such
as fraud detection, where may be very small, the mixture
datasetSmneeds to be very large.
To answer Q4 regarding the impact of using an incorrect
value0> , we repeated these experiments with 0=
+, for2f0:002;0:004;0:006;0:008;0:010g. Figure 8
plots the change in false positive rate and recall as a functionof0. Two points are plotted for each combination
of0and dataset, the change in Recall and the change in
FPR. We observe that the recall increases slightly (in the
range from 0.01 to 0.05). However, the false positive rate
increases by much larger amounts (from 0.01 to 0.336). This
demonstrates that it is very important to determine the value
ofaccurately.
7. Summary
We have taken a step toward open category detection with
guarantees by providing a PAC-style guarantee on the prob-
ability of detecting 1of the aliens on the test data. This
is the ﬁrst such guarantee under any similarly general con-
ditions. We have shown that this guarantee is satisﬁed in
our experiments, although the guarantee is somewhat loose,
especially on small training sets. Obtaining a guarantee re-
quires more data than standard PAC guarantees on expected
prediction accuracy. This is because we must estimate the
qquantile of the alien anomaly score distribution, where
qis typically quite small. Nonetheless, our experiments
show that our algorithm gives good recall performance and
non-trivial false alarm rates on datasets of reasonable size.
It is important to note that the very formulation of a PAC-
style guarantee on the probability of detecting aliens re-
quires assuming that the aliens are drawn from a well-
deﬁned distribution Da. While this is appropriate in some
applications, such as the insect survey application described
in the introduction, it is not appropriate for adversarial set-
tings. In such settings, a PAC-style guarantee does not make
sense, and some other form of safety guarantee needs to be
formulated.
To obtain the guarantee, we employ two training datasets:
a clean dataset that contains no aliens and an (unlabeled)
contaminated dataset that contains a known fraction of
aliens. An important theoretical problem for future research
is to develop a method that can estimate a tight upper bound
on^> . We believe this is possible, but we have not yet
found a method that guarantees that ^> .
Our guarantee requires more data as becomes small. For-
tunately, when is small, it may be possible in some appli-
cations to afford lower recall rates, since the frequency of
aliens will be smaller. However, in safety-critical applica-
tions where a single undetected alien poses a serious threat,
there is little recourse other than to collect more data or
allow for higher false positive rates.
Acknowledgements
This research was supported by a gift from Huawei, Inc.,
and grants from the Future of Life Institute and the NSF
Grant 1514550. Any opinions, ﬁndings, and conclusions
Open Category Detection with PAC Guarantees
or recommendations expressed in this material are those of
the author(s) and do not necessarily reﬂect the views of the
sponsors.
A. Proof for Theorem 1
Suppose there are nrandom variables which are i.i.d. from
the distribution with CDF Fand let ^Fnbe the empirical
CDF calculated from this sample. Then Massart (1990)
shows that
P(pnsup
xj^Fn(x)F(x)j>)2 exp(22)(2)
holds without any restriction on . Making use of this,
and assuming we use the same sample size nfor both the
mixture dataset and the clean data set, for any 2(0;1q),
we seek to determine how large nneeds to be in order to
guarantee that with probability at least 1our quantile
estimate ^qsatisﬁesFa(^q)q+. To achieve this, we
want to have
P(sup
xj^Fa(x)Fa(x)j>):
We have
P(sup
xj^Fa(x)Fa(x)j>)
=P(sup
xj^Fm(x)(1)^F0(x)

Fm(x)(1)F0(x)
j>)
=P(sup
xj1
(^Fm(x)Fm(x))
1
(^F0(x)F0(x))j>)
P((1
sup
xj^Fm(x)Fm(x)j+
1
sup
xj^F0(x)F0(x)j)>)
P(f1
sup
xj^Fm(x)Fm(x)j>1
2g
[f1
sup
xj^F0(x)F0(x)j>1
2g)
=P(fsup
xj^Fm(x)Fm(x)j>
2g
[fsup
xj^F0(x)F0(x)j>
2g):
Making use of (2), when
n>1
2ln2
1p
1(1
)2(2
)2;we will have
P(sup
xj^Fm(x)Fm(x)j>
2)1p
1;
P(sup
xj^F0(x)F0(x)j>
2)1p
1:
In this case we will have
P(sup
xj^Fa(x)Fa(x)j>)
1P(fsup
xj^Fm(x)Fm(x)j
2g
\fsup
xj^F0(x)F0(x)j
2g)
1(11 +p
1)2
=:
Now we have with probability at least 1,
j^Fa(x)Fa(x)j;8x2R:
If this inequality holds, then for any value ^qsuch that
^Fa(^q)q, we have
Fa(^q)^Fa(^q) +q+:
So we have with probability at least 1, any ^qsatisfying
^Fa(^q)qwill satisfyFa(^q)q+.
B. Proof for Corollary 1
If0, and if we write
F0
a(x) =Fm(x)(10)F0(x)
0;
thenF0
ais still a legal CDF, because
F0
a(1) = 0; F0
a(1) = 1;
and it is easy to show that F0
ais monotonically nondecreas-
ing.
But
F0
a(x)Fa(x) =(0)(Fm(x)F0(x))
00;8x2R;
and because of this, if we let ^0
qdenote the threshold we
get from using 0, we will have Fa(^0
q)F0
a(^0
q). By
the proof of previous theorem, we know that when n >
1
2ln2
1p1(1
)2(20
0)2, we have with probability at least
1,F0
a(^0
q)q+, and thus we have Fa(^0
q)q+.
References
Barlow, RE and Brunk, HD. The isotonic regression prob-
lem and its dual. Journal of the American Statistical
Association , 67(337):140–147, 1972.
Open Category Detection with PAC Guarantees
Bendale, A. and Boult, T. E. Towards open set deep net-
works. In 2016 IEEE Conference on Computer Vision
and Pattern Recognition (CVPR) , pp. 1563–1572, June
2016.
Cevikalp, H. and Triggs, B. Efﬁcient object detection using
cascades of nearest convex model classiﬁers. In 2012
IEEE Conference on Computer Vision and Pattern Recog-
nition , pp. 3138–3145, June 2012.
Chow, C. On optimum recognition error and reject tradeoff.
IEEE Transactions on Information Theory , 16(1):41–46,
Jan 1970. ISSN 0018-9448.
Da, Qing, Yu, Yang, and Zhou, Zhi-Hua. Learning with
augmented class by exploiting unlabeled data. In Pro-
ceedings of the Twenty-Eighth AAAI Conference on Artiﬁ-
cial Intelligence , AAAI’14, pp. 1760–1766. AAAI Press,
2014.
Emmott, Andrew F, Das, Shubhomoy, Dietterich, Thomas,
Fern, Alan, and Wong, Weng-Keen. Systematic construc-
tion of anomaly detection benchmarks from real data. In
Proceedings of the ACM SIGKDD workshop on outlier
detection and description , pp. 16–21. ACM, 2013.
Geifman, Yonatan and El-Yaniv, Ran. Selective classiﬁca-
tion for deep neural networks. In Guyon, I., Luxburg,
U. V ., Bengio, S., Wallach, H., Fergus, R., Vishwanathan,
S., and Garnett, R. (eds.), Advances in Neural Informa-
tion Processing Systems 30 , pp. 4885–4894. Curran As-
sociates, Inc., 2017.
Heﬂin, B., Scheirer, W., and Boult, T. E. Detecting and
classifying scars, marks, and tattoos found in the wild. In
2012 IEEE Fifth International Conference on Biometrics:
Theory, Applications and Systems (BTAS) , pp. 31–38,
Sept 2012.
Hendrycks, Dan and Gimpel, Kevin. A baseline for de-
tecting misclassiﬁed and out-of-distribution examples in
neural networks. In Proceedings of International Confer-
ence on Learning Representations , 2017.
Jin, Hongliang, Liu, Qingshan, and Lu, Hanqing. Face
detection using one-class-based support vectors. In Sixth
IEEE International Conference on Automatic Face and
Gesture Recognition, 2004. Proceedings. , pp. 457–462,
May 2004.
Liang, Shiyu, Li, Yixuan, and Srikant, R. Enhancing the
reliability of out-of-distribution image detection in neural
networks. International Conference on Learning Repre-
sentations , 2018.
Liu, Fei Tony, Ting, Kai Ming, and Zhou, Zhi-Hua. Isolation
forest. In Data Mining, 2008. ICDM’08. Eighth IEEE
International Conference on , pp. 413–422. IEEE, 2008.Lytle, David A, Mart ´ınez-Mu ˜noz, Gonzalo, Zhang, Wei,
Larios, Natalia, Shapiro, Linda, Paasch, Robert, Mold-
enke, Andrew, Mortensen, Eric N, Todorovic, Sinisa, and
Dietterich, Thomas G. Automated processing and identi-
ﬁcation of benthic invertebrate samples. Journal of the
North American Benthological Society , 29(3):867–874,
2010.
Manevitz, Larry M. and Yousef, Malik. One-class svms for
document classiﬁcation. J. Mach. Learn. Res. , 2:139–154,
March 2002. ISSN 1532-4435.
Massart, P. The tight constant in the dvoretzky-kiefer-
wolfowitz inequality. The Annals of Probability , 18(3):
1269–1283, 1990. ISSN 00911798.
Mendes J ´unior, Pedro R., de Souza, Roberto M., Werneck,
Rafael de O., Stein, Bernardo V ., Pazinato, Daniel V .,
de Almeida, Waldir R., Penatti, Ot ´avio A. B., Torres,
Ricardo da S., and Rocha, Anderson. Nearest neighbors
distance ratio open-set classiﬁer. Machine Learning , 106
(3):359–386, Mar 2017. ISSN 1573-0565.
Pevn ´y, Tom ´aˇs. Loda: Lightweight on-line detector of
anomalies. Machine Learning , (November 2014), 2015.
Pietraszek, Tadeusz. Optimizing abstaining classiﬁers using
roc analysis. In Proceedings of the 22Nd International
Conference on Machine Learning , ICML ’05, pp. 665–
672, New York, NY , USA, 2005. ACM. ISBN 1-59593-
180-5.
Pritsos, Dimitrios A. and Stamatatos, Efstathios. Open-
Set Classiﬁcation for Automated Genre Identiﬁcation , pp.
207–217. Springer Berlin Heidelberg, Berlin, Heidelberg,
2013. ISBN 978-3-642-36973-5.
Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., and
Boult, T. E. Toward open set recognition. IEEE Transac-
tions on Pattern Analysis and Machine Intelligence , 35
(7):1757–1772, July 2013. ISSN 0162-8828.
Scheirer, W. J., Jain, L. P., and Boult, T. E. Probability
models for open set recognition. IEEE Transactions on
Pattern Analysis and Machine Intelligence , 36(11):2317–
2324, Nov 2014. ISSN 0162-8828.
Sch¨olkopf, Bernhard, Platt, John C., Shawe-Taylor, John C.,
Smola, Alex J., and Williamson, Robert C. Estimating
the support of a high-dimensional distribution. Neural
Comput. , 13(7):1443–1471, July 2001. ISSN 0899-7667.
Shu, Lei, Xu, Hu, and Liu, Bing. DOC: deep open classiﬁ-
cation of text documents. CoRR , abs/1709.08716, 2017.
Tax, D.M.J. and Duin, R.P.W. Growing a multi-class classi-
ﬁer with a reject option. Pattern Recognition Letters , 29
(10):1565 – 1570, 2008. ISSN 0167-8655.
Open Category Detection with PAC Guarantees
Wegkamp, Marten H. Lasso type classiﬁers with a reject
option. 2007.
Wu, M. and Ye, J. A small sphere and large margin ap-
proach for novelty detection using training data with out-
liers. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence , 31(11):2088–2092, Nov 2009. ISSN
0162-8828.
Zhou, Xiang Sean and Huang, Thomas S. Relevance feed-
back in image retrieval: A comprehensive review. Mul-
timedia Systems , 8(6):536–544, Apr 2003. ISSN 1432-
1882.

Si Liu
Risheek Garrepalli
Thomas G Dietterich
Alan Fern
Dan Hendrycks


Probing BERT’s priors with serial reproduction chains.

Probing BERT’s priors with serial reproduction chains
Takateru Yamakoshi1;2, Thomas L. Grifﬁths1, Robert D. Hawkins1
1Princeton University,2The University of Tokyo
{takateru,tomg,rdhawkins}@princeton.edu
Abstract
Sampling is a promising bottom-up method for
exposing what generative models have learned
about language, but it remains unclear how to
generate representative samples from popular
masked language models (MLMs) like BERT.
The MLM objective yields a dependency net-
work with no guarantee of consistent condi-
tional distributions, posing a problem for naive
approaches. Drawing from theories of iter-
ated learning in cognitive science, we explore
the use of serial reproduction chains to sam-
ple from BERT’s priors. In particular, we ob-
serve that a unique and consistent estimator of
the ground-truth joint distribution is given by
a Generative Stochastic Network (GSN) sam-
pler, which randomly selects which token to
mask and reconstruct on each step. We show
that the lexical and syntactic statistics of sen-
tences from GSN chains closely match the
ground-truth corpus distribution and perform
better than other methods in a large corpus of
naturalness judgments. Our ﬁndings establish
a ﬁrmer theoretical foundation for bottom-up
probing and highlight richer deviations from
human priors1.
1 Introduction
Large neural language models have become the
representational backbone of natural language pro-
cessing. By learning to predict words from their
context, these models have induced surprisingly
human-like linguistic knowledge, from syntactic
structure (Linzen and Baroni, 2021; Tenney et al.,
2019; Warstadt et al., 2019) and subtle lexical bi-
ases (Hawkins et al., 2020) to more insidious so-
cial biases and stereotypes (Caliskan et al., 2017;
Garg et al., 2018). At the same time, efforts to
probe these models have revealed signiﬁcant de-
viations from natural language (Braverman et al.,
2020; Holtzman et al., 2019; Dasgupta et al., 2020).
1Code and data are available at https://github.
com/taka-yamakoshi/TelephoneGame
chain 2chain 1
food was running short, and winters were colder.
time was running short, and winters were colder.
time was running out, and winters were colder.Figure 1: We use a serial reproduction method to probe
BERT’s prior over possible sentences (visualization of
reproduction chains obtained by running t-sne on sen-
tence embeddings; chains are color-coded and fade to
black across their burn-in period).
Observations of incoherent or “weird” behavior
may often be amusing, as when a generated recipe
begins with “1/4 pounds of bones or fresh bread”
(Shane, 2019), but also pose signiﬁcant dangers in
real-world settings (Bender et al., 2021).
These deviations present a core theoretical and
methodological puzzle for computational linguis-
tics. How do we elicit and characterize the full
prior2that a particular model has learned over pos-
sible sentences in a language? A dominant ap-
proach has been to design benchmark suites that
2We use the term prior to refer to graded linguistic knowl-
edge assigning probabilities to all possible sentences. While
we focus on text, this prior is also the foundation for more
grounded, pragmatic language use.arXiv:2202.12226v2  [cs.CL]  18 Mar 2022
Type of unnaturalness Example
word-level morphological Higher education school xurdivided into six institutions.
phrase-levelsyntacticSwallowing hard, Verity stared at the these , desperately wanting
to see if they congealed.
semanticThe west section is a ﬁg octagon .
A private apartment with nothing but hot cooled water.
predicationHe already costumes his relationship with my mother carefully.
V oices rapped on the incremental door .
sentence-levelout-of-context Like a cataract , Horatius responds, “You are better than me.”
self-contradictory The newspaper is published weekly andbiannually .
pragmaticShe grew up with three sisters andten sisters .
It should apply between the extreme andthe extreme .
Table 1: Examples of sentences sampled from BERT’s prior that received low naturalness ratings from our partic-
ipants, including sources forms of unnaturalness like predicability or category errors (e.g. doors typically do not
have the property of “incrementality”), semantic incoherence (“hot cooled water”), or contradictory constructions
(especially for longer sentences). More examples can be found in table S2 and in the online supplement.
probe theoretically important aspects of the prior,
and compare model behavior to human behavior
on those tasks (e.g. Warstadt et al., 2020; Ettinger,
2020). Yet this approach can be restrictive and
piecemeal: it is not clear ahead of time which
tasks will be most diagnostic, and many sources of
“weirdness” are not easily operationalized (Kurib-
ayashi et al., 2021).
A more holistic, bottom-up alternative is to di-
rectly examine samples from the model’s prior and
compare them against those from human priors.
However, many successful models do not explicitly
expose this distribution, and many generation meth-
ods optimize the “best” sentences rather than the-
oretically meaningful or representative ones. For
example, masked language models (MLMs) like
BERT (Devlin et al., 2018) are dependency net-
works (Heckerman et al., 2000; Toutanova et al.,
2003), trained to efﬁciently learn an independent
collection of conditional distributions without en-
forcing consistency between them. In other words,
these conditionals may not correspond to any co-
herent joint distribution at all, leading recent work
to focus on other score-based sampling objectives
(Goyal et al., 2021).
Here, we explore the use of serial reproduction
chains (see Fig. 1) to overcome these challenges.
While a naive (pseudo-)Gibbs sampler is indeed
problematic for MLMs, the literature on Generative
Stochastic Networks (GSNs; Bengio et al., 2014)
has formally shown that a simple algorithmic vari-
ant we call GSN sampling produces a stationary
distribution that is, in fact, a unique and consis-tent estimator of the ground-truth joint distribution.
Furthermore, while the independent conditionals
learned by dependency networks may be arbitrarily
inconsistent in theory, empirical work has found
that these deviations tend to be negligible in prac-
tice, especially on larger datasets (Heckerman et al.,
2000; Neville and Jensen, 2007). Thus, we argue
that it is both theoretically and empirically justiﬁed
to take these samples as uniquely representative of
the model’s prior over language.
We begin in Section 2 by introducing the serial
reproduction approach and clarifying the problem
of re-constructing a joint distribution from a depen-
dency network. We then validate that our chains are
well-behaved (Section 3) and compare the statis-
tics of samples from BERT’s prior to the lexical
and syntactic statistics of its ground-truth training
corpus to measure distributional similarity (Section
4). Finally, in Section 5, we present a large-scale
behavioral study eliciting naturalness judgments
from human speakers and identify features of the
generated sentences which most strongly predict
human ratings of “weirdness” (see Table 1). We
ﬁnd that GSN samples closely approximate the
ground-truth distribution and are judged to be more
natural than other methods, while also revealing
areas of improvement that have been difﬁcult to
quantify with top-down benchmarks.
2 Approach
2.1 Serial reproduction
Our approach is inspired by serial reproduction
games like Telephone, where an initial message is
Bayes net (acyclic)
dependency net (cyclic)LM MLM
Figure 2: While autoregressive language models (LMs)
are Bayes nets, masked language models (MLMs) are
dependency networks with cyclic dependencies.
gradually relayed along a chain from one speaker
to the next. At each step, the message is changed
subtly as a result of noisy transmission and recon-
struction, and the ﬁnal version of the message often
differs drastically from the ﬁrst. This serial repro-
duction method, initially introduced to psychology
by Bartlett (1932), has become an invaluable tool
for revealing human inductive biases (Xu and Grif-
ﬁths, 2010; Langlois et al., 2021; Sanborn et al.,
2010; Harrison et al., 2020). Because reconstruct-
ing a noisy message is guided by the listener’s prior
expectations, such chains eventually converge to a
stationary distribution that is equivalent to the pop-
ulation’s prior, reﬂecting what people expect others
to say (Kalish et al., 2007; Grifﬁths and Kalish,
2007; Beppu and Grifﬁths, 2009). For example,
Meylan et al. (2021) recently evaluated the ability
of neural language models to predict the changes
made to sentences by human participants at each
step of a serial reproduction chain. Thus, while
serial reproduction is commonly used to probe hu-
man priors, and to compare models against human
data, it is not yet in wide use for probing the models
themselves.
2.2 BERT as a dependency network
There has been considerable confusion in the recent
literature over how to interpret the MLM objective
used to train models like BERT, and how to inter-
pret samples from such models. Wang and Cho
(2019) initially observed that BERT was a Markov
Random Field (MRF) and proposed a Gibbs sam-
pler that iteratively masks and reconstructs differ-
ent siteskby sampling from the conditional given
the tokens at all other sites ^P(wkjwk). As ob-
served by Goyal et al. (2021), however, this pro-
cedure does not actually correspond to inference
in the MRF. Unlike auto-regression language mod-
els (LMs) like GPT-3 (Brown et al., 2020), which
deﬁne an acyclic dependency graph (or Bayes net)from left-to-right, MLMs have cyclic dependencies
(see Fig. 2) and are therefore usefully interpreted as
dependency networks rather than Bayes networks
(Heckerman et al., 2000). Because dependency net-
works estimate independent conditionals, there is
no guarantee that these conditionals are consistent
(i.e. they may violate Bayes rule) and therefore do
not represent a coherent joint distribution.
Still, it is possible to re-construct a joint dis-
tributions from these conditionals. For example,
Heckerman et al. (2000) proved that if sites are
visited in a ﬁxed order, a (pseudo-)Gibbs chain
similar to the one used by Wang and Cho (2019)
does converge to a stationary distribution that is
a well-formed joint. The problem is that differ-
ent orders may yield different joint distributions,
making it difﬁcult to interpret any distributions as
deﬁnitive. This ambiguity was resolved by the Gen-
erative Stochastic Network framework proposed by
Bengio et al. (2014). Instead of visiting sites in
a ﬁxed order, a GSN sampler randomly chooses
which site to visit at each step (with replacement),
thus preserving aperiodicity and ergodicity. Specif-
ically, this algorithm begins by initializing with
a sequencefw0
1;:::;w0
ng. At each step tof the
chain, we randomly choose a site k21;:::;n to
mask out, and we sample a new value wt+1
kfrom
the conditional distribution P(wkjwt
k)with the
othern1sites ﬁxed.
A key theorem of Bengio et al. (2013, 2014)
proves that the stationary distribution arising from
the GSN sampler deﬁnes a unique joint distribu-
tion, and furthermore, this stationary distribution
is a consistent estimator of the ground-truth joint
distribution3. Importantly, this stationary distribu-
tion differs from the one given by the Metropolis-
Hastings (MH) approach suggested by Goyal et al.
(2021), which uses the GSN sampler as a proposal
distribution but accepts or rejects proposals based
on an energy-based pseudo-likelihood deﬁned by
the sum of the conditional scores at each location
(Salazar et al., 2020). This MH sampler instead
converges to an implicit stationary distribution de-
ﬁned by the energy objective4.
3Technically, the proof only holds if the dependency net-
work was trained using consistent estimators for the condition-
als, which is the case for the cross-entropy loss used by BERT;
see also McAllester (2019).
4Although our focus is on evaluation rather than algorith-
mic performance characteristics, we note that because GSN
sampling does not require calculating energy scores to deter-
mine the acceptance probability for each sample, it is signiﬁ-
cantly faster, especially for longer sequences.
2.3 Mixture kernels
In practice, Markov chain sampling methods have
many failure modes. Most prominently, because
samples in the chains are not independent, it is
challenging to guarantee convergence to a station-
ary distribution, and the chain is easily “stuck” in
local regions of the sample space (Gelman et al.,
1992). Typically, samples from a burn-in period
(e.g. the ﬁrst mepochs) are discarded to reduce
dependence on the initial state, and a lagbetween
samples (e.g. recording only every lepochs) is in-
troduced to reduce auto-correlation. However, the
problem is particularly severe for language models
like BERT where there are strong mutual dependen-
cies between words at different sites. For example,
once the chain reaches a tri-gram like ‘Papua New
Guinea’, it is unlikely to change any single word
while keeping the other words constant. To ensure
ergodicity, we use a mixture kernel introducing a
small constant probability ( = 0:001) of return-
ing to the initial distribution of [MASK] tokens on
each epoch, allowing the chain to burn in again.
3 Validating the stationary distribution
In this section, we validate that the samples pro-
duced by our serial reproduction method are repre-
sentative of the stationary prior distribution. More
speciﬁcally, we consider two basic properties of the
chain: convergence andindependence . For these
analyses, we consider samples from the pretrained
bert-base-uncased model with 12 layers, 12
heads, and 110M parameters5.
3.1 Convergence
We begin by checking the convergence time for
chains generated by GSN sampling. Theoretical
bounds derived for serial reproduction chains give a
convergence time of nlogn, wherenis the number
of sites (see Rafferty et al., 2014). To check these
convergence bounds in practice, we set n= 21
and select 20 sentences from Wikipedia to serve as
initial states, and run 10 chains initialized at each
sentence. We ensured that half of these sentences
have high initial probability (under BERT’s energy
score) and half have low initial probability. We
ﬁnd that these distributions indeed begin to quickly
mix in probability (see Figure S1). Because longer
sentences may require a longer burn-in time, we
conservatively set our burn-in window to m=
1000 epochs for our subsequent experiments.
5https://huggingface.co/bert-base-uncased3.2 Independence
Second, we want to roughly ensure independence
of samples, so that the statistics of our distribution
of samples isn’t simply reﬂecting auto-correlation
in the chain. For a worst-case analysis of a local
minimum, suppose P(wijwi)< (0<  < 1)
for alli2[1;:::;k ], wherekis the sentence length
in tokens. Then the probability of re-sampling the
same sentence is roughly < knafternepochs.
We can solve for the number of epochs nwe need
to bound the probability of re-sampling the exact
same sentence under for a given worst-case . For
example, if = 0:99and we want to ensure that
the probability of re-sampling the same sentence is
below a threshold = 0:01, thenn= 47 epochs
will likely sufﬁce. Ensuring complete turnover in
the worst case scenario requires much longer lags,
i.e.[1(1)k]n<.
To evaluate the extent to which these cases arise
in practice, we examine auto-correlation rates on
longer chains (50,000 epochs). We calculate cor-
relations between the energy scores at each epoch
as a proxy for the state: when the chain gets stuck
re-sampling the same sentence, the same scores
appear repeatedly. We ﬁnd that auto-correlation is
generally high, but our mixture kernel prevents the
worst local minima for both the MH chain (Goyal
et al., 2021) and our GSN chain (see Fig. S2),
although we still found higher auto-correlation
rates for the MH chain. To further examine these
minima, we examined edit rates: the number of
changes made to the sentence within an epoch.
Without the mixture kernel, we observe long re-
gions of consistently low edit rates (e.g. in some
cases, 5000 epochs in a row of exactly the same
sentence) which disappear under the mixture kernel
(see Fig. S3).
Based on these observations, we set the lag to
l= 500 epochs to maintain relatively high inde-
pendence between samples.
4 Distributional comparisons
In this section, we examine the extent to which
higher-order statistics of sentences from BERT’s
prior are well-calibrated to the data it was trained
on. This kind of comparison provides a richer sense
of what the model has learned or failed to learn
than traditional scalar metrics like perplexity (Taka-
hashi and Tanaka-Ishii, 2017; Meister and Cotterell,
2021; Takahashi and Tanaka-Ishii, 2019; Pillutla
et al., 2021).
10−610−410−2
100101102103
(log) rank of word(log) frequency of wordcorpus GSN mh
100101102103
100101102103
rank of word (BERT distribution)rank of word (corpus distribution)GSN MHFigure 3: The lexical frequencies of our GSN samples (A) closely match the Zipﬁan distribution of the corpus and
(B) closely correlate with the corresponding frequencies of the corpus distribution.
4.1 Corpus preparation
The version of BERT we analyzed in the previ-
ous section was trained on a combination of two
corpora: Wikipedia andBookCorpus . In order to
make valid comparisons between human priors and
machine priors, we needed to closely match BERT-
generated sentences with a comparable subset of
human-generated sentences from these combined
corpora. There are two technical challenges we
must overcome to ensure comparable samples, con-
cerning the sentencizer andtokenizer steps.
First, because our unit of comparison is the sen-
tence , we needed to control for any artifacts that
may be induced by how we determine what sen-
tences are (e.g. if our Wikipedia sentences were
systematically split on abbreviations, skewing the
distribution toward fragments). We therefore ap-
plied the same punkt sentencizer to create our
distribution of Wikipedia sentences and to check
our BERT samples for cases where the generated
sequence contained multiple sentences or ended
with a colon or semicolon.
Second, we needed a tokenizer that equates sen-
tence length. Because bi-directional models like
BERT operate over sequences of ﬁxed length, all
samples drawn from a single chain have the same
number of tokens.
Critically, however, BERT chains are deﬁned
over sequences of WordPiece tokens, so once these
sequences are decoded back into natural language
text, they may yield sentences of varying length,depending on how the sub-word elements are com-
bined together6(see Fig. S5). We solve this align-
ment problem by using the WordPiece tokenizer to
extract sentences of ﬁxed sub-word token length
from our text corpora, yielding equivalence classes
of corpus sentences that are all tokenized to the
same number of WordPiece tokens. We ran GSN
and MH chains over sentences of n= 11 tokens,
representing the modal lengths of sentences in
BookCorpus (see Fig. S4). We obtained 5,000 in-
dependent sentences from each sampling method
after applying our conservative burn-in and lag,
and combined the Wikipedia and BookCorpus sen-
tences together into a single corpus that is represen-
tative of BERT’s training regime.
4.2 Lexical distributions
We begin by comparing the lexical frequency statis-
tics of our samples from BERT against the ground-
truth corpus statistics. First, we note that the rela-
tionship between rank and frequency of tokens in
the GSN sampling matches the Zipﬁan distribution
of its training corpus better than those produced by
MH sampling (see Fig. 3A). However, it is possible
to produce the same overall distribution without
6One additional complexity is that the mapping between
WordPiece tokens and word tokens is non-injective. There
exist multiple sequences of sub-word tokens that render to
the same word (e.g. the WordPiece vocabulary contains a
token for the full word ‘missing’ but it is also able to generate
‘missing’ by combining the sub-word tokens ‘miss’+‘#ing’).
However, these cases are rare.
NUMCCONJADJADVAUXADPPROPNDETPRONVERBNOUNPUNCT
0.0 0.1 0.2 0.3
frequency of part of speechsource corpus GSN MH
negprtadvclnpadvmodnsubjpassxcompauxpassnummodacompccompattrconjccposscompoundamoddobjadvmodpobjnsubj
0.0 0.1 0.2 0.3
frequency of dependencysource corpus GSN MHFigure 4: The relative frequencies of different parts of speech (left) and dependencies (right) in the ground-truth
training corpora closely matched for GSN samples. In all cases, the GSN frequencies fell closer to the ground-truth
than the MH frequencies.
matching the empirical frequencies of individual
words. We next examined the respective ranks of
each word across the two distributions. Overall,
the word ranks in the GSN samples had a strong
Spearman rank correlation of r= 0:75with the
word ranks in the ground-truth corpus; the MH
samples had a signiﬁcantly lower correlation of
r= 0:48(Pearsonz= 17;p < 0:001, Fig 3B).
Most disagreements lay in the tails where frequency
estimates are particularly poor (e.g. many words
only appeared once in our collection of samples).
Indeed, among words with greater than 10 occur-
rences, the correlation improved to r= 0:83for
GSN andr= 0:65for MH.
To understand this relationship further, we con-
ducted an error analysis of lexical items which
were systematically over- or under-produced by
BERT relative to its training corpus. We found
that certain punctuation tokens (e.g. parentheses)
were over-represented in both the GSN samples
and the MH samples, while contractions like ’s
and’dwere under-represented. The MH samples
speciﬁcally over-produced proper names such as
Nina andJones . Finally, due to the use of sub-word
representations, we found a long tail of morpho-
logically complex words that did not appear at all
in the training corpus (e.g. names like Kyftenberg
or Streckenstein and seemingly invented scientiﬁc
terms like lymphoplasmic, neopomphorus, or pyra-
nolamines).4.3 Syntactic distributions
While the lexical distributions were overall well-
matched for GSN samples, our error analysis sug-
gested potential structure in the deviations. In
other words, entire grammatical constructions may
be over- or under-represented, not just particular
words. To investigate these patterns, we used the
spacy library to extract the parts of speech and
dependency relations that are present within each
sentence. We are then able to examine, in aggre-
gate, whether certain classes of constructions are
disproportionately responsible for deviations. Our
ﬁndings are shown in Fig. 4. Overall, the distri-
butions are close, but several areas of misalign-
ment emerge. For parts of speech, we observe
that the GSN sampler is slightly over-producing
nouns (and proper nouns) while under-producing
verbs and prepositions. We also observe that it
is over-producing noun-related dependencies (e.g.
compound nouns and appositional modiﬁers, which
are noun phrases modifying other noun phrases, as
in “Bill, my brother, visited town”). This pattern
suggests that BERT’s prior may be skewed toward
(simpler) noun phrases while neglecting more com-
plex constructions.
4.4 Sentence complexity
One hypothesis raised by comparing distributions
of syntactic features is that BERT may be regu-
larizing the complex structure of its input toward
0.000.250.500.751.00
0 10 20 30
dependency lengthCDFcombined
corporaGSN MHFigure 5: Cumulative probability distribution of depen-
dency lengths across sentences from BERT chains and
from the training corpus.
simpler constructions. To test this hypothesis, we
operationalize syntactic complexity using a mea-
sure known as the average dependency length of
a sentence (Futrell et al., 2015; Grodner and Gib-
son, 2005). This measure captures the (linear) dis-
tance between syntactically related words, which
increases with more complex embedded phrase
structures. We found that the distribution of de-
pendency distances in the sentences produced by
GSN sampling is overall more similar to those in
its training corpus than the MH (Fig. 5), although
closer analysis suggests it is still skewed slightly
simpler (see Fig. S6).
5 Human judgments
Finally, while our corpus comparisons highlighted
particular ways in which samples from BERT’s
prior were well-calibrated to the high-level statis-
tics of its training distribution, it is unclear whether
these agreements or deviations ‘matter’ in terms of
naturalness. In this section, we elicit human natu-
ralness judgments in order to provide a more holis-
tic measure of potential ‘weirdness’ with BERT
sentences.
5.1 Experimental methods
We recruited 1016 ﬂuent English speakers on the
Proliﬁc platform and asked them to judge the natu-
ralness of 4040 unique sentences from three length
classes: short (11 tokens), medium (21 tokens),
and long (37 tokens). 1675 of these sentences were
from the stationary state of the different chains,
2339 were from the burn-in phase (i.e. <1000
epochs), and the remainder were baseline sentences
(149 from Wikipedia, 48 from a 5-gram model, and
42 from an LSTM model; see Appendix for details).Each participant was shown a sequence of 25 sen-
tences in randomized order, balanced across differ-
ent properties of the stimulus set7. On each trial,
one of these sentences appeared with a slider rang-
ing from 0 (“very weird”) to 100 (“completely nat-
ural”)8. After excluding 8 participants who failed
the attention check (i.e. failed to rate a scram-
bled sentence below the midpoint of the scale and
a human-generated sentence above the midpoint),
we were left with an average of 7.3 responses per
sentence.
5.2 Behavioral results
We begin by comparing the naturalness of sen-
tences from the stationary GSN distribution to
other baselines (see Fig. 6), using a linear regres-
sion model predicting trial-by-trial judgments as
a function of categorical variables encoding sen-
tence length (short, medium, long) and the source
of the sentence (Wikipedia, GSN, MH, LSTM, or
n-gram). First, we ﬁnd that the naturalness of sen-
tences from GSN declines by 14 points at longer
sentence lengths, p < 0:001, while the natural-
ness of Wikipedia sentences is unaffected by length
(interaction term, p <0:001), consistent with re-
sults reported by Ippolito et al. (2020). Further-
more, among short sentences, where we included
additional baselines, we ﬁnd that GSN sentences
tend to be rated as slightly less natural than sen-
tences from Wikipedia (+10 points, p<0:001) but
more natural than those produced by an n-gram
model (-52 points, p < 0:001), LSTM model (-
25 points,p<0:001); or MH sampling from the
same BERT conditionals (-15 points, p <0:001;
see Table S1). MH samples also deteriorate sig-
niﬁcantly in naturalness for longer sentences com-
pared to GSN samples ( p < 0:001). Finally, we
examine naturalness ratings across the the burn-in
period, ﬁnding that ratings decline steadily across
the board as the chain takes additional steps (linear
term:t(7297) =12:4;p < 0:001), suggesting
gradual deviation away from the initial distribu-
tion of Wikipedia sentences toward the stationary
distribution (shown as the green and grey regions,
respectively, in Fig. S7).
7In a later batch, we increased the number of sentences
per participant to 40. The task was approximately 10 minutes
and participants were paid $2.50, for an average compensation
rate of $15/hr.
8See Clark et al. (2021) for a discussion of the merits of
phrasing the question in terms of naturalness instead of asking
participants to judge whether it was produced by a human or
machine.
short medium longwiki
GSN
MH
lstm
ngram
wiki
GSN
MH
wiki
GSN
MH0255075100mean naturalnessFigure 6: Empirical naturalness ratings elicited from
the stationary GSN distribution, compared to different
baselines at different sentence lengths. Error bars are
bootstrapped 95% CIs.
5.3 Predicting naturalness
Given that sentences from the stationary GSN dis-
tribution are judged to be less natural than human-
generated sentences overall, we are interested in
explaining why. Which properties of these sen-
tences make them sound strange? We approach this
problem by training a regression model to predict
human judgments from attributes of each sentence.
We include all part of speech tag counts and depen-
dency counts, as well as the sentence probability
scored under BERT, and the sentence length. We
use a cross-validated backwards feature selection
procedure to select the most predictive set of these
features for a linear regression (Kuhn and Johnson,
2013)9.
The best-ﬁtting model used 26 features and
achieved an (adjusted) R2= 0:21. The only fea-
tures associated with signiﬁcantly lower ratings
were the use of adpositions (e.g. before ,after )
and coordinating conjunctions. Importantly, we
found that including a categorical variable of cor-
pus (i.e. Wikipedia vs. GSN) signiﬁcantly im-
proved model ﬁt even after controlling for all other
features,2(1) = 7135 ;p < 0:001, suggesting
that sources of “weirdness” are not being captured
by typical statistics. We show some of these low-
naturalness sentences in Table 1 and S2.
6 Discussion
6.1 Probing through generation
A core idea of our serial reproduction approach
is to use generation as a window into a model’s
prior over language. While a variety of metrics
9Speciﬁcally, we used the lmStepAIC procedure imple-
mented in the caret R package, with k= 10 folds.and techniques have been proposed to quantify the
“quality” of generation, especially in the domains of
open-ended text generation and dialogue systems
(Caccia et al., 2020; Li et al., 2020; Guidotti et al.,
2018; Celikyilmaz et al., 2020), these metrics have
typically been applied to compare speciﬁc genera-
tion algorithms and operationalize speciﬁc pitfalls,
such as incoherence, excess repetition, or lack of
diversity. Consequently, it has been difﬁcult to dis-
entangle the extent to which deviations resulting
from generations are an artifact of speciﬁc decod-
ing algorithms (e.g. greedy search vs. beam search)
or run deeper, into the prior itself. For the purposes
of probing, we suggest that it is important to ask
not only how to generate the highest-scoring sen-
tences but how to generate sentences that may be
interpreted as representative of the model’s prior, as
formal results on GSNs have effectively provided.
6.2 GSN vs. energy-based objectives
We found that the prior distribution yielded by the
GSN sampler more closely approximated the lexi-
cal and syntactic distributions of the ground-truth
corpus and also sounded more “natural” to humans
than the samples yielded by MH. These results are
in contrast to ﬁndings by Goyal et al. (2021), show-
ing that MH produced high-quality BLEU scores
on a Machine Translation (MT) task compared to
a degenerate (pseudo-)Gibbs sampler. There are
several possible reasons for this discrepancy. One
possibility may be task-speciﬁc: while we focused
on unconditional generation, Goyal et al. (2021)
focused on a neural machine translation (MT) task,
where sentence generation was always conditioned
on a high-quality source text and thus remained
within a constrained region of sentence space. An-
other possibility is that we ran substantially longer
chains (50,000 epochs compared to only 33 epochs)
and the pitfalls of MH sampling only emerged later
in the chain.
More broadly, our corpus comparisons and hu-
man evaluations suggest serious limitations of sim-
ple “quality” metrics like energy values. We found
that the best-scoring states were often degenerate
local minima with mutually supporting n-grams
(such as repetitive phases and names like “Papua
New Guinea”). Indeed, there was only a loose re-
lationship between energy scores and participants’
judgments in our study, with many poorer-scoring
sentences judged to be more natural than better-
scoring sentences (e.g. overall, the distribution
of Wikipedia sentences tended to be much lower-
scoring under the energy function despite being
rated as more natural). We empirically validated
that the stationary distribution of the GSN chain
successfully approximates even higher-order statis-
tics of the ground-truth corpus, suggesting that the
raw conditionals of the dependency network may
implicitly acquire the joint distribution, without
requiring guarantees of consistency.
6.3 Other architectures
Serial reproduction methods are particularly useful
for probing models that do not directly generate
samples from their prior. For auto-regressive mod-
els like GPT-2, these samples are obtained more
directly by running the model forward (and, in-
deed, ancestral sampling produces text that better
balances the precision-recall tradeoff than other al-
gorithms; Pillutla et al., 2021). While we focused
on BERT, this method may be particularly use-
ful for encoder-decoder architectures like BART
(Lewis et al., 2020) which more closely resemble
the human Telephone Game task, requiring full re-
construction of the entire sentence from noisy input
rather than reconstruction of a single missing word.
Indeed, these architectures may overcome an im-
portant limitation of serial reproduction with BERT:
because these chains operate over a ﬁxed sequence
length, the resulting prior is not over all of language
but only over sentences with the given number of
WordPiece tokens. Finally, while we focused on
unconditional generation, the GSN sampler also
generalizes straightforwardly to conditional gen-
eration, where a subset of sites are ﬁxed and the
masked site is chosen from the remaining set.
6.4 Conclusions
Serial reproduction paradigms have been central
for exposing human priors in the cognitive sci-
ences. In this paper, we drew upon the theory
of iterated learning and of Generative Stochastic
Networks (GSNs) to expose the priors of large neu-
ral language models, which are often similarly in-
scrutable. We hope future work will consider other
points of contact between these areas and draw
more extensively from the theory developed to un-
derstand dependency networks. More broadly, as
language models become increasingly adaptive and
deployed in increasingly unconstrained settings,
bottom-up probing has the potential to reveal a
broader spectrum of “weirdness” than top-down
evaluative benchmarks.Acknowledgements
This work was supported by NSF grant #1911835
to RDH. We are grateful to Jay McClelland, Adele
Goldberg, and Stephan Meylan for helpful con-
versations, and to three anonymous reviewers for
feedback that improved our work.
References
Frederic Charles Bartlett. 1932. Remembering: A study
in experimental and social psychology . Cambridge
University Press.
Emily M Bender, Timnit Gebru, Angelina McMillan-
Major, and Shmargaret Shmitchell. 2021. On the
dangers of stochastic parrots: Can language models
be too big? In Proceedings of the 2021 ACM Confer-
ence on Fairness, Accountability, and Transparency ,
pages 610–623.
Yoshua Bengio, Eric Laufer, Guillaume Alain, and
Jason Yosinski. 2014. Deep generative stochastic
networks trainable by backprop. In International
Conference on Machine Learning , pages 226–234.
PMLR.
Yoshua Bengio, Li Yao, Guillaume Alain, and Pas-
cal Vincent. 2013. Generalized denoising auto-
encoders as generative models. Advances in neural
information processing systems , 26.
Aaron Beppu and Thomas Grifﬁths. 2009. Iterated
learning and the cultural ratchet. In Proceedings of
the Annual Meeting of the Cognitive Science Society ,
volume 31.
Mark Braverman, Xinyi Chen, Sham Kakade, Karthik
Narasimhan, Cyril Zhang, and Yi Zhang. 2020. Cali-
bration, entropy rates, and memory in language mod-
els. In International Conference on Machine Learn-
ing, pages 1089–1099. PMLR.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, et al. 2020. Language models are few-shot
learners. Advances in Neural Information Process-
ing Systems, 34 .
Massimo Caccia, Lucas Caccia, William Fedus, Hugo
Larochelle, Joelle Pineau, and Laurent Charlin.
2020. Language GANs falling short. International
Conference on Learning Representations .
Aylin Caliskan, Joanna J Bryson, and Arvind
Narayanan. 2017. Semantics derived automatically
from language corpora contain human-like biases.
Science , 356(6334):183–186.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao.
2020. Evaluation of text generation: A survey.
arXiv preprint arXiv:2006.14799 .
Elizabeth Clark, Tal August, Soﬁa Serrano, Nikita
Haduong, Suchin Gururangan, and Noah A. Smith.
2021. All that’s ‘human’ is not gold: Evaluating hu-
man evaluation of generated text. In Proceedings of
the 59th Annual Meeting of the Association for Com-
putational Linguistics. , pages 7282–7296.
Ishita Dasgupta, Demi Guo, Samuel J Gershman, and
Noah D Goodman. 2020. Analyzing machine-
learned representations: A natural language case
study. Cognitive Science , 44(12):e12925.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. BERT: Pre-training of
deep bidirectional transformers for language under-
standing. In Proceedings of NAACL-HLT .
Allyson Ettinger. 2020. What bert is not: Lessons from
a new suite of psycholinguistic diagnostics for lan-
guage models. Transactions of the Association for
Computational Linguistics , 8:34–48.
Richard Futrell, Kyle Mahowald, and Edward Gibson.
2015. Large-scale evidence of dependency length
minimization in 37 languages. Proceedings of
the National Academy of Sciences , 112(33):10336–
10341.
Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and
James Zou. 2018. Word embeddings quantify
100 years of gender and ethnic stereotypes. Pro-
ceedings of the National Academy of Sciences ,
115(16):E3635–E3644.
Andrew Gelman, Donald B Rubin, et al. 1992. In-
ference from iterative simulation using multiple se-
quences. Statistical science , 7(4):457–472.
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick.
2021. Exposing the implicit energy networks behind
masked language models via Metropolis–Hastings.
arXiv preprint arXiv:2106.02736 .
Thomas L Grifﬁths and Michael L Kalish. 2007. Lan-
guage evolution by iterated learning with Bayesian
agents. Cognitive Science , 31(3):441–480.
Daniel Grodner and Edward Gibson. 2005. Conse-
quences of the serial nature of linguistic input for
sentenial complexity. Cognitive Science , 29(2):261–
290.
Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri,
Franco Turini, Fosca Giannotti, and Dino Pedreschi.
2018. A survey of methods for explaining black box
models. ACM computing surveys (CSUR) , 51(5):1–
42.
Kristina Gulordava, Piotr Bojanowski, Edouard Grave,
Tal Linzen, and Marco Baroni. 2018. Colorless
green recurrent networks dream hierarchically. In
Proceedings of NAACL-HLT , page 1195–1205.
Peter Harrison, Raja Marjieh, Federico Adolﬁ, Pol van
Rijn, Manuel Anglada-Tort, Ofer Tchernichovski,
Pauline Larrouy-Maestri, and Nori Jacoby. 2020.Gibbs sampling with people. Advances in Neural
Information Processing Systems , 33.
Robert Hawkins, Takateru Yamakoshi, Thomas Grif-
ﬁths, and Adele Goldberg. 2020. Investigating rep-
resentations of verb bias in neural language models.
InProceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP) ,
pages 4653–4663.
Kenneth Heaﬁeld. 2011. Kenlm: Faster and smaller
language model queries. In Proceedings of the sixth
workshop on statistical machine translation , pages
187–197.
David Heckerman, David Maxwell Chickering,
Christopher Meek, Robert Rounthwaite, and Carl
Kadie. 2000. Dependency networks for inference,
collaborative ﬁltering, and data visualization. Jour-
nal of Machine Learning Research , 1(Oct):49–75.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and
Yejin Choi. 2019. The curious case of neural text de-
generation. International Conference on Learning
Representations .
Daphne Ippolito, Daniel Duckworth, Chris Callison-
Burch, and Douglas Eck. 2020. Automatic detec-
tion of generated text is easiest when humans are
fooled. In Proceedings of the 58th Annual Meet-
ing of the Association for Computational Linguistics ,
pages 1808–1822.
Michael L Kalish, Thomas L Grifﬁths, and Stephan
Lewandowsky. 2007. Iterated learning: Intergener-
ational knowledge transmission reveals inductive bi-
ases. Psychonomic Bulletin & Review , 14(2):288–
294.
Reinhard Kneser and Hermann Ney. 1995. Improved
backing-off for m-gram language modeling. In 1995
international conference on acoustics, speech, and
signal processing , volume 1, pages 181–184. IEEE.
Max Kuhn and Kjell Johnson. 2013. An introduction
to feature selection. In Applied predictive modeling ,
pages 487–519. Springer.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo
Yoshida, Masayuki Asahara, and Kentaro Inui. 2021.
Lower perplexity is not always human-like. In Pro-
ceedings of the 59th Annual Meeting of the Asso-
ciation for Computational Linguistics , pages 5203–
5217.
Thomas A Langlois, Nori Jacoby, Jordan W Suchow,
and Thomas L Grifﬁths. 2021. Serial reproduction
reveals the geometry of visuospatial representations.
Proceedings of the National Academy of Sciences ,
118(13).
Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer
Levy, Veselin Stoyanov, and Luke Zettlemoyer.
2020. BART: Denoising sequence-to-sequence pre-
training for natural language generation, translation,
and comprehension. In Proceedings of the 58th An-
nual Meeting of the Association for Computational
Linguistics , pages 7871–7880.
Margaret Li, Stephen Roller, Ilia Kulikov, Sean
Welleck, Y-Lan Boureau, Kyunghyun Cho, and Ja-
son Weston. 2020. Don’t say that! making inconsis-
tent dialogue unlikely with unlikelihood training. In
Proceedings of the 58th Annual Meeting of the Asso-
ciation for Computational Linguistics , pages 4715–
4728.
Tal Linzen and Marco Baroni. 2021. Syntactic struc-
ture from deep learning. Annual Review of Linguis-
tics, 7(1).
David McAllester. 2019. A consistency theorem for
BERT. Retrieved November 1, 2021 from https:
//machinethoughts.wordpress.com/2019/
07/14/a-consistency-theorem-for-bert/ .
Clara Meister and Ryan Cotterell. 2021. Language
model evaluation beyond perplexity. In Proceedings
of the 59th Annual Meeting of the ACL , pages 5328–
5339.
Stephan C Meylan, Sathvik Nair, and Thomas L Grif-
ﬁths. 2021. Evaluating models of robust word
recognition with serial reproduction. Cognition ,
210:104553.
Jennifer Neville and David Jensen. 2007. Relational
dependency networks. Journal of Machine Learning
Research , 8(3):653–692.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers,
John Thickstun, Sean Welleck, Yejin Choi, and Zaid
Harchaoui. 2021. Mauve: Measuring the gap be-
tween neural text and human text using divergence
frontiers. Advances in Neural Information Process-
ing Systems , 34.
Anna N Rafferty, Thomas L Grifﬁths, and Dan Klein.
2014. Analyzing the rate at which languages lose
the inﬂuence of a common ancestor. Cognitive Sci-
ence, 38(7):1406–1431.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Ka-
trin Kirchhoff. 2020. Masked language model scor-
ing. In Proceedings of the 58th Annual Meeting
of the Association for Computational Linguistics ,
pages 2699–2712.
Adam N Sanborn, Thomas L Grifﬁths, and Richard M
Shiffrin. 2010. Uncovering mental representations
with Markov chain Monte Carlo. Cognitive psychol-
ogy, 60(2):63–106.
Janelle Shane. 2019. You look like a thing and I love
you. Hachette UK.
Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2017.
Do neural nets learn statistical laws behind natural
language? PloS one , 12(12):e0189326.Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2019.
Evaluating computational language models with
scaling properties of natural language. Computa-
tional Linguistics , 45(3):481–513.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019.
BERT rediscovers the classical NLP pipeline. In
Proceedings of ACL , page 4593–4601.
Kristina Toutanova, Dan Klein, Christopher D Man-
ning, and Yoram Singer. 2003. Feature-rich part-of-
speech tagging with a cyclic dependency network.
InProceedings of the 2003 Human Language Tech-
nology Conference of the North American Chapter
of the Association for Computational Linguistics ,
pages 252–259.
Alex Wang and Kyunghyun Cho. 2019. BERT has a
mouth, and it must speak: BERT as a Markov Ran-
dom Field language model. In Proceedings of the
Workshop on Methods for Optimizing and Evaluat-
ing Neural Language Generation (NeuralGen) , page
30–36.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mo-
hananey, Wei Peng, Sheng-Fu Wang, and Samuel R
Bowman. 2020. Blimp: The benchmark of linguis-
tic minimal pairs for english. Transactions of the As-
sociation for Computational Linguistics , 8:377–392.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow-
man. 2019. Neural network acceptability judgments.
Transactions of the Association for Computational
Linguistics , 7:625–641.
Jing Xu and Thomas L Grifﬁths. 2010. A rational anal-
ysis of the effects of memory biases on serial repro-
duction. Cognitive psychology , 60(2):107–126.
Appendix A: Baseline details
Wikipedia sentences were randomly selected from
the full sentencized corpus English Wikipedia that
tokenized to 12, 21, and 37 WordPiece tokens for
the short, medium, and long conditions, respec-
tively. These sentences were also chosen to span a
broad range of sentence probabilities under BERT
(i.e.logP(p1;:::;p n) =P
klogP(pkjpk)).
For our ngram baseline, we trained a 5-gram
model with Kneser-Ney smoothing (Kneser and
Ney, 1995) on English Wikipedia using the kenlm
library (Heaﬁeld, 2011), and generated sentences
of length 10 by sampling from the resulting condi-
tional distributions. Because this model stripped
punctuation, and was therefore unable to emit an
“end of sentence” token, we expected it to serve as
a lower bound on the naturalness scale.
For our LSTM baseline, we used the network
pre-trained by Gulordava et al. (2018) on English
Wikipedia. This model was trained to emit an end
of sentence ( <eos> ) token, allowing us to rejec-
tion sample to obtain sentences that were exactly 10
words long with no unknown words (i.e. <unk> to-
kens). Because it was not trained with a <start>
token, however, we needed to initialize it with the
initial word of the sentence. We randomly selected
this initial word from a small set of common sen-
tence openers (e.g. the,a,it,his,her). As a
result of our initial token selection, this model does
not precisely sample from its true prior over sen-
tences. Thus, it is best viewed as another baseline
of sentences rather than as a careful architectural
comparison.
Because we were asking participants to judge
the naturalness of complete sentences , we did not
want to include samples which clearly violated sen-
tencehood, as these would not be informative (e.g.
fragments from Wikipedia that were incorrectly
sentencized and ended with an abbreviation, bibli-
ographic text like “korsakov (1976) r.s.,” or table
markdown with pipes like “ jajbj”). We automat-
ically removed any sentences containing pipes or
ending with colons or semicolons, as these were
associated with sentencizer inconsistency, as well
as sequences that contained multiple sentences (ac-
cording to our sentencizer). Finally, the authors
took a manual pass to exclude other non-sentential
fragments from the stimulus set.
Appendix B: Corpus details
We downloaded cleaned Wikipedia data provided
by GluonNLP (https://github.com/dmlc/gluon-
nlp/tree/master/scripts/datasets/pretrain_corpus),
and BookCorpus data from HuggingFace Datasets
(https://huggingface.co/datasets/bookcorpus).
−60−40−200
0100200300
epochsentence scoreFigure S1: We examine the convergence time by initial-
izing different chains at different classes of sentences
(red is high probability under BERT’s energy function,
blue is low probability). Faint lines show smoothed tra-
jectories for individual chains and error bars are boot-
strapped 95% conﬁdence intervals across chains.
no mixture
MH mixture
GSN mixture0.00.20.40.6
010002000300040005000
lag (epochs)auto−correlation
Figure S2: MCMC methods like GSN and MH sam-
pling tend to get stuck in local regions with high auto-
correlation. We ﬁnd that a minimal autocorrelation is
achievable with lower lag (500 epochs between sam-
ples) using a mixture kernel with a constant probabil-
ity of resetting the chain. Error ribbons are 95% conﬁ-
dence intervals.
term estimate std.error statistic p.value
1 (Intercept) 67.33 1.14 59.08 <0:001
2 short vs. long (GSN) -14.49 1.60 -9.08 <0:001
3 short vs. medium (GSN) -10.21 1.60 -6.39 <0:001
5 GSN vs. LSTM (short) -28.60 2.04 -14.05 <0:001
6 GSN vs. MH (short) -14.76 1.59 -9.26 <0:001
7 GSN vs. ngram (short) -54.26 2.00 -27.07 <0:001
8 GSN vs. wiki (short) 10.40 1.70 6.13 <0:001
13 interaction (short vs. long; GSN vs. MH) -12.31 2.23 -5.51 <0:001
14 interaction (short vs. medium; GSN vs. MH) -7.33 2.23 -3.29 <0:001
17 interaction (short vs. long; GSN vs. wiki) 11.22 2.39 4.70 <0:001
18 interaction (short vs. medium; GSN vs. wiki) 5.56 2.37 2.35 0.02
Table S1: Fixed effect estimates for regression on human scores. Length class and source are dummy coded with
short lengths and GSN as baselines.
base mixture
01000020000300004000050000010000200003000040000500000.000.250.500.751.00
stepedit rate
Figure S3: Without mixing in a constant probability of returning to the initial distribution, the GSN chain (and MH
chain, not shown) goes through periods of stasis with low edit rates (red curves), contributing to high autocorrela-
tions.
Figure S4: Empirical distribution of sentence lengths in Wikipedia and BookCorpus training corpora, after Word-
Piece tokenization. For our corpus comparisons, we selected the modal Wikipedia sentence length of 21 tokens and
the modal BookCorpus length of 11 tokens. For our human judgment experiment, we included baseline sentences
only from Wikipedia for shorter (12 tokens) and longer sentences (37 tokens), with roughly equal prevalence in
the corpus (orange dots).
types of unnaturalness examples
character-level He preened on a Ldrink of copper.
phrase-level semanticThe little wattled songbird, also called the Chink Warbler,
Orange Garver or Quickcumber is a socially luscious and
habituated bird species.
sentence-levelconstruction There were two hours before he made the walk .
out-of-context word No need to focus on bicycling .
self-contradictoryThe symbols ()read as ()and()are written as
( ), not as ().
repetitionThe college of arts and sciences, adjacent to the business
school, is majoring in business.
He saw Cronus and Cronus, Cronus and James Cronus he
saw Cronus and Cronus and Cronus and Cronus Cronus
when he saw Cronus.
Table S2: More examples of sentences from BERT’s prior with low naturalness ratings.
sub-word tokens
word tokens
BERT chains corpus textFigure S5: There is a misalignment between the space
of sentences obtainable by a BERT chain of a ﬁxed to-
ken length (in sub-word tokens) and natural language
sentences of a ﬁxed length (in words). We consider
the distribution of corpus sentences that are obtainable
from a ﬁxed-length BERT chain, which may decode to
different lengths in natural text (black arrows).
0.0000.0250.0500.0750.100
01020304050
dependency lengthdensitycombined
corporagibbs mh
Figure S6: Dependency distances are similar for sen-
tences sampled from BERT’s prior and sentences from
its training corpus, but the BERT distribution is more
bimodal and tends to skew simpler.
long medium shorthigh low
0 2 4 60 2 4 60 2 4 6406080
406080
log1p(step)empirical_stat_GSN_earlyFigure S7: Sentences gradually drift away from the initial distribution across the burn-in period. Light green
region represents the 95% conﬁdence interval for the mean naturalness of Wikipedia sentences while grey region
represents the same interval around the stationary distribution of the converged chain. Top row represents chains
that are initialized at high-probability states, while bottom row is initialized in low-probability states.