Distinguishing rule- and exemplar-based generalization in learning systems.

Distinguishing rule- and exemplar-based generalization
in learning systems
Ishita Dasgupta* 1 @Erin Grant* 2Thomas L. Grifﬁths1
Abstract
Machine learning systems often do not share the
same inductive biases as humans and, as a re-
sult, extrapolate or generalize in ways that are
inconsistent with our expectations. The trade-
off between exemplar- and rule-based general-
ization has been studied extensively in cognitive
psychology; in this work, we present a proto-
col inspired by these experimental approaches to
probe the inductive biases that control this trade-
off in category-learning systems. We isolate two
such inductive biases: feature-level bias (differ-
ences in which features are more readily learned)
and exemplar or rule bias (differences in how
these learned features are used for generalization).
We ﬁnd that standard neural network models are
feature-biased and exemplar-based, and discuss
the implications of these ﬁndings for machine
learning research on systematic generalization,
fairness, and data augmentation.
1. Introduction
Extrapolation or generalization—decisions on unseen
datapoints—is always underdetermined by data; which par-
ticular extrapolation behavior a machine learning (ML) sys-
tem exhibits is determined by its inductive biases (Mitchell,
1980). When those inductive biases are opaque—as is often
the case with many modern ML systems (Geirhos et al.,
2020; D’Amour et al., 2020)—we can instead turn to empir-
ical investigation of the behavior of a system to reveal the
system’s implicit inductive biases. Cognitive psychology
provides a rich basis for experimental designs to study the
often-opaque human cognitive system via its external behav-
*Equal contribution1Departments of Psychology & Computer
Science, Princeton University2Department of Electrical Engineer-
ing & Computer Sciences, UC Berkeley.@Now at DeepMind.
Correspondence to: Ishita Dasgupta <idg@deepmind.com >,
Erin Grant <eringrant@berkeley.edu >.
Proceedings of the 39thInternational Conference on Machine
Learning , Baltimore, Maryland, USA, PMLR 162, 2022. Copy-
right 2022 by the author(s).training examples extrapolation
“dax” “fep”?
Figure 1. Example of a data condition: Data often underdeter-
mines a decision boundary; here, it is unclear whether shape or
color determines object label (“dax” vs “fep”). How a learner
extrapolates to new stimuli reveals inductive bias.
ior; these designs can be leveraged to distinguish between
competing hypotheses about a machine learning system’s
inductive biases as well ( e.g., Ritter et al., 2017b; Lake
et al., 2018; Dasgupta et al., 2019).
We draw on cognitive psychology to construct a protocol
that isolates the inductive biases determining how an ML
system generalizes feature-based categories such as those in
Fig. ( 1). A key property of such categorization problems is
the presence of a distractor dimension that does not play a
causal role in the underlying category boundary; the ground
truth categorization is determined by a discriminant dimen-
sion. Such problems are ubiquitous in machine learning
applications ( e.g., Beery et al., 2018), where learned asso-
ciations between the distractor and the categorization label
are termed “spurious” (Arjovsky et al., 2019). The tendency
to acquire (potentially harmful) spurious associations is an
example of a downstream consequence of implicit inductive
bias, and so characterizing such implicit inductive biases is
of both theoretical and practical interest.
We use abstract problem settings such as that in Fig. ( 1) to
identify and isolate two distinct inductive biases underlying
feature-based category learning. The ﬁrst, feature-level bias ,
expresses a preference for some features over others to sup-
port a decision boundary ( e.g., preferring shape over color).
The second, exemplar bias —vs. rule bias —expresses a pref-
erence for feature-dense (vs. feature-sparse) decision bound-
aries ( e.g., a boundary informed by both shape and color, vs.
only one of the two features). Our protocol presents data
conditions that manipulate feature co-occurrences observed
during training such that the resulting extrapolation behavior
is diagnostic of these inductive biases in the learner.
Code at https://github.com/eringrant/
icml-2022-rules-vs-exemplars .
Distinguishing rule- and exemplar-based generalization in learning systems
The experimental setup underlying our training and testing
conditions is similar to existing works in “combinatorial gen-
eralization” (Andreas et al., 2016; Johnson et al., 2017) and
“subgroup fairness” (Sagawa et al., 2020a;b). Our work also
makes several independent contributions: We identify and
isolate two distinct inductive biases that affect extrapolation
of feature-based categories, and we examine these across
models in an expository points-in-a-plane setting, as well
as in more naturalistic text and image domains. We demon-
strate that existing measures of feature co-occurrence and
extrapolation behavior (“spurious correlation” and “worst-
group accuracy,” Sagawa et al., 2020b) are insufﬁcient to
characterize these inductive biases. Finally, we consider
the normative question: What extrapolation behavior is de-
sirable for a given application ? We provide a preliminary
answer by discussing the relevance of the inductive biases
we identify to related work in systematic generalization,
fairness, and data augmentation.
2. Inductive biases in category learning
We start by introducing the two inductive biases of interest.
Feature-level bias characterizes which feature a system
ﬁnds easier orharder to learn and thus which feature a
system will utilize when both are associated with the cat-
egory label. This kind of feature-level bias has been stud-
ied extensively in human cognition (Landau et al., 1988;
Hudson Kam & Newport, 2005), and speciﬁc feature-level
biases—mostly notably the “shape-bias,” the tendency to
generalize image category labels according to shape rather
than according to color or texture—have been revisited in
the context of recent neural network models (Ritter et al.,
2017a; Hermann et al., 2019; Geirhos et al., 2018). We
examine feature-level bias for arbitrary features, as well as
demonstrate how this bias interacts with—but is distinct
from—another kind of inductive bias, to be discussed next.
Exemplar (orrule)bias characterizes how a system uses
features to inform decisions by trading off exemplar- and
rule-based generalization . A rule-based decision is made
on the basis of minimal features that support the cate-
gory boundary ( e.g., Ashby & Townsend, 1986), while
an exemplar-based decision-maker generalizes on the basis
of similarity to category exemplars ( e.g., Shepard & Chang,
1963), invoking many or all features that underlie a category.
Extensive empirical work in cognitive psychology has found
evidence of both kinds of generalization in humans (Nosof-
sky et al., 1989; Rips, 1989; Allen & Brooks, 1991; Smith
& Sloman, 1994). This trade-off can be understood as a
continuum that varies the number of features employed to
discriminate between categories (Pothos, 2005).
Feature-level bias and exemplar bias are practically rel-
evant because they describe how a learning system uses
features to extrapolate, and different problem settings callfor different ways of doing so. An exemplar-based system
that depends on all features, and is not invariant to any of
them, suffers when not all feature combinations are observed
and systematic generalization to unobserved combinations
is expected (Lake et al., 2018; Marcus, 2018; Arjovsky et al.,
2019). On the other hand, a rule-based system that applies
the same category decision rules across all data regions
might over-generalize, which is undesirable in naturally oc-
curring long-tailed distributions (Feldman & Zhang, 2020;
Feldman, 2020; Brown et al., 2020). Diagnosing exemplar
vs. rule bias is therefore of both theoretical and practical
interest. In Section ( 6), we give a concrete example in a
fairness setting—where certain regions of the data support is
underrepresented but we want comparable accuracy on these
regions nonetheless—in which understanding the inductive
biases of the learning system allows for a data intervention
that improves performance.
We now build intuitions for how the category learning
paradigm in Fig. (2) isolates feature-level bias and ex-
emplar bias . The stimuli Fig. ( 2) vary along two feature
dimensions, shape and color. Color determines the label of
an object ( i.e.,green objects are “dax”; purple are “fep”, us-
ing arbitrary names to emphasize that the category is novel
to humans as well as to ML systems). Shape is unrelated
to the underlying category structure and acts as a distractor.
Participants (either humans or artiﬁcial learning systems)
are independently placed in three different conditions— cue
conﬂict ,zero shot , orpartial exposure —that vary in cov-
erage of the feature space. After observing the training
examples , the participant is presented with an extrapola-
tion test consisting of an example outside the support of
feature combinations observed during training ( i.e.,they
must classify the green circle as a “dax” or a “fep”). We
explain below how differences in classiﬁcation behavior on
this extrapolation test isolate feature-level bias as well as
exemplar-vs-rule bias, but ﬁrst: We encourage the reader to
try the experiment themselves to examine their intuitions.
Cue conﬂict (CC, top row, Fig. ( 2)). The data presented
in this condition confound color and shape ( i.e.,color and
shape are equally predictive of the category boundary). How
a system generalizes here directly measures its feature-level
bias towards color or shape.
Characteristic behavior (right half of Fig. ( 2)). Since hu-
mans have an established shape bias (Landau et al., 1988),
we expect that humans will classify the test item according
to the object that shares its shape, not its color; in this case,
as a “fep.” However, this inductive bias is independent of
whether a reasoner is rule- or exemplar-based; neither has
ana priori propensity for features, both are equally likely
to classify the test item as a “dax” or a “fep.”
Zero shot (ZS, middle row, Fig. ( 2)). This condition
requires extrapolation to a new feature value “zero-shot”
Distinguishing rule- and exemplar-based generalization in learning systems
observations ratio of predictions
condition training examples extrapolation humans rule-based exemplar-based
(shape-biased ) ( no feature bias ) ( no feature bias )
cue
conﬂict“dax” “fep” ?
“dax” “fep” “dax” “fep” “dax” “fep”
zero
shot“dax” “fep” ?
“dax” “fep” “dax” “fep” “dax” “fep”
partial
exposure“dax” “fep” “fep” ?
“dax” “fep” “dax” “fep” “dax” “fep”
Figure 2. Illustrative category learning experiment: Training examples from the 3 independent training conditions, the extrapolation
test, and characteristic behavior for learners with different inductive biases. We formalize the training conditions in Fig. ( 3).
(i.e.,without prior exposure). This setting is often used to
examine out-of-domain (OOD) and compositional general-
ization in machine learning (Xian et al., 2018). Behavior
in this condition reveals whether the model has learned the
discriminating features and whether it can extrapolate to
new feature values, and thus acts as a baseline.
Characteristic behavior (right half, Fig. ( 2)). Rule- and
exemplar-based behavior in this condition is confounded. A
rule-based learner infers the minimal rule that color deter-
mines label, does not assign any predictive value to shape,
and therefore classiﬁes the extrapolation stimulus based on
color as a “dax.” An exemplar-based learner categorizes
based on the similarity along all feature dimensions of the
extrapolation stimulus to category exemplars. Neither train-
ing exemplars have any overlap with the test stimulus along
the shape dimension, but the “dax” overlaps along the color
dimension, and the learner categorizes it as a “dax.”
Partial exposure (PE, bottom row, Fig. ( 2)). Compared to
zero shot, participants in this condition also receive “par-
tial exposure” to a new feature value ( i.e., circle ) along
the shape dimension. The extrapolation test in this con-
dition is most similar to combinatorial zero-shot general-
ization (e.g., Lake & Baroni, 2018a), where the learner
is exposed independently to all feature values but has to
generalize to a new combination.
Characteristic behavior (right half of Fig. ( 2)). This setting
meaningfully distinguishes rule- and exemplar-based gen-
eralization. To understand this distinction, we contrast this
condition to the cue-conﬂict condition. The addition of the
purple diamond-shaped “fep” means the learner has seen
both a diamond and a circle labeled “fep”. A rule-based sys-
tem takes this as direct evidence that shape is notpredictive
of category label and classiﬁes the extrapolation stimulus
on the basis of color as a “dax.” This is typically also how
humans extrapolate. This additional training example, how-ever, does not impact an exemplar-based system, since it
does not share any features with the extrapolation stimu-
lus. The exemplar-based reasoner classiﬁes on the basis
of feature-overlap with training exemplars and is therefore
indifferent, exactly as in the cue-conﬂict condition.
3. A protocol for measuring inductive bias
We embed the structure of the category learning problem dis-
cussed in Section ( 2) into a statistical learning problem that
can be applied across domains to test black-box learners.
Problem setting. We consider the oracle compositional
setting of Andreas (2019) in which inputs are a composi-
tion of categorical attributes with two latent binary features,
zdisc;zdist;2f0;1gthat jointly determine the observation
x2X  Rdvia some mapping g:f0;1g2!X ; see
Fig. ( 3). We consider the binary classiﬁcation task of ﬁtting
a model ^f:X!f 0;1gfrom a given model family Fto
predict a class for each observation. One of the underlying
features, the discriminant ,zdisc, deﬁnes the decision bound-
ary; the other one, the distractor ,zdist, is not independently
predictive of the label.
This speciﬁes a generative process x;zdisc;zdistp(xj
zdisc;zdist)p(zdisc;zdist).p(xjzdisc;zdist)is either gener-
ated ( e.g.,in Section ( 4)), or the empirical distribution of the
subset of datapoints xwith the corresponding underlying
feature values (assuming access to these annotations, e.g., in
Sections ( 5) and ( 6)).p(zdisc;zdist)is varied across training
conditions, as outlined below.
Training conditions. The upper-right quadrant in all subﬁg-
ures of Fig. ( 3), for which p(zdisc= 1;zdist= 1) = 1 , acts
as a hold-out set on which we can evaluate generalization
to an unseen combination of attribute values. We produce
multiple training conditions with the remaining three quad-
rants of data by manipulating p(zdisc;zdist). All the analyses
Distinguishing rule- and exemplar-based generalization in learning systems
training condition
feature space cue conﬂict zero shot partial exposure extrapolation
0= 0:5,1= 0:50= 1:0,1= 0:00= 0:0,1= 0:00= 0:5,1= 0:0 1= 1:0
discriminant (color)distractor (shape)
Figure 3. Formalizing the illustrative experiment: The experiment from Fig. ( 2) expressed in terms of the
formalism in Section ( 3) with color as zdistand shape as zdisc. Background colors indicate the true category.0 0:5 101
10
Figure 4. Spurious
correlation (Eq. (3)).
in this work compare model extrapolation to the held-out
test quadrant across various training conditions.
To equalize the class base rates we balance all training con-
ditions across the discriminant; i.e.,we enforcep(zdisc=
0) =p(zdisc= 1) = 0:5. We also ﬁx the number of dat-
apoints across all conditions at N; With these constraints,
we can control p(zdisc;zdist)via two degrees of freedom:
0=p(zdist= 1jzdisc= 0) (this implicitly ﬁxes
p(zdist= 0jzdisc= 0) = 10to balance the dataset);
and1=p(zdist= 1jzdisc= 1) .The three conditions in
Section ( 2), as well as the held-out test set, correspond to
particular settings of 0and1(shown in Fig. ( 3), more in
Appendix ( A.2)).
Measuring inductive bias. We measure feature-level bias
as deviation from chance performance in the CC condition.
Exemplar bias is measured as the difference between per-
formance in the partial-exposure condition and zero-shot
condition—-no difference indicates rule-based generaliza-
tion, the magnitude of the difference measures exemplar
bias. Formally, for a given model family F, let^fZSdenote
the result of selecting a model from Fby training in the
zero-shot condition, and similarly ^fPEand^fCC. We deﬁne
feature-level bias ( FLB ) and exemplar vs. rule propensity
(EvR ) as:
FLB(F) =E[(A(y;^fCC(x))]0:5; (1)
EvR(F) =E[A(y;^fZS(x))]E[A(y;^fPE(x))] (2)
where the expectation is taken with respect to the data dis-
tribution under the extrapolation region ( p(x;yj0=
1;1= 1) ), andAis the 0-1 accuracy. FLB takes values
between -0.5 and 0.5 (indicating bias toward zdistorzdisc,
respectively); 0 represents no feature bias. EvR takes val-
ues between 0 and 1 (indicating rule bias and exemplar bias,
respectively).
Related formalisms and spurious correlation. This bi-
nary formulation of discriminant and distractor features has
previously been studied in the context of spurious correla-tion (Sagawa et al., 2020b). Rather than independently vary-
ing occupancy in the four quadrants, Sagawa et al. (2020b)
directly manipulate the (spurious) linear correlation between
the distractor and the discriminant features ( pmaj). In com-
binatorial feature spaces, a scalar spurious correlation in-
sufﬁciently speciﬁes the data distribution. The linear cor-
relation coefﬁcient between zdiscandzdist—henceforth
spurious correlation —can be written in terms of 0and1
via=01
2and=0+1
2as
(0;1) =p
(1): (3)
Different combinations of 0and1give equal(see the
contours in Fig. ( 4), with markings for points along the equi-
correlation contour from partial exposure ( 0= 0:5;1=
0:0,= 0:58)); while nonetheless producing qualitatively
different extrapolation behavior, as we demonstrate in later
sections. This indicates that sensitivity to spurious corre-
lation insufﬁciently speciﬁes extrapolation behavior. We
argue for a formulation like ours—based on manipulating
feature combinations —that can tease apart distinct inductive
biases at the level of what features a system ﬁnds easier to
learn ( FLB ) as well as how to use these features to inform
a decision boundary ( EvR ).
4. 2-D classiﬁcation example
To illustrate our framework in a simple statistical learning
problem and quantitatively conﬁrm the intuitions outlined
in Section ( 2), we consider a two-dimensional classiﬁcation
problem. The feature dimensions are orthogonal bases in
2D space, and we deﬁne the data generating procedure as
p(xjzdisc;zdist) =N(;1:0) ; (4)
=[2zdisc1;2zdist1];
where, as speciﬁed in Section ( 3),zdisc;zdist;2 f0;1g,
p(zdisc;zdist)is determined by the training condition. zdisc
determines class labels, zdistis a distractor, is ﬁxed at 3,
andN= 300 datapoints are in each class. The group with
zdist=zdisc= 1is assigned the test set.
Distinguishing rule- and exemplar-based generalization in learning systems
cue conﬂict (CC) zero shot (ZS) partial exposure (PE)
GLM
505
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
GP
505
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
NN
5 0 5505
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.0
5 0 5
0.0 0.2 0.4 0.6 0.8 1.00.00.20.40.60.81.05 0 5
(a)Decision boundaries averaged across 20 runs. Training datapoints
are green or purple by label; test are white.GLM:linGLM:l1 GLM:l2 GLM:
GP:ﬁt GP:0.5 GP:8.0
NN:16h1dNN:4h4d NN:2h1d00:20:4
model familyFEvR(F)
(b)EvR reﬂects exemplar-vs-rule propensity both within and across model families. The
EvR across model families, computed across 20 runs, error bars represent 95% CIs. The GLMs
are largely rule-based and show low EvR . Even within GLMs, sparsity regularization gives
lower EvR . GPs are largely exemplar-based and show high EvR . Even within GPs, more
‘local’ GPs with lower lengthscales have higher EvR . NNs lie in-between, with larger NNs
giving higher EvR .
Figure 5. Simple 2-D classiﬁcation (Section ( 4)) The speciﬁc model used in (a) are bolded in (b).
4.1. Model families and nomenclature.
Neural network (NN): We train feedforward rectiﬁed lin-
ear unit (ReLU) classiﬁers with varying numbers of hidden
layers and hidden units. We use the scikit-learn implemen-
tation with default parameters, run 20 times for conﬁdence
intervals.
Generalized linear model (GLM): Parametric models al-
low us to formalize the feature-sparsity that characterizes
rule-based learners. Linear logistic regression is sparse by
deﬁnition (it has access to only linear features). We general-
ize this model by expanding the feature space to include a
nonlinear interaction and examine L1 and L2 regulariza-
tion in a GLM over this altered feature space.
Gaussian process (GP): Non-parametric kernel methods
allow us to formalize exemplar-based generalization, where
generalizations are made on the basis of feature-dense simi-
larity to training data. We examine the performance of GPs
with radial basis function (RBF) kernels. We ﬁt the kernel
lengthscale using gradient descent on the log marginal like-
lihood of the data (Rasmussen, 2003) (giving 5:2) as well as
vary it (adjusting “locality” in decision boundaries); GP:8.0
denotes a GP with lengthscale value of 8.0.
We can implement explicit rule- and exemplar-based mod-
els in the synthetic setting since we know the features over
which to build parametric or similarity-based models respec-
tively, so we use it to validate our measures. In most appli-
cation domains (including those in Sections ( 5) and ( 6)) fea-
ture learning is automated (Hinton & Salakhutdinov, 2006),
making it difﬁcult to specify the corresponding GLM or GP.4.2. Comparing cue conﬂict, zero shot, and partial
exposure
We consider one model from each class: NN with 1 hidden
layer of 2 units (NN:2h1d); linear GLM (GLM:lin); RBF
GP with ﬁtted lengthscale (GP:ﬁt). The decision boundaries
learned by these models are shown in Fig. ( 5a).zdist;zdisc
are equivalent by design, and permit no feature-level bias,
so cue conﬂict is exactly at chance. This lets us focus on
validating our novel protocol for measuring EvR without
confounds. We generalize to cases with feature-level bias
in later sections. The GLM, sparse and therefore rule-based
by deﬁnition, can only learn a linear boundary. It is there-
fore unaffected by the distractor dimension, showing no
difference in extrapolation behavior between zero shot and
partial exposure (zero EvR ). On the other hand, the GP is
exemplar-based by deﬁnition and displays a high EvR . The
NN shows an intermediate EvR , more rule-based than the
purely-exemplar-based GP but not entirely rule-based like
the GLM.
4.3. The inﬂuence of model properties on EvR
We ﬁrst examine EvR in our control model classes (GLMs
and GPs) to validate that it tracks rule- vs exemplar-based
extrapolation, followed by analyses of various NNs.
Regularized GLMs: EvR reduces with rule propensity.
A key property of rule propensity is sparsity in feature space.
A linear GLM (GLM:lin) is sparse by deﬁnition, we examine
a GLM on an expanded feature set so we can manipulate
this sparsity. The additional feature /zdistzdiscis
the product of the observed features and normalizing by .
We compute EvR for this GLM with different regularizers
Distinguishing rule- and exemplar-based generalization in learning systems
(regularization weight 1.0), shown in Fig. ( 5b).
GLM with no regularization (GLM: ) displays a signiﬁcant
EvR . L2 regularization reduces it but L1 (which directly
induces feature sparsity1) brings it to zero (or perfectly
rule-based). This demonstrates that a low EvR tracks rule
propensity via feature-level sparsity.
Lengthscales in GPs: EvR increases with exemplar
propensity. A sufﬁcient condition for exemplar propen-
sity is the locality of decision boundaries. We can directly
manipulate this in a GPs with its lengthscale. We evaluate
EvR in GPs with RBF kernels of different lengthscales in
Fig. ( 5b). We ﬁnd that the EvR is lowest with high length-
scales and grows as the lengthscale reduces, demonstrating
that a high EvR tracks exemplar propensity via locality of
decision boundaries.
NNs: The necessary but insufﬁcient role of expressivity.
The results from GLMs and GPs indicate that some ways
to reduce expressivity (L1 regularization in GLMs and high
lengthscale in RBF GPs) encourage rule propensity over
exemplar propensity (thereby a lower EvR ). We manipulate
the most common variable in NN expressivity—its size.
We increase the width of an NN with ﬁxed depth of 1
(Fig. ( 5b)) and ﬁnd that the EvR increases. A deep NN
with the same number of units, however, exhibits compa-
rable EvR to a wide network. Deeper networks with the
same number of units are more expressive than wide ones
(Raghu et al., 2017), indicating that excess expressivity,
while necessary, is not the sole driver of EvR .
4.4.EvR is distinct from sensitivity to spurious
correlation
A crucial difference between the zero shot and the par-
tial exposure conditions is that the partial exposure con-
dition creates a (spurious) correlation = 0:58between
zdistandzdisc. Is sensitivity to this spurious correlation ( )
the sole the driver of the difference in performances be-
tween the partial exposure and zero shot conditions, i.e.,of
theEvR ? We show that this is not the case; the EvR is
measuring something distinct. As described in Section ( 3),
there are multiple data-settings with the same . We con-
sider training conditions speciﬁed by other 0;1that have
the sameas the partial exposure condition (dots along
the solid contour in Fig. ( 4)). We ﬁnd that performance
on the extrapolation quadrant after training on these new
data distributions is much higher (and closer to zero shot
performance) than when trained on the partial exposure
condition—even though is exactly the same. This indi-
cates that performance on the partial exposure condition
(normalized by zero shot performance to give the EvR ) is
1Weight sparsity from L1-regularizer is equivalent to feature-
sparsity only in special cases, including GLM.uniquely indicative of something different from sensitiv-
ity to spurious correlations—it measures the inductive bias
toward exemplar-vs-rule based extrapolation.
We can reduce in different ways by increasing 1or by re-
ducing0. We ﬁnd that these are not equivalent and result in
different extrapolation behaviors ( e.g., increasing1gives
more rule-based generalization than reducing 0; see results
for the 2-D classiﬁcation setting in Appendix ( C.3.1 ) and
for the vision domain in Fig. ( 7c)). This has implications
for data manipulation methods ( e.g., subsampling or aug-
mentation) that manipulate this to control extrapolation.
This further supports that spurious correlation alone cannot
explain extrapolation behavior, highlighting the importance
ofFLB andEvR that measure behavior under different
feature combinations in training.
Conclusions. EvR tracks exemplar- and rule-based extrap-
olation, as validated on interpretable models such as GLMs
and GPs. In particular, EvR decreases with reductions in ex-
pressivity mediated by regularization and lengthscale, and,
in NNs, also decreases with (some kinds of) expressivity.
Finally, sensitivity to spurious correlation cannot explain
theEvR .
5. IMDb text classiﬁcation
discriminant
(sentiment)distractor
(contains ‘ﬁlm’)
Figure 6. Example stimuli
from the IMDb dataset.In this section, we demon-
strate our protocol on a
standard text classiﬁcation
task: sentiment analysis on-
the Internet Movie Database
Movie Reviews (IMDb)
dataset (Maas et al., 2011).
Selecting features. The sen-
timent label (“positive” or
“negative”) is the discriminant
zdisc. We manufacture an or-
thogonal distractor zdistas the
presence or absence of a word
that occurs in roughly 50% of the sentences in the dataset
and does not occur more frequently for either positive or
negative reviews. Some examples are “ﬁlm” and “you”: we
use the word “ﬁlm” (see Fig. ( 6)).
Models. We train a single layer LSTM (20 hidden units;
default hyperparameters) on each condition and test on the
held-out quadrant. We exclude models that do not reach
80% validation accuracy.
Feature-level bias. The distractor zdistis easier to learn
than the discriminant zdisc, as reﬂected in the CC condition
(19.7%, FLB =0:3).
Exemplar bias. We see good performance in ZS (84%):
Despite never having seen the word “ﬁlm,” the system
Distinguishing rule- and exemplar-based generalization in learning systems
can generalize to reviews containing it. The performance
in PE drops signiﬁcantly (30.1%) giving a large EvR
(EvR = 0:54), indicating exemplar-based reasoning. As
such, the exemplar-based tendency to utilize an additional
unnecessary feature ( e.g., the presence of the word “ﬁlm”)
hurts performance on the extrapolation quadrant.
6. CelebA image classiﬁcation
We now test our protocol on a standard classiﬁcation
task on a large-scale image dataset, CelebFaces Attributes
(CelebA) (Liu et al., 2015). Each image in this dataset is
labeled with 40 binary attributes, each of which can be as-
signed discriminant or distractor. We examine FLB and
EvR for standard models across different feature pairs, and
discuss the practical implications of our ﬁndings.
Selecting features. We select feature pairs that split the data
roughly evenly and thus maximizing the number of training
datapoints in each quadrant. We carry out our analyses
across a range of feature pairs; an example is depicted in
Fig. ( 7a), and further details are in the Appendix.
Models. We train ResNets of various depths ( f10;18;34g)
and widths (f2;4;8;16;32;64g) on 6 different choices for
feature pairs, with standard hyperparameters. We limit our
analyses to networks that achieve at least 75% validation
accuracy (on held-out samples from its own training distri-
bution) to ensure that, despite differences in data variability
across training conditions, all models learn a meaningful
decision boundary.
Feature-level bias. There is a wide range of FLB across
feature pairs; e.g.,“male” is easier to learn than “high cheek-
bones” giving high FLB , and “mouth open” and “wearing
lipstick” are equally difﬁcult and give FLB of close to 0.
FLB values for each feature pair were consistent across
ResNet widths and depths.
Exemplar-rule bias. We observe good ZS performance:
the models can generalize to new feature values outside
the training support. We see a wide range of EvR across
feature pairs, Fig. ( 7b). Across all feature pairs, the EvR is
non-negative: generalization in the PE condition is always
worse (or not signiﬁcantly better) than in the ZS condition.
Further, we see a linear correlation between EvR andFLB
in logit space across feature pairs. EvR therefore depends
on how easy or hard the features are to learn. The key,
however, is that this regression of the EvR ontoFLB has a
positive intercept: there is a positive EvR even for feature
pairs with no FLB . That is, we see lower performance in
PE compared to ZS (a nonzero EvR , exemplar propensity)
even when FLB is controlled for.
We ﬁnd no differences in EvR across ResNet widths and
depths: Fig. ( 7b) plots EvR andFLB averaged over ResNetsizes (model-speciﬁc results in Appendix). One explanation
is that the features in CelebA are complex; to learn these, we
need reasonably high model expressivity, and differences
in parameter count do not further modulate EvR . This is
consistent with ﬁndings in Section ( 4) where expressivity
is necessary but not sufﬁcient for increases in EvR : we see
a jump in EvR going from NN:2h1d to NN:16h1d, but no
further change going to the even more expressive NN:4h4d.
Controlling spurious correlation. We replicate the ﬁnd-
ings in Section ( 4): the EvR cannot be explained by sen-
sitivity to spurious correlation . This is demonstrated
in Fig. ( 7c), where we substitute performance in the PE
condition with performance in a different data condition
(0= 0:825;0= 0:25) with the same = 0:58as in
the PE condition. We ﬁnd none of the effects discussed
above, indicating that the PE condition is measuring some-
thing unique—exemplar-vs-rule propensity—which is not
accounted for by sensitivity to spurious correlation. Fur-
ther,EvR does not increase with model expressivity, unlike
sensitivity to spurious correlation (Sagawa et al., 2020b).
Practical implications of the EvR .The nonzero EvR
(i.e.,exemplar bias) reveals that models are better at ex-
trapolating zero-shot to a new feature value than when they
have partial exposure to that feature value even though the
additional data need not change the learned decision bound-
ary. In particular, the training examples added in PE can
be classiﬁed with the decision function from ZS without
incurring additional training loss. A rule-based system rec-
ognizes this and bases its generalization on the minimal
features that support the category boundary. However, an
exemplar-based model changes its decision boundary in
response to this additional data.
PE-approximating data distributions ( 00:5;10:0)
occur naturally. For example, as (Sagawa et al., 2020b)
observe, “blond” “male”s are under-represented in CelebA.
Consistent with the rest of our results, we ﬁnd better classi-
ﬁcation for the extrapolation quadrant (blond males) if we
discard data from an adjacent quadrant (blond non-males, or
non-blond males) simulating the zero-shot condition, as op-
posed to the PE condition if such data is included: ResNet10,
width 2, gives ZS = 75:123:09%;PE = 60:227:27%
forzdisc=“male” (discard blond non-males to get ZS)
andZS = 68:163:34%;PE = 49:783:76% for
zdisc=“blond” (discard non-blond males to get ZS).
These results demonstrate the practical impact of under-
standing the exemplar-vs-rule bias in a model: an exem-
plar biased model (like the ResNet here) generalizes poorly
in combinatorial settings, and can be made to generalize
better by discarding an entire quadrant of data. Previous
sub-sampling approaches (Sagawa et al., 2020b; Haixiang
et al., 2017) do not manipulate feature combinations and
only manipulate spurious correlations. The aforementioned
Distinguishing rule- and exemplar-based generalization in learning systems
discriminant
(wearing lipstick)distractor
(mouth open)
(a)Example CelebA stimuli; we
test 6 discriminant-distractor pairs
(with sufﬁcient coverage of quad-
rants in Fig. ( 3)) of 6 features.2 1 0 1 2012no feature-level bias
rule-based
FLB(F)EvR(F)
(b) Main effect , logit scale: Exemplar-vs-rule propensity
(EvR ) trends linear with non-zero intercept (ﬁt in red) in
feature-level bias ( FLB ).2 1 0 1 2012no feature-level bias
FLB(F)[ZSequi-](F)
(c) Control , logit scale: Performance deviation from ZS un-
der equi-correlation interpolation is close to zero; linear ﬁt
(red) has intercept at zero.
Figure 7. CelebA results. Stimuli and results on various feature pairings from the CelebA domain (Section ( 6)). Error bars represent
95% conﬁdence intervals across ResNets of various sizes. See ﬁgure sub-captions and main text for details.
analyses (Fig. ( 4)) and results (Fig. ( 7c)) demonstrate that
this underspeciﬁes extrapolation behavior.
7. Related work and future directions
Model design for systematic generalization. Rule-
based generalization permits systematic extrapolation in
combinatorial domains. This systematicity has been found
lacking in neural networks (Lake & Baroni, 2018b; Barrett
et al., 2018), leading to renewed interest in hybrid symbolic–
connectionist methods ( e.g., Garnelo & Shanahan, 2019).
However, works proposing new methods usually do not
examine how feature co-occurrences modulate the system-
aticity of extrapolation. Using our protocol to examine
exemplar- vs. rule-based generalization in these models is a
promising future direction.
Learning causal features. Rule-based generalization, is
equivalent to learning causal features under the assumption
that the causal model is the simplest model that explains the
data. Recent work has investigated data settings that sepa-
rate causal features from spurious ones ( e.g., Arjovsky et al.,
2019).We showed that a model with exemplar propensity
makes more rule-based extrapolations for certain training
feature combinations ( i.e.,zero shot vs. partial exposure).
Investigating how feature coverage impacts causal general-
ization is a fruitful future direction.
Similarity-based generalization and kernels. We use
similarity-based kernels ( e.g., radial basis function (RBF))
to exemplify exemplar-based extrapolation. Recent work
has interpreted neural networks as kernel regression (Jacot
et al., 2018). Using a kernel framing to formalize the causes
of exemplar bias is an exciting future direction.
Data augmentation. TheEvR measure allows us to
demonstrate that increased data variation in the form of
feature coverage worsens systematic generalization. The
negative effect of data variation on generalization has beendocumented for adversarial augmentations (Raghunathan
et al., 2020). We show that this can persist even when aug-
mentation is not adversarial, rendering it generally relevant
for the design of data augmentations.
8. Conclusions
Taking inspiration from—and going beyond—psychological
studies, we design a behavioral protocol to distinguish the
effects of two inductive biases (feature-level bias and ex-
emplar bias) that is easily applicable to any classiﬁcation
domain. This follows in a promising line of recent work
that analyses and interprets deep learning systems based
on their external behavior (Ritter et al., 2017b; Dasgupta
et al., 2019). It complements other approaches that follow
in the neuroscience tradition of analyzing internal represen-
tations (Zeiler & Fergus, 2014; Karpathy et al., 2015) or
make approximations of these internal workings to support
theoretical results (Jacot et al., 2018; Allen-Zhu et al., 2019).
The behavioral approach has the advantage that it makes no
assumptions about the model, allowing comparisons across
systems that differ in design.
Both rule- and exemplar-based extrapolation are valuable
depending on domain, underscoring the importance of di-
agnosing feature-level bias and exemplar bias. Moreover,
studying this trade-off allows us to demonstrate an important
phenomenon: We ﬁnd that more feature coverage (as in par-
tial exposure compared to zero shot) hurts generalization for
exemplar-based models. This has implications for methods
that manipulate data distributions to improve performance
(e.g.,data subsampling (Haixiang et al., 2017), data augmen-
tation (Perez & Wang, 2017), and contrastive learning (Chen
et al., 2020)). Since an exemplar-based model tends to ac-
quire spurious associations, our measures have the potential
to be useful as diagnostics in application settings where
the goal is to control model behavior on non-representative
factors (( e.g., Mitchell et al., 2019)).
Distinguishing rule- and exemplar-based generalization in learning systems
A limitation of the present work is that we do not provide a
conclusive answer as to what properties of a model family in-
ﬂuence both feature-level bias and exemplar bias. A broader
study of these factors and theoretical work formalizing this
effect are exciting avenues for future work.
Acknowledgements and Funding Sources
We thank our anonymous reviewers for their feedback. This
work was supported by ONR Grant #N00014-18-1-2873
and the DARPA L2M program.
References
Allen, S. W. and Brooks, L. R. Specializing the operation
of an explicit rule. Journal of experimental psychology:
General , 120(1):3, 1991.
Allen-Zhu, Z., Li, Y ., and Song, Z. A convergence theory for
deep learning via over-parameterization. In International
Conference on Machine Learning , pp. 242–252. PMLR,
2019.
Andreas, J. Measuring compositionality in representation
learning. arXiv preprint arXiv:1902.07181 , 2019.
Andreas, J., Rohrbach, M., Darrell, T., and Klein, D. Neural
module networks. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR) ,
pp. 39–48, 2016. URL https://arxiv.org/abs/
1511.02799 .
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-
Paz, D. Invariant risk minimization. arXiv preprint
arXiv:1907.02893 , 2019.
Ashby, F. G. and Townsend, J. T. Varieties of perceptual
independence. Psychological Review , 93(2):154, 1986.
Barrett, D., Hill, F., Santoro, A., Morcos, A., and Lillicrap,
T. Measuring abstract reasoning in neural networks. In
International Conference on Machine Learning , pp. 511–
520, 2018. URL https://arxiv.org/abs/1807.
04225 .
Beery, S., Van Horn, G., and Perona, P. Recognition in terra
incognita. In Proceedings of the European Conference
on Computer Vision (ECCV) , pp. 456–473, 2018. URL
https://arxiv.org/abs/1807.04975 .
Brown, G., Bun, M., Feldman, V ., Smith, A., and Talwar,
K. When is memorization of irrelevant training data
necessary for high-accuracy learning? arXiv preprint
arXiv:2012.06421 , 2020.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A
simple framework for contrastive learning of visual rep-resentations. In International conference on machine
learning , pp. 1597–1607. PMLR, 2020.
D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Ali-
panahi, B., Beutel, A., Chen, C., Deaton, J., Eisen-
stein, J., Hoffman, M. D., et al. Underspeciﬁcation
presents challenges for credibility in modern machine
learning. arXiv preprint arXiv:2011.03395 , 2020. URL
https://arxiv.org/abs/2011.03395 .
Dasgupta, I., Guo, D., Gershman, S. J., and Goodman, N. D.
Analyzing machine-learned representations: A natural
language case study. arXiv preprint arXiv:1909.05885 ,
2019.
Feldman, V . Does learning require memorization? a short
tale about a long tail. In Proceedings of the 52nd Annual
ACM SIGACT Symposium on Theory of Computing , pp.
954–959, 2020.
Feldman, V . and Zhang, C. What neural networks mem-
orize and why: Discovering the long tail via inﬂuence
estimation. Advances in Neural Information Processing
Systems , 33, 2020.
Garnelo, M. and Shanahan, M. Reconciling deep learning
with symbolic artiﬁcial intelligence: representing
objects and relations. Current Opinion in Behav-
ioral Sciences , 29:17–23, 2019. ISSN 2352-1546.
doi: https://doi.org/10.1016/j.cobeha.2018.12.010.
URL https://www.sciencedirect.com/
science/article/pii/S2352154618301943 .
Artiﬁcial Intelligence.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich-
mann, F. A., and Brendel, W. Imagenet-trained cnns are
biased towards texture; increasing shape bias improves ac-
curacy and robustness. arXiv preprint arXiv:1811.12231 ,
2018.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren-
del, W., Bethge, M., and Wichmann, F. A. Shortcut learn-
ing in deep neural networks. Nature Machine Intelligence ,
2(11):665–673, 2020.
Haixiang, G., Yijing, L., Shang, J., Mingyun, G., Yuanyue,
H., and Bing, G. Learning from class-imbalanced data:
Review of methods and applications. Expert Systems with
Applications , 73:220–239, 2017.
Hermann, K. L., Chen, T., and Kornblith, S. The origins
and prevalence of texture bias in convolutional neural
networks. arXiv preprint arXiv:1911.09071 , 2019.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the di-
mensionality of data with neural networks. science , 313
(5786):504–507, 2006.
Distinguishing rule- and exemplar-based generalization in learning systems
Hudson Kam, C. L. and Newport, E. L. Regularizing unpre-
dictable variation: The roles of adult and child learners in
language formation and change. Language learning and
development , 1(2):151–195, 2005.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel:
Convergence and generalization in neural networks. In
Advances in neural information processing systems , pp.
8571–8580, 2018.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L.,
Lawrence Zitnick, C., and Girshick, R. Clevr: A diag-
nostic dataset for compositional language and elementary
visual reasoning. Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition , pp. 2901–
2910, 2017.
Karpathy, A., Johnson, J., and Fei-Fei, L. Visualizing and
understanding recurrent networks. Proceedings of the 3rd
International Conference on Learning Representations,
Workshop Track , 2015.
Lake, B. and Baroni, M. Generalization without systematic-
ity: On the compositional skills of sequence-to-sequence
recurrent networks. In International Conference on Ma-
chine Learning , pp. 2873–2882. PMLR, 2018a.
Lake, B. M. and Baroni, M. Still not systematic after all
these years: On the compositional skills of sequence-
to-sequence recurrent networks. Proceedings of the 6th
International Conference on Learning Representations,
Workshop Track , 2018b.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh-
man, S. J. Building machines that learn and think like
people. Behavioral and Brain Sciences , 40, 2018.
Landau, B., Smith, L. B., and Jones, S. S. The importance
of shape in early lexical learning. Cognitive development ,
3(3):299–321, 1988.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face
attributes in the wild. In Proceedings of International
Conference on Computer Vision (ICCV) , December 2015.
Maas, A., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y ., and
Potts, C. Learning word vectors for sentiment analysis.
InProceedings of the 49th annual meeting of the asso-
ciation for computational linguistics: Human language
technologies , pp. 142–150, 2011.
Marcus, G. Deep learning: A critical appraisal. arXiv
preprint arXiv:1801.00631 , 2018.
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman,
L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T.
Model cards for model reporting. In Proceedings of the
conference on fairness, accountability, and transparency ,
pp. 220–229, 2019.Mitchell, T. M. The need for biases in learning generaliza-
tions . Department of Computer Science, Laboratory for
Computer Science Research, 1980.
Nosofsky, R. M., Clark, S. E., and Shin, H. J. Rules and ex-
emplars in categorization, identiﬁcation, and recognition.
Journal of Experimental Psychology: Learning, memory,
and cognition , 15(2):282, 1989.
Perez, L. and Wang, J. The effectiveness of data augmen-
tation in image classiﬁcation using deep learning. arXiv
preprint arXiv:1712.04621 , 2017.
Pothos, E. M. The rules versus similarity distinction. Be-
havioral and brain sciences , 28(1):1, 2005.
Raghu, M., Poole, B., Kleinberg, J., Ganguli, S., and Sohl-
Dickstein, J. On the expressive power of deep neural
networks. In international conference on machine learn-
ing, pp. 2847–2854. PMLR, 2017.
Raghunathan, A., Xie, S. M., Yang, F., Duchi, J., and Liang,
P. Understanding and mitigating the tradeoff between ro-
bustness and accuracy. arXiv preprint arXiv:2002.10716 ,
2020.
Rasmussen, C. E. Gaussian processes in machine learn-
ing. In Summer school on machine learning , pp. 63–71.
Springer, 2003.
Rips, L. J. Similarity, typicality, and categorization. 1989.
Ritter, S., Barrett, D. G., Santoro, A., and Botvinick, M. M.
Cognitive psychology for deep neural networks: A shape
bias case study. International Conference on Machine
Learning , pp. 2940–2949, 2017a.
Ritter, S., Barrett, D. G., Santoro, A., and Botvinick, M. M.
Cognitive psychology for deep neural networks: A shape
bias case study. In International conference on machine
learning , pp. 2940–2949. PMLR, 2017b. URL https:
//arxiv.org/abs/1706.08606 .
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P.
Distributionally robust neural networks. In International
Conference on Learning Representations , 2020a. URL
https://arxiv.org/abs/1911.08731 .
Sagawa, S., Raghunathan, A., Koh, P. W., and Liang, P. An
investigation of why overparameterization exacerbates
spurious correlations. In International Conference on
Machine Learning , pp. 8346–8356. PMLR, 2020b.
Shepard, R. N. and Chang, J.-J. Stimulus generalization in
the learning of classiﬁcations. Journal of Experimental
Psychology , 65(1):94, 1963.
Distinguishing rule- and exemplar-based generalization in learning systems
Smith, E. E. and Sloman, S. A. Similarity-versus rule-based
categorization. Memory & Cognition , 22(4):377–386,
1994.
Xian, Y ., Lampert, C. H., Schiele, B., and Akata, Z. Zero-
shot learning—a comprehensive evaluation of the good,
the bad and the ugly. IEEE transactions on pattern anal-
ysis and machine intelligence , 41(9):2251–2265, 2018.
Zeiler, M. D. and Fergus, R. Visualizing and understand-
ing convolutional networks. European conference on
computer vision , pp. 818–833, 2014.
Distinguishing rule- and exemplar-based generalization in learning systems
training condition
feature space cue conﬂict zero shot partial exposure extrapolation
0= 0:5,1= 0:50= 1:0,1= 0:00= 0:0,1= 0:00= 0:5,1= 0:00= 1:0,1= 1:0
discriminant (color)distractor (shape)
5 0 5505
x1x2
discriminant (x1>0)distractor (x2>0)
5 0 5505
5 0 5505
5 0 5505
5 0 5505
Figure 8. We expand on Fig. ( 3) from the main text by including a realization of the abstract training conditions in the simple 2D
points-in-a-plane setting. ( Top)Formalizing the illustrative experiment: The experiment from Fig. ( 2) expressed in terms of the
formalism in Section ( 3) with zdist=color and zdisc=shape. Background colors indicate true category boundary. ( Bottom ) The
conditions realized via a binarization of continuous feature values. Here, the discriminant is binarized as x1>0and the distractor as
x2>0; this setting is further investigated in Section ( 4). Color here depicts the label but is not part of the input.
A. Additional formalizations
A.1. Generalizing the framework from two binary attributes to many categorical attributes
In the most general terms, we consider a setting in which each observation x2X is underlied by ncategorical variables
z1;:::;zn2f0;:::;CgwithC2Z+, henceforth attributes whose concatenation z= (z1;:::;zn)determines the
observable input xvia some mapping g:Zn
0+!X . We consider the binary classiﬁcation task of ﬁtting a model
^f:X!f 0;1gfrom a given model family Fto predict a binary label for each input. A subset of the attributes in z, without
loss of generality (z0;:::;zi), is taken to deﬁne the decision boundary, while the remaining attributes, zi+1;:::;zn, are
assumed to not be independently predictive of the true classiﬁcation y2f0;1g. We therefore denote the discriminant ,
zdisc= (z0;:::;zi), and the distractor zdist= (zi+1;:::;zn). For simplicity, we assume that the attributes are binary
(i.e.,C= 2andzi2f0;1g;8i), and that the discriminant attributes must be jointly active for the classiﬁcation to change
from the null class y= 0(i.e.,y= 1()zdisc=1); the latter simpliﬁcation allows us to redeﬁne zdisc=z0^^zi
andzdist=zi+1^^zn, which is equivalent to the earlier discussion of the illustrative two-attribute case.
A.2. Training conditions expressed in terms of the joint distribution
We express the training conditions displayed in Fig. ( 3) and realized in Figure 6 in terms of the joint distribution instead of
the parameters 0,1.
1.The cue-conﬂict condition the upper left and lower right quadrants in Figure 6 and deﬁnes the distribution of attributes
as
pcc(zdisc= 0;zdist= 1) =0:5pcc(zdisc= 1;zdist= 1) =0
pcc(zdisc= 0;zdist= 0) =0pcc(zdisc= 1;zdist= 0) =0:5.
2.The zero-shot condition populates the bottom left and right quadrants in Figure 6 and deﬁnes the distribution of
attributes as
pzs(zdisc= 0;zdist= 1) =0pzs(zdisc= 1;zdist= 1) =0
pzs(zdisc= 0;zdist= 0) =0:5pzs(zdisc= 1;zdist= 0) =0:5.
Distinguishing rule- and exemplar-based generalization in learning systems
3.The partial-exposure condition populates all quadrants but the upper right in Figure 6 and deﬁnes the distribution of
attributes as
ppe(zdisc= 0;zdist= 1) =0:25ppe(zdisc= 1;zdist= 1) =0
ppe(zdisc= 0;zdist= 0) =0:25ppe(zdisc= 1;zdist= 0) =0:5.
B. More CelebA results
We include model-speciﬁc results, split by ResNet depth and width, in Fig. ( 9). We ﬁnd no systematic relationship between
EvR and depth or width.
012
−2 −1 0 1 2
logit FLBlogit EVRwidth
2
4
8
16
32
48
64ResNet10
012
−2 −1 0 1 2
logit FLBlogit EVRwidth
2
4
8
16
32
48
64ResNet18
012
−2 −1 0 1 2
logit FLBlogit EVRwidth
2
4
8
16
32
48
64ResNet34
Figure 9. CelebA EvR andFLB across feature pairs, averaged across 30 runs, split by depth and width of ResNet.
C. Spurious correlation underdetermines feature distributions
The partial-exposure condition ( 0= 0:5,1= 0:0) in Section ( 2) results in a spurious correlation between the discriminant
zdiscand the distractor zdist(= 0:58). To examine behavior in a wider range of data settings, we vary 0and1as
described in Section ( 3), thereby also changing the degree of spurious correlation.
I. Interpolation towards zero shot. We interpolate 0from 0:5towards 0:0, keeping1= 0:0. This moves us closer to
0=1= 0:0, where we have no exposure to zdisc= 1in training. Intuitively, we are reducing the exposure to the new
distractor feature value from the partial-exposure condition.
II. Interpolation to full exposure. We interpolate 1from 0:0towards 0:5, keeping0= 0:5. This moves us closer to
0=1= 0:5, where we have equal exposure to all quadrants in training. Here, rather than reducing the exposure to the
new distractor feature value, we are equalizing the exposure to it across the discriminant dimension.
III. Interpolation with matched correlation. We report results on this in Sections ( 4) and ( 6). As also depicted in Fig. ( 4),
we generate training conditions by changing 0and1such that we follow a -contour away from the partial-exposure
condition (0= 0:5;1= 0:0,= 0:58): solid contour in Fig. ( 4). We also match the spurious correlation across the two
interpolations in Appendix ( C)A and B: Fig. ( 11) shows these additional -contours as dashed lines.
These different interpolations are depicted in Fig. ( 11a) with different shape/colors.
C.1. Generating interpolation points
We generate points along all three interpolation lines: from partial exposure towards zero shot (( C)I); from partial exposure
towards full exposure (( C)II); and the equi-correlation line originating from partial exposure (( C)III). The interpolating
points along each line are selected to balance spurious correlation and feature exposure. In particular, we follow the
following procedure:
1.We choose a point that interpolates towards full exposure. We do this by choosing a value of 1between 0:0and0:5,
FE. This gives a data setting, along with a corresponding spurious correlation, , computed via Eq. (3):
0= 0:5 ; 1=FE; =
0:5;FE
:
2.We generate a corresponding point that interpolates towards zero shot. Given the data setting above, we set 1= 0:0
and compute the 0to produce the same as the full-exposure interpolations in Step 1. This gives the data setting:
0=ZS
FE
; 1= 0:0 ; =
ZS
FE
;0:0
=
0:5;FE
:
Distinguishing rule- and exemplar-based generalization in learning systems
partial exposure interpolant #1 interpolant #2
0= 0:5,1= 0:0,
= 0:580= 0:32,1= 0:0,
= 0:4360= 0:125,1= 0:0,
= 0:258
zero-shot
interp. ( C)I
5 0 5505
5 0 5505
5 0 5505
0= 0:5,1= 0:0,
= 0:580= 0:5,1= 0:1,
= 0:4360= 0:5,1= 0:25,
= 0:258
full-exposure
interp. ( C)II
5 0 5505
5 0 5505
5 0 5505
0= 0:5,1= 0:0,
= 0:580= 0:66,1= 0:1,
= 0:580= 0:825,1= 0:25,
= 0:58
equi-correlation
interp. ( C)III
5 0 5505
5 0 5505
5 0 5505
Figure 10. We visualize several of the interpolants used for the interpolation analyses.
3.Finally, we also derive the equi-correlation interpolation from the full-exposure interpolation as follows. We retain 1
from the full-exposure condition, but recompute the 0such that the correlation matches the spurious correlation of
the pure
glspec (= 0:58). This gives an additional data setting:
0=EQ
FE
; 1=FE; =(0:5;0:0) = 0:58:
Note that, despite there being three different interpolation lines, the speciﬁc interpolants we use are constrained along a
single degree of freedom—choosing FE(Step 1). The data settings for zero shot (Step 2) and equi-correlation (Step 3) are
derived from this value.
C.2. Speciﬁc interpolation values used
For all data settings, we generate points along the interpolation lines using the procedure in Appendix ( C.1).
For the simple 2D classiﬁcation setting, we examine two interpolants. In this simple domain, we keep the interpolation
distances small, since we expect changes in extrapolation behavior even from small changes.
interpolant 1 interpolant 2
01  01
interpolation to zero shot (( C)I) 0:481 0:0 0:563 0:32 0:0 0:436
interpolation to full exposure (( C)II) 0:5 0:01 0:563 0:5 0:1 0:436
equi-correlation interpolation (( C)III) 0:519 0:01 0:58 0:661 0:1 0:58
For CelebA, we increase the interpolation distance to reﬂect the wider range of natural data distributions among feature
Distinguishing rule- and exemplar-based generalization in learning systems
0 0:5 101
10zero-shot interp. ( C)A
full-exposure interp. ( C)B
equi-correlation interp. ( C)C0 0:5 101
10zero-shot interp. ( C)A
full-exposure interp. ( C)B
equi-correlation interp. ( C)C
0 0:5 101
10zero-shot interp. ( C)A
full-exposure interp. ( C)B
equi-correlation interp. ( C)C
0
0:5
101

1
0
zero-shot interp. (
C)Afull-exposure interp. (
C)Bequi-correlation interp. (
C)C(a)A heatmap of the spurious correlation (Eq. (3)), showing different interpolations.
The partial-exposure condition is identiﬁed with a star outline; points along the three
interpolation line are identiﬁed with ﬁlled shapes.0 0:13 0:2500:20:4
interpolation distance
q
(0:50)2+2
1[ZSequi-]
(b)Interpolations for NN:16h1d on 2D
classiﬁcation of points-in-a-plane.00:130:250:380:500:20:4
interpolation distance
q
(0:50)2+2
1
(c) Interpolations, ResNet-18-8,
CelebA (“wearing lipstick”,
“mouth open”)
Figure 11. Interpolations away from the PE: changes in extrapolation behavior under data distribution with the same spurious correlation
as in PE, as well as different ways to change spurious correlation.
pairs. The data these interpolation values generate is visualized as the equivalent points-in-a-plane setting in Figure 7.
interpolant 1 interpolant 2
01  01
interpolation to zero shot (( C)I) 0:32 0:0 0:436 0:125 0:0 0:258
interpolation to full exposure (( C)II) 0:5 0:1 0:436 0:5 0:25 0:258
equi-correlation interpolation (( C)III) 0:66 0:1 0:58 0:825 0:25 0:58
C.3. Interpolation analyses
C.3.1. I N THE 2-D CLASSIFICATION EXAMPLE
In the simple setting from Section ( 4), we vary0,1for an NN model (NN:16h1d, the NN with lowest EvR level overall).
Results are in Fig. ( 11b) and discussed below.
EvR6=sensitivity to spurious correlation. As also discussed in the main text, along the equi-correlation interpolation
line, the “effective EvR ” drops drastically ( i.e.,the learner generalizes in more rule-based manner) despite no change in
spurious correlation.
Implications for controlling extrapolation. Despite both having the same , interpolating towards full-exposure increases
theEvR more than towards zero-shot. This further supports that spurious correlation cannot fully characterize extrapolation
behavior. This shows that different ways to reducehave different effects on extrapolation, and has important implications
for data manipulation methods ( e.g., subsampling or augmentation) that aim to directly control this .
C.3.2. I NCELEB A
We see the same effects as in the linear setting: as also discussed in the main text, we see a much smaller gap to the ZS
condition despite no change in spurious correlation. We don’t ﬁnd clear effects distinguishing different ways to reduce
spurious correlation (interpolation to zero shot (( C)I) and interpolation to full exposure (( C)II)).