From convolutional neural networks to models of higher level cognition (and back again).

Ann. N.Y . Acad. Sci. ISSN 0077-8923
ANNALS OF THE NEW YORK ACADEMY OF SCIENCES
Special Issue: The Year in Cognitive Neuroscience
Review
From convolutional neural networks to models of
higher-level cognition (and back again)
Ruairidh M. Battleday,1,aJoshua C. Peterson,1,aand Thomas L. Grifﬁths1,2
1Department of Computer Science, Princeton University, Princeton, New Jersey.2Department of Psychology, Princeton
University, Princeton, New Jersey
Addresses for correspondence: Ruairidh M. Battleday and Joshua C. Peterson, Department of Computer Science, Princeton
University, 35 Olden Street, Princeton, NJ 08540. battleday@princeton.edu and joshuacp@princeton.edu
The remarkable successes of convolutional neural networks (CNNs) in modern computer vision are by now well
known, and they are increasingly being explored as computational models of the human visual system. In this
paper,weaskwhetherCNNsmightalsoprovideabasisformodelinghigher-levelcognition,focusingonthecore
phenomenaofsimilarityandcategorization.ThemostimportantadvancecomesfromtheabilityofCNNstolearn
high-dimensional representations of complex naturalistic images, substantially extending the scope of traditional
cognitivemodelsthatwerepreviouslyonlyevaluatedwithsimpleartificialstimuli.Inallcases,themostsuccessful
combinationsarisewhenCNNrepresentationsareusedwithcognitivemodelsthathavethecapacitytotransform
themtobetterfithumanbehavior.Oneconsequenceoftheseinsightsisatoolkitfortheintegrationofcognitively
motivatedconstraintsbackintoCNNtrainingparadigmsincomputervisionandmachinelearning,andwereview
cases where this leads to improved performance. A second consequence is a roadmap for how CNNs and cogni-
tive models can be more fully integrated in the future, allowing for flexible end-to-end algorithms that can learn
representationsfromdatawhilestillretainingthestructuredbehaviorcharacteristicofhumancognition.
Keywords: categorization;cognitivemodeling;convolut ionalneuralnetworks;similarity;vision
Introduction
The first demonstration of the potential impact
of deep learning came from the field of com-
puter vision, with the unprecedented success of a
deep convolutional neural network (CNN) called
“AlexNet” on the ImageNet challenge—a large-
scale natural image classification task.1Through
a number of scientific and engineering advances,
researchers were able to train much larger artificial
neural networks than previously, and in doing so
apply them to a broader and more difficult range
of tasks. Beyond making networks deeper, suc-
cess was found to depend on matching the right
types of architectural and functional constraints
aBothoftheseauthorscontributedequally.to different kinds of task—collectively known as
inductive biases , or the aspects of model design
that influence the generalization performance of a
learning algorithm beyond the data themselves.2
For computer vision tasks involving images, the
i n d u c t i v eb i a sp r o v i d e db yd e e ps t a c k so fl e a r n -
able convolutional filters has proved instrumen-
tal in reaching state-of-the-art performance on
a large number of computer vision benchmarks
relatedtocognition,suchasimageclassification,1,3,4
image segmentation and object recognition,5,6and
visual question answering;7–10as well as in visual
modules for algorithms solving more integrated
tasks.11DevelopmentsinCNNstructureandappli-
cations continue at an ever increasing rate in the
computer vision and machine learning commu-
nities, to the point where it has become rare to
hear of a statistical tool or application related to
doi: 10.1111/nyas.14593
1 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and
distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modiﬁcations or adaptations are made.

Convolutional networks and higher-level cognition Battleday et al.
image processing that does not include a CNN
component.
In addition to solving problems in computer
vision, there has been increasing recognition that
CNNs could also serve as models of the human
visual system.12CNNs have repeatedly proved to
b et h eb e s tp r e d i c t i v em o d e lo fn e u r a la n dv o x e l
responses to visual stimuli in primate electrophys-
iologyandhumanimagingstudies,withthedegree
of approximation roughly improving with CNN
performance.13–22Interestingly, the level of corti-
cal and CNN hierarchy often appears to match,
with shallow CNN layers better predicting earlier
visual areas and deeper CNN layers better predict-
inghighervisualareas.15,17,18
CNNs represent an equally interesting opportu-
nityforthestudyofhigher-levelcognition.Tha tis,
they may allow us to extend computational mod-
els of cognition from simple artificial stimuli to
more complex naturalistic stimuli representative
of the complex visual world in which our cog-
nitive abilities arise. In particular, cognitive psy-
chology has provided a range of influential com-
putational models that examine the fundamental
cognitive phenomena of similarity and categoriza-
tion, which typically abstract over details of neu-
ral implementation, and instead focus on explain-
ing human behavior in terms of the computational
problems posed by the environment and cognitive
processesthatsolveit.23,24However,well-motivated
considerations—namely, having good representa-
tions of stimuli on which to base quantitative
analysis—have limited these models to examin-
ing behavior from laboratory-based experimental
paradigms using simple artificial stimuli. The rich
representations produced by CNNs solving related
computational tasks suggest they are a good can-
didate to extend this modeling framework further,
to more naturalistic stimuli. If effective, this would
allo wustost udyh uma nbeha vio rinmo r eecologi-
cally relevant settings—a goal that must be an ulti-
mateaimofanyresearchprogramintothemindand
brain.
Inthisreview,weexaminehowCNNscanbeused
toextendtherangeoftraditionalmodelsofhigher-
levelcognition,and,inturn,whetherinsightsfrom
studies of cognition can improve the performance
of CNNs in solving related computer vision and
machine learning tasks (Fig. 1). We begin with an
introduction to CNNs, and then show how rep-resentations from these networks can be directly
integratedintoexistingcognitivemodelsofsimilar-
ity and categorization as tools for feature learning
overdiverserangesofnaturalisticstimuli.Although
off-the-shelf CNN representations prove our best
available substrate for modeling behavior, a grow-
ingnumberofcognitivemodelshavebeenproposed
thatincreasetheircorrespondencetopsychological
representations using simple mathematical trans-
formations.Beyondcapturinghumanbehaviorbet-
ter,thesemodelingregimesgiveustoolstointegrate
more structured, cognitive constraints into normal
CNN training paradigms, and we review a num-
berofexampleswherethishasledtoimprovedper-
formance. Finally, we give an account of how deep
learningandhigher-levelcognitivemodelingcanbe
morefullyintegratedinthefuture,inordertoobtain
theadvantagesofbothframeworksandprovidethe
nextgenerationofcognitivemodels.
Deep and convolutional neural networks
Artificialneuralnetworkshavealongconnectionto
cognitive neuroscience, both as models of neurons
and as a brain-compatible computing paradigm.
Interest began with the investigation of the com-
putationalpropertiesofsimplifiedmodelsofsingle
neurons in the mid-20th century,25,26inspired by
theoriesaboutthebiologyofneurons.27,28Onegoal
wastoemulatethecognitivepropertiesofthebrain
thathadremainedelusiveforsymbolicapproaches:
good performance on inductive learning tasks,
robustness to error in the input or computation,
and graceful degradation of performance with
alteration of the type of input or task. A second
waveofresearchwasdrivenbylearningalgorithms
thatallowedresearcherstotrainthehiddenweights
in multilayer networks automatically, including
the “backpropagation” algorithm.29Learning these
hidden weights allowed networks to solve non-
l i n e a rp r o b l e m s ,w i t hr e s e a r c hm o v i n gq u i c k l y
fromdemonstrationson“toy”logicalproblemslike
implementing the XOR function to more cogni-
tively interesting tasks. Key to the important series
of successes in this era was the identification of
d i ff e r e n ti n d u c t i v eb i a s e sa si n s t r u m e n t a lt os o l v -
ing different kinds of problems—for example, the
needforrecurrentloopsintime-dependenttasks,30
certainintermediateformsinlinguistics,31andcon-
volutionalfiltersinimageclassificationproblems.32
2 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
Figure 1. Overview. Traditional studies have used simple artificial stimuli that can be mathematically represented unambigu-
ously as the substrate for models of higher-level cognition (left pathway). CNNs can be used to supply representations for more
complexnaturalisticimages,whichcanbe furthermodifiedtobetterreflecthumanjudgmentsbeforebeinginputintothesame
kinds of cognitive model (middle pathways).50,64,73End-to-end models offer the opportunity to solve both of these problems
simultaneouslyandlearnarepresentationfornaturalisticstimulithatsatisfiestheconstraintsinherentinhigher-levelcognitive
models(rightpathway).136
Manyothercomponentsofthemoderndeeplearn-
ing framework arose during this period from the
collaboration of psychologists, neuroscientists, and
thecomputationalvisioncommunity,includingthe
development of hierarchical feed-forward visual
models based on stacks of nonlinear feature maps
andpoolingbetweenlayers.33–36The third wave of research into artificial neural
networks,nowknownasthe“deeplearningrevolu-
tion,” was heralded in 2012 by the unprecedented
performance of a CNN with several hidden lay-
ers, “AlexNet,” on a difficult task of natural image
classification.1Not only did the large improve-
ment in classification accuracy between AlexNet
3 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
andpreviousstate-of-the-artclassifierssignalinter-
e s t,b u tal soth era p ids u b seq ue n tim p r o v e m e n tsin
performanceseemedtoannouncethathuman-level
performance on human-relevant tasks might be in
sight.AlthoughmostofthecomponentsofAlexNet
came from the first two waves of interest in artifi-
cialneuralnetworks,ittookanumberofengineer-
ing and scientific advances to render performance
at this scale achievable. First, these successes came
hand-in-handwithexponentiallyincreasedcompu-
tationalpowerandthereleaseofverylargedatasets
(e.g.,ImageNet).1,3Havingmoredataandthecom-
putational power to approximate more complex
functionsmeanttherangeoftasksthatdeepneural
networks could learn to perform was significantly
extended. But this belies the second, major influ-
ence, which is the scientific and societal side of the
revolution. Initial successes drew the attention of a
l a r g ec o m m u n i t yv e r s e di ns t a t i s t i c sa n dc o m p u t a -
tion, and the sheer breadth of architectures, learn-
ingrules,objectivefunctions,anddatasetstestedby
thecommunitybecameanimportantfactorinitself.
Focusisnowlevelednotatlaboratoryproblems,but
at real-world problem-solving environments, such
as the prediction and generation of text,37perfor-
manceonvideogames,11andlarge-scalecategoriza-
tionofnaturalimages.1,3
Thedeepneuralnetworksthathaveprovidedthe
closest correspondence with brain and behavioral
data using visual stimuli have been trained in the
contextofclassificationbysupervisedlearning,12,38
which refers to training a learning algorithm to
improve its predictions by comparing them to the
known correct answer—a form of teaching, or
“supervision.” This comparison is given explicit
formusinganobjectivefunction,andthealgorithm
must use the error signal it returns to maximize
its performance. In classification tasks, the input is
usuallysomenaturalisticstimulus,suchasanimage
or text, encoded as a matrix or vector of activa-
tion values, and the output most commonly a cat-
e g o r yl a b e l .D e e pn e u r a ln e t w o r k ss o l v et h i sp r o b -
lembysuccessivelytransformingactivationsusinga
sequentialstackoffeed-forwardlayers,endingwith
acategorylabel(Fig.2).
The basic computing element is a single unit, or
“neuron,” which computes a weighted sum of its
inputsandthenpassesthissumthroughanonlinear
activation function—most commonly the rectified
linear unit, or, “ReLU,” which outputs all negativeinput as zero, and all positive input without trans-
formation.Theseoutputsarethentreatedasactiva-
tions for the next layer of neurons. All mathemat-
ical operations are differentiable, and the weights
between computing units are model parameters.
This means that networks can be trained to make
betterpredictionsusingstochasticgradientdescent,
in which the derivative of each weight can be cal-
culated with respect to the output error, and its
value updated accordingly over a number of itera-
tions.Foramoredetailedreviewoftheelementsof
supervised learning and deep neural networks, see
Ref.38.
Theseprinciplesformthebasisofthedeeplearn-
ing paradigm. However, they leave unspecified
the exact connectivity and arrangement of layers
of computing units. Discovering the connectivity,
a r c h i t e c t u r e ,a n do b j e c t i v ef u n c t i o no fan e t w o r k
thatprovidestherightinductivebiasestoallownet-
workstoperformwellisakeypartofdeeplearning
research. The special operation that gives CNNs
theirname,convolution,isonesuchinductivebias.
Typically, the lower layers of CNNs comprise many
local feature detectors, each of which is repeated
many times across the two-dimensional input.
Mathematically, this corresponds to convolution:
sliding a candidate visual pattern across a set of
activations from the layer below, outputting a “fea-
ture map” that shows how strongly the pattern was
detected recorded at each location (Fig. 2). The
pattern might correspond to an oriented Gabor
filter, a surface, or even a face.17,39,40In a typical
CNN, many convolutional filters would be passed
across activation values at each layer, outputting
a stack of feature maps. The key to solving the
classification problem is to learn the weights of
these convolutional filters—that is, the patterns
that the filters are detecting—from the training
data.
Thereareanumberofotherimportantlayertypes
inmodernCNNs.Forclassification,theexactloca-
tionoffeatures(orcombinationsoffeatures)isoften
less important than their presence—a property
termed “local translation invariance.” This allows
as p a t i a ld o w n - s a m p l i n gt h a ti sp a r t i a l l ya c h i e v e d
byanotherspeciallayerknownasa“poolinglayer,”
whichcontainsneuronsthatsimplyoutputtheaver-
ageormaximumofapatchofinputvalues.Another
major concern when training learning algorithms
withmanyweightsistheproblemofoverfitting:the
4 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
Figure 2. AbasicCNNarchitectureusedforimageclassification.(AandB)Convolutionalfiltersaremovedacrosstheactivations
oflayersbelow,outputtingfeaturemaps.(C)AtypicalCNNarchitecture,basedonAlexNet.1Froma32 ×32RGBimage,thefirst
convolutional block learns weights for 32 feature maps, followed by two max pooling layers. The output is a vector of category
probabilities; the category with the maximum of these values is taken as the output label. For computer vision tasks, the filters
thatarelearnedinthefirstfewlayerstypicallycorrespondtomoregeneral,low-levelimagefeatures,suchasorientedGaborsand
coloredblobs.Thedeeperlayerstendtocorrespondtomoretask -specific,high-levelfeatures,suchasfacesorhumanoranimal
figures.17,39,40
algorithm learns a solution that will not generalize
w e l lb e c a u s ei ti st o os p e c i fi ct ot h et r a i n i n gs e ta t
hand.Inmachinelearning,thestatisticaltechnique
of “regularization” is used to protect against this
phenomenon,byeitherincludingatermintheover-
all loss function that penalizes very large weights,
and therefore extreme solutions, or by including
regularizationlayerswithinthenetwork.“Dropout”
layers, which probabilistically drop the output of
some units during training,41,42are often used for
this purpose, meaning the network is forced not
to rely on very specific coactivation patterns, but
instead seek representations that are tolerant to
noiseandbetterdistributedacrossnodes.
In the last few layers of the network, fully con-
nected feedforward layers are often implemented.
This gives extra capacity for representational flex-
ibilityandcross-classcomparison.Finally,theclas-
sification layers consist of a linear decision rule for
eachcategory,convertedintoaprobabilitythroughthesoftmaxfunction:
p/parenleftbig
Categoryi |x/parenrightbig
=exp{x·wi}/summationtext
i/primeexp{x·wi/prime},
where xis the input representation from the final
layer and wiis the weight vector for category i.
Duringtraining,theseoutputprobabilitiesarecom-
paredtotheknownoutputlabel,typicallyusingthe
cross-entropylossfunction,andthenetworkback-
propagates the error between these as the super-
vised learning signal. The description we have
presentedherecharacterizestheprototypicalorper-
haps archetypal CNN, and over the last decade,
t h e r eh a v eb e e nm a n ys u c c e s s f u lm o d i fi c a t i o n so f
thisarchitecturalframeworkandtrainingregime—
for example, allowing lower layers to directly con-
nect to higher ones.43For reviews of more recent
developments,seeRefs.38,44,and45.
Viewedaltogether,theeffectofpassinganimage
signal through a CNN is to learn the nonlinear
5 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
transformation of the input that supports the best
linear separation between classes, with the aim of
learning a mapping that will generalize well to
other,relatedstimuli.Thefeedforwardconnections
betweenstacksoflayersmeanthatateachsuccessive
layer,increasinglyabstractstimulusrepresentations
are formed. As we shall see below, all of these fea-
turesmakeCNNsapromisingcandidateforextend-
ing cognitive models of similarity and categoriza-
tionstonewstimulusdomains.
Integrating CNN representations into models
of higher cognition
One success from the study of higher-level cog-
nition has been the development of a range of
high-precision mathematical models for various
aspects of human behavior,46and in particular a
well-explored set of these models exists for the
fundamental cognitive phenomena of similarity
and categorization.47–49Although these modeling
frameworks have driven an impressive range of
theoretical and empirical developments, they have
been most successful in a narrow range of cog-
nitive settings—namely, human behavior in highly
controlled experimental paradigms involving sim-
ple artificial stimuli.50,51This focus has been moti-
vated by sensible aims. First, using simple and
user-definedstimulifacilitatesthegeneralscientific
method of isolating well-defined aspects of behav-
iorandphysiology,inordertoevaluatetheoriesand
c o m p u ta t i o n a lm od e l sa tbo thth em e c h a n i s t i ca n d
functional levels. Second, most models of higher-
level cognition require as input stimulus represen-
ta ti o n sth a tm u s t“ s ta n di n ”f o ri n a c c e s s i b l em e n tal
ones, and simple artificial stimuli can often be rep-
resented in a straightforward manner. Using CNN
representationstoextendthesemodelstonaturalis-
ticstimulimorerepresentativeofthecomplexvisual
worldis,then,anobviousandattractivetarget.
The classic technique for deriving representa-
tio nsusedincogni tivemodelsism ul tidimensio nal
scaling(MDS),whichseeksaspatialrepresentation
such that the distance between stimuli is inversely
r e l a t e dt op a i r w i s es t i m u l u sc o m p a r i s o n s ,s u c ha s
similarity judgments.52,53The recovered dimen-
sionsareoftenfoundtobehighlyintuitive,andrep-
resentations of stimuli, therefore, psychologically
meaningful.However,MDShastwokeylimitations.
T h efi r s ti st h a ti tr e q u i r e ss i m i l a r i t yj u d g m e n t s
betweenpairsof stimuli, meaning data collectionrapidly becomes unachievable as the desired stim-
ulussetgrows.Thesecondisthatthereisnomech-
anism to generalize the inferred representation to
novelitems,whichusuallycome withoutanaccom-
panying set of similarity judgments with what we
have already observed. Partially as a means of cir-
cumventing these limitations, studies using larger
stimulussetshavedesignedsimpleartificialstimuli
that can be directly represented using a low num-
berofobviousperceptualdimensionsofvariation.54
Ononehand,theseapproachesresultinstraightfor-
ward stimulus representation and precise manipu-
lation of participants’ knowledge of stimuli—both
ofwhichallowexperimentaldesignandanalysisto
b ef o c u s e do np r o b i n gt h ed i ff e r e n c eb e t w e e nc a n -
didate modelingstrategies. On theotherhand, this
meansthatthestimulithathavebeenusedaretyp-
ically far removed from the types that are used for
reasoning about using our natural cognitive abili-
tiesandthatmoststimulateourdesiretomodelthe
mind(Fig.3).
The hierarchy of representations learned by
CNNs for complex naturalistic inputs can help to
closethisgap.Theserepresentationsaretheoutputs
ofintermediatefunctionsinacomposition(i.e.,hid-
denlayers),andthusconstitutemodelsofrepresen-
tation learning—although often only implicitly in
service of the training objective such as classifica-
tion,andnotnecessarilydesignedtomimichuman
perception.Further,becausethesenetworksareable
to learn from millions of data points, and because
they are validated on held out data, they generalize
broadly, and can be used to characterize the con-
tent of diverse images. For example, CNNs trained
exclusivelytoclassifynaturalistic,uncurateddigital
photographsofobjectsconstituteimpressiveexpla-
nationsofhumanshapesensitivityfornovel,artifi-
cialshapestimulithataredevoidofcontext.55Taken
together, these properties provide a compelling a
prioriargumentthattrainingsuchnetworkstosolve
human-relevanttasksoverlargedatasetsofnatural-
isticstimulimayproducegeneralpurposerepresen-
tations,notunlikehumanones,thatcanbeusedas
a basis for cognitive modeling. However, just how
similartheserepresentationsaretohumanpsycho-
logical representations, and the nature of their dis-
crepancies,isonlyrecentlycomingintofocus.
A first empirical indication that CNN represen-
tationsmightbeofcognitiveinterestinthiscontext
camefromastudybyLake etal.,whoexaminedhow
6 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
Figure 3. Representativestimulifromseminalpsychologicalstudiesof categorization.Toprow:typicalartificialstimulirepre-
sentativeofthoseusedintraditionalstudiesofcognition.Bottomrow:themathematicalrepresentationofthesestimulithatare
inputintocognitivemodels.ReproducedfromRef.50.
well CNN activations at different layers predicted
human category typicality ratings on novel images
fromeightImageNetcategories.56Theytestedthree
CNNs pretrained on ImageNet against the state-
of-the-art classifier from computer vision prior to
CNNs (scale-invariant feature transform, “SIFT”
features57derivedfromImageNet,followedbysup-
portvectormachine,“SVM,”classifiers).58Thebest
predictive performance came from the final clas-
sification layer, where the average category acti-
vations from CNNs correlated very highly with
human typicality ratings. The SIFT +SVM base-
line, by contrast, scored poorly. Furthermore, for
the categories where the CNNs performed poorly,
there was some indication that this was due to the
training image distribution from ImageNet being
particularly skewed, and not as representative of
t h er e a lw o r l d .F i n a l l y ,t h ep r e d i c t i v ea b i l i t yo f
CNNsdecreasedmonotonicallywithdistancefrom
this final layer, with the shallowest layers uncorre-
lated to the human judgments, indicating that, like
human learners,59CNNs learn to use complex and
category-specific features for natural stimuli as the
basisofcategorytypicalityjudgments.
In the remainder of this section, we summarize
morerecentresultsevaluatingCNNrepresentations
in models of similarity and categorization. In gen-
eral, we find that while CNN representations per-formsurprisinglywellincapturinghumanbehavior
when used in conjunction with traditional cogni-
tivemodels,theycanalwaysbemadebetterbysim-
ple mathematicaltransformations.When relatedto
eachother,thesemethodsconstitutethebeginnings
of a modern toolkit for representing more natural-
isticstimuli,basedondataandtheoryfromthecog-
nitivesciences,statistics,andcomputerscience.
Similarity
Judgments of stimulus similarity have long been
of interest to psychologists, given both their intu-
itiveimportancetonumerouscognitiveprocesses60
and their apparent correspondence to law-like
human generalization behavior.61However, their
most enduring influence in the field comes from
their utility in revealing the structure of psycholog-
ical representations —the comparatively high-level
internal representations of external stimuli that are
initially processed as raw sensory data. The pre-
cise structure of these representations is impor-
tant to understand, because it shapes the down-
stream inferences that they support. For example,
representing animals as a function of their rela-
tive size and color may lead to considerably dif-
ferent generalizations than representing them via
their muscle mass and position on the food chain.
7 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
Similarity judgments provide strong clues as to
which representations people might be employing,
whichareotherwisenotdirectlyobservable. More-
o v e r ,c o m p a r e dt om e a s u r e m e n t so fn e u r a la c t i v i t y
in visual brain areas, human similarity judgments
exhibit additional structure and comparably little
measurement noise.62Previous work often makes
the assumption that similarity is a relatively sim-
ple metric that operates over pairs of stimuli in the
representational structure (e.g., Euclidean distance
between points in a metric space or Hamming dis-
tance for nodes in a hierarchy). Holding a given
metric fixed, algorithms, such as multidimensional
scaling,agglomerativeclustering,andadditiveclus-
tering, allow researchers to infer continuous and
discrete feature representations that best explain
patternsofobservedhumansimilarityjudgments.63
Applying these methods has been highly produc-
tive, often revealing crucial latent mental structure
in sets of stimuli, and providing relatively unbi-
ased candidates for stimulus representations that
helpexplaindownstreamcognitiveprocesses.Since
these methods infer representations directly, they
do not require a model of representation learn-
ing (i.e., an explanation of how the representations
cametobe,orwhy).Theadvantageofthisapproach
is that it allows researchers to immediately begin
to further study the processes that operate over
these representations; but it is also a disadvantage,
becauseitlacksexactlythatlevelofexplanation.
To begin to evaluate the precise correspondence
between representations from CNNs and humans,
Peterson et al.compared similarity scores (inner
products) for pairs of images in the hidden-layer
feature space of a CNN to average pairwise simi-
larity ratings from people.64Across six sets of 120
images (720 in total) and over more than 400,000
total judgments, CNN representations were found
to give a much better account of human behav-
ior than traditionalcomputer vision methods, cap-
turing approximately half of the explainable vari-
ance in similarity ratings for five out of six image
sets.Whilegivingareasonablygoodoff-the-shelffit,
whatseemedtobemissingintheCNNrepresenta-
tions was the taxonomic structure humans employ
(e.g., making a pronounced distinction between
primates and nonprimates in the case of images
of animals), as revealed by hierarchical clustering
(Fig. 4). Next, Peterson et al.showed that learn-
ing a set of dimensional weights that are applied tothe inner product calculation in the CNN feature
spaceproducedsimilaritiesthatcapturednearlyall
of the remaining explainable variance in human
judgments in most cases.64This formulation fol-
lows one long tradition in cognitive psychology of
modelingsimilarityastheresultofacomparisonof
objects in terms of a set of features,65–67expressed
inmatrix-vectornotationasfollows:
S/parenleftbig
x,y/parenrightbig
=xTWy,
where xandyarevectorsoffeaturesassociatedwith
twostimuli, xandy,and Wisadiagonalmatrixthat
servestoweighteachdimensioninthecomparison.
Learning these weight parameters can be thought
of as improving the similarity metric that oper-
ates over these representations in a way that bet-
termimicshumanjudgments—inparticular,apply-
ingattentionalweightstoeachfeature.Whenthese
weights are constrained to be non-negative, it can
also be thought of as a fixed linear transformation
(axis-aligned scaling) of the original feature space,
producing a modestly altered representation. The
morehuman-likesimilaritiesobtainedthroughthis
method also largely reproduced the latent hierar-
chicalstructurethatwasnotpreviouslyobservedin
theCNNrepresentation(Fig.4).Sincethismethod
wasvalidatedonout-of-sampleimagepairs,itpro-
vides one of the first generalizable formulas for
automatically producing human-like psychological
representations for arbitrary naturalistic images:
correct the CNN representation once, then apply
thelearnedtransformationtoallfutureimagesthat
areinputtothenetwork.
Several subsequent studies have extended this
method to a wider range of mathematical trans-
formations, and in each case shown that doing so
allows CNN representations to afford better and
more interpretable models of human behavior.
Attarian et al.showed the fit to human judg-
ments can be improved further by allowing the
transformation matrix, W,t ot a k ei n c r e a s i n g l y
unconstrained forms.68They applied these trans-
formationstoadatasetofsimilarityjudgmentsover
natural images of birds69that take a slightly dif-
ferent form from the absolute ratings described in
thestudiesabove:givenaqueryimage,participants
m u s ta n s w e rw h i c ho ft w o( o rm o r e )r e f e r e n c e
imagesismostsimilar.Usingthenonexponentiated
Luce–Shepard choice rule,52,70the probability of
8 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
Figure 4. Transforming CNN representations using similarity judgments. (A) Representations of images derived from human
similarityjudgmentsusingMDSexhibitmeaningfulvariationandsegregation(leftpanel).UsingMDStoexaminethesimilarity
structure of raw CNN representations shows they fail to capture these relationships (center panel). Improving the fit of CNN
representationstohumansimilarityjudgmentsrecoversthisstructure(rightpanel).(B)Dendrogramsofimagerepresentations
also display meaningful hierarchical categorical structure (top panel) that is not present in raw CNN representations (middle
panel) but that is recovered by modifying them using the learned similarity transformation (bottom panel). Reproduced from
Ref.64.
anysinglejudgmentbecomesasfollows:
p/parenleftbig
yismoresimilarto x|x,y,z/parenrightbig
=S/parenleftbig
x,y/parenrightbig
S/parenleftbig
x,y/parenrightbig
+S(x,z).
Attarianet al.modeled each similarity compari-
sonasfollows:
S/parenleftbig
x,y/parenrightbig
=f(x)TWf/parenleftbig
y/parenrightbig
,
where xandyare CNN representations of images
xandy,a n df(·) is a linear function that acts to
reduce their dimensionality. The authors use prin-
cipalcomponentsanalysis(PCA)forthiscompres-sion,suchthatthetransformationmatrixactsonthe
compressedspaceof kdimensions(i.e., W∈Rk×k).
By choosing different constraints on W,A t t a r -
ianet al.allowed a range of linear transforma-
tions on the compressed CNN representations.68
They compare the untransformed but compressed
CNN representations ( W=I), a dilation of com-
pressed representations encoded in non-negative
diagonal matrix characterized by a vector of diago-
nalcomponents, w,withandwithoutregularization
(W=diag(|w|)), a symmetric matrix ( Wij=Wji),
and a full, unconstrained matrix. In general, they
9 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
Figure 5. Increasing the flexibility of the linear transforma-
tion of CNN representations improves fit to human similar-
ityjudgments.Astheconstraintsonthetransformationmatrix
arerelaxed(seelegend,bottomtotop),theaccuracyofmodel
predictionsincreases.Increasingthenumberofprincipalcom-
ponents used to represent the compressed CNN representa-
tionsalsoimprovesperformanceandwidensthegapsbetween
modelsubtypes.Errorbarsrepresent ±1SEMoverfivecross-
validationfolds.ReproducedfromRef.68.
findthateachofthesesuccessiverelaxationsallows
for a closer fit to the human judgments (Fig. 5),
replicating and extending the findings in Ref. 64.
Within these data are two other findings of note.
The first is that regularization did not significantly
i m p r o v et h ep e r f o r m a n c eo ft h ed i l a t i o n ,a n dt h e
more complex models performed well on held-out
triples.Thisindicatesthesetransformationsdonot
leadtooverfittingdespitelargeratiosofparameters
to judgments—perhaps because of the functional
restrictionstolineartransformations.Thesecondis
that allowing similarity judgments to be asymmet-
ri cs i gn i fi c a n tl yi m p r o v e sth em od e l .T h i si si nl i n e
with the same finding from classical debates about
thesimilarityinthepsychologicalliterature.65
Jhaet al.proposed a model that jointly reduces
the dimensionality of CNN representations and
allows for more complicated linear transforma-
tions by using a bottleneck layer.71This lower-
dimensional, fully connected linear layer is added
on top of the final representational layer of a pre-
trained CNN. It serves to learn a projection of
images into a lower dimensional space such that
the similarity between two images is given by the
innerproductoftheselowerdimensionalandtrans-
formedrepresentations.Aseverycomponentisdif-
ferentiable, it can be trained by backpropagation
usingsimilarityjudgmentsthemselves,allowingthedimensionality reduction to preserve information
strictly relevant to similarity comparisons. This is
incontrasttoPCA,whichaimstocapturethevari-
a n c eo ft h eo r i g i n a ld a t ai na nu n s u p e r v i s e dm a n -
ner. In the context of the similarity transformation
above, we can think of this as transforming and
reducing the CNN representation space according
tosomek×ddimensionalmatrix, V,wheredisthe
dimensionalityoftherawCNNrepresentation(i.e.,
V∈Rk×d).
UsingthedatasetsoriginallygiveninRef.64,Jha
etal.foundthatsurprisinglyfewbottleneckdimen-
sions are needed to offer a good approximation
of human similarity judgments.71As the number
of bottleneck dimensions increases, performance
approaches that given by the full transformation
overthe212-dimensionalspaceoftheoriginalCNN
representation. However, this increase is not lin-
ear,andanelbowtoperformancebenefitsisseenat
around24–26dimensions(Fig.6,topleft).ThePCA
projections almost always perform worse than the
bottleneckapproach,corroboratingtheideathatthe
information relevant to similarity judgments is an
important but not exclusive component of higher-
levelCNNrepresentations(Fig.6,topright).
Varyingthesizeofthebottlenecklayeroffersthe
opportunity to explore what information from the
CNN representations is used to model similarity
judgments at different levels of granularity (Fig. 6,
middle). Jha et al.found the principal components
ofthebottleneckrepresentationsreflectedintuitive
dimensions of variation in each original dataset,
for example, separating images based on broad
and continuous categorical distinctions, such as
the animacy of objects, or the number of wheels.71
Dendrograms constructed from bottleneck repre-
sentations revealed that increasing the size of the
bottleneck introduces finer levels of distinction
(Fig. 6, bottom), mirroring the hierarchical aspect
ofclusteringobservedinhumancognitionandthat
found by parallel methods for the interpretation of
CNNrepresentations.72
O n ew a yt ov i e wt h em e t h o d sd e t a i l e da b o v ei s
as conducting fine tuning and transfer learning.39
CNNs are first trained on large naturalistic image
datasets, in order to ensure they learn relevant
and general image features and do not overfit,
and then these features are modified using sim-
ilarity judgments to better highlight or extract
the statistical information humans rely on when
10 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
Figure 6. Exploringtheeffectofdimensionalityreductiononmodelingsimilarityjudgments.Toprow:CNNrepresentationsfor
anumberofimagedatasetswerereducedbyusingsimilarityjudgments(left)orPCA(right).Performanceusingallrepresenta-
tionsandasimpledimensionalreweightingareshownasadashed lineforeachdataset.Middlerow:fixingasmallbottlenecksize
andprojectingbottleneckrepresentationsusingPCAallowsinterpretationoftheinformationbeingencodedbythenetworkand
extractedforthesimilaritycomparison.Bottomrow:dendrogramsfortheanimaldatasetbasedonrepresentationsfrombottle-
necksizesoftwoandsixshowtheCNNrepresentation,andreductioncapturessimilarityinformationaccordinginahierarchical
manner.H,herps;B,birds;P,primates;R,rodents;WC,wildcats;G,grazers;E,dogs,bears,andlargeanimals.Reproducedfrom
Ref.71.
11 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
m a k i n gt h es a m et y p e so fc o m p a r i s o n .As e c o n d
set of methods for refining CNNs in this man-
ner has adapted pretrained CNNs to model MDS
coordinates for images, derived from similarity
judgments.73–75I ft h eC N Ni sa b l et ol e a r naf a i t h -
ful and psychologically meaningful MDS embed-
ding, it can then be used to generate approxi-
mate coordinates for potentially infinite numbers
of novel stimuli, overcoming the inherent lack of
generalization from traditional MDS. In two stud-
ies, Sanders and Nosofsky73,75examined similarity
judgments on a dataset comprising 360 images of
subtypes of igneous, metamorphic, and sedimen-
tary rocks.76First, they pretrained models on Ima-
g e N e tt od e r i v eb r o a d l yg e n e r a l i z a b l eb a s ef e a t u r e s
andpreventoverfittingtotheirownsmallerdataset.
Then, they generated a set of MDS coordinates for
a subset of geological images, replaced the upper-
most layers of CNNs with unlearned weights, and
restarted training using the MDS co-ordinates as
output targets for each image. If only the new lay-
ers are trained, this is known as “transfer learn-
ing”; if all layers are trained with a small learn-
ingrate,“fine-tuning.”39Aftertraining,theirCNNs
produced approximate MDS coordinates, and in
bothcases,theauthorsfindareasonablyhighcorre-
lationbetweentheCNN-andhuman-derivedMDS
coordinates for a held out set of images. In the
second study, Sanders and Nosofsky assessed the
generalizability of the raw MDS space and found
that the majority of MDS dimensions had a high
degree of subjective and objective correspondence
with a second MDS space for a novel set of images
fromthesamegeologicalcategories.75Theremain-
ingdimensions,however,werelesseasytorelateand
exhibitedlower correlationscores. TheCNN-MDS
approximationsalsoshowedreasonablyhighcorre-
lations with the “true” MDS coordinates over the
four dimensions that were best preserved between
original and novel image sets. The approximations
were,however ,quitepoorfortheremainingfour .
T h em e t h o d so u t l i n e da b o v eh a v eb e e na b l et o
use behavioral data related to similarity to fine-
tune CNN representations such that they better
reflect human similarity structure. Peterson et al.
investigated whether CNNs could be fine-tuned in
thismannersimplybyusingmoregeneralcategory
labels.77T h eI m a g e N e tl a b e l st h a tC N N sa r en o r -
mally trained on correspond to Rosch’s “subordi-
nate” level.78,79These are relatively specific, com-prising, for example, breeds of dog (e.g., “dalma-
tian”). Peterson et al.trained CNNs on a subset
of these labels for which the super-class label was
alsoprovided,80labelsthatcorrespondedtoRosch’s
“basic” level (e.g., “dog”).77Forcing the CNN to
learnatransformationthatgroupedstimuliaccord-
ing to both levels of this hierarchy was enough to
lead to representations that both captured human
similarity judgments better and exhibited much
morestructuredandhuman-liketaxonomichierar-
chies. The authors also found that CNNs that were
trained or supplemented with basic-level super-
vision provided a stronger match to the “basic-
level bias” described in human generalization, so
called because given a new label and an associ-
atedimage,peoplewilltendtogeneralizethislabel
to other images at this level of description.59,79As
withrecoveringtheshapebiasdescribedinRef.55,
ensuring that CNNs reproduce such biases is yet
another avenue for assessing and improving their
correspondencewithhumans.
Theabovestudiesprovideevidencethat,beyond
comparable task performance, CNNs themselves
m a yp r o v i d eau s e f u ls o u r c eo fi m a g ef e a t u r e st h a t
c a nb eu s e dt oc a p t u r ea s p e c t so fh u m a np s y c h o -
logical representations of naturalistic images. They
also demonstrate that cognitively motivated strate-
gies for modifying these representations can lead
to better accounts of human behavior. Developing
these strategies and integrating them more directly
with the feature learning stage are important next
steps for these frameworks, as well as examining
important failure cases, in which the abstract cor-
respondencesbetweenstimulithathumansidentify
arenotwellcapturedbyCNNs.81,82
Categorization
A promising second candidate for the direct
integration of CNN representations is the study
of categorization—perhaps the most fundamental
cognitive phenomenon following from similarity.
Theproblemofcategorizationcanbeformulatedas
d e c i d i n gh o wt oa s s i g nan o v e ls t i m u l u st oan e w
or existing category, and it makes intuitive sense
that this is in turn related its similarity with pre-
viously encountered category members. Although
it seems like something has been lost—now only
as u b s e to fs t i m u l ii sb e i n gc o n s i d e r e df o ra n y
given comparison—categories add structure that
12 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
allows us to model within- and between-category
interactions, and therefore a richer and more pre-
cise range of generalizations. Categories form the
basisofmostcognitiveframeworksforunderstand-
ing the brain and mind, and virtually all computa-
tional paradigms begin with a definition or expla-
nationofthem.83
Theories of categorization began with Plato and
Aristotle, who posited that categories were best
represented by a single ideal form or all previ-
ouslyencounteredmembers,respectively.84,85More
recently,thestudy ofcategorizationhastrackedthe
development of the cognitive sciences in general,
beginning with the idea that people use rules or
definitionstocategorizestimuli.86,87However intu-
itivethisideaseems,itwaschallengedinthe1970s
by the work of Eleanor Rosch and colleagues, who
showed that natural categories often lack central,
definingfeaturesandexhibitgradedcategorystruc-
ture better characterized by a relation of family
resemblance.88–90Strongevidenceinsupportofthis
viewcamefromanumberofempiricalfindings,in
whichastimuluscouldbemoreorless“typical,”59,79
unseenprototypical stimuli were preferred to other
unseen stimuli from the category during recogni-
tion tasks,91and that some levels of label (e.g., the
“subordinate” level) worked better than others in
explainingcategoryjudgments.78
Up until recently, the role of cognitive model-
inghasbeentoproposehowthesimilaritybetween
a stimulus and category should be computed in
order to capture these behavioral effects, both in
terms of the representation of existing category
members and the formulation of the comparison
itself.Twoinfluentialmodelingstrategieshavebeen
developed in this context. The first, known as a
prototype model,assumesthatcategoriesarerepre-
sented by their average or central tendency.92The
second,knownasan exemplar model,assumesthat
categories are represented by all of their known
members.93,94These strategies can be unified by
identifyingthecomputationaltaskofcategorization
asprobabilitydensityestimation,48andinparticular
asinferringtheprobabilityaparticipantwillchoose
ap a r t i c u l a rc a t e g o r yl a b e lg i v e na ni m a g e .T h i s
morepreciseframingledtothefine-grainedexplo-
ration of model predictions in laboratory settings,
using sets of artificial stimuli that were designed
to differentiate model performance, with exemplar
modelsoftenwinningout.92–105Sincethen,extrap-olating the full probabilistic reframing of catego-
rization has resulted in a number of other statis-
tical strategies that can account for human behav-
ior in a wider range of settings, such as learning
mixture density estimators composed of a number
of (sub)prototypes,106,107sharing exemplars across
categories,49and allowing the categorization strat-
egy to adapt flexibly to the number of stimuli
observed.24,49
Battleday etal.50investigatedwhetherthesemod-
els could be extended to more naturalistic settings
byusingCNNrepresentationstoexaminetheirpre-
dictions over a wide range of natural images. They
first collected over 500,000 human categorizations
on 1000 images from each of the 10 categories in
thetestsubsetofCIF AR-10,108abehavioraldataset
they call CIFAR-10H. For each image, they gener-
ated a range of machine learning representations
(Fig. 7), including raw pixel representations, fea-
tures derived from computer vision and engineer-
ing (histograms of oriented gradients; HOG), the
activationsofthelatentlayerofagenerativeadver-
sarialnetworkthatusesconvolutions(BiGAN),and
theupper-layeractivationsoftwoCNNs—AlexNet1
andDenseNet.43Finally,theygeneratedpredictions
from a range of prototype and exemplar models
using the machine learning representations. These
strategies draw on a second influential tradition
in modeling similarity: this time as an exponen-
tially decreasing function of distance in some psy-
chologicalspace.61Prototypeandexemplarmodels
specify this similarity calculation mathematically,
and convert it into a probability by relating it to
the inverse distance to the category prototype or
summedinversedistancestoallcategorymembers,
respectively:
S/parenleftbig
y,C/parenrightbig
Prototype=exp−d/parenleftbig
y,μc/parenrightbig
,
S/parenleftbig
y,C/parenrightbig
Exemplar=/summationdisplay
x∈Cexp−βd/parenleftbig
y,x/parenrightbig
,
whereCis a category, yis a stimulus, yis the mul-
tidimensional representation of that stimulus, and
S(·,·)isoursimilarityfunction.Thisrepresentation
is compared with the category mean for prototype
models or all known category members for exem-
plar models. The similarity function, S,i sadditive:
if a category is represented by a vector of stimuli
(as in the exemplar model), Scomputes the sum of
13 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
Figure 7 . Thefeaturebasisusedformodelingcategorizationsofnaturalstimuliaffectsoverallmodelperformancemorethancate-
gorizationstrategy.Toprow:two-dimensionallineardiscriminantanalysisprojectionsoftherepresentationsfromeachcomputer
visionmethod.Thefeaturebasesacrossthex-axisroughlytrackthedevelopmentofcomputervision:rawpixels,hand-engineered
features(HOG),thelatentspaceofagenerativenetworkthatusesconvolutions(BiGAN),andabasic(AlexNet)andmoreadvanced
(DenseNet) CNN. Bottom row: categorization models using different prototype and exemplar strategies were trained on each of
thesefeaturebases,withmodelflexibilitybeingmoreobviouslyrelatedtooverallmodelperformancethancategorizationstrategy
(i.e.,prototypeorexemplar).BaselineswereprovidedbytakingthesoftmaxprobabilitiesfromthefinalCNNlayerasthesimilarity
measurement.ReproducedfromRef.50.
similaritiesbetween yandtherepresentation, x,for
each stimulus in C. Finally, the exemplar model
contains a “specificity” parameter, β,t h a ts c a l e s
all distance calculations by the same amount prior
to exponentiation. This acts to sharpen or lessen
the influence of exemplars on subsequent category
judgments, and therefore allow the model to con-
trolsforoverallstimulusdiscriminabilityintherel-
evantpsychologicalspace.94
Battleday et al.50found that CNNs provided
the best representational basis for modeling the
human categorization judgments, outperforming
deepunsupervisedandtraditionalcomputervision
methods.Indeed,thechoiceofstimulusrepresenta-
tionaffectedoverallperformancetoamuchgreater
extentthanthechoiceofcategorizationmodel.This
is particularly interesting given the main focus in
categorizationmodelinghasbeenthecategorization
strategy, whereas the unambiguous nature of sim-
pleartificialstimulihasallowedtheirrepresentation
to remain fixed. The loss of such neat factor-
ization when analyzing naturalistic stimuli estab-
lishesaneed—atleastinitially—tofocusontheless
well-integrated question of perceptual representa-tion. When restricting analysis to the CNN rep-
resentations, categorization models with more free
parameters—andthereforemoreflexibilitytotrans-
form CNN representations—performed the best,
and better than competitive baselines. This indi-
cates that the CNN representations also naturally
contain latent information relevant to the kinds of
features humans use to make category judgments,
whichcanbefurtherrefinedusingsimplelinearand
quadratic transformations. Another unexpected
finding was that prototype and exemplar models
performedroughlythesameacrossavariedrangeof
imagerepresentations,contrastingwithwhatmight
be expected based on previous laboratory work in
whichexemplarmodelsareneededtocapturearti-
ficialcategorieswithmorecomplexstructure.
Thereareknown cases in which prototype
a n de x e m p l a rm o d e l sm a k es i m i l a rp r e d i c t i o n s ,
for example, if category representations are well-
captured by simple Gaussians, but how these situ-
ationsrelatetohigh-dimensionalrepresentationsof
complex stimuli remains unclear. Battleday et al.50
conductedasimulationstudythatinvestigatedhow
the number of dimensions and training samples
14 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
affected the different modeling strategies. They
foundthatwhileindeedinlow-dimensionalspaces
exemplarmodelsoutperformprototypemodels,no
such difference exists in high-dimensional repre-
sentational spaces after training with very large
numbers of stimuli. This is the regime in which
CNN-based representations of naturalistic images
allowustooperate.Furthertheoreticalinvestigation
is needed to frame the import of these depen-
dencies, as well as to make precise the relation-
shipbetweenthesizeandnatureofrepresentational
space.
An u m b e ro ft h es t u d i e si n v e s t i g a t i n gC N N s
in the context of similarity outlined above have
used categorization as a downstream application
to test the utility of the representational spaces
their methods derive. For example, Peterson et al.
noted that people also found it much easier to
learn novel categories of natural images when
these were constructed based on the similarity
between CNN representations rather than simi-
larity between HOG/SIFT computer vision engi-
neering features.64When these CNN representa-
tions had been transformed to better align with
human similarity data, categories were even eas-
ier to learn. Second, Sanders and Nosofsky exam-
inedcategorizationsforaheld-outsetofdatataken
over 120 novel images of geological samples, taken
from the same 30 categories that they used to
train CNNs to approximate MDS coordinates (see
above).73When a low-parameter exemplar model
was supplied with the CNN-MDS representations,
it was able to model these human categorizations
v e ryw e ll .I nafu rth e rs t u d y ,Sa n d e r sa n dN oso fs k y
also tested an exemplar model on this categoriza-
tion data, but this time compared its performance
using a number of CNN representations,including
asetmodifiedtobetterreflecthuman(dis)similarity
judgements.75TheyfoundtheCNN-derivedMDS-
approximations allowed much better performance
thanthesimple,off-the-shelfCNNimagerepresen-
tations, and could be further improved by train-
ing the CNN to also predict “missing” dimensions,
appendedtothenormalMDSco-ordinates.76
A convergent set of findings comes from the
closely related field of object recognition. Annis
et al.109showed that CNNs pretrained on Ima-
geNet generated intuitive, correctly clustered, and
robustrepresentationsofentirelynovelexperimen-
tal objects, including Greebles,110Ziggerins,111andSheinbugs .112These more complex artificial stimuli
exhibitnaturalisticqualitiesandaredifficulttorep-
resentbyafewobviousdimensionsorMDS,andyet
ensure participants have no prior experience with
the object category. Annis et al.then used these
CNN representations to model a set of data113in
which participants had to decide whether a second
object,possiblyrotatedorscaled,wasthesameasa
first,separatedbyadelay.109Thesimilaritybetween
thefirstandsecondobjectswasusedastheinputtoa
hierarchicalBayesianevidenceaccumulationmodel
inordertomodelparticipantsjudgmentsandreac-
tiontimes.Aswiththecategorizationstudiesabove,
Anniset al.found that using CNN representations
together with the cognitive model outperformed a
number of baselines, that the choice of CNN rep-
resentation was the major determinant of perfor-
mance, and that the best-performing model could
beimprovedbymodifyingtheCNNrepresentations
to also be robust to rotations. They also explored a
rangeofmethodsforfindingtransformation,com-
plementing the analysis conducted by Refs. 64 and
68forsimilarity.SeealsoRef.114,foranothersuc-
cessfulapplicationofCNNsandevidenceaccumu-
lation models, this time in modeling the decisions
and reaction times of novices and experts in medi-
calimageclassification.
T h e r ea r ean u m b e ro fo t h e rs t u d i e st h a ti n v e s -
tigate aspects of the relationship between image
categorization, cognitive models, and CNNs. As
withcategorytypicalityjudgments,56thepredictive
benefit of CNN representations for modeling cat-
egorizations was found to increase with increas-
ing depth.115On the other hand, representations
from shallower CNN layers provided the best basis
for capturing visual biases observed in in particu-
larpigeonsclassifyingcardiograms,116suggesting a
roleforCNNsincomparativestudiesofvisualcate-
gorizationacrossothertypesofvisualsystem.
Along with typicality and similarity, CNNs
appear our best available source of representations
for modeling behavior over naturalistic stimuli
in another fundamental cognitive domain—
categorization. In general, studies have found two
approachessuccessfulinimprovingtheserepresen-
tationsfurther:adaptingthemtobetterfitsimilarity
judgments beforeapplying categorization models,
or by designing flexible categorization models that
transform CNN representations further during
parameter fitting. More generally, it appears the
15 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
role of stimulus feature learning can no longer be
withheld from formal accounts of categorization
overmorecomplextypesofstimuliandconsidering
both together will be necessary for characterizing
human category judgments. This echoes a long-
accepted perspective in the machine learning
literature and has been called for previously in the
cognitivesciences.117
Improving deep neural networks using
concepts from cognitive modeling
In the studies above, CNNs have been primarily
usedtoproviderepresentationsofstimulitowhich
traditional models of higher cognition can then be
applied. We have also seen that it has always been
possible to make these representations better by
transforming them to fit held-out behavioral data.
InapplyingandmodifyingCNNrepresentationsfor
this new set of problems, a number of insights and
models have been developed that are also valid to
themachinelearningandcomputervisioncontexts
in which CNNs were originally developed. In this
section,wediscusshowthisknowledgecanbeinte-
grated back into normal CNN training paradigms
inordertomakethembettercognitivemodels,with
thesimpleaimofimprovingtheirperformance.
A central goal of the transformations above was
tobetteremulatethesimilaritystructureandgraded
categorymembershipassumedbycognitivemodels
and exhibited in human categorization data. This
type of structure has been a central and organiz-
ing feature of categorization theory in general and
hasbeenfoundtowidelyapplytonaturalcategories
ingeneral,includingthetypesofstimuliCNNsare
typically applied to.59,79,88,89In the field of natural
imageclassification,however,emulatingsuchstruc-
ture is not normally viewed as a priority. Rather,
the “top-one” accuracy of a classifier is prioritized,
achieved through associating a “ground-truth” or
“hard”labelwitheachimagethatassignsittoasin-
glecategorywithnoroomforuncertainty;theCNN
must learn a feature space that supports such n-
ary discrimination. This modeling choice has until
nowbeenunderstandable—untilrecently,thechal-
lenge for classifiers was simply getting the most
likely image label right. It is only now, when CNN
accuracy is beginning to asymptote at near-human
performance, that secondary features of CNN per-
formance are beginning to come under scrutiny—
for example, their poor generalization to out-of-training-distribution images, and their fragility in
thefaceofadversarialattacks.118–122Therehasbeen
b r o a da p p r e c i a t i o nt h a tt ot a c k l et h e s ec h a l l e n g e s
image classification models must be better able to
deal with noise, and one means of achieving this
is by endowing them with better perceptual mod-
elsoftheworld.Indeed,thereissomeevidencethat
whenhumanparticipantsaretrainedtolabelstim-
uliwithsinglecategoriesbasedonstimulusfeatures,
as opposed to making predictive inferences about
features or building a generativemodel of the data,
they are less likely to learn the kinds of category
abstractionthatarethoughttounderliegradedtyp-
icalityjudgments.56,123,124
Aprimemotivefortryingtointegratecognitively
motivatedcategorystructurebackintoCNNlearn-
ing paradigms is, then, that it may help provide a
solution to this new frontier of problems. Peterson
et al.125attempted to do so indirectly by training
a range of CNNs to predict the human categoriza-
tionjudgments(“human”or“soft”labels)fromthe
CIFAR-10H dataset. They found that these CNNs
performed significantly better on a number of out-
of-sample natural image sets than CNNs trained
o nC I F A R - 1 0 ,w h i l es c o r i n gt h es a m eo nt h eh a r d -
label validation set (Fig. 8, left and center). As test
imagesweredrawnfromincreasinglyout-of-sample
datasets(i.e.,decreasinglysimilar),thebenefitfrom
usinghumanlabelsincreased,indicatingthesenet-
works were learning more perceptually relevant
category distributions. This was also reflected in
their higher second-best accuracy scores: whether
their second-best guess for an image corresponded
to humans (Fig. 8, right). The authors also sub-
jected these networks to various forms of adver-
sarial attack and found that CNNs trained with
humanlabelsweremuchmoreresilient.Itappeared
as though training with human labels endowed
networks with more tolerance to noise and more
graceful degradation: exactly the current aims of
t h ec o m p u t e rv i s i o nc o m m u n i t y ,a n dt h o s ep r o p -
erties originally sought by early artificial intelli-
genceresearchers.Finally,Peterson etal.125showed
that CNNs trained with human labels invariably
performed better than alternative strategies, which
either incorporate random label noise or train on
convex combinations of image-label pairs.126This
showsthatthestructurecontainedinhumanlabels
is helpful for classification beyond the regulariza-
tioneffectsofaddingtrainingnoise.
16 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
Figure 8. ImprovingthegeneralizationabilitiesofCNNsusinghumanuncertainty.Astestimagescomefromincreasinglyout-
of-training-sample distributions, CNNs trained on soft labels derived from human uncertainty increasingly outperform their
traditional hard-label counterparts(interms ofaccuracyandloss). Thedifference indistributional benefits tousinghardlabels
isreflectedintheconsistentbenefitsregardingsecond-bestaccuracyscores.ReproducedfromRef.125.
As e c o n da r e aw h e r ec o g n i t i v em o d e l sh a v e
inspiredsuccessfulcomputervisionmodelsisinthe
context of few-shot, zero-shot, and semisupervised
learning—allvariantsofimageclassificationtasksin
which CNNs are trained with a much smaller sub-
setofthedatathannormal.Thesereductionsinthe
sizeandcoverageofthetrainingsetmeansstronger
ormoresuitableinductivebiasesmustbebuiltinto
CNNs in order to prevent overfitting and make
successful generalizations from fewer data. Here,
t h es t r a t e g i e st h a th a v eb e e ns u c c e s s f u l directly
incorporate graded category structure into their
trainingparadigmbyusingprobabilisticconstraints
on a CNN’s classification function, many of which
comedirectlyfromtherangeofcategorizationmod-
elsreviewedabove.
In each episode of a few-shot learning task, a
classifier must identify the label of a query image
given a support set of image-label pairs. Within
the support set, there are a fixed number of exam-
ples from a subset of possible categories, and the
classification algorithm must construct categorical
knowledge from this small number of examples in
order to identify which category the query image
belongs to. The number of “shots” refers to the
number of training examples per category, and the
“way”referstothenumberofcategories:forexam-
ple, “one-shot five-way” means identifying which
of five possible category labels best applies to the
query image based on one labeled example from
eachcategory.127,128Bytrainingovermanyofthese
episodes,asuccessfulclassificationalgorithmlearns
howtousetheexamplesinthesupportsetstogen-
eralize categorical knowledge to the query exam-
pleovermanydifferentcombinationsofimagesand
categories.Intheprototypicalnetworks(PNs)ofSnell etal.,
isotropicGaussiancategorydistributionswithiden-
tity covariance matrices are used over final layer
representations of CNNs in order to classify novel
stimuli.129By representing categories in this man-
ner, PNs reduce classification to the calculation of
(Euclidean)distancesbetweencategoryprototypes,
drawingonanequivalencebetweenprototypemod-
els and Gaussian classifiers first described in Ref.
48. PNs use a standard CNN to transform images
into deep representations in the regular manner,
such that the CNN weights are shared across sup-
port set members and episodes. The category pro-
totype for each episode is the empirical average of
labeled category in the support set, and the role
of the CNN during training is to learn the best
transformation to support prototype construction.
Simply adapting the final-layer comparison in this
wayledtostate-of-the-artperformanceonfew-shot
classification over the Omniglot character recogni-
tion task,127mini-ImageNet,128a n de v e nf o rz e r o -
shot learning—in which only a vector of metadata
isgivenaboutcategories,withnoexamples—onthe
CaltechUCSDbirddataset.130
Following the development of cognitive catego-
rization models as increasingly complex strategies
for probability density estimation, Scott et al.were
able to improve on PNs by using a wider range
of distributions to model category structure, an
approachtheycallstochasticprototypeembeddings
(SPE).131Each image was still used to generate a
deep CNN representation that would be combined
to model a prototype over each of the categories in
the support set. However, Scott et al.also learned
an additional output representation for each image
to model category variance, allowing for more
17 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
flexible categories that followed axis-aligned ellip-
soidal Gaussian distributions. They also altered
the prototype embedding function to be stochas-
tic, using a probabilistic model that incorporates
a global noise distribution over latent prototypes.
Thesetw omodifica tio nsallo w edtheCNNtolea rn
embeddings that supported more accurate and
robust distributions more likely to handle uncer-
taintyarisingfrom,forexample,out-of-distribution
inputs due to measurement noise and label uncer-
tainty due to overlapping classes. For one-shot
classification tasks in low-dimensional embedding
spaces on Omniglot, the SPE model outperformed
the deterministic PN and achieved state-of-the-art
results.Ontheotherhand,whenfivetrainingexam-
pleswereusedineachepisodeinstead,orinhigher
dimensions,theuncertaintyoverprototypesproved
less important, and the models performed simi-
larly.Theirmostimpressivefindingwasinthecon-
text of data corruption. On an N-digit version of
MNIST,132the SPE consistently performed better
than a number of baselines, including PNs, when
input images were randomly corrupted, validating
the idea that SPEs learn categorical structures that
aremorerobusttoperceptualnoise.
Allenet al.allowed for even more complex cat-
egory structure with infinite mixture prototypes
(IMP), which model each category as a (possibly
infinite) mixture of Gaussian distributions.133The
authors built this model based on two key insights
from cognitive models of categorization: first, the
idea that complex categories might be best rep-
resented by more than one cluster—an interpola-
tion toward exemplar models had been previously
exploredincognitivescience;106,107andsecond,the
idea that the number of category clusters should
be free to grow as needed and the nonparametric
Bayesian machinery to support it.24,49Combining
these insights led to a differentiable model for gen-
erating an arbitrary number of subprototypes for a
givencategory,whichcouldthenbeusedforsubse-
quentlearningandclassificationtasks.Forfew-shot
character classification tasks in Omniglot, the IMP
performedsimilarlytoPNsandnearestneighbors.
Theequivalencebetweenthesemethodsisunsur-
prising,asdistributionsovercharactersarelikelyto
take some unimodal form well captured by a sin-
gle prototype. However, as the classes being mod-
eled became more complex, the IMP performed
muchbetter.Onfive-shotfive-waytasksfrom mini-ImageNet, which is based on natural image cate-
g o r i e st h a ta r el i k e l yt oh a v em o r ec o m p l e xa n d
multimodal structure, the IMP achieved state-of-
the-art results. The IMP also performed extremely
well on ten-way ten-shot alphabetrecognition on
Omniglot, around 25% better than PNs. This is
again because alphabets of individual characters
likely constitute complex distributions, with mul-
tiple modes corresponding to clusters of charac-
ter types and styles. Allen et al.demonstrated that
learningthesemorecomplexdistributionswasalso
beneficialfortransferlearning.133Theyshowedthat
when the IMP is trained on alphabet recognition
tasksandtestedoncharacterrecognitiontasksfrom
Omniglot,itoutperformsaPNtrainedjustonchar-
acter recognition tasks. This is a significant result,
because it implies something extra about individ-
ual characters has been learned by modeling their
distribution within a specific alphabet, and that
theinference model can be responsible for cap-
turing this—in this case by maintaining flexibility
over clusters that can vary with the task instance
at hand. Even more impressively, when particular
charactersareheldoutfromtrainingatthealphabet
level, the IMP successfully generalized its knowl-
e d g et or e c o g n i z et h e ma sp a r to ft h ea l p h a b e t
at test time, to a much greater extent than PNs.
These positive transfer learning results were also
found to hold for a tiered version of ImageNet,134
where fine-grained and multimodal structure cap-
tured by the IMP from image distributions at the
level of complex classes (i.e., when using super-
category labels) transferred well to image distri-
butions under from lower-level category labels. In
both cases, the PN trained on superclasses and
testedonsubclassesperformsmuchworse.Because
it uses a soft-clustering scheme that allows unla-
beled examples to act as supports, Allen et al.were
abletoapplytheIMPtosemisupervisedversionsof
Omniglotand mini-ImageNet,inwhichonlyasub-
setofsupportimagescomewithtraininglabels,and
in learning the unsupervised regime inferred clus-
ters of Omniglot. In the semisupervised tasks, the
IMP achieved state-of-the-art performance, and in
theunsupervisedtask,itprovidedmeaningfulclus-
tersofcharacters.
The above studies are an encouraging sign that
integrating extra information from learners with
pressures to learn generative and predictive visual
categories into CNNs through training objectives
18 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
and datasets is a promising route to strengthen-
ingtheirgeneralizabilityandrobustness.Whendata
a r ep l e n t i f u l ,i nt h ec o n t e x to fn o r m a li m a g ec l a s -
sification, adding this information indirectly—for
example, in the form of augmented image labels—
islikelytobethemostvaluablestrategy,inorderto
bestexploittheextremelyflexiblelearningcapacity
o fC N N s .W h e nd a t aa r es c a r c e ,h o w e v e r ,s t r o n g e r
inductive biases that can be directly incorporated
into the classification function used by CNNs have
led to better performance. Indeed, learning and
makingrobustinferencesfromlimitedinformation
is a hallmark of human cognition,135and so these
arenaturalsettingstoconsiderintegratinghuman-
likeconstraints.
End-to-end cognitive models
So far, we have seen CNN representations extend
traditional models of higher-level cognition to
more ecologically valid stimuli, and cognitive con-
siderations and constraints modify CNN training
paradigms such that they become better perform-
ers.Theremainingchallengeistobringtheseframe-
works even closer together, and fully leverage the
b e n e fi t so fb o t ha p p r o a c h e st ob u i l dt h en e x tg e n -
erationofcognitivemodels.
Amainclueabouthowtodothiscomesfromthe
investigationsalreadyoutlinedabove.Inthesestud-
i e s ,t h ep r i m a rya d v a n t a g eo fC N N sw a st h e i ra b i l -
ity to learn rich representations of a diverse range
of visual stimuli. This ability arises from their flex-
ible,gradient-basedtrainingregimes,aswellasthe
specialfunctionsofeachlayer,andtheseproperties
were not modified at all during their use in cog-
nitive models or reapplication to computer vision
tasks. Instead, there was a search over CNN rep-
resentations to find the best support for a behav-
ioral dataset, followed by investigating which of a
range of cognitive models were able to best fit the
data.Fromacognitiveperspective,thisamountsto
taking a two-step search, identifying the best com-
bination for the representation of stimuli and the
representation of higher-level cognitive structures
suchascategories.But,thereisnoreasontoseparate
this process. Provided our cognitive models can be
made differentiable, the same training regime can
be applied to both CNN and cognitive parameters
instead, allowing the CNN and cognitive compo-
nents to jointly learn which features of images are
Figure 9. Deepcategorizationmodelslearncategory-specific
stimulus embeddings. Top and middle rows: t-distributed
stochastic neighbor embeddings138of representations from a
deep prototype (top) and deep GMM with 25 centers (bot-
tom),withlocationsofprototypes andsubprototypes marked,
respectively. Bottom row: performance in the GMM increases
with the number of centers until around 10 −25, then asymp-
totes.ReproducedfromRef.136.
necessary to support the best application of a cog-
nitivemodeltohumanbehavioraldata.
A first attempt in this direction was made by
Singhet al., for a range of the categorization mod-
els reviewed above.136Fortunately, the functional
19 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
formsofprototype,Gaussian-mixture(GMM),and
exemplar categorization models can be made dif-
ferentiable with only slight modifications, and are
similarenoughtonormalCNNclassifiersthatthey
can simply replace them as the top layer of an
otherwise normal CNN. In this setup, the loca-
tion of the prototypes, subprototypes, and exem-
plars, respectively, are free parameters, which the
models must learn from some training subset of
l a b e l e di m a g e s .A su s u a l ,t h er o l eo ft h el o w e r
CNN layers is to learn the best feature represen-
tation to support these kinds of category compari-
son.Singh et al.trained a prototype model, GMMs
with2−2500centers,andanexemplarmodelacross
two CNN architectures.136Their input data were
thetrainingsubsetofCIFAR-10:50,000image-label
pairs, subdivided into 10 categories (i.e., image-
hard-labelpairs).Theytestedmodelgeneralization
totheCIFAR-10validationset:10,000image-hard-
labelpairs,aswellastheCIFAR-10Hdataset:10,000
image-soft-label pairs (i.e., distributions derived
fromhuman categorizations).The first finding was
that, again, choice of CNN architecture was the
major determinant of performanceand generaliza-
tion.Thesecondwasthatdespiteperformingsimi-
larlyonthehardlabelvalidationdata,thecognitive
modelsperformed muchbetterinpredictinghuman
categorizations than their CNN baselines. Varying
the number of centers in the GMM revealed that
the optimal number of centers—subprototypes—
wasaround25,whichoverallprovidedthebestper-
formance on human data (Fig. 9). Beyond model-
ingthehumandatawell,thefactthattheseend-to-
endmodelswereabletooutperformCNNbaselines
o nh u m a nd a t ae v e nw h e nt r a i n e de x c l u s i v e l yw i t h
hard labels is an important validation of the func-
tionalformsarrivedatbythecognitivepsychology
literature,aswellasthedevelopmentofcategoriza-
tion models beyond prototypes and exemplars to
mixture models and beyond. It remains to be seen
whatfurtherimprovementsarisewhensuchmodels
aretrainedonhumanbehavioraldata,inadditionto
beingtestedonit.
Conclusion
A major success from traditional studies of higher-
level cognition was the development of a range
of computational models that make high-precision
predictions about fundamental human behaviors.
To apply these models, it was necessary to developstatistical techniques to represent stimuli in a psy-
chologicallymeaningfulmanner,whichinturnlim-
ited their application to simple artificial stimuli. In
thisreview,wehaveshownthatwecanuseCNNsas
the basis for a new set of statistical techniques that
allow the representation of more naturalistic and
varied stimuli. Indeed, these methods differ from
techniques like MDS in a quantitative ,r a t h e rt h a n
qualitativeway.Bothsetsuseadifferentiableobjec-
tive function to iteratively optimize the statistical
fit of a multidimensional vector to a set of human
decisions, as well as some degree of hyperparame-
ter search (e.g., the number of dimensions in MDS
versuslayersizeinaCNN).Themaindifferenceis,
t h e n ,t h es h e e rn u m b e ro fp a r a m e t e r si n v o l v e di n
learningarepresentationthatcanbegeneralizedto
anarbitrarynumberofnaturalisticstimuli.Itmakes
sensethatthisprocessbeconstrainedbothbyinfor-
mationaboutthedistributionofthesestimuliinthe
natural world and the structure of human knowl-
edger egar dingthem,andweha vehighligh tedho w
t h eC N Na n dc o g n i t i v ec o m p o n e n t so fm o r ei n t e -
gratedmodelscanbeusedtobringthesesetsofcon-
straintsclosertogether.Alongwiththefundamental
scientific interest in extending analysis of behavior
toamorecomplexandecologicallyvaliddomain,as
wellasthegeneralaimofincreasingtheprecisionof
oureffortstomodelhigher-levelcognition,thisnew
setofmodelshastheaddedbenefitofbeingdirectly
applicable to the wealth of psychologically relevant
datanowavailableonline.137
Acknowledgments
This work was conducted under Grant Number
1718550 from the National Science Foundation.
R.M.B.draftedthemanuscript.J.C.P.createdthefig-
ures.T.L.G.conceivedthepaperdesign.Allauthors
e d i t e dt h em a n u s c r i p t .T h ea u t h o r sw o u l dl i k et o
thank Thomas Palmeri and Jason Chow for their
helpfulcommentsinreviewingthemanuscript.
Competing interests
Theauthorsdeclarenocompetinginterests.
References
1. Krizhevsky,A.,I.Sutskever&G.E.Hinton.2012.ImageNet
classification with deep convolutional neural networks. In
Advances in Neural Information Processing Systems 1097–
1105.
20 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
2. Baxter,J.2000.Amodelofinductivebiaslearning. J. Artif.
Intell. Res. 12:149–198.
3. Russakovsky, O. et al.2015. ImageNet large scale visual
recognitionchallenge. Int. J. Comput. Vis. 115:211–252.
4. Duta,I.C.,L.Liu,F.Zhu&L.Shao.2020.Pyramidalconvo-
lution:rethinkingconvolutionalneuralnetworksforvisual
recognition. arXivpreprintarXiv:2006.11538 .
5. Lin, T.-Y. et al.2014. Microsoft COCO: common objects
in context. In European Conference on Computer Vision
740–755.
6. Qiao, S., L.-C. Chen & A. Yuille. 2020. DetectoRS: detect-
ing objects with recursive feature pyramid and switchable
atrousconvolution. arXivpreprintarXiv:2006.02334 .
7. Goyal, Y., T. Khot, D. Summers-Stay, et al.2017. Making
the V in VQA matter: elevating the role of image under-
standinginvisualquestionanswering.In Proceedingsofthe
IEEEConferenceonComputerVisionandPatternRecogni-
tion6904–6913.
8 . J i a n g ,H . ,I .M i s r a ,M .R o h r b a c h , et al.2020. Defense of
grid features for visual question answering. In Proceedings
of the IEEE/CVF Conference on Computer Vision and Pat-
ternRecognition 10267–10276.
9. Johnson, J. et al.2017. CLEVR: a diagnostic dataset for
compositional language and elementary visual reasoning.
InProceedings of the IEEE Conference on Computer Vision
andPatternRecognition 2901–2910.
1 0 . P e r e z ,E . ,F .S t r u b ,H .D eV r i e s , et al.2017. Film: visual
reasoningwithageneralconditioninglayer. arXivpreprint
arXiv:1709.07871 .
11. Mnih, V. et al.2015. Human-level control through deep
reinforcementlearning. Nature 518:529–533.
12. Kriegeskorte,N.2015.Deepneuralnetworks:anewframe-
workformodelingbiologicalvisionandbraininformation
processing. A nnu.Rev .V is.Sci. 1:417–446.
13. Khaligh-Razavi, S.-M., L. Henriksson, K. Kay & N.
Kriegeskorte. 2017. Fixed versus mixed RSA: explaining
visualrepresentationsbyfixedandmixedfeaturesetsfrom
shallow and deep computational models. J. Math. Psychol.
76:184–197.
14. Bashivan,P.,K.Kar&J.J.DiCarlo.2019.Neuralpopulation
controlviadeepimagesynthesis. Science 364:eaav9436.
15. Hong, H., D.L. Yamins, N.J. Majaj & J.J. DiCarlo. 2016.
Explicit information for category-orthogonal object prop-
ertiesincreasesalongtheventralstream. Nat.Neurosci. 19:
613–622.
16. Agrawal,P.,D.Stansbury,J.Malik&J.L.Gallant.2014.Pix-
elstovoxels:modelingvisualrepresentationinthehuman
brain.arXivpreprintarXiv:1407.5104 .
17. Güçlü, U. & M.A. van Gerven. 2015. Deep neural net-
works reveal a gradient in the complexity of neural repre-
sentationsacrosstheventralstream. J.Neurosci. 35:10005–
10014.
18. Khaligh-Razavi, S.-M. & N. Kriegeskorte. 2014. Deep
supervised, but not unsupervised, models may explain IT
corticalrepresentation. PLoSComput.Biol. 10:e1003915.
19. Schrimpf, M. et al.2018. Brain-Score: which artificial
neural network for object recognition is most brain-like?
https://doi.org/10.1101/40700720. Yamins,D.L. etal.2014.Performance-optimizedhierarchi-
calmodelspredictneuralresponsesinhighervisualcortex.
Proc.N a tl.A cad.Sci.USA 111:8619–8624.
21. Cadieu, C.F. et al.2013. The neural representation bench-
mark and its evaluation on brain and machine. arXiv
preprintarXiv:1301.3530 .
22. Cadieu, C.F. et al.2014. Deep neural networks rival the
representation of primate IT cortex for core visual object
recognition. PLoSComput.Biol. 10:e1003963.
23. Marr,D.1982. Vision.SanFrancisco,CA:W.H.Freeman.
24. Anderson,J.R.1991.Theadaptivenatureofhumancatego-
rization.Psychol.Rev. 98:409–429.
25. McCulloch,W.S.&W.Pitts.1943.Alogicalcalculusofthe
ideasimmanentinnervousactivity. Bull.Math.Biophys. 5:
115–133.
26. Rosenblatt, F. 1958. The perceptron: a probabilistic model
forinformationstorageandorganizationinthebrain. Psy-
chol.Rev.65:386–408.
27. Hebb, D.O. 1949. The Organization of Behavior: A Neu-
ropsychologicalTheory .W iley.
28. Hodgkin,A.L.&A.F.Huxley.1952.Aquantitativedescrip-
tionofmembranecurrentanditsapplicationtoconduction
andexcitationinnerve. J. Physiol. 117:500–544.
29. Rumelhart, D.E., G.E. Hinton & R.J. Wilson. 1986. Learn-
ing representations by back-propagating errors. Nature
323:533–536.
30. Elman, J.L. 1990. Finding structure in time. Cogn. Sci. 14:
179–211.
31. Rumelhart, D. & J. McClelland. 1986. Parallel Distributed
Processing: Explorations intheMicrostructure ofCognition .
Cambridge,MA:MITPress.
32. LeCun, Y. et al.1989. Backpropagation applied to hand-
written zip code recognition. Neural Comput. 1:541–
551.
33. Riesenhuber,M.&T.Poggio.1999.Hierarchicalmodelsof
objectrecognitionincortex. Nat. Neurosci. 2:1019–1025.
34. Fukushima, K. & S. Miyake. 1982. Competition and Coop-
era tioninN euralN ets .Springer.
35. Denker, J.S. et al.1989. Neural network recognizer for
hand-written zip code digits. In Advances in Neural Infor-
mationProcessingSystems 323–331.
36. Mozer, M.C. 1987. Early Parallel Processing in Reading:
AC o n n e c t i o n i s tA p p r o a c h .L a w r e n c eE r l b a u mA s s o c i a t e s ,
Inc.
37. Radford,A. etal.2019.Languagemodelsareunsupervised
multitask learners. OpenAIblog 1:9.
38. LeCun, Y., Y. Bengio & G. Hinton. 2015. Deep learning.
Nature 521:436–444.
39. Yosinski, J., J. Clune, Y. Bengio & H. Lipson. 2014. How
transferable are features in deep neural networks? In
Advances in Neural Information Processing Systems 3320–
3328.
40. Zeiler, M.D. & R. Fergus. 2014. Visualizing and under-
standing convolutional networks. In European Conference
onComputerVision 818–833.
4 1 .S r i v a s t a v a ,N . ,G .H i n t o n ,A .K r i z h e v s k y , et al.2014.
Dropout: a simple way to prevent neural networks from
overfitting. J. Mach. Learn. Res. 15:1929–1958.
21 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
42. Hinton, G.E., N. Srivastava, A. Krizhevsky, et al.2012.
Improvingneuralnetworksbypreventingco-adaptationof
featuredetectors. arXivpreprintarXiv:1207.0580 .
43. Huang, G., Z. Liu & K.Q. Weinberger. 2016. Densely
connected convolutional networks. arXiv preprint arXiv:
1608.06993 .
44. Rawat, W. & Z. Wang. 2017. Deep convolutional neural
networksforimageclassification:acomprehensivereview.
NeuralComput. 29:2352–2449.
45. Khan, A., A. Sohail, U. Zahoora & A.S. Qureshi. 2020. A
survey of the recent architectures of deep convolutional
neuralnetworks. Artif. Intell. Rev. 53:5455–5516.
46. Sun, R. 2008. The Cambridge Handbook of Computational
Psychology .CambridgeUniversityPress.
47. Griffiths, T.L. & J.B. Tenenbaum. 2001. Randomness and
coincidences:reconcilingintuitionandprobabilitytheory.
InProceedingsoftheTwenty-ThirdAnnualConferenceofthe
Cognitive Science Society .J . D .M o o r e&K .S t e n n i n g ,E d s . :
370–375. Mahwah,NJ &London, UK:Lawrence Erlbaum
Associates.
48. Ashby,F.G.&L.A.Alfonso-Reese.1995.Categorizationas
probability density estimation. J. Math. Psychol. 39:216–
233.
49. Canini, K.R. & T.L. Griffiths. 2011. A nonparametric
Bayesian model of multi-level category learning. In Pro-
ceedings of the 25th AAAI Conference on Artificial Intelli-
gence.
50. Battleday, R.M., J.C. Peterson & T.L. Griffiths. 2020. Cap-
turinghumancategorizationofnaturalimagesbycombin-
ing deep networks and cognitive models. Nat. Commun.
11:5418.
51. Nosofsky,R.M.1992.Similarityscalingandcognitivepro-
cessmodels. Annu.Rev.Psychol. 43:25–53.
52. Shepard,R.N.1957.Stimulusandresponsegeneralization:
a stochastic model relating generalization to distance in
psychologicalspace. Psychometrika 22:325–345.
53. Kruskal,J.B.1964.Multidimensionalscalingbyoptimizing
goodness of fit to a nonmetric hypothesis. Psychometrika
29:1–27.
54. Ashby,F.G.,G.Boynton&W.W.Lee.1994.Categorization
responsetimewithmultidimensionalstimuli. Percept.Psy-
chophys. 55:11–27.
55. Kubilius,J.,S.Bracci&H.P.OpdeBeeck.2016.Deepneu-
r a ln e t w o r k sa sac o m p u t a t i o n a lm o d e lf o rh u m a ns h a p e
sensitivity. PLoSComput.Biol. 12:1–26.
56. Lake,B.M.,W.Zaremba,R.Fergus&T.M.Gureckis.2015.
Deep neural networks predict category typicality ratings
for images. In Proceedings of the 37th Annual Conference
oftheCognitiveScienceSociety .
57. Lowe, D.G. 1999. Object recognition from local scale-
invariantfeatures.InProceedingsofthe7thIEEEInterna-
tionalConferenceonComputerVision1150–1157.
58. Cortes, C. & V. Vapnik. 1995. Support-vector networks.
Mach. Learn. 20:273–297.
59. Rosch,E.,C.Simpson&R.S.Miller.1976.Structuralbases
oftypicalityeffects. J. Exp. Psychol. Hum. Percept. Perform.
2:491–502.
60. Attneave,F.1950.Dimensionsofsimilarity. Am.J.Psychol.
63:516–556.61. Shepard, R.N. 1987. Towards a universal law of gener-
alization for psychological science. Science 237:1317–
1323.
62. Mur, M. et al.2013. Human object-similarity judgments
reflectandtranscendtheprimate-ITobjectrepresentation.
Front.Psychol. 4:128.
63. Shepard,R.N.1980.Multidimensionalscaling,tree-fitting,
andclustering. Science 210:390–398.
64. Peterson,J.C.,J.T.Abbott&T.L.Griffiths.2018.Evaluating
(andimproving)thecorrespondencebetweendeepneural
networksandhumanrepresentations. Cogn.Sci. 42:2648–
2669.
65. Tversky, A. 1977. Features of similarity. Psychol. Rev. 84:
327–352.
66. Shepard, R.N. & P. Arabie. 1979. Additive clustering: rep-
resentationofsimilaritiesascombinationsofdiscreteover-
lappingproperties. Psychol.Rev. 86:87–123.
67. Navarro, D.J. & M.D. Lee. 2004. Common and distinctive
features in stimulus similarity: a modified version of the
contrastmodel. Psychon.Bull.Rev. 11:961–974.
68. Attarian,M.,B.D.Roads&M.C.Mozer.2020.Transform-
ingneuralnetworkvisualrepresentationstopredicthuman
judgmentsofsimilarity. arXivpreprintarXiv:2010.06512 .
69. Roads, B.D. & M.C. Mozer. 2021. Predicting the ease of
human category learning using radial basis function net-
works.NeuralComput. 33:376–397.
70. Luce, R.D. 1959. Individual Choice Behavior .N e wY o r k :
JohnWiley.
71. Jha, A., J. Peterson & T.L. Griffiths. 2020. Extracting
low-dimensional psychological representations from con-
volutional neural networks. arXiv preprint arXiv:2005.
14363.
72. Saxe,A.,J.McClelland&S.Ganguli.2013.Learninghierar-
chical category structure in deep neural networks. In Pro-
ceedingsofthe35thAnnualConferenceoftheCognitiveSci-
enceSociety .
73. Sanders, C.A. & R.M. Nosofsky. 2018. Using deep learn-
ing representations of complex natural stimuli as input to
psychologicalmodelsofclassification.In Proceedingsofthe
2018ConferenceoftheCognitiveScienceSociety,Madison .
74. Bechberger, L. & K.-U. Kühnberger. 2019. Generalizing
psychological similarity spaces to unseen stimuli. arXiv
preprintarXiv:1908.09260 .
75. Sanders, C.A. & R.M. Nosofsky. 2020. Training deep net-
works to construct a psychological feature space for a
natural-object category domain. Comput. Brain Behav .3:
229–251.
76. Nosofsky,R.M.,C.A.Sanders,B.J.Meagher&B.J.Douglas.
2018.Towardthedevelopmentofafeature-spacerepresen-
tation for a complex natural category domain. Behav. Res.
Methods 50:530–556.
77. Peterson, J.C., P. Soulos, A. Nematzadeh & T.L. Griffiths.
2018. Learning hierarchical visual representations in deep
neural networks using hierarchical linguistic labels. arXiv
preprintarXiv:1805.07647 .
78. Rosch,E.1978. CognitionandCategorization .Hillsdale,NJ:
Erlbaum.
79. Rosch, E. 1973. Cognitive Development and Acquisition of
Language .Elsevier.
22 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Battleday et al. Convolutional networks and higher-level cognition
80. Wang, P. & G.W. Cottrell. 2015. Basic level categoriza-
tion facilitates visual object recognition. arXiv preprint
arXiv:1511.04103 .
81. Rosenfeld, A., M.D. Solbach & J.K. Tsotsos. 2018. Totally
looks like-how humans compare, compared to machines.
InProceedings of the IEEE Conference on Computer Vision
andPatternRecognitionWorkshops 1961–1964.
82. Rosenfeld, A., R. Zemel & J.K. Tsotsos. 2019. High-level
perceptual similarity is enabled by learning diverse tasks.
arXivpreprintarXiv:1903.10920 .
83. Cohen,H.&C.Lefebvre.2005. HandbookofCategorization
inCognitiveScience .ElsevierScience.
84. Plato.1968. TheRepublic .N ewY ork:BasicBooks.
85. Aristotle. 1984. Categories . Princeton, NJ: Princeton Uni-
versityPress.
86. Hull, C.L. 1920. Quantitative aspects of evolution of con-
cepts:anexperimentalstudy. Psychol.Monogr. 28:i–86.
87. Bruner,J.S.,J.J.Goodnow&G.A.Austin.1956. AS t ud yo f
Thinking.N ewY ork:W iley.
88. Rosch, E. 1973. Natural categories. Cogn. Psychol. 4:328–
350.
89. Rosch,E.&C.B.Mervis.1975.Familyresemblances:stud-
iesintheinternalstructureofcategories. Cogn.Psychol. 7:
573–605.
90. Wittgenstein, L. 2009. Philosophical Investigations .J o h n
Wiley & Sons.
91. Posner,M.I.&S.W.Keele.1968.Onthegenesisofabstract
ideas.J. Exp. Psychol. 77:353–363.
92. Reed, S.K. 1972. Pattern recognition and categorization.
Cogn.Psychol. 3:393–407.
93. Medin,D.L.&M.M.Schaffer.1978.Contexttheoryofclas-
sificationlearning. Psychol.Rev. 85:207–238.
94. Nosofsky, R.M. 1986. Attention, similarity, and the
identification–categorization relationship. J. Exp. Psychol.
Gen. 115:39–61.
95. Whittlesea,B.W.1987.Preservationofspecificexperiences
intherepresentationofgeneralknowledge. J.Exp.Psychol.
Learn.Mem.Cogn. 13:3–17.
96. Nosofsky, R.M., S.E. Clark & H.J. Shin. 1989. Rules and
exemplars in categorization, identification, and recog-
nition.J. Exp. Psychol. Learn. Mem. Cogn. 15:282–
304.
97. Nosofsky, R.M. 1984. Choice, similarity, and the context
theoryofclassification. J. Exp. Psychol. Learn. Mem. Cogn.
10:104–114.
98. Fried,L.S.&K.J.Holyoak.1984.Inductionofcategorydis-
tributions: a framework for classification learning. J. Exp.
Psychol.Learn.Mem.Cogn. 10:234–257.
99. Aha, D.W. & R.L. Goldstone. 1992. Concept learning and
flexibleweighting.InProceedingsofthe14thAnnualCon-
ferenceoftheCognitiveScienceSociety.
100. McKinley, S.C. & R.M. Nosofsky. 1995. Investigations of
exemplar and decision bound models in large, ill-defined
categorystructures. J.Exp.Psychol.Hum.Percept.Perform.
21:128–148.
101. Nosofsky, R.M. 1988. Similarity, frequency, and category
representations. J.Exp.Psychol.Learn.Mem.Cogn. 14:54–
65.102. Jüttner,M.&I.Rentschler.2000.Scale-invariantsuperior-
ityoffovealvisioninperceptualcategorization. Eur.J.Neu-
rosci. 12:353–359.
103. Palmeri, T.J. & R.M. Nosofsky. 2001. Central tendencies,
extreme points, and prototype enhancement effects in ill-
defined perceptual categorization. Quart. J. Exp. Psychol.
Sect.A 54:197–235.
104. Stewart,N.&N.Chater.2002.Theeffectofcategoryvari-
ability in perceptual categorization. J. Exp. Psychol. Learn.
Mem. Cogn. 28:893–907.
105. Maddox, W.T. & F.G. Ashby. 1993. Comparing decision
boundandexemplarmodelsofcategorization. Attent. Per-
cept.Psychophys. 53:49–70.
106. Rosseel,Y.2002.Mixturemodelsofcategorization. J.Math.
Psychol. 46:178–210.
107. Vanpaemel, W., G. Storms & B. Ons. 2005. A varying
abstractionmodelforcategorization.InProceedingsofthe
AnnualConferenceoftheCognitiveScienceSociety2277–
2282.
108. Krizhevsky,A.&G.Hinton.2009.Learningmultiplelayers
offeaturesfromtinyimages.Technicalreport.Universityof
Toronto.
109. Annis,J.,I.Gauthier&T.J.Palmeri.2020.Combiningcon-
volutionalneuralnetworksandcognitivemodelstopredict
novelobjectrecognitioninhumans. J. Exp. Psychol. Learn.
Mem. Cogn .https://doi.org/10.1037/xlm0000968.
110. Gauthier, I. & M.J. Tarr. 1997. Becoming a “Greeble”
expert: exploring mechanisms for face recognition. Vis.
Res.37:1673–1682.
111. Richler,J.J.,J.B.Wilmer&I.Gauthier.2017.Generalobject
recognition is specific: evidence from novel and familiar
objects.Cognition 166:42–55.
112. Wong, A.C.-N., T.J. Palmeri, B.P. Rogers, et al.2009.
Beyondshape:howyoulearnaboutobjectsaffectshowthey
arerepresentedinvisualcortex. PLoSOne 4:e8405.
113. Richler, J.J. et al.2019. Individual differences in object
recognition. Psychol.Rev. 126:226–251.
114. Holmes,W.R.,P.O’Daniels&J.S.Trueblood.2020.Ajoint
deep neural network and evidence accumulation model-
ing approach to human decision-making with naturalistic
images.Comput.BrainBehav. 3:1–12.
115. Battleday,R.M.,J.C.Peterson&T.L.Griffiths.2017.Mod-
eling human categorization of natural images using deep
featurerepresentations. arXivpreprintarXiv:1711.04855 .
116. Guest, O. & B.C. Love. 2019. Levels of representation in a
deep learning model of categorization. https://doi.org/10.
1101/626374.
117. Schyns, P.G., R.L. Goldstone & J. Thibaut. 1998. Develop-
ment of features in object concepts. Behav. Brain Sci. 21:
1–54.
118. Recht, B., R. Roelofs, L. Schmidt & V. Shankar. 2018.
Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv
preprintarXiv:1806.00451 .
119. Kurakin, A., I. Goodfellow & S. Bengio. 2016. Adver-
sarial examples in the physical world. arXiv preprint
arXiv:1607.02533 .
120. Szegedy,C. etal.2013.Intriguingpropertiesofneuralnet-
works.arXivpreprintarXiv:1312.6199 .
23 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.
Convolutional networks and higher-level cognition Battleday et al.
121. Goodfellow, I.J., J. Shlens & C. Szegedy. 2014. Explain-
ing and harnessing adversarial examples. arXiv preprint
arXiv:1412.6572 .
122. Nguyen, A., J. Yosinski & J. Clune. 2015. Deep neural net-
works are easily fooled: high confidence predictions for
unrecognizable images. In Proceedings of the IEEE Confer-
enceonComputerVisionandPatternRecognition 427–436.
123. Yamauchi,T.&A.B.Markman.1998.Categorylearningby
inferenceandclassification. J. Mem. Lang. 39:124–148.
124. Yamauchi, T., B.C. Love & A.B. Markman. 2002. Learning
nonlinearly separable categories by inference and classifi-
cation.J. Exp. Psychol. Learn. Mem. Cogn. 28:585–593.
125. Peterson, J.C., R.M. Battleday, T.L. Griffiths & O. Rus-
sakovsky. 2019. Human uncertainty makes classification
morerobust.In ProceedingsoftheIEEEInternationalCon-
ferenceonComputerVision 9617–9626.
126. Zhang, H., M. Cisse, Y.N. Dauphin & D. Lopez-Paz. 2017.
mixup:beyondempiricalriskminimization. arXivpreprint
arXiv:1710.09412 .
127. Lake, B.M., R. Salakhutdinov, J. Gross & J. Tenenbaum.
2011. One shot learning of simple visual concepts. In Pro-
ceedings of the Annual Meeting of the Cognitive Science
Society.
128. Vinyals, O., C. Blundell, T. Lillicrap, et al.2016. Matching
networksforoneshotlearning.In AdvancesinNeuralInfor-
mationProcessingSystems 3630–3638.129. Snell, J., K. Swersky & R. Zemel. 2017. Prototypical net-
works for few-shot learning. In Advances in Neural Infor-
mationProcessingSystems 4077–4087.
130. Welinder,P. etal.2010.Caltech-UCSDbirds200.
131. Scott, T.R., K. Ridgeway & M.C. Mozer. 2019. Stochas-
tic prototype embeddings. arXiv preprint arXiv:1909.
11702.
132. Oh, S.J. et al.2018. Modeling uncertainty with
hedged instance embedding. arXiv preprint arXiv:1810.
00319.
133. Allen,K.R.,E.Shelhamer,H.Shin&J.B.Tenenbaum.2019.
Infinite mixture prototypes for few-shot learning. arXiv
preprintarXiv:1902.04552 .
134. Ren,M. etal.2018.Meta-learningforsemi-supervisedfew-
shotclassification. arXivpreprintarXiv:1803.00676 .
135. Lake,B.M.,T.D.Ullman,J.B.Tenenbaum&S.J.Gershman.
2017. Building machines that learn and think like people.
Behav.BrainSci. 40:e253.
136. Singh, P., J.C. Peterson, R.M. Battleday & T.L. Grif-
fiths. 2020. End-to-end deep prototype and exemplar
models for predicting human behavior. arXiv preprint
arXiv:2007.08723 .
137. Griffiths, T.L. 2015. Manifesto for a new (computational)
cognitiverevolution. Cognition 135:21–23.
138. Maaten, L.V.D. & G. Hinton. 2008. Visualizing data using
t-SNE.J. Mach. Learn. Res. 9:2579–2605.
24 Ann. N.Y. Acad. Sci. xxxx (2021) 1–24 © 2021 New York Academy of Sciences.

Ruairidh M Battleday
Joshua C Peterson
Thomas L Griffiths