Cognitive science as a source of forward and inverse models of human decisions for robotics and control.

AnnualReviewofControl,Robotics,and
AutonomousSystems
Cognitive Science as a Source
of Forward and Inverse Modelsof Human Decisions forRobotics and Control
Mark K.Ho1and Thomas L.Griffiths1,2
1Department of Computer Science,Princeton University,Princeton,New Jersey,USA
2Department of Psychology,Princeton University,Princeton,New Jersey,USA;
email: tomg@princeton.edu
Annu.Rev.Control Robot.Auton.Syst.2022.5:33–53
First published as a Review in Advance on
November 16,2021
TheAnnualReviewofControl,Robotics,and
AutonomousSystems is online at
control.annualreviews.org
https://doi.org/10.1146/annurev-control-042920-
015547
Copyright © 2022 by Annual Reviews.
All rights reservedKeywords
cognitive science,robotics,psychology,decision-making,theory of mind
Abstract
Those designing autonomous systems that interact with humans will
invariably face questions about how humans think and make decisions.
Fortunately, computational cognitive science offers insight into humandecision-making using tools that will be familiar to those with backgrounds
in optimization and control (e.g., probability theory, statistical machine
learning, and reinforcement learning). Here, we review some of this work,focusing on how cognitive science can provide forward models of human
decision-making and inverse models of how humans think about others’
decision-making. We highlight relevant recent developments, including
approachesthatsynthesizeblackboxandtheory-drivenmodeling,accounts
that recast heuristics and biases as forms of bounded optimality, andmodels that characterize human theory of mind and communication in
decision-theoretic terms. In doing so, we aim to provide readers with a
glimpseoftherangeofframeworks,methodologies,andactionableinsightsthat lie at the intersection of cognitive science and control research.
33
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

1. INTRODUCTION
As robots and other automated systems are beginning to become more integrated into human
lives,engineersfaceanewproblem:designingthesesystemstoeffectivelyandsafelyinteractwith
people.Partofthechallengeisthathumansarethemselvesautonomousagents,makingdecisions
andactinginwaysthatintroducepotentiallyunpredictabledynamicsintotheenvironment.Even
more challenging, humans change their behavior in response to the actions of the system they
are interacting with,meaning that the engineer has to consider not just how to predict and inter-
pret human behavior, but also how the behavior of the system that they are designing might be
predicted and interpreted by humans in turn.
As cognitive scientists, we have had many enjoyable conversations with engineers about how
to solve these problems. Typically these conversations begin with an email or a knock on the
door requesting the most up-to-date model of human behavior in a format that can be easily
integratedintoacontrol-theoreticframework.Wedisappointourcolleaguesbytellingthemthat
unfortunatelynosuchmodelexists,butthenexcitethemwithhowmuchprogresshasbeenmade
toward this goal and all of the research possibilities that this entails. Our intent in this article is
to offer our readers a chance to follow the same emotional trajectory, highlighting the ways in
whichwethinkcontemporarycognitivesciencecanprovidetoolsthatmaybeusefultoengineers
designing systems that interact with humans and identifying some of the exciting possibilities for
future research in this area.
Speakingbroadly,computationalmodelsdevelopedbycognitivescientistsoffersolutionstoat
least two of the problems that engineers face ( Figure 1 ).First,they provide forward models that
canbeusedtogeneratepredictionsabouthumanbehaviorbasedonassumptionsaboutthebeliefs,
goals,anddesiresofhumanagents.Thesemodelscanbeusefulforanticipatingwhatapersonwill
do in a given situation and hence provide a way to enrich the modeling of the environment in
which an automated system operates. These forward models can also be used as an ingredient
in inverse models, which infer the beliefs, goals, and desires of humans based on their actions—
a necessary step for any agents that seek to coordinate their behavior with humans, collaborate
effectively,provide assistance,or cooperate as they pursue common goals.
Beliefs
Desires
HabitsHeuristics
Emotions
ValuesAttention
Planning
BiasesDecisionsInverse models
(inference)
Forward models
(prediction)
Figure 1
Forward and inverse models from cognitive science.In this article,we review work in cognitive science on
forward models of how humans make decisions and inverse models that humans use to reason about other
agents.Cognitive scientists are increasingly using computational tools such as probability theory,
reinforcement learning,and statistical machine learning to characterize forward and inverse models of
human decision-making.This provides opportunities for cross-talk and collaboration between cognitive
science and control research.
34 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

Cognitive science also offers insight into the way that humans solve exactly this inverse prob-
lem. People routinely make inferences about the beliefs, goals, and desires of other people, a
process that has been extensively studied by psychologists (1–3) and is increasingly captured in
computationalmodels(4–6).Thesemodelsareusefulnotonlyasasourceofinsightforengineers
seeking to re-create this capacity to draw inferences from the actions of others,but also as a toolfor anticipating how the actions of an automated system will be interpreted by a human (7, 8).
Research in human–robot interaction has begun to make use of these ideas, designing systems
that act in a way that is more legible to humans (9–11),a line of work that we also review.
In considering these two ways that computational models of cognition can be used by engi-
neers designing automated systems—as both forward and inverse models—we also highlight the
ways in which recent work in computational cognitive science has emphasized formalisms thatwill bevery familiar toresearcherscoming froma backgroundof optimization and control.Cog-
nitive scientists increasingly use ideas from probability theory, statistical machine learning, and
reinforcement learning in specifying models of human cognition (12). This creates an opportu-nity to develop a common language for describing the behavior of both humans and machines,
and supports easier integration of insights from cognitive science into control.
Given the vast scope of human behavior, it is necessary to limit our review to a specific sub-
domain of human activity. To that end, we focus on models of human decision-making, broadly
construed.Thedecisionspeoplemakerevealtheirpreferencesanddeterminetheiractions,which
iskey tothe design of interactive systems.They alsoprovidea rich territoryfor researchers,with
formal models of human decision-making going back almost 300 years (13).Our goal is to sum-
marize the current state of the art in predicting and interpreting human decisions in a form thatis immediately actionable by designers of automated systems.
Theremainderofthearticleissplitintotwoparts.Inthefirstpart,wefocusonforwardmodels,
considering the criteria for useful computational models of human decision-making and summa-rizingrecentresearchthataimstosatisfythesecriteria.Wethenturntoinversemodels,describing
theproblemofinverseinference,summarizingthekeyideasfromthepsychologicalliterature,ex-
plaining how this has been translated into formal models, and highlighting some of the ways inwhich these ideas have been applied within robotics.We close with a brief discussion of some of
the remaining open questions in these areas and possibilities for future research.
2. FORWARD MODELS OF HUMAN DECISION-MAKING
For a theory of human decision-making to be useful to an engineer designing a system that hashuman behavior as a component,that theory should have two properties: It should be generaliz-
able,meaningthatitcanbeappliedinanycontextinwhichtheengineerneedstobeabletomakepredictions about how people will act, and it should be accurate, producing good predictions
about human behavior in that context. The development of theories of human decision-making
has historically tended to alternate between these criteria,making progress on one at the cost ofthe other ( Figure 2 ).
The earliest formal theories of human decision-making made the strong assumption that hu-
mansarerational,inthesenseofpursuingactionsthatareintheirself-interestandincompliance
with axioms that can be widely agreed to characterize rational behavior (19,20).The impressive
resultofthisinvestigationisthatthepreferencesofarationalagentcanbecharacterizedbyautil-ityfunctionthatassignsanumericalvaluetoeachpossibleoutcome,andwhenfacedwithdecisions
thatinvolveuncertainty,thatagentshouldpursuetheoptionthathasthehighestexpectedutility.
This theory—which we refer to as expected utilitytheory—fulfills the goal of generalizability.
To predict the actions that a rational agent will take in a new environment, it is necessary only
www.annualreviews.org CognitiveScienceasaSourceofModels 35
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

a* = arg
amax E[u | a]
π* = arg
πmax  E[u | π] – cost( π)Anchoring and
adjustmentAvailability biasConﬁrmation bias
Framing effectsProspe
cttheoryPlanning fallacyab
d c
Figure 2
Four approaches to studying human decision-making.( a) Early formal theories of decision-making assumed
humans were expected utility maximizers.( b) The heuristics and biases research program initiated in the
1970s by Tversky & Kahneman (14) demonstrated that people systematically violate basic predictions of
expected utility theory.This resulted in a focus on the heuristics that people tend to use in specific situations
as opposed to general theories.( c) Decision-making models represented by neural networks,combined with
large data sets of human choice behavior and informed by psychological theory (e.g.,15,16),provide a way
to make predictions about human decisions (see Section 2.1).( d) The theoretical framework of resource
rationality (17,18) aims to provide a general formal theory that accounts for people’s heuristics and biases.
Specifically,resource rationality proposes that human decision-making reflects expected utility maximization
subject to computational costs and cognitive limitations (see Section 2.2).Emojis designed by OpenMoji,the
open source emoji and icon project,and used under a CC BY-SA license ( https://creativecommons.org/
licenses/by-sa/4.0 ).
to identify the utility assigned to different outcomes; the decisions that agent will take can then
be derived directly from these quantities. The tools that are used for deriving this behavior are
exactlythetoolsthatareusedinoptimizationandcontrol,aswetypicallyseektodefineagentsthat
arerational.Asaconsequence,humanrationalityisacommonassumptionininteractivesystems,
albeit with some allowance for stochasticity (e.g.,21).It is also a common assumption in the kind
of choice models that are widely used in econometrics (e.g.,22).
The only problem with assuming that humans are rational is that this assumption turns out
to be false. Starting in the 1970s, psychologists (led by Daniel Kahneman and Amos Tversky)
began to document the ways in which people’s decisions violate the axioms that are assumed in
rationalmodels(14).Thisledtoaswingtowardamorequalitativepsychologyofhumandecision-
making,whichemphasizedidentifyingalloftheheuristicshortcutsthatpeopleseemtousewhen
making decisions,and the behavioral biases that result.The outcome of this process is a long list
ofthethingsthatpeopledowronginspecificsituations.Thisissomethingthatmightpotentially
36 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

increase the accuracy of our models,but since these behaviors are specific to particular scenarios
anditishardtoknowwhichheuristicmightdominateinanewsetting,thisaccuracycomesatthe
cost of generalizability.
This qualitative approach to understanding human decision-making was complemented
by efforts to formalize the cognitive processes that people engage in when making decisions.Kahneman and Tversky developed prospect theory, which extends expected utility theory by
allowing different functions characterizing the subjective value of gains and losses and recog-
nizing that the probabilities of events may also be subjectively transformed (23). Subsequentwork has introduced further nuances to this theory (e.g., 24), together with hypotheses for how
to formalize ideas about different heuristics that people might follow (e.g., 25) as well as other
cognitive factors,such as the salience of different options (e.g.,26).
Recent work in psychology and neuroscience has drilled down even further into the cognitive
processesthatmightaccountforthesebehaviors.Oneprominentlineofworkfocusesontheidea
thatpeoplemakedecisionsbyaccumulatingevidencethatoneoptionisbetterthanthealternatives(27).Thereisongoingdebateabouttheprecisemechanismsbywhichsuchaprocesscouldoperate
(e.g., 28), but these accumulator models have also received support from results in neuroscience
thatseemedtoshowareasinthebrainthatengageinevidenceaccumulation(e.g.,29,butsee30).
For the engineer, the precise cognitive and neural mechanisms underlying people’s decisions
might matter less than what those decisions actually are and how good predictions about human
decision-making can be generated in other contexts—our two criteria of accuracy and generaliz-
ability. To this end, we focus on two recent developments that increase the potential for models
of human decision-making to be used effectively as forward models in control settings.The firstis the potential to use ideas from machine learning, combined with the availability of large data
sets on human behavior, to develop more accurate models of human decision-making. The sec-
ond is recent efforts to revisit the notion of rationality, with the goal of obtaining a theoreticalframework that has the same generalizability as expected utility theory while incorporating what
we know about human cognitive limitations in a way that supports greater accuracy.
2.1. Developing More Accurate Models of Human Decisions
Using Machine Learning
The recent success of machine learning in many domains raises the possibility that such systems
may be able to better predict human decisions than the theories of choice developed by psychol-ogists and economists. For the engineer seeking a forward model of human decision-making,
it may be tempting to collect a data set of human decisions and train an off-the-shelf machine
learning method to predict people’s behavior.This possibility has been explored extensively overthe last decade, showing that machine learning has a great deal of promise in this area, but also
that the performance of these systems can be significantly improved by the injection of some
psychological insight.
A first extensive comparison of psychological models against machine learning systems for
predicting human decisions occurred in the 2015 Choice Prediction Competition (31). The
competition employed a standard risky-choice paradigm that has been used extensively to study
human decisions, informing the development of many of the models summarized above. In this
task,participantsmakeachoicebetween twogambles.Ineach gamble,differentoutcomes—herecorresponding to actual monetary gains and losses—occur with different probabilities.The pairs
of gambles can be described by an 11-dimensional vector that summarizes the payoffs and their
probabilities. The task is to map this 11-dimensional vector to a probability of choosing onegamble over the other,with the goal of getting this probability as close as possible to the choice
www.annualreviews.org CognitiveScienceasaSourceofModels 37
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

probabilities of a group of human participants. In the competition, the choice probabilities for
90 such pairs of gambles were provided,and the goal was to predict the corresponding probabil-
ities for a held-out test set.
The results of the 2015 Choice Prediction Competition showed that psychological models—
that is, those developed by psychologists and economists—tended to outperform off-the-shelfmachine learning methods. The best-performing model instantiated a set of heuristics that had
been identified in the psychological literature. Subsequent work showed that machine learning
methods could improve on this performance, but only when provided with features that weremotivated by psychological theory (32,33).
Tomachinelearningpractitioners,theseresultsmaynotcomeasabigsurprise.Thisprediction
problem has a relatively large number of features compared with the amount of available data(90 pairs of gambles). The success of psychological models can be interpreted as an instance of
the bias–variance trade-off (34), with the small amounts of data involved meaning that models
with carefully crafted inductive biases are most likely to be successful. However, the other sideof that trade-off is the expectation that as the amount of data increases, we should expect to see
improved performance from machine learning models with weaker inductive biases.
Consistent with this hypothesis,applications of machine learning to predicting other kinds of
humandecisionshaveshowngreatersuccess.Withmoreinstancesofmoreconstrainedproblems,
machine learning methods can outperform psychological models,and it has even been suggested
that they offer an upper bound on the amount of variance we can expect to account for (35,36).
Indeed,in a subsequent Choice Prediction Competition where models were trained on 210 pairs
of gambles, relatively generic machine learning models were able to outperform psychologicaltheories (37).
Based on this insight, recent work has collected and analyzed a risky-choice data set that in-
volvesordersofmagnitudemoreproblemsthantheoriginalChoicePredictionCompetition(15,38). In this data set, human participants made decisions for more than 10,000 pairs of gambles.
The size of the data set makes it possible to systematically evaluate existing models of choice and
to use machine learning to exhaustively explore the space of possible theories. Different modelsof choice can be expressed in terms of constraints on the functional form of a predictive model.
For example, under the expected utility theory, we can take the probability that people choose
a gamble to be proportional to exp{/summationtext
ipiu(xi)}, wherepiis the probability of the outcome xiand
u(·) is a utility function.By taking an arbitrary differentiable form for this utility function—such
asanartificialneuralnetwork—wecanemploystandardtoolsforautomaticdifferentiationtouse
gradientdescentinordertooptimizetheformofthisfunctionagainsthumandata.Thisapproachgeneralizes to other psychological theories.For example,prospect theory corresponds to assum-
ing that the choice probability is proportional to exp{/summationtext
iπ(pi)u(xi)}, where π(·) is a probability
weighting function.
Petersonetal.(15)usedthisapproachtoidentifytheoptimalfunctionalformforvariousclas-
sic theories of choice, and also evaluated unconstrained artificial neural networks for predictingpeople’s decisions (see Figure 3 ). The results showed that when using the entire data set of dif-
ferentpairsofgambles,anunconstrainedneuralnetworksystematicallyoutperformedallexisting
psychologicaltheories.However,theyalsoshowedthatequivalentperformancecouldbeobtainedby defining a model based on a mixture of these classic theories, and that this model achieved a
highlevelofpredictiveperformancefarfasterthananunconstrainedneuralnetwork.Theresults
of this analysis suggest that, given enough data, we can obtain better forward models of humandecisionsusingmachinelearning,butalsothatthesemodelsarelikelytobeenhancedbydrawing
on psychological theory when possible.
38 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

Neural network with expected utility
Neural network with prospect theory
Neural network with cumulative prospect theory
Value-based neural network
Context-dependent neural network
BEAST
Previous largest data set0.040
0.035
0.030
0.025
0.020
0.015
0.010Test set mean sqaared error
10 20 30 40 50 60 70 80 90 100
Percentage of training data used
Figure 3
Performance of machine learning models predicting people’s decisions in a risky-choice task.Neural
networks constrained to a functional form consistent with classic theories of decision-making,such as
expected utility,prospect theory,and cumulative prospect theory,are compared against networks that
directly estimate the value of a gamble from its features (value based) or directly predict people’s choices
based on the features of both gambles (context dependent).The vertical axis shows the mean squared error
inpredicting theprobabilitywith which peoplechooseaparticular gamble,and thehorizontal axisshowsthe
percentage of the training data (approximately 10,000 pairs of gambles) that was used.The previous largest
data set for risky choice (37) is also shown.Given enough data,all neural network models outperform the
best models in their class proposed by human psychologists and economists,but this requires orders of
magnitude more data than have previously been collected.For comparison,the dotted line shows the
performance of the Best Estimate and Sampling Tools (BEAST) model (31),which won the 2015 Choice
Prediction Competition.Data are from Reference 15.
A further challenge of using off-the-shelf machine learning methods when developing for-
ward models of human decision-making is that these methods often result in models that are
uninterpretable. Previous work in this area has relied on post hoc analysis of models to identify
features that are psychologically interpretable (e.g., 36). Agrawal et al. (16) recently outlined
an alternative approach that uses an off-the-shelf machine learning model to critique a more
interpretable model until that interpretable model yields similar performance. This approach
was applied to a large data set of human decisions that is likely to be of interest to researchers
working on autonomous systems: the Moral Machine project (39).This data set consists of more
than 10 million human decisions about what an autonomous vehicle should do when faced with
an inevitable collision, where the only available choice is about which group of pedestrians the
vehicle will collide with—a version of theclassic trolleyproblem (40).Having an uninterpretable
model that predicts these choices is not particularly useful for designing autonomous systems,
but Agrawal et al. (16) showed that their approach can be used to identify the features that a
predictivemodelhaddiscovered,makingthosefeaturesexplicitinawaythatislikelytobeuseful
when deciding how to design and regulate autonomous vehicles.
2.2. Developing More Generalizable Models via Resource Rationality
Despite the promise of machine learning to improve the accuracy of models of human decisions,
generalizabilityisstilllikelytobeachallenge.Machinelearningsystemsaretypicallytrainedina
specificdomain and can face difficultywhen appliedin another related domain.Making thiskind
of generalization requires extracting the causal principles that underlie people’s decisions.In this
section,we consider an approach that has the potential to do just that.
www.annualreviews.org CognitiveScienceasaSourceofModels 39
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

The classical notion of rationality was our prime example of a theory that satisfies the goal
of generalizability—for any new situation, it is possible to derive predictions about behavior.
However,this classical theory falls short not just because it fails to empirically capture aspects of
how people make decisions, but also because it represents an unrealistic ideal for any intelligent
system with finite computational resources. This is a long-standing idea, going all the way backtotheclassicworkofHerbertSimon(41)onboundedrationality.However,thisideahasrecently
begun to receive a more comprehensive mathematical and empirical treatment.
The classical theory of rational action via maximizing expected utility does not take into ac-
count the computational cost of selecting that action. As a consequence, it is easy to imagine an
agent that is trying to follow the prescriptions of this theory ending up paralyzed as it tries to
compute all of the possible outcomes and their probabilities. To address this, researchers in theartificialintelligenceliteraturehavesoughtamorerealisticcriterionforrationalactionforagents
with finite computational resources.The outcome of this investigation is the theory of bounded
optimality, which focuses not on the optimal action that an agent should take but rather on theoptimalalgorithmanagentshouldfollowinordertoselectthataction(42,43).Thistheoryexplic-
itly trades off the expected utility of finally taking an action with the computational cost involved
in getting to that point.
For cognitive scientists, bounded optimality offers a way to theorize about the optimal cog-
nitive processes that intelligent agents should engage in when trying to make a decision (44,45).
Becauseitemphasizesrationaluseofthecognitiveresourcesanagentisabletoapply,thisapproach
has been referred to as resource-rational analysis (17, 18). Considering how an agent should ra-
tionally deploy its cognitive resources provides a way to explain why people may choose to adoptparticular heuristics—even if those heuristics result in systematic biases—and to make generaliz-
able predictions about the kinds of cognitive strategies that we expect people to engage in.
Recent research has recast some of the classic heuristics discovered by psychologists from the
perspective of the rational use of cognitive resources. For example, focusing on extreme events
when considering the outcomes of a decision is a strategy that can minimize the variance of esti-
matesofexpectedutilitybasedonsmallsamples,eventhoughitintroducesabiastothoseestimates(46).Thinkinginthesetermsallowsustopotentiallybegintoreconcilethevariousheuristicsand
biases identified by psychologists into a broader mathematical theory.
To the engineer,resource rationality offers the potential to make better predictions about hu-
man behavior in a way that incorporates realistic assumptions about the cognitive limitations of
human agents. Generalizability results from the fact that deriving an optimal resource-rational
strategycanbeformulatedasasequentialdecisionproblem.Anagenttryingtomakeadecisionisgoing to execute the sequence of computations that provide information about the possible out-
comesoftheiractions,atsomepointselectinganactiontoperformbasedonthisinformation.The
sequentialdecisionproblemherecorrespondstothechoiceofthatsequenceofcomputations:We
can construe each computation as a kind of mental action,ending with the decision that we have
done enough computation and are ready to act as the end of the sequence.
Expressed in these terms, it is possible to see that we can use familiar tools such as Markov
decisionprocesses(MDPs)toformalizetheinternaldecision-makingwedoabouthowtodeploy
our cognitive resources. Solving the resulting MDPs—referred to as meta-level MDPs (47)—provides a way to derive predictions about behavior. Crucially, this approach carries with it the
same generality as the classic theory of rational action.It simply moves rationality up to the level
ofthechoiceofhowtodeploycognitiveresourcesandrequiresustobeexplicitaboutwhatthoseresources might be.
To provide a concrete example, one recent paper (48) used this approach to examine how to
modelattentionallocationinasimpledecision-makingtask.Inthistask,peoplearepresentedwith
40 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

three objects—in this case, snack foods—and asked to decide which they prefer. While they are
doingthis,theirgazeisrecordedusinganeyetracker.Peopleshowaconsistentpatternofbehavior
inthistask.Forexample,theyspendmoretimelookingatitemsthattheyassignahighersubjective
value.Thesepatternsofbehaviorcanbeexplainedbyassumingthatpeoplearetryingtoestimate
thesubjectivevalueofeachitem,andthateachmomenttheyspendlookingatanoptionprovidesa sample from a Gaussian distribution centered on that value.The problem of deciding whether
to sample can then be formulated as an MDP, and the resulting policy generates predictions
about which objects people look at and in what sequence. We spend more time looking at itemswith higher subjective value because those are the items that are most relevant to our ultimate
decision.
Describing cognitive processes in terms of the solutions to MDPs has the virtue of charac-
terizing human cognition using formal tools that are likely to be familiar to those working in
control theory.Other work on resource rationality has likewise employed formalisms that will be
familiartoengineers.Forexample,onelineofworkfocusesontheinformation-theoreticcostsofmaintainingmentalrepresentationsatagivendegreeofprecision(49,50).Thisapproachalsohas
connections with work in economics that explains apparently irrational aspects of human choice
in terms of rational inattention, where information-theoretic costs are assumed to apply to theprecision of the signal an agent uses to inform a decision (51,52).
Resourcerationalityoffersanewsetoftoolsforcapturinghumanbehaviorwithgreaterpreci-
sion in a way that is compatible with standard modeling techniques used in robotics and control.
Beingabletomakegeneralizablepredictionsabouthowlongitwilltakepeopletomakeadecision,
whatinformationtheyaregoingtoseekwhenmakingthatdecision,andwhatkindsofinformationtheyarelikelynottoincludewhenmakingadecisioncanfacilitatethedesignofhuman–machine
interfaces. Furthermore, it is possible to use this kind of approach to engage with the question
of how to improve human decision-making: If we assume that people are rational but resourcelimited,wecanthinkabouthowassistiverobotsmightmodifytheenvironmentsinwhichhumans
are making decisions to allow them to make better use of those resources.
3. INVERSE MODELS OF HUMAN DECISION-MAKING
While forward models can help us to make predictions about what people will do given theirpreferences, this is typically not all we need to know in order to design systems that are able to
interact effectively with humans. In an ideal world, autonomous systems would effectively helpfulfill people’s needs and desires, which can change from person to person and situation to situ-
ation. We need a way to infer those needs and desires from people’s behavior—inverse models.
Furthermore,peopleareadaptive;theyoftenchangetheirbehaviorinresponsetoasystembasedon their best guess as to how it functions and may even expect the system to do the same. So it
is not enough to be able to make inferences about people’s needs and desires; we also need to
anticipate the inferences that people will be making in turn.
Developing systems that can solve these problems is a daunting challenge, but fortunately,
we can take inspiration from existing systems that must regularly interact with humans: other
humans. And while nearly everyone has had to deal with other humans, cognitive science offers
an extensive,systematic understanding of how we effectively solve the problem of understanding
and interacting with others in our everyday lives.
In this section, we turn our focus toward what cognitive science has to say about people’s in-
verse models of cognition and action. One of the most remarkable capacities that humans have
is the ability to understand the hidden mental states that give rise to other people’s observablebehavior (3, 53). This ability is often referred to as theory of mind, and cognitive scientists have
www.annualreviews.org CognitiveScienceasaSourceofModels 41
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

studied it in adults (3),children (54),infants (55),and even other species (1).Theory of mind has
played a key role in our evolution as a social species capable of large-scale culture,coordination,
andcooperation,anditsdevelopmentwithinthefirstyearoflifeisamajormilestonethatenables
us to comprehend and participate in the social world (2).
One exciting development in the recent study of theory of mind has been the application of
ideas from economics, artificial intelligence, and control theory to characterizing mental state
inference in computational terms. These approaches are reminiscent of methods familiar to en-
gineers, such as imitation learning, inverse reinforcement learning, and apprenticeship learning(56–58).However,inmodelingthevarietiesofhumansocialinferenceandinteraction,theydepart
from and extend these ideas in numerous ways.This presents an exciting opportunity for collab-
orationbetweenthecognitivesciencesandengineeringbyprovidingnewperspectivesoninversedecision-making but in a shared conceptual and technical framework.
Here, we focus on two lines of research on inverse models at the intersection of artificial in-
telligenceandcognitivescience.Thefirstisworkoncatalogingandsystematizingtheconceptualprimitives involved in mental state inference—for example,how people reason about mental en-
tities like beliefs,desires,intentions,emotions,and so on.The second is work on inverse models
inthecontextofteachingandcommunication,whichareamongthemostbasictypesofsocialin-teractionsthatalsoexposethecomplexityofhowhumansusetheoryofmindproductively.Along
the way, we discuss cases in which ideas from cognitive science have already been applied to the
design of automated systems, limitations of existing approaches, and the possibilities for future
research and applications.
3.1. Identifying the Building Blocks of Theory of Mind
Putinsimplecomputationalterms,theoryofmindisaninferenceproblem.Thatis,givenlimitedobservations of a process (e.g., a person’s behavior), the task is to identify the hidden variables
that produced those observations (e.g.,the person’s thoughts,desires,or feelings).Of course,thisrequires having not only concepts like thoughts and desires but also an understanding of how
theseelementscombinetoproducebehavior.Thiscanbeunderstoodinroughanalogytoanother
machine learning problem: parsing natural language. For example, inferring the parse tree of aparticular sentence is jointly constrained by knowledge of primitive types of words (e.g., nouns,
verbs, and prepositions) and a grammar of how words tend to be combined. Recent models of
theory of mind can be understood in terms of this linguistic metaphor: To explain how humansparse the behavior of other agents, we must understand the mental state primitives and mental
state grammars that dictate how they are combined.
Incidentally, we have already covered one possible theory of how people parse behavior: ex-
pectedutilitytheory(19,20).Takenasagenerativemodelofpeople’sintentionalaction,expected
utilitytheorypositsthatothershavebeliefsaboutthestateoftheworld(e.g.,thebeliefthatthere
is a burger joint down the street) and desires that certain states of the world be realized (e.g.,thedesiretoeataburgerforlunch)andthatpeopleactrationallytorealizetheirdesiresgiventheirbe-
liefs(e.g.,theactofwalkingdownthestreettotheburgerjoint).Asanaccountoftheoryofmind,
inverse expected utility theory makes several generalizable predictions that have been confirmed
with human experiments.For example,adults,children,and infants can reason about how others
integrateinformationaboutgoalsandactioncosts(4,59,60),featuresofdifferentchoices(61),thestatistics of the environment (5), and limited perception of the environment (62). Findings such
as these have led to the proposal that human common sense psychology consists of a naive util-
ity calculus where we abstractly reason about other decision-makers as utility-maximizing agents(6).
42 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

At a broad level,the formal tools used to characterize inverse utility theory will be familiar to
those from a control background, as they build on standard formalisms such as MDPs, partially
observable MDPs (POMDPs), and inverse reinforcement learning (63). However, there are
interesting differences between how they are applied as models of human inference and their
typical engineering applications.For example,cognitive models tend to assume that people havehighly structured representations of others’ mental states (e.g., discrete objects), whereas in
engineering applications the state representations are less structured [e.g.,weights on a vector of
continuousfeatures(58)].Thisreflectsthefactthatcognitivescientistsaimtoexplainhowpeoplecan rapidly draw inferences based on only a few observations and a large base of background
knowledge, while engineers are often trying to analyze large data sets of expert trajectories with
minimal fine-tuning. As a result, cognitive scientists tend to use Bayesian methods, while engi-neersarelikelytobemorefamiliarwithmethodsdesignedtoscaletolargedatasets.Animportant
direction for future work is developing methods that cut across these different research agendas
and can replicate the sophistication of human theory of mind with tractable implementations.
Expected utility theory captures an important dimension of how humans parse others’ behav-
iorintermsofbeliefs,desires,andintentions.Butpsychologistshavealsolongstudiedothertypes
ofmentalstatesthatpeoplereasonabout,suchasemotions,habits,norms,rules,values,andsocialaffinities,tonameonlyafew(3).Whilethetraditionalframeworksofexpectedutilityandinverse
reinforcement learning have not typically focused on these kinds of mental states, cognitive sci-
entistshavemadegreatstridesinextendingtheformalismtostudythesetypesofrepresentations.
For example, inverse planning models have been combined with models of habits (64), emotion
and appraisal (65–67), responsibility judgments (68), values and norms in moral dilemmas (69),and social groups (70, 71). Although we do not typically think of robots as having these types of
internal states, systems that are expected to interact with humans invariably need a basic under-
standingofthecompleterepertoireofpsychologicalstatesthataffectourindividualandcollectivebehavior.
The models described so far generally rely on assuming the standard formulation of rational
actionasoptimizingautilityfunctiondefinedoverstatesofanMDP.Forinstance,theycanexpresstheideaofreachingagoalstatewhileattemptingtominimizecostsalongtheway,buttheycannot
generally express history-dependent or temporally specified constraints, such as going to a goal
state only after accomplishing a subgoal. Recent work in both cognitive science and computerscience has attempted to remedy this by introducing more flexible logics to express an agent’s
utilities,including linear temporal logic (72–74),finite state machines (75),and simple programs
composed of subprocesses (76). These methods are powerful because they can express rationalcognition and action in a rich, compositional manner that may reflect human intuitions about
agency.At the same time,this expressiveness comes at the cost of more complex and costly infer-
ence, and identifying appropriate constraints and settings in which tractable inference methods
can be applied is an active area of research.
A complementary method for sidestepping strong assumptions about the structure of rational
action is to try to learn a theory of mind directly from data without recourse to predefined
structural priors (e.g., using a neural network). Rabinowitz et al. (77) took this approach by
generating a large data set of behaviors from synthetic agents with different goals, utilities, andperceptual abilities, and then using meta-learning to train networks with no prior conception of
theory of mind or rationality to predict features of future behavior. Their networks were able
to acquire a low-dimensional embedding of behaviors and agent types capable of re-creatingseveral qualitative findings associated with theory of mind, such as inferences about goals, costs,
perceptions, and, to a certain extent, false beliefs. This work is an important demonstration of
how relatively standard machine learning methods can learn theory of mind–like representations
www.annualreviews.org CognitiveScienceasaSourceofModels 43
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

given enough data and computation. However, studies by Nematzadeh et al. (78) have indicated
that standard neural networks are limited in their capacity to explicitly represent false beliefs
because they do not distinguish between appearance and reality,which psychologists consider to
be a defining feature of theory of mind (54). This suggests that some structural priors about the
natureofrationalactionarelikelyneededtocapturetheconceptualprimitivesandgrammarthatconstitute human theory of mind.
3.2. Identifying Generalizable Principles of Communication and Teaching
The previous section focused on theory of mind as a pure inference problem,but theory of mindalso influences how people act and interact. Building on ideas originally explored by philoso-
phers (79–81), developmental psychologists and linguists have extensively studied the principles
underlying how people use theory of mind to learn from others and communicate (2, 82, 83). Akey idea is that when people communicate (by saying words, making gestures, providing exam-
ples,etc.),theydosowithcommunicativegoalslikemodifyingthereceiver’smentalstateorfuture
actions. The person receiving these communicative signals can then reason about these goals toflexibly and efficiently draw inferences about what the sender meant to convey. Note that this
process requires both the receiver and sender to have a capacity for theory of mind and, more
specifically, a capacity for recursive theory of mind, in which one agent reasons about anotheragent reasoning about the original agent (and potentially up to higher levels of recursion).Com-
putational research over the past few years has formalized and extended many of these findings
withintheframeworkofprobabilisticinferenceanddecision-making(84,85).Here,weprovideabroad overview of these developments.
One approach to studying communication in computational terms is the Bayesian pedagogy
and cooperative communication framework (84, 86, 87), which characterizes how a teacher who
presentsdataandalearnerwhointerpretsdatashouldcoordinatetocommunicateefficientlyand
successfully.Toillustratethisidea,supposeyouwantedtoteachsomeonetheconceptofevennum-bersbygivingthemaseriesofexamples.Someexampleswillbebetterthanothers—forinstance,
{2,2,2} is technically a sequence of even numbers,but it is not very informative,whereas {2,4,6}
is clearly more helpful.Additionally,if the person receiving these examples knows you are beinghelpful,theycandrawevenstrongerinferencesbasedonwhattheyareshown.Bayesianpedagogy
modelsformalizethisintuitionabouthelpful,informativeexamplesintermsofrecursivelydefined
teacher–learnerequations,whosefixedpointsareoptimalteacher–learnercommunicationproto-cols.This approach has been successful in characterizing how both adults and children teach and
learn concepts during pedagogical interactions (88,89).Additionally,recent theoretical work has
established a direct correspondence between optimal transport problems and the equations thatcharacterizeteacher–learnerfixedpoints(90,91).Thisopensthedoorforalgorithmicinsightsto
be shared between engineering disciplines, such as operations research and the study of optimal
cooperative communication in humans.
Cooperativecommunicationmodelscapturesettingsinwhichateacherhasasole,explicitgoal
of being helpful and informative. However, communication is not always so clear-cut, and cog-
nitive scientists are modeling more complex communicative situations within the rational speech
act(RSA)framework(85).Forexample,peoplemayhavegoalsotherthanbeinginformative,like
being succinct or inoffensive, and they may expect others to perform joint inference over thesenoncommunicative goals. This process can give rise to linguistic phenomena as varied as hyper-
bole [“This kettle cost $1,000!”(92)] and politeness [“Your poem wasn’t bad!”(93)].And while it
maybeextremetoexpectautomatedsystemstounderstandironichumor,theywilllikelyneedtorecognize more mundane forms of indirect speech.
44 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

More broadly,RSA can characterize how the context surrounding communicative acts shapes
their meaning. For example, the statement “It’s warm today” has different consequences if said
during the spring as opposed to the winter (e.g.,you would likely wear shorts in the first but not
the second case) (94). Computational cognitive models of how humans flexibly reason about a
sharedcontextareakeypartofunderstandinggeneralstatementsabouttheworld(95).Similarly,the ability to establish contexts as communicative (e.g.,intentionally getting someone’s attention
inordertoconveysomeinformation)isanessentialprecursortothetypesofcooperativecommu-
nication interactions discussed above (96,97).Formal models that combine RSA with sequentialdecision-makingandinferenceaboutwhetherpartnershaveinformativegoalscanprovideastart-
ing point for implementing such flexible reasoning about communicative context in autonomous
systems (98,99).
3.3. Applying Human Inverse Models to the Design of Autonomous Systems
Ideasfromthecomputationalcognitivescienceofmentalstateinferenceandcommunicationhavealready begun to inform research in human–robot interaction and reinforcement learning. Thisshowcases the potential for scaling up probabilistic models of cognition to complex, real-world
domains, while also revealing novel insights about communication and teaching. For example,
work in motion planning has led to robots capable of legible motion (9, 11), in which action se-quencesaremodifiedtoallowhumanstomorequicklyandsuccessfullyunderstandarobot’sgoals.
Related work by Ho et al. (10) showed that when learning from human demonstrations, inverse
reinforcementlearningalgorithmscanbenefitfrombeingshownintentionallycommunicativeex-pertbehaviors( Figure 4 ).Thesemodificationsbybothrobotsandhumansaredirectlyanalogous
to the approach taken in the Bayesian pedagogy framework and demonstrate how consideration
of communicative goals can facilitate human–machine teaching,learning,and cooperation.
Insights from cognitive science can also inform the design of learning algorithms themselves.
For example, humans readily use rewards and punishments to modify the behavior of other ani-malsandevenotherhumans,andthereisgoodreasontobelievethatsimilarprinciplesofmental
state inference and communication apply to these interactions (101). In a series of experiments
with humans interacting with different learning algorithms, Ho et al. (102, 103) demonstratedpreciselythis.Specifically,theyfoundthatpeopledonotuserewardsinamannerconsistentwith
thestandardinterpretationasaquantitytodirectlymaximize,asistypicallydoneinreinforcement
learning.Rather,they expect learners to reason about a teacher’s pedagogical goals and interpretrewards as signaling information about whether an agent is headed in the right direction during
the learning process. Such findings help motivate the development of learning algorithms that
interpretrewardsinmoresophisticatedwaysandattempttoinferpeople’steachingstrategiesandgoals (104,105).
Along similar lines, MacGlashan et al. (106) found that the structure of a human teacher’s
feedbackdependsonanagent’scurrentstageoflearning—asimpleformofcontext.Inabehavioralstudy, the authors had human participants interact with either a completely naive agent or an
expert agent that knew the optimal path to a goal. When the naive agent took a suboptimal but
moderatelygoodaction,participantsprovidedahighreward,whiletheygavetheexpertagentwho
should have known better a low or even negative reward for taking the same action.The logic of
this strategy closely resembles that of the advantage function,which reflects the relative value ofeach action in a state under the agent’s current policy (107). This insight motivated the design
of the Convergent Actor–Critic by Humans (COACH) algorithm,which treats human feedback
as an advantage signal. Subsequent work has also successfully applied COACH to training deeplearning agents (108).
www.annualreviews.org CognitiveScienceasaSourceofModels 45
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

–2.5 –2.0 –1.5 –1.0 –0.5 0.0 –3.0Doing
ShowingUnderlying
reward
function
Doing
(MLIRL)
Showing
(MLIRL)
Reward values
Figure 4
Learning from humans with communicative intent.Ho et al.(10) recruited human participants to perform grid navigation tasks that
required reaching a goal state while not losing points.Each column represents one trial.Row 1 represents the true underlying reward
function,where white tiles are 0 points,red tiles are −2 points,and the yellow goal tile is 10 points.Participants could not directly view
the reward values,but were shown colors on the grid (orange,purple,and blue) and told the value of each color (e.g.,that orange and
purple are safe).Row 2 shows the visible layout of each grid,and each black line represents one participant’s trajectory on the task when
they were told only to do the task.Row 3 shows the same grid and trajectories for participants told to do the task as well as show the
reward function to an anonymous observer.Rows 4 and 5 show the reward weights estimated by maximum-likelihood inverse
reinforcement learning (MLIRL) (100) when given the doing versus showing demonstrations.Agents trained with the showing
demonstrations obtained better estimates of the underlying reward function than those trained with the doing demonstrations.Figure
adapted from Reference 10.
Additionally, researchers in control and robotics have developed unifying frameworks for
modeling cooperative interactions likely to be faced in human–robot applications. For instance,
cooperative inverse reinforcement learning (109) defines a general class of games from which
specific cooperative strategies can be derived (legible motion, requests for information, etc.)
based on a particular setup. Similarly, reward-rational learning (110) has been proposed as a
frameworkforcharacterizingdifferenttypesofhuman–robotinteractionproblems(e.g.,learning
from demonstrations, feedback, or examples) in terms of inference about human preferences
that may be implicit.Combining these general computational frameworks with insights from the
cognitive science of human decision-making will be an important direction for future research.
4. OPPORTUNITIES FOR FURTHER RESEARCH
The research we have reviewed is only a starting point for exploring potential applications of
cognitivesciencetoengineering,robotics,andcontrol.Thereareanumberofexcitingdirections
based on the ideas we have covered.Here,we focus on three broad future directions.
46 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

4.1. Inverse Resource Rationality
As we have noted,cognitive scientists have proposed that people reason about others as expected
utility maximizers. Additionally, we discussed how expected utility theory is inaccurate and how
resource rationality is a promising alternative framework.This raises the obvious possibility thatpeople understand that others have limited cognitive resources and therefore reason about them
not as pure utility maximizers but as resource-rational utility maximizers. Importantly, progress
here will depend on the continued development of plausible forward resource-rational models,especially models of planning (e.g.,111,112).Nonetheless,several lines of research have already
begun to explore inverse resource rationality. For example, research on perspective-taking and
communication has shown that people can flexibly reason about the division of cognitive labor
required for efficient communication (113).Additionally,recent work has demonstrated how hu-
mans can infer preferences by jointly reasoning about the time it takes to make a decision andthe decision itself (114),while other studies have shown how people reason about suboptimal or
inconsistent planning (115–117). These ideas have begun to be incorporated into inverse rein-
forcement learning settings (118). As with models based on inverse expected utility theory, un-derstanding people’s inverse models of resource-rational processes will be essential for how they
interpret how machines make decisions and can also serve as inspiration for how machines inter-
pret and interact with humans.
4.2. Black Box Versus Theory-Driven Models for Inferring Intent
Inourdiscussionofforwardandinversemodels,weencounteredthreedifferentexamplesofhow
machine learning tools can complement traditional psychological theory building.This included
work on how humans make risky choices,work on how humans make moral decisions,and workon learning theory of mind concepts without hard-coding conceptual primitives or a principle of
rationality.These approaches have illustrated how large data sets can be combined with machine
learning tools to search the vast space of cognitive models in an efficient manner.However,theyalso illustrated some of the limitations of a purely black box approach and the benefits of also
incorporatingexplicit,interpretabletheoriesandstructuredpriorknowledge.Thus,animportant
direction for future work will be developing methods that seamlessly integrate the benefits ofeachapproach—ontheonehand,thescalabilityofblackboxmethods,andontheotherhand,the
efficiency and interpretability of explicit psychological theories.
4.3. Schemas and Mechanisms for Human–Robot Interaction
Unlike settings involving a single agent, interactions between two or more agents are challeng-
ing to design because they are fundamentally ill defined: Each agent might have their own goals
and beliefs,which means there may not exist a single yardstick by which to measure whether the
engineering problem has been solved. This has occasionally prompted researchers to ask them-
selves,“If multi-agent learning is the answer,what is the question?”(119).
We propose that cognitive science is uniquely positioned to provide guidance on the question
that multiagent learning answers,in the form of schemas and mechanisms grounded in the types
of interactions that humans are adapted for. We have already encountered one example of this:
The interaction of a teacher and learner engaged in cooperative communication can serve as atemplate for developing robots capable of legible action and value alignment (9, 109). Beyond
this, there are many other types of human interactions and sociocognitive mechanisms that we
have not discussed that could inspire future research.For example,humans form joint intentions
www.annualreviews.org CognitiveScienceasaSourceofModels 47
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

toachievesharedgoals,andthisunderliesourabilitytocooperativelysolvenovelproblems(120).
At a broader scale, human interactions are often shaped by norms, which can be understood as
shared behavioral tendencies that generalize across interactions with different agents in a pop-
ulation. Norms and normative cognition have been extensively studied in cognitive science and
psychology,andresearchershavebeguntoexploretheseprocessescomputationally(121).Finally,anadditionalbenefitofdesigningautonomoussystemsaroundhowhumansactuallyinteractisthe
potential for new insights into those very interactions, leading to further collaboration between
the cognitive sciences and engineering disciplines.
5. CONCLUSION
Atthemoment,thereisnounifiedmodelofhumancognitionanddecision-makingthatengineerscandrawonwhendesigningtheirsystems.Nonetheless,cognitivesciencehasmuchtoofferthosedesigning autonomous systems that interact with humans. In particular, cognitive science has a
rich trove of theories and methods for systematically studying how humans think, decide, and
interact with one another, and these discoveries are increasingly being couched in formal termsthat are familiar to researchers in robotics and control. Here, we have discussed several recent
frameworks and methodologies, such as the synthesis of black box and theory-driven methods,
resource-rationaldecision-making,cooperativecognition,andRSAtheory.Wethensurveyedhowthey have been applied to derive insights into the mechanisms and principles underlying people’s
forwardandinversemodelsofdecision-making.Asautonomoussystemscontinuetobecomemore
commonplaceinpeople’severydaylives,weexpectengineerswillalsoneedtothinksystematicallyabout how the humans their systems encounter will make decisions. We hope this review can
provide clarity on the types of actionable insights and open questions that sit at the intersection
of cognitive science and control research.
DISCLOSURE STATEMENT
The authors are not aware of any affiliations, memberships, funding, or financial holdings thatmight be perceived as affecting the objectivity of this review.
ACKNOWLEDGMENTS
ThisworkwasfundedbytheNationalScienceFoundation(grant1545126),theJohnTempletonFoundation (grant 61454), and the Air Force Office of Scientific Research (grant FA 9550-18-
1-0077).
LITERATURE CITED
1. PremackD,WoodruffG.1978.Doesthechimpanzeehaveatheoryofmind? Behav.BrainSci. 1:515–26
2. Tomasello M,Carpenter M,Call J,Behne T,Moll H.2005.Understanding and sharing intentions: the
origins of cultural cognition. Behav.BrainSci. 28:675–90
3. MalleBF.2008.Thefundamentaltools,andpossiblyuniversals,ofhumansocialcognition.In Handbook
ofMotivationandCognitionAcrossCultures ,ed.R Sorrentino,S Yamaguchi,pp.267–96.San Diego,CA:
Academic
4. BakerCL,SaxeR,TenenbaumJB.2009.Actionunderstandingasinverseplanning. Cognition 113:329–49
5. LucasCG,GriffithsTL,XuF,FawcettC,GopnikA,etal.2014.Thechildaseconometrician:arational
model of preference understanding in children. PLOSONE 9:e92160
6. Jara-Ettinger J, Gweon H, Schulz LE, Tenenbaum JB. 2016. The naïve utility calculus: computational
principles underlying commonsense psychology. TrendsCogn.Sci. 20:589–604
7. Scassellati B.2002.Theory of mind for a humanoid robot. Auton.Robots 12:13–24
48 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

8. Breazeal C.2003.Toward sociable robots. Robot.Auton.Syst. 42:167–75
9. Dragan AD, Lee KC, Srinivasa SS. 2013. Legibility and predictability of robot motion. In 2013 8th
ACM/IEEEInternationalConferenceonHuman-RobotInteraction(HRI) ,pp.301–8.Piscataway,NJ: IEEE
10. Ho MK,Littman M,MacGlashan J,Cushman F,Austerweil JL.2016.Showing versus doing: teaching
by demonstration. In Advances in Neural Information Processing Systems 29 , ed. DD Lee, M Sugiyama,
UV Luxburg,I Guyon,R Garnett,pp.3027–35.Red Hook,NY: Curran
11. Fisac JF, Gates MA, Hamrick JB, Liu C, Hadfield-Menell D, et al. 2020. Pragmatic-pedagogic value
alignment.In RoboticsResearch:The18thInternationalSymposiumISRR ,ed.NAmato,GHager,SThomas,
M Torres-Torriti,pp.49–57.Cham,Switz.: Springer
12. SunR,ed.2008. TheCambridgeHandbookofComputationalPsychology .Cambridge,UK:CambridgeUniv.
Press
13. Bernoulli D.1738.Exposition of a new theory on the measurement of risk. Econometrica 22:22–36
14. TverskyA,KahnemanD.1974.Judgmentunderuncertainty:heuristicsandbiases. Science185:1124–31
15. Peterson JC, Bourgin D, Agrawal M, Reichman D, Griffiths TL. 2021. Using large-scale experiments
and machine learning to discover theories of human decision-making. Science372:1209–14
16. Agrawal M,Peterson JC,Griffiths TL.2020.Scaling up psychology via scientific regret minimization.
PNAS117:8825–35
17. Griffiths TL,Lieder F,Goodman ND.2015.Rational use of cognitive resources: levels of analysis be-
tween the computational and the algorithmic. Top.Cogn.Sci. 7:217–29
18. LiederF,GriffithsTL.2020.Resource-rationalanalysis:understandinghumancognitionastheoptimal
use of limited computational resources. Behav.BrainSci. 43:e1
19. VonNeumannJ,MorgensternO.1944. TheoryofGamesandEconomicBehavior .Princeton,NJ:Princeton
Univ.Press
20. Savage LJ.1972. TheFoundationsofStatistics .New York: Courier
21. Ziebart BD,Maas A,Bagnell JA,Dey AK.2008.Maximum entropy inverse reinforcement learning.In
ProceedingsoftheTwenty-ThirdAAAIConferenceonArtificialIntelligence , Vol. 3, pp. 1433–38. Palo Alto,
CA: AAAI Press
22. McFaddenD.1973.Conditionallogitanalysisofqualitativechoicebehavior.In FrontiersinEconometrics ,
ed.P Zarembka,pp.105–35.New York: Academic
23. KahnemanD,TverskyA.1979.Prospecttheory:ananalysisofdecisionsunderrisk. Econometrica 47:263–
92
24. Tversky A,Kahneman D.1992.Advances in prospecttheory: cumulativerepresentation of uncertainty.
J.RiskUncertain. 5:297–323
25. Gigerenzer G,Todd PM.1999. SimpleHeuristicsThatMakeUsSmart .Oxford,UK: Oxford Univ.Press
26. Bordalo P,Gennaioli N,Shleifer A.2012.Salience theory of choice under risk. Q.J.Econ. 127:1243–85
27. RatcliffR,SmithPL,BrownSD,McKoonG.2016.Diffusiondecisionmodel:currentissuesandhistory.
TrendsCogn.Sci. 20:260–81
28. TsetsosK,GaoJ,McClellandJL,UsherM.2012.Usingtime-varyingevidencetotestmodelsofdecision
dynamics: bounded diffusion versus the leaky competing accumulator model. Front.Neurosci. 6:79
29. Kiani R,Shadlen MN.2009.Representation of confidence associated with a decision by neurons in the
parietal cortex. Science324:759–64
30. LatimerKW,YatesJL,MeisterML,HukAC,PillowJW.2015.Single-trialspiketrainsinparietalcortex
reveal discrete steps during decision-making. Science349:184–87
31. Erev I,Ert E,Plonsky O,Cohen D,Cohen O.2017.From anomalies to forecasts: toward a descriptive
model of decisions under risk,under ambiguity,and from experience. Psychol.Rev. 124:369–409
32. NotiG,LeviE,KolumbusY,DanielyA.2016.Behavior-basedmachine-learning:ahybridapproachfor
predicting human decision making.arXiv:1611.10228 [cs.LG]
33. PlonskyO,ErevI,HazanT,TennenholtzM.2017.Psychologicalforest:predictinghumanbehavior.In
ProceedingsoftheThirty-FirstAAAIConferenceonArtificialIntelligence ,pp.656–62.Palo Alto,CA: AAAI
Press
34. Geman S, Bienenstock E, Doursat R. 1992. Neural networks and the bias/variance dilemma. Neural
Comput.4:1–58
www.annualreviews.org CognitiveScienceasaSourceofModels 49
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

35. Fudenberg D, Kleinberg J, Liang A, Mullainathan S. 2019. Measuring the completeness of theories.
arXiv:1910.07022 [econ.TH]
36. PeysakhovichA,NaeckerJ.2017.Usingmethodsfrommachinelearning toevaluatebehavioralmodels
of choice under risk and ambiguity. J.Econ.Behav.Organ. 133:373–84
37. Plonsky O, Apel R, Ert E, Tennenholtz M, Bourgin D, et al. 2019. Predicting human decisions with
behavioral theories and machine learning.arXiv:1904.06866 [cs.AI]
38. Bourgin DD, Peterson JC, Reichman D, Russell SJ, Griffiths TL. 2019. Cognitive model priors for
predicting human decisions. In Proceedings of the 36th International Conference on Machine Learning ,e d .
K Chaudhuri,R Salakhutdinov,pp.5133–41.Proc.Mach.Learn.Res.97.N.p.: PMLR
39. Awad E, Dsouza S, Kim R, Schulz J, Henrich J, et al. 2018. The moral machine experiment. Nature
563:59–64
40. Thomson JJ.1976.Killing,letting die,and the trolley problem. Monist59:204–17
41. Simon HA.1955.A behavioral model of rational choice. Q.J.Econ. 69:99–118
42. Horvitz EJ. 1987. Reasoning about beliefs and actions under computational resource constraints. In
ProceedingsoftheThirdConferenceonUncertaintyinArtificialIntelligence ,pp.429–47.Arlington,VA:AUAI
Press
43. Russell S,Wefald E.1991.Principles of metareasoning. Artif.Intell. 49:361–95
44. Gershman SJ,Horvitz EJ,Tenenbaum JB.2015.Computational rationality: a converging paradigm for
intelligence in brains,minds,and machines. Science349:273–78
45. LewisRL,HowesA,SinghS.2014.Computationalrationality:linkingmechanismandbehaviorthrough
bounded utility maximization. Top.Cogn.Sci. 6:279–311
46. LiederF,GriffithsTL,HsuM.2018.Overrepresentationofextremeeventsindecisionmakingreflects
rational use of cognitive resources. Psychol.Rev. 125:1–32
47. Hay N, Russell S, Tolpin D, Shimony S. 2012. Selecting computations: theory and applications. In
Proceedings of the 28th Conference on Uncertainty in Artificial Intelligence , ed. N de Freitas, K Murphy,
pp.346–55.Arlington,VA: AUAI Press
48. CallawayF,RangelA,GriffithsTL.2021.Fixationpatternsinsimplechoicereflectoptimalinformation
sampling.PLOSComput.Biol. 17:e1008863
49. OrtegaDA,BraunPA.2011.Information,utilityandboundedrationality.In ArtificialGeneralIntelligence:
4thInternationalConference,AGI2011 ,ed.JSchmidhuber,KRThórisson,MLooks,pp.269–74.Berlin:
Springer
50. BhuiR,GershmanSJ.2018.Decisionbysamplingimplementsefficientcodingofpsychoeconomicfunc-
tions.Psychol.Rev. 125:985–1001
51. Sims CA.2003.Implications of rational inattention. J.Monet.Econ. 50:665–90
52. Gershman SJ,Bhui R.2020.Rationally inattentive intertemporal choice. Nat.Commun. 11:3365
53. GergelyG,CsibraG.2003.Teleologicalreasoningininfancy:thenaïvetheoryofrationalaction. Trends
Cogn.Sci. 7:287–92
54. Flavell JH.2004.Theory-of-mind development: retrospect and prospect. Merrill-PalmerQ. 50:274–90
55. Gergely G, Nádasdy Z, Csibra G, Bíró S. 1995. Taking the intentional stance at 12 months of age.
Cognition 56:165–93
56. AbbeelP,NgAY.2004.Apprenticeshiplearningviainversereinforcementlearning.In Proceedingsofthe
Twenty-FirstInternationalConferenceonMachineLearning ,pap.1.NewY ork:A CM
57. Argall BD,Chernova S,Veloso M,Browning B.2009.A survey of robot learning from demonstration.
Robot.Auton.Syst. 57:469–83
58. Arora S,Doshi P.2021.A survey of inverse reinforcement learning: challenges,methods and progress.
Artif.Intell. 297:103500
59. Jara-Ettinger J,Gweon H,Tenenbaum JB,Schulz LE.2015.Children’s understanding of the costs and
rewards underlying rational action. Cognition 140:14–23
60. Liu S, Ullman TD, Tenenbaum JB, Spelke ES. 2017. Ten-month-old infants infer the value of goals
from the costs of actions. Science358:1038–41
61. Jern A, Lucas CG, Kemp C. 2017. People learn other people’s preferences through inverse decision-
making.Cognition 168:46–64
50 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

62. Baker CL, Jara-Ettinger J, Saxe R, Tenenbaum JB. 2017. Rational quantitative attribution of beliefs,
desires and percepts in human mentalizing. Nat.Hum.Behav. 1:0064
63. Jara-EttingerJ.2019.Theoryofmindasinversereinforcementlearning. Curr.Opin.Behav.Sci. 29:105–
10
64. Gershman SJ,Gerstenberg T,Baker CL,Cushman FA.2016.Plans,habits,and theory of mind. PLOS
ONE11:e0162246
65. Ong DC,Zaki J,Goodman ND.2015.Affective cognition: exploring lay theories of emotion. Cognition
143:141–62
66. SaxeR,HoulihanSD.2017.Formalizingemotionconceptswithin aBayesianmodeloftheoryofmind.
Curr.Opin.Psychol. 17:15–21
67. Ong DC,Zaki J,Goodman ND.2019.Computational models of emotion inference in theory of mind:
a review and roadmap. Top.Cogn.Sci. 11:338–57
68. Gerstenberg T, Ullman TD, Nagel J, Kleiman-Weiner M, Lagnado DA, Tenenbaum JB. 2018. Lucky
or clever? From expectations to responsibility judgments. Cognition 177:122–41
69. Kleiman-WeinerM,GerstenbergT,LevineS,TenenbaumJB.2015.Inferenceofintentionandpermis-
sibilityinmoraldecisionmaking.In Proceedingsofthe37thAnnualMeetingoftheCognitiveScienceSociety ,
ed.DNoelle,RDale,ASWarlaumont,JYoshimi,TMatlock,etal.,pp.1123–28.RedHook,NY:Curran
70. LauT,PouncyHT,GershmanSJ,CikaraM.2018.Discoveringsocialgroupsvialatentstructurelearn-
ing.J.Exp.Psychol.Gen. 147:1881–91
71. ShumM,Kleiman-WeinerM,LittmanML,TenenbaumJB.2019.Theoryofminds:understandingbe-
havioringroupsthroughinverseplanning.In ProceedingsoftheThirty-ThirdAAAIConferenceonArtificial
Intelligence ,pp.6163–70.Palo Alto,CA: AAAI Press
72. Littman ML, Topcu U, Fu J, Isbell C, Wen M, MacGlashan J. 2017. Environment-independent task
specifications via GLTL.arXiv:1704.04341 [cs.AI]
73. Velez-Ginorio J, Siegel MH, Tenenbaum JB, Jara-Ettinger J. 2017. Interpreting actions by attribut-
ing compositional desires. In Proceedings of the 39th Annual Meeting of the Cognitive Science Society ,e d .
G Gunzelmann,A Howes,T Tenbrink,J Davelaar,pp.1284–89.Red Hook,NY: Curran
74. Vazquez-Chanlatte M, Jha S, Tiwari A, Ho MK, Seshia S. 2018. Learning task specifications from
demonstrations. In Advances in Neural Information Processing Systems 31 , ed. S Bengio, H Wallach,
H Larochelle,K Grauman,N Cesa-Bianchi,R Garnett,pp.5367–77.Red Hook,NY: Curran
75. Icarte RT,Klassen T,Valenzano R,McIlraith S.2018.Using reward machines for high-level taskspec-
ification and decompositionin reinforcement learning.In Proceedingsofthe35thInternationalConference
onMachineLearning ,ed.J Dy,A Krause,pp.2107–16.Proc.Mach.Learn.Res.80.N.p.: PMLR
76. Ho MK, Sanborn S, Callaway F, Bourgin D, Griffiths T. 2018. Human priors in hierarchical program
induction. In 2018 Conference on Cognitive Computational Neuroscience , pap. PS-1A.38. College Station,
TX: CCN
77. Rabinowitz N,Perbet F,Song F,Zhang C,Eslami SA,Botvinick M.2018.Machine theory of mind.In
Proceedingsofthe35thInternationalConferenceonMachineLearning ,ed.JDy,AKrause,pp.4218–27.Proc.
Mach.Learn.Res.80.N.p.: PMLR
78. Nematzadeh A, Burns K, Grant E, Gopnik A, Griffiths T. 2018. Evaluating theory of mind in ques-
tion answering.In Proceedingsofthe2018ConferenceonEmpiricalMethodsinNaturalLanguageProcessing ,
pp.2392–400.Brussels: Assoc.Comput.Linguist.
79. Wittgenstein L.1953. PhilosophicalInvestigations .New York: Macmillan
80. Grice HP.1957.Meaning. Philos.Rev. 66:377–88
81. Sperber D, Wilson D. 1986. Relevance: Communication and Cognition . Cambridge, MA: Harvard Univ.
Press
82. Clark HH.1996. UsingLanguage .Cambridge,MA: Cambridge Univ.Press
83. Csibra G,Gergely G.2009.Natural pedagogy. TrendsCogn.Sci. 13:148–53
84. Shafto P,Goodman ND,Griffiths TL.2014.A rational account of pedagogical reasoning: teaching by,
and learning from,examples. Cogn.Psychol. 71:55–89
85. Goodman ND, Frank MC. 2016. Pragmatic language interpretation as probabilistic inference. Trends
Cogn.Sci. 20:818–29
www.annualreviews.org CognitiveScienceasaSourceofModels 51
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

86. Shafto P,Goodman ND.2008.Teaching games: statistical sampling assumptions for learning in peda-
gogical situations.In Proceedingsofthe30thAnnualMeetingoftheCognitiveScienceSociety ,ed.BC Love,
K McRae,VM Sloutsky,pp.1632–37.Red Hook,NY: Curran
87. Landrum AR,Eaves BS Jr.,Shafto P.2015.Learning to trust and trusting to learn: a theoretical frame-
work.TrendsCogn.Sci. 19:109–11
88. BonawitzE,ShaftoP,GweonH,GoodmanND,SpelkeE,SchulzL.2011.Thedouble-edgedswordof
pedagogy: instruction limits spontaneous exploration and discovery. Cognition 120:322–30
89. Bridgers S, Jara-Ettinger J, Gweon H. 2020. Young children consider the expected utility of others’
learning to decide what to teach. Nat.Hum.Behav. 4:144–52
90. Wang P, Wang J, Paranamana P, Shafto P. 2020. A mathematical theory of cooperative communica-
tion. InAdvances in Neural Information Processing Systems 33 , ed. H Larochelle, M Ranzato, R Hadsell,
MF Balcan,H Lin,pp.17582–93.Red Hook,NY: Curran
91. Shafto P, Wang J, Wang P. 2021. Cooperative communication as belief transport. Trends Cogn. Sci.
25:826–28
92. Kao JT, Wu JY, Bergen L, Goodman ND. 2014. Nonliteral understanding of number words. PNAS
111:12002–7
93. Yoon EJ, Tessler MH, Goodman ND, Frank MC. 2020. Polite speech emerges from competing social
goals.OpenMind 4:71–87
94. Tessler MH,Lopez-Brau M,Goodman ND.2017.Warm (for winter): comparison class understanding
in vague language.In ProceedingsofICCM2017:15thInternationalConferenceonCognitiveModeling ,ed.
MK van Vugt,AP Banks,WG Kennedy,pp.193–98.Coventry,UK: Univ.Warwick
95. Tessler MH,Goodman ND.2019.The language of generalization. Psychol.Rev. 126:395–436
96. Csibra G.2010.Recognizing communicative intentions in infancy. MindLang. 25:141–68
97. Scott-Phillips T.2014. SpeakingOurMinds:WhyHumanCommunicationIsDifferent,andHowLanguage
EvolvedtoMakeItSpecial .New York: Macmillan Int.High.Educ.
98. Shafto P, Eaves B, Navarro DJ, Perfors A. 2012. Epistemic trust: modeling children’s reasoning about
others’ knowledge and intent. Dev.Sci.15:436–47
99. HoMK,CushmanF,LittmanML,AusterweilJL.2021.Communicationinaction:planningandinter-
preting communicative demonstrations. J.Exp.Psychol.Gen. 150:2246–72
100. MacGlashanJ,LittmanML.2015.Betweenimitationandintentionlearning.In ProceedingsoftheTwenty-
FourthInternationalJointConferenceonArtificialIntelligence ,pp.3692–98.Palo Alto,CA: AAAI Press
101. Ho MK, MacGlashan J, Littman ML, Cushman F. 2017. Social is special: a normative framework for
teaching with and learning from evaluative feedback. Cognition 167:91–106
102. Ho MK, Littman ML, Cushman F, Austerweil JL. 2015. Teaching with rewards and punishments: re-
inforcement or communication? In Proceedingsofthe37thAnnualMeetingoftheCognitiveScienceSociety ,
ed.DNoelle,RDale,ASWarlaumont,JYoshimi,TMatlock,etal.,pp.920–25.RedHook,NY:Curran
103. HoMK,CushmanF,LittmanML,AusterweilJL.2019.Peopleteachwithrewardsandpunishmentsas
communication,not reinforcements. J.Exp.Psychol.Gen. 148:520–49
104. Loftin R, MacGlashan J, Peng B, Taylor M, Littman M, et al. 2014. A strategy-aware technique for
learningbehaviorsfromdiscretehumanfeedback.In ProceedingsoftheTwenty-EighthAAAIConferenceon
ArtificialIntelligence ,pp.937–43.Palo Alto,CA: AAAI Press
105. Hadfield-Menell D,Milli S,Abbeel P,Russell S,Dragan AD.2017.Inverse reward design.In Advances
inNeuralInformationProcessingSystems30 ,ed.I Guyon,UV Luxburg,S Bengio,H Wallach,R Fergus,
et al.,pp.6768–77.Red Hook,NY: Curran
106. MacGlashan J, Ho MK, Loftin R, Peng B, Wang G, et al. 2017. Interactive learning from policy-
dependent human feedback. In Proceedings of the 34th International Conference on Machine Learning ,e d .
D Precup,YW Teh,pp.2285–94.Proc.Mach.Learn.Res.70.N.p.: PMLR
107. Baird L. 1995. Residual algorithms: reinforcement learning with function approximation. In Machine
Learning:ProceedingsoftheTwelfthInternationalConferenceonMachineLearning ,ed.APrieditis,SRussell,
pp.30–37.San Francisco: Morgan Kaufmann
108. Arumugam D, Lee JK, Saskin S, Littman ML. 2019. Deep reinforcement learning from policy-
dependent human feedback.arXiv:1902.04257 [cs.LG]
52 Ho Griffiths
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

109. Hadfield-Menell D,Dragan A,Abbeel P,Russell S.2016.Cooperative inverse reinforcement learning.
InAdvancesinNeuralInformationProcessingSystems29 , ed. D Lee, M Sugiyama, U Luxburg, I Guyon,
R Garnett,pp.3916–24.Red Hook,NY: Curran
110. Jeon HJ, Milli S, Dragan A. 2020. Reward-rational (implicit) choice: a unifying formalism for reward
learning.In AdvancesinNeuralInformationProcessingSystems33 ,ed.HLarochelle,MRanzato,RHadsell,
MF Balcan,H Lin,pp.4415–26.Red Hook,NY: Curran
111. CorreaCG,HoMK,CallawayF,GriffithsTL.2020.Resource-rationaltaskdecompositiontominimize
planning costs. In Proceedings of the 42ndAnnualMeeting of the Cognitive Science Society , ed. S Denison,
M Mack,Y Xu,B Armstrong,pp.2974–80.Red Hook,NY: Curran
112. HoMK,AbelD,CorreaCG,LittmanML,CohenJD,GriffithsTL.2021.Controlofmentalrepresen-
tations in human planning.arXiv:2105.06948 [cs.AI]
113. Hawkins RD,Gweon H,Goodman ND.2021.The division of labor in communication: speakers help
listeners account for asymmetries in visual perspective. Cogn.Sci. 45:e12926
114. Gates V, Callaway F, Ho MK, Griffiths T. 2021. A rational model of people’s inferences about others’
preferences based on response times. Cognition 217:104885
115. Evans O,Stuhlmüller A,Goodman N.2016.Learning the preferences of ignorant,inconsistent agents.
InProceedingsoftheThirtiethAAAIConferenceonArtificialIntelligence ,pp.323–29.Palo Alto,CA: AAAI
Press
116. BerkeM,Jara-EttingerJ.2021.Thinkingaboutthinkingthroughinversereasoning.PsyArXiv. https://
doi.org/10.31234/osf.io/r25qn
117. Alanqary A, Lin GZ, Le J, Zhi-Xuan T, Masinghka V, Tenenbaum J. 2021. Modeling the mistakes of
boundedlyrationalagentswithinaBayesiantheoryofmind.In Proceedingsofthe43rdAnnualMeetingof
theCognitiveScienceSociety ,p.1013.Red Hook,NY: Curran (Abstr.)
118. Zhi-Xuan T, Mann J, Silver T, Tenenbaum J, Masinghka V. 2020. Online Bayesian goal inference
for boundedly rational planning agents. In Advances in Neural Information Processing Systems 33 ,e d .
H Larochelle,M Ranzato,R Hadsell,MF Balcan,H Lin,pp.19238–50.Red Hook,NY: Curran
119. Shoham Y, Powers R, Grenager T. 2007. If multi-agent learning is the answer, what is the question?
Artif.Intell. 171:365–77
120. Kleiman-Weiner M, Ho MK, Austerweil JL, Littman ML, Tenenbaum JB. 2016. Coordinate to co-
operate or compete: abstract goals and joint intentions in social interaction. In Proceedings of the 38th
AnnualMeetingoftheCognitiveScienceSociety ,ed.A Papafragou,D Grodner,D Mirman,JC Trueswell,
pp.1679–84.Red Hook,NY: Curran
121. Hawkins RXD,Goodman ND,Goldstone RL.2019.The emergence of social norms and conventions.
TrendsCogn.Sci. 23:158–69
www.annualreviews.org CognitiveScienceasaSourceofModels 53
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

AS05_TOC ARjats.cls January 27,2022 15:37
Annual Review of
Control, Robotics,
and Autonomous
Systems
Volume 5,2022Contents
An Historical Perspective on the Control of Robotic Manipulators
MarkW.Spong ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 1
Cognitive Science as a Source of Forward and Inverse Models of
Human Decisions for Robotics and Control
MarkK.HoandThomasL.Griffiths pppppppppppppppppppppppppppppppppppppppppppppppppppppppp 33
Internal Models in Control,Bioengineering,and Neuroscience
MichelangeloBin,JieHuang,AlbertoIsidori,LorenzoMarconi,MatteoMischiati,
andEduardoSontag pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 55
Behavior Trees in Robot Control Systems
PetterÖgrenandChristopherI.Sprague pppppppppppppppppppppppppppppppppppppppppppppppppppp 81
Methods for Robot Behavior Adaptation for Cognitive
Neurorehabilitation
AlyssaKubotaandLaurelD.Riek pppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 109
Grappling Spacecraft
CarlGlenHenshaw,SamanthaGlassner,BoNaasz,andBrianRoberts ppppppppppppppppp 137
Design and Control of Drones
MarkW.Mueller,SeungJaeLee,andRaffaelloD’Andrea ppppppppppppppppppppppppppppppp 161
Contact and Physical Interaction
NevilleHogan ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 179
Multirobot Control Strategies for Collective Transport
HamedFarivarnejadandSpringBerman ppppppppppppppppppppppppppppppppppppppppppppppppp 205
Observer Design for Nonlinear Systems with Equivariance
RobertMahony,PietervanGoor,andTarekHamel ppppppppppppppppppppppppppppppppppppppp 221
Partially Observable Markov Decision Processes and Robotics
HannaKurniawati pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 253
Increasingly Intelligent Micromachines
Tian-YunHuang,HongriGu,andBradleyJ.Nelson ppppppppppppppppppppppppppppppppppppp 279
Magnetic Micro- and Nanoagents for Monitoring Enzymatic Activity
In Vivo
MichaelG.Christiansen,MatejVizovišek,andSimoneSchuerle ppppppppppppppppppppppppp 311
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.

AS05_TOC ARjats.cls January 27,2022 15:37
From Theoretical Work to Clinical Translation: Progress in
Concentric Tube Robots
ZisosMitros,S.M.HadiSadati,RossHenry,LyndonDaCruz,
andChristosBergeles ppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 335
Medical Robotics: Opportunities in China
YaoGuo,WeidongChen,JieZhao,andGuang-ZhongYang ppppppppppppppppppppppppppppp 361
Probabilistic Model Checking and Autonomy
MartaKwiatkowska,GethinNorman,andDavidParker pppppppppppppppppppppppppppppppp 385
Safe Learning in Robotics: From Learning-Based Control to Safe
Reinforcement Learning
LukasBrunke,MelissaGreeff,AdamW.Hall,ZhaocongYuan,SiqiZhou,
JacopoPanerati,andAngelaP.Schoellig pppppppppppppppppppppppppppppppppppppppppppppppp 411
Secure Networked Control Systems
HenrikSandberg,VijayGupta,andKarlH.Johansson ppppppppppppppppppppppppppppppppppp 445
Energy-Aware Controllability of Complex Networks
GiacomoBaggio,FabioPasqualetti,andSandroZampieri pppppppppppppppppppppppppppppppp 465
Control of Microparticle Assembly
XunTangandMarthaA.Grover pppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 491
Stimuli-Responsive Polymers for Soft Robotics
YusenZhao,MutianHua,YichenYan,ShuwangWu,YousifAlsaid,
andXiminHe pppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 515
Control of the Stefan System and Applications: A Tutorial
ShumonKogaandMiroslavKrstic ppppppppppppppppppppppppppppppppppppppppppppppppppppppppp 547
Turbulence and Control of Wind Farms
CarlR.Shapiro,GenevieveM.Starke,andDenniceF.Gayme ppppppppppppppppppppppppppp 579
Autonomous Airborne Wind Energy Systems: Accomplishments
and Challenges
LorenzoFagiano,ManfredQuack,FlorianBauer,LodeCarnel,andEspenOland pppppp603
Analysis and Control of Autonomous Mobility-on-Demand Systems
GioeleZardini,NicolasLanzetti,MarcoPavone,andEmilioFrazzoli ppppppppppppppppppp 633
Control as an Enabler for Electrified Mobility
AndrewG.AlleyneandChristopherT.Aksland ppppppppppppppppppppppppppppppppppppppppppp 659
Stability and Control of Power Grids
TaoLiu,YueSong,LipengZhu,andDavidJ.Hill pppppppppppppppppppppppppppppppppppppppp 689
Errata
An online log of corrections to AnnualReviewofControl,Robotics,andAutonomous
Systemsarticles may be found at http://www.annualreviews.org/errata/control
Annu. Rev. Control Robot. Auton. Syst. 2022.5:33-53. Downloaded from www.annualreviews.org
 Access provided by Princeton University Library on 11/11/22. For personal use only.