The Value of Abstraction.

The Value of Abstraction
Mark K. Ho
Princeton UniversityDavid Abel
Brown University
Thomas L. Gri ths
Princeton UniversityMichael L. Littman
Brown University
Agents that can make better use of computation, experience, time, and memory can solve
a greater range of problems more e ectively. A crucial ingredient for managing such ﬁnite
resources is intelligently chosen abstract representations . But, how do abstractions facilitate
problem solving under limited resources? What makes an abstraction useful? To answer such
questions, we review several trends in recent reinforcement-learning research that provide in-
sight into how abstractions interact with learning and decision making. During learning, ab-
straction can guide exploration and generalization as well as facilitate e cient tradeo s—e.g.,
time spent learning versus the quality of a solution. During computation, good abstractions
provide simpliﬁed models for computation while also preserving relevant information about
decision-theoretic quantities. These features of abstraction are not only key for scaling up
artiﬁcial problem solving, but can also shed light on what pressures shape the use of abstract
representations in humans and other organisms.
Keywords: abstraction, reinforcement learning, bounded rationality, planning, problem
solving, rational analysis
Highlights
Agents with ﬁnite space, time, and data can beneﬁt
from abstract representations.
Both humans and artiﬁcial agents rely on abstractions
to solve complex problems.
Recent work in reinforcement learning elucidates what
makes abstractions useful.
Abstractions guide learning, facilitate tradeo s, and
simplify computation.
This structure provides a basis for a rational analysis
of abstract representations.
Pre-print June 2019—to appear in Current Opinion in Be-
havioral Sciences themed issue on Artiﬁcial Intelligence.
Edited by Matt Botvinick and Sam Gershman.
We would like to thank the anonymous reviewers for their com-
ments that greatly improved previous versions of this paper. This
material is based upon work supported by the NSF under Grant No.
1544924.
Correspondence concerning this article should be addressed to
Michael L. Littman, Brown University, Providence, RI. Email:
mlittman@cs.brown.edu.Introduction
In spite of bounds on space, time, and data, people are
able to make good decisions in complex scenarios. What en-
ables us to do so? And what might equip artiﬁcial systems to
do the same? One essential ingredient for making complex
problem solving tractable is a capacity for exploiting abstract
representation .
To illustrate why abstractions are needed for adaptive de-
cision making, consider hiking in a forest (Figure 1). A river
runs from mountains in the north down south through the
forest, and you have set up camp in a clearing just east of the
river. As the sun begins to set, you ﬁnd yourself west of the
river and want to return to camp. How do you accomplish
this task? You might reason, “Since I am on the west side of
the river , I probably want to cross the river to get to the east
side of the river and then walk towards my campsite .” Such
a compact thought is possible because you have abstract rep-
resentations for that support decision making: These include
the general notion of being in a location relative to the axis of
the river, or the rough ideas of crossing the river and moving
towards the campsite.
Without abstractions, decision making would be conﬁned
to speciﬁc, low-level states and actions—e.g., “At the cur-
rent state, rotate your left leg 25, place it down beside the
rock on the path, then swing your arm forward...”. In a huge
and complex scenario like navigating through a forest, this
approach is both cognitively unwieldy and, in a fundamental
2 MARK K. HO, DA VID ABEL, THOMAS L. GRIFFITHS, MICHAEL L. LITTMAN
Figure 1 . An environment, an agent, and the agent’s abstract representation of a domain.
sense, computationally intractable (Bellman, 1957; Bylan-
der, 1994). The di culty is due to decision-making costs
that arise from two interrelated sources. First, agents need to
learn about their environment, which costs time and experi-
ence (e.g., via exploration) (Kakade, 2003). Second, agents
need to compute decision-theoretic quantities, which costs
memory and thought (e.g., via planning) (Littman, Dean, &
Kaelbling, 1995). As problems grow, these costs grow ex-
ponentially. Abstractions are essential because they enable
decision makers to manage the growth of these costs.
Within psychology and neuroscience, abstraction plays
a key role in the hierarchical learning and organization
of motor sequences (Lashley, 1951; Rosenbaum, Kenny,
& Derr, 1983; Rosenbaum, Inho , & Gordon, 1984),
habits (Botvinick, 2008; Dezfouli & Balleine, 2013; Ribas-
Fernandes et al., 2011), and planning (Solway et al., 2014;
Botvinick & Weinstein, 2014). For this reason, researchers
in artiﬁcial intelligence (AI) and reinforcement learning (RL)
have long been interested in how to leverage abstraction
to provide human-like solutions to human-level problems
(Dayan & Hinton, 1993; Sacerdoti, 1974; Dietterich, 2000;
Givan, Dean, & Greig, 2003; Andre & Russell, 2002; Barto
& Mahadevan, 2003). In this paper, we discuss work in AI
and RL that helps elucidate how abstractions support e cient
decision making in both people and machines.
Research in AI and RL focuses on two broad types of ab-
straction. First, state abstractions treat certain conﬁgurations
of the environment as similar by aggregating them or assign-
ing them shared features. For instance, in the hiking exam-
ple,being west of the river is not a single concrete conﬁgura-
tion of the environment, but rather an abstract representation
that covers many concrete states (e.g., being di erent dis-
tances west of the river). Second, temporal abstractions (of-
ten called “options”) are temporally extended macro-actionsthat describe a general course of action. For example, the
idea of crossing the bridge is not a single speciﬁc action, but
rather captures many possible courses of action.
State and temporal abstractions enable compact represen-
tation of a domain. For instance, Figure 2a represents the
hiking example as a graph where nodes are speciﬁc conﬁg-
urations of the environment (i.e., ground states) and edges
are transitions to new conﬁgurations resulting from taking
actions. The appropriate abstractions could induce the sim-
pler problem representation in Figure 2b. This new model
ignores irrelevant details like the color of speciﬁc trees while
capturing useful distinctions such as your location relative to
the river. As a result, it supports e cient learning and re-
quires little thought to reason about compared to the original
version.
The foregoing discussion sketches out how abstractions
can support e cient decision making. But how could it
work in practice? Here, we dive into recent work in AI and
RL that provides some answers. Speciﬁcally, we focus on
three ways in which abstractions have been shown to facil-
itate e cient and scalable decision making. First, abstrac-
tions guide exploration and generalization by systematically
modifying the distribution of learning experiences. In par-
ticular, abstractions can guide how agents explore and gen-
eralize based on environmental structure or representational
simplicity. Second, abstractions facilitate e cient tradeo s
in learning . For example, they enable learning time and op-
timality to be exchanged as well as support optimal transfer
between tasks. Finally, abstraction is essential for simplifying
computation . For instance, abstractions inﬂuence the cost of
computing a good plan by inducing simpler or more complex
planning models. In the following sections, we discuss each
of these beneﬁts of abstraction.
THE V ALUE OF ABSTRACTION 3
…………
(a)
 (b)
Figure 2 . (a) A domain in all of its complexity, and (b) the representation induced by an e ective set of state and temporal
abstractions for this domain. The representation in (b) facilitates e cient learning and computation, unlike the one in (a).
Abstractions Guide Exploration and Generalization
Suppose during your camping trip you want to learn to
ﬁsh. At ﬁrst, you try a few spots along the river at random.
But soon, you notice a pattern: Certain areas have more veg-
etation, others have less. This information provides a new
and more e cient way for you to organize your ﬁshing at-
tempts. For example, you could ﬁsh areas with high and low
vegetation to gain a range of experiences about how it a ects
your catch. Or, on your ﬁrst trip you may learn that your best
catch was in high vegetation areas, so that on your second
trip to a di erent river, you seek out similar areas. Here, the
abstract concept of river vegetation guides your exploration
in the current ﬁshing task and tracks a generalizable feature
relevant to future tasks, which both allow you to make better
use of your limited time and experience.
In RL, abstractions similarly facilitate e cient learning
by guiding exploration and generalization. But what is the
basis of this guidance? Put another way, the concept of veg-
etation is useful, but what determines the identiﬁcation of
such a concept in general? Below, we discuss two ways in
which abstractions can guide learning: domain structure and
representational simplicity.
Domain Structure
Many domains have meaningful structure based on the
topology of state connections. In the hiking example, be-
ing west of the river is an intuitive abstract state because all
the locations west of the river are more connected to one an-
other and less connected to locations east of the river. Early
research in RL identiﬁed a number strategies for learning
abstractions based on clustering or graph-theoretic notions
like “bottleneck” states (¸ Sim¸ sek & Barto, 2004). More re-
cent methods generalize such approaches to learn abstrac-
tions based on the state connectivity induced by a domain’sstructure. For instance, the successor representation (Dayan,
1993; Gershman, Moore, Todd, Norman, & Sederberg, 2012;
Momennejad et al., 2017) and successor features (Barreto et
al., 2017; Kulkarni, Saeedi, Gautam, & Gershman, 2016) are
state abstractions based on how likely an agent is to visit
other states or features. A similar idea underpins eigen-
options (Machado, Bellemare, & Bowling, 2017; Machado,
Rosenbaum, et al., 2017; Mahadevan, 2005). Speciﬁcally,
both successor representations and eigen-options involve de-
composing an a domain’s structure into a set of “principal
components” that compactly represent how certain regions
of a domain are connected or will be visited.
Thus, being west of the river andbeing east of the river
could emerge as successor-based state abstractions due to the
connectivity and visitation of their respective ground states.
A corresponding eigen-option would express the temporally
abstract action of going from the west side to the east side
since action sequences falling in this category capture a prin-
cipal source of variance in connectivity. Notably, abstrac-
tions based on state connectivity encode information about
the dynamics of a domain separate from its reward structure.
These reward-agnostic representations can be learned either
oine or during task learning, but, in either case, they pro-
vide an especially powerful and e cient way to adapt when
rewards change or when transferring to tasks with related dy-
namics.
Representational Simplicity
Abstractions also guide exploration and generalization
based on representational simplicity. Simplicity plays a key
role in understanding representations in humans (Chater &
Vitányi, 2003; Sanborn, Bourgin, Chang, & Gri ths, 2018)
and machines (Li & Vitányi, 2008). For ﬁnite decision mak-
ers, simplicity is important because it enables compression ,
the representation of information using less physical space
4 MARK K. HO, DA VID ABEL, THOMAS L. GRIFFITHS, MICHAEL L. LITTMAN
(e.g., memory).
In RL, a number of methods have been developed that
leverage a bias towards representational simplicity. For
example, curiosity-driven learning (Schmidhuber, 1991;
Oudeyer, Kaplan, & Hafner, 2007; Bellemare et al., 2016;
Pathak, Agrawal, Efros, & Darrell, 2017; Ostrovski, Belle-
mare, Oord, & Munos, 2017; Kulkarni, Narasimhan, Saeedi,
& Tenenbaum, 2016) pits a motivation to learn a simple,
compressed representation of the world against a desire for
new experiences that do not easily ﬁt within that represen-
tation. Placing these two processes in competition with one
another prevents the drive for simplicity from compressing
everything into a single undi erentiated representation (the
ultimate abstraction) that cannot support high quality deci-
sions and instead pushes the system to adaptively explore a
range of qualitatively di erent parts of a task. Doing so ul-
timately allows the architecture to both represent the world
compactly and explore it e ciently.
Simplicity also plays a role in the Option-Critic frame-
work (Bacon, Harb, & Precup, 2017; Harb, Bacon, Klis-
sarov, & Precup, 2018; Liu, Machado, Tesauro, & Camp-
bell, 2017), a neural network architecture that simultaneously
learns abstract macro-actions and the value of low-level ac-
tions that constitute them. By concurrently learning at multi-
ple levels of abstraction, the agent acquires a representation
that leads to better transfer to new tasks. The process at play
is closely related to the notion of a “blessing of abstraction”
in causal learning (Goodman, Ullman, & Tenenbaum, 2011)
and regularization in supervised learning (Bishop, 2006): A
bias towards simple, unifying representations not only saves
on representational space, but also prevents overﬁtting to a
particular task. An important consequence of this bias is that
the agent is able to transfer to new tasks more e ciently.
Abstractions Facilitate E cient Tradeo s
Decision making and learning involve tradeo s when
space, time, and data are limited. But, it is important that
these tradeo s be made eciently (i.e., without unnecessary
costs). Here, we discuss how abstractions can e ciently
trade o learning time and optimality, as well as minimize
negative transfer between tasks.
Learning Time and Optimality
Imagine that every morning you wake up at your campsite
and need to navigate to the bridge crossing the river. What
kind of abstract states would be useful? It depends on how
much time you have to spend learning. One option is to learn
every tree, rock, and shortcut through the forest in great de-
tail. Doing so has the beneﬁt of ensuring you learn a highly
ecient route to the bridge. An alternative strategy is to learn
a less detailed representation of the forest that takes you to
an obvious landmark—e.g., the river—and then take a simple
path to the bridge. This second approach may be more costlyin terms of the physical actions taken or time navigating, but
is simpler and faster to learn. Depending on how much time
and data are available, it may be better for the agent to pursue
the second strategy than the ﬁrst.
Learning the simpler but suboptimal policy depends on the
resolution of the state abstractions used—i.e., the amount of
detail one is willing to keep or ignore about the environment.
In RL, a number of approaches use abstraction in this man-
ner to e ciently balance optimality and learning time (Jiang,
Kulesza, & Singh, 2015; Ortner, 2013). For instance, if ab-
stractions are represented by aggregating ground states into
clusters, adaptively modulating the size of the clusters and
granularity of the state representation controls this trade-
o(Taiga, Courville, & Bellemare, 2018; Bellemare et al.,
2016; Mandel, Liu, Brunskill, & Popovic, 2016).
Optimizing Transfer between Tasks
Abstractions also facilitate e ective transfer from one task
to another. Transfer can be positive (Sutton, Precup, &
Singh, 1999; Konidaris & Barto, 2007; Bacon et al., 2017;
Topin et al., 2015), but it can also be negative (Jong, Hes-
ter, & Stone, 2008). Ideally, an agent would learn abstrac-
tions that avoid negative transfer as much as possible while
increasing the possibility of positive transfer. One setting
in which this approach is feasible is in lifelong learning , in
which an agent is assumed to be continuously receiving tasks
from a distribution of tasks. Given separate phases of discov-
ering and using temporal abstractions, it is possible to ensure
that any learned options only help later learning and decision
making (Brunskill & Li, 2014). Used properly, temporal ab-
stractions can also guarantee that the amount of data needed
for learning is reduced even within a single task (Fruit &
Lazaric, 2017; Fruit, Pirotta, Lazaric, & Brunskill, 2017).
For agents with limited time, such abstractions can play a
critical role in ensuring e cient and e ective use of data.
Abstractions Simplify Computation
Computation is costly for people (Gri ths, Lieder, &
Goodman, 2015) as well as artiﬁcial agents (Russell, 1997).
For example, if you were to plan a hike back to your camp-
site, you could simulate possible routes from your current
state or “work backwards” from your ﬁnal destination. These
cognitive operations require time and mental e ort. How-
ever, the total cost of computation depends directly on the
model such operations occur over (Givan et al., 2003). Plan-
ning can be easier given a good abstract representation (e.g.,
compare Figure 2a and Figure 2b). Recent work in RL lever-
ages this intuition to e ciently scale planning to complex do-
mains. Here, we focus on two broad approaches: algorithm-
speciﬁc abstractions , in which an abstraction scheme is tai-
lored to take advantage of particular features of a planning
algorithm, and end-to-end approaches, in which learning ab-
THE V ALUE OF ABSTRACTION 5
stractions for planning occurs in the context of a more gen-
eral learning architecture.
Algorithm-Speciﬁc Abstractions
Algorithms such as Monte Carlo Tree Search (Abramson,
1987; Coulom, 2006; Browne et al., 2012) are based on the
notion of forward simulation : From an initial state, candidate
plans are generated, evaluated, and the best is chosen. The
major problem for these algorithms is branching: Each addi-
tional timestep into the future increases the number of plans
to evaluate multiplicatively. To keep branching manageable,
state-aggregation techniques simplify models by only stor-
ing relevant information. For instance, by treating all states
with the same optimal action as identical, branching due to
stochasticity in action outcomes can be reduced (Hostetler,
Fern, & Dietterich, 2014; Anand, Grover, Mausam, & Singla,
2015; Anand, Noothigattu, Mausam, & Singla, 2016). Along
similar lines, the model used for planning can be simpliﬁed
by aggregating states that lead to the same possible outcomes
within a few time steps (Jiang, Singh, & Lewis, 2014).
An alternative to forward simulation is backwards in-
duction , which involves starting from possible future states
and propagating information backwards to possible precur-
sor states. Abstractions can also assist computation in back-
ward induction. For instance, value iteration (Bellman, 1957)
repeatedly “sweeps” over a set of states and backs up infor-
mation to each precursor state. Since each iteration involves
looking at every state in the planning model, it is extremely
sensitive to the number of possible states. However, by in-
corporating temporal abstractions into the planning model,
more information can be propagated further backwards on
each sweep (Mann & Mannor, 2014) and it becomes possi-
ble to more e ciently trade o error in the approximation of
future rewards with computation (Mann, Mannor, & Precup,
2015).
Finally, relaxing the goal of planning provides opportuni-
ties to leverage powerful abstractions. For instance, plan fea-
sibility seeks to determine whether a plan can be constructed
that satisﬁes some set of constraints. Since the goal is no
longer optimality but simply satisﬁability , relevant aspects of
the task can be translated into symbolic logic and e ciently
tested (Konidaris, Kaelbling, & Lozano-Perez, 2014, 2018;
McDermott et al., 1998).
End-to-end E cient Planning
Another approach is to learn planning abstractions while
also learning everything else about a task in an end-to-end
manner with neural networks. Using abstractions for plan-
ning is especially important because planning in a ground
state space (e.g., the pixels of a video game) is only possi-
ble for short time horizons (Oh, Guo, Lee, Lewis, & Singh,
2015). To address these limitations, architectures have been
developed that learn and use abstract representations forplanning. In particular, the abstract representation needs
to be trained to predict future rewards, thus preserving the
most relevant information for decision making in the plan-
ning model (Oh, Singh, & Lee, 2017; Silver et al., 2017).
Conclusion
Abstractions are key for e ciently scaling decision mak-
ing. In particular, recent research in RL demonstrates how
state and temporal abstractions guide exploration and gener-
alization, facilitate e cient tradeo s, and simplify computa-
tions. In the context of learning and decision making with
limited resources, abstractions enable agents to strategically
manage ﬁnite space, time, and data, which is necessary for
scaling problem solving to complex tasks.
For psychologists, this analysis raises questions about
why biological agents like humans have certain types of ab-
stract representations. For instance, humans and other or-
ganisms may have certain abstractions not only because they
facilitate e ective exploration and generalization but because
they support fast planning or modulate critical tradeo s dur-
ing learning. Future work in both computer science and psy-
chology will need to identify other pressures that can shape
abstraction learning—such as the need to communicate and
coordinate with others—to o er a clearer understanding of
the value of abstraction.
Conﬂict of Interest Statement
The authors declare no conﬂict of interest.
Annotated References
(**) (Bacon et al., 2017) - Proposes option-critic ar-
chitecture that demonstrates the e ectiveness of jointly
learning abstract and low-level policies.
(*) (Jiang et al., 2015) - Analyzes how state aggrega-
tion learning schemes can be used to e ciently trade
odata and solution quality, which is key when data is
costly or ﬁnite.
(*) (Machado, Rosenbaum, et al., 2017) - Combines
learning eigen-options with learning successor repre-
sentations, which both leverage the transition structure
of a domain for learning abstract representations.
(**) (Pathak et al., 2017) - Implements curiosity as an
intrinsic reward signal based on an agent’s error when
predicting the outcome of actions to enable e ective
representation learning even in the absence of external
rewards.
(*) (Oh et al., 2017) - Proposes the Value Predic-
tion Network architecture that learns an internal model
composed of abstract states for e cient model-based
planning.
6 MARK K. HO, DA VID ABEL, THOMAS L. GRIFFITHS, MICHAEL L. LITTMAN
(*) (Barreto et al., 2017) - Introduces a generalization
of successor representations (Dayan, 1993) that en-
ables e cient transfer across tasks with shared causal
structure.
References
Abramson, B. D. (1987). The Expected-Outcome Model of Two-
Player Games (Unpublished doctoral dissertation). Columbia
University, New York, NY , USA.
Anand, A., Grover, A., Mausam, & Singla, P. (2015). ASAP-UCT:
Abstraction of State-Action Pairs in UCT. Twenty-Fourth Inter-
national Joint Conference on Artiﬁcial Intelligence .
Anand, A., Noothigattu, R., Mausam, & Singla, P. (2016). OGA-
UCT: On-the-Go Abstractions in UCT. Twenty-Sixth Interna-
tional Conference on Automated Planning and Scheduling .
Andre, D., & Russell, S. J. (2002). State Abstraction for Pro-
grammable Reinforcement Learning Agents. In Eighteenth na-
tional conference on artiﬁcial intelligence (pp. 119–125). Menlo
Park, CA, USA: American Association for Artiﬁcial Intelli-
gence.
Bacon, P.-L., Harb, J., & Precup, D. (2017). The option-critic
architecture. In Proceedings of the conference aaai (pp. 1726–
1734).
Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Has-
selt, H. P., & Silver, D. (2017). Successor features for transfer
in reinforcement learning. In Advances in Neural Information
Processing Systems (pp. 4055–4065).
Barto, A. G., & Mahadevan, S. (2003). Recent advances in hierar-
chical reinforcement learning. Discrete event dynamic systems ,
13(1-2), 41–77.
Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton,
D., & Munos, R. (2016). Unifying count-based exploration and
intrinsic motivation. In Advances in Neural Information Pro-
cessing Systems (pp. 1471–1479).
Bellman, R. (1957). Dynamic programming . Princeton University
Press.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning .
Springer.
Botvinick, M. M. (2008). Hierarchical models of behavior and pre-
frontal function. Trends in Cognitive Sciences ,12(5), 201–208.
doi: 10.1016 /J.TICS.2008.02.009
Botvinick, M. M., & Weinstein, A. (2014). Model-based hierarchi-
cal reinforcement learning and human action control. Philosoph-
ical Transactions of the Royal Society B: Biological Sciences ,
369(1655), 20130480–20130480. doi: 10.1098 /rstb.2013.0480
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling,
P. I., Rohlfshagen, P., . . . Colton, S. (2012). A survey of monte
carlo tree search methods. IEEE Transactions on Computational
Intelligence and AI in games ,4(1), 1–43.
Brunskill, E., & Li, L. (2014). PAC-inspired option discovery in
lifelong reinforcement learning. In International conference on
machine learning (pp. 316–324).
Bylander, T. (1994). The computational complexity of proposi-
tional STRIPS planning. Artiﬁcial Intelligence ,69, 161–204.
Chater, N., & Vitányi, P. (2003). Simplicity: a unifying principle in
cognitive science? Trends in Cognitive Sciences ,7(1), 19–22.
doi: 10.1016 /S1364-6613(02)00005-0Coulom, R. (2006). E cient Selectivity and Backup Operators in
Monte-Carlo Tree Search. In International conference on com-
puters and games (pp. 72–83).
Dayan, P. (1993). Improving generalization for temporal di erence
learning: The successor representation. Neural Computation ,
5(4), 613–624.
Dayan, P., & Hinton, G. E. (1993). Feudal reinforcement learn-
ing. In Advances in Neural Information Processing Systems (pp.
271–278).
Dezfouli, A., & Balleine, B. W. (2013). Actions, Action Sequences
and Habits: Evidence That Goal-Directed and Habitual Action
Control Are Hierarchically Organized. PLoS Computational Bi-
ology ,9(12), e1003364. doi: 10.1371 /journal.pcbi.1003364
Dietterich, T. G. (2000). Hierarchical reinforcement learning with
the maxq value function decomposition. Journal of Artiﬁcial
Intelligence Research ,13, 227–303.
Fruit, R., & Lazaric, A. (2017). Exploration–exploitation in MDPs
with options. arXiv preprint arXiv:1703.08667 .
Fruit, R., Pirotta, M., Lazaric, A., & Brunskill, E. (2017). Regret
minimization in MDPs with options without prior knowledge. In
Advances in Neural Information Processing Systems (pp. 3169–
3179).
Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A., &
Sederberg, P. B. (2012). The successor representation and tem-
poral context. Neural Computation ,24(6), 1553–1568.
Givan, R., Dean, T., & Greig, M. (2003). Equivalence notions and
model minimization in markov decision processes. Artiﬁcial In-
telligence ,147(1-2), 163–223.
Goodman, N. D., Ullman, T. D., & Tenenbaum, J. B. (2011). Learn-
ing a theory of causality. Psychological review ,118(1), 110.
Griths, T. L., Lieder, F., & Goodman, N. D. (2015). Rational Use
of Cognitive Resources: Levels of Analysis Between the Com-
putational and the Algorithmic. Topics in Cognitive Science ,
7(2), 217–229. doi: 10.1111 /tops.12142
Harb, J., Bacon, P.-L., Klissarov, M., & Precup, D. (2018). When
Waiting Is Not an Option: Learning Options With a Deliberation
Cost. Thirty-Second AAAI Conference on Artiﬁcial Intelligence .
Hostetler, J., Fern, A., & Dietterich, T. (2014). State Aggregation
in Monte Carlo Tree Search. AAAI 2014 , 7.
Jiang, N., Kulesza, A., & Singh, S. (2015). Abstraction selection
in model-based reinforcement learning. In F. Bach & D. Blei
(Eds.), Proceedings of the 32nd international conference on ma-
chine learning (V ol. 37, pp. 179–188). Lille, France: PMLR.
Jiang, N., Singh, S., & Lewis, R. (2014). Improving UCT planning
via approximate homomorphisms. In Proceedings of the 2014
international conference on autonomous agents and multi-agent
systems (pp. 1289–1296).
Jong, N. K., Hester, T., & Stone, P. (2008). The utility of tempo-
ral abstraction in reinforcement learning. In Proceedings of the
Conference on Autonomous Agents and Multiagent Systems (pp.
299–306).
Kakade, S. M. (2003). On the sample complexity of reinforcement
learning (Unpublished doctoral dissertation). University of Lon-
don, England.
Konidaris, G., & Barto, A. G. (2007). Building portable options:
Skill transfer in reinforcement learning. In Proceedings of the
International Joint Conference on Artiﬁcial Intelligence (V ol. 7,
pp. 895–900).
THE V ALUE OF ABSTRACTION 7
Konidaris, G., Kaelbling, L. P., & Lozano-Perez, T. (2014). Con-
structing symbolic representations for high-level planning. In
Aaai (pp. 1932–1938).
Konidaris, G., Kaelbling, L. P., & Lozano-Perez, T. (2018). From
Skills to Symbols: Learning Symbolic Representations for Ab-
stract High-Level Planning. Journal of Artiﬁcial Intelligence Re-
search ,61, 215–289.
Kulkarni, T. D., Narasimhan, K., Saeedi, A., & Tenenbaum, J.
(2016). Hierarchical deep reinforcement learning: Integrating
temporal abstraction and intrinsic motivation. In Advances in
Neural Information Processing Systems (pp. 3675–3683).
Kulkarni, T. D., Saeedi, A., Gautam, S., & Gershman, S. J.
(2016). Deep successor reinforcement learning. arXiv preprint
arXiv:1606.02396 .
Lashley, K. S. (1951). The problem of serial order in behavior.
InCerebral mechanisms in behavior; the hixon symposium. (pp.
112–146). Oxford, England: Wiley.
Li, M., & Vitányi, P. (2008). An Introduction to Kolmogorov Com-
plexity and Its Applications . New York, NY: Springer New York.
doi: 10.1007 /978-0-387-49820-1
Littman, M. L., Dean, T. L., & Kaelbling, L. P. (1995). On the com-
plexity of solving markov decision problems. In Proceedings of
the eleventh conference on uncertainty in artiﬁcial intelligence
(pp. 394–402).
Liu, M., Machado, M. C., Tesauro, G., & Campbell, M.
(2017). The eigenoption-critic framework. arXiv preprint
arXiv:1712.04065 .
Machado, M. C., Bellemare, M. G., & Bowling, M. (2017). A lapla-
cian framework for option discovery in reinforcement learning.
arXiv preprint arXiv:1703.00956 .
Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., &
Campbell, M. (2017). Eigenoption discovery through the deep
successor representation. arXiv preprint arXiv:1710.11089 .
Mahadevan, S. (2005). Proto-value functions: Developmental re-
inforcement learning. In Proceedings of the 22nd international
conference on machine learning (pp. 553–560).
Mandel, T., Liu, Y.-E., Brunskill, E., & Popovic, Z. (2016). E -
cient bayesian clustering for reinforcement learning. Proceed-
ings of the International Joint Conference on Artiﬁcial Intelli-
gence .
Mann, T., & Mannor, S. (2014). Scaling up approximate value
iteration with options: Better policies with fewer iterations. In
International conference on machine learning (pp. 127–135).
Mann, T., Mannor, S., & Precup, D. (2015). Approximate value
iteration with temporally extended actions. Journal of Artiﬁcial
Intelligence Research ,53, 375–438.
McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A.,
Veloso, M., . . . Wilkins, D. (1998). Pddl-the planning domain
deﬁnition language.
Momennejad, I., Russek, E. M., Cheong, J. H., Botvinick, M. M.,
Daw, N. D., & Gershman, S. J. (2017). The successor rep-
resentation in human reinforcement learning. Nature Human
Behaviour ,1(9), 680–692. doi: 10.1038 /s41562-017-0180-8
Oh, J., Guo, X., Lee, H., Lewis, R. L., & Singh, S. (2015).
Action-conditional video prediction using deep networks in atari
games. In Advances in Neural Information Processing Systems
(pp. 2863–2871).
Oh, J., Singh, S., & Lee, H. (2017). Value prediction network. InAdvances in Neural Information Processing Systems (pp. 6118–
6128).
Ortner, R. (2013). Adaptive aggregation for reinforcement learning
in average reward Markov decision processes. Annals of Oper-
ations Research ,208(1), 321–336.
Ostrovski, G., Bellemare, M. G., Oord, A., & Munos, R. (2017).
Count-based exploration with neural density models. In Inter-
national conference on machine learning (pp. 2721–2730).
Oudeyer, P.-Y ., Kaplan, F., & Hafner, V. V . (2007). Intrinsic
motivation systems for autonomous mental development. IEEE
transactions on evolutionary computation ,11(2), 265–286.
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017).
Curiosity-driven Exploration by Self-supervised Prediction. In
Proceedings of the 34th international conference on machine
learning - volume 70 (pp. 2778–2787). JMLR.
Ribas-Fernandes, J., Solway, A., Diuk, C., McGuire, J., Barto, A.,
Niv, Y ., & Botvinick, M. (2011). A Neural Signature of Hier-
archical Reinforcement Learning. Neuron ,71(2), 370–379. doi:
10.1016 /J.NEURON.2011.05.042
Rosenbaum, D. A., Inho , A. W., & Gordon, A. M. (1984). Choos-
ing between movement sequences: A hierarchical editor model.
Journal of Experimental Psychology: General ,113(3), 372–
393. doi: 10.1037 /0096-3445.113.3.372
Rosenbaum, D. A., Kenny, S. B., & Derr, M. A. (1983). Hierar-
chical control of rapid movement sequences. (V ol. 9) (No. 1).
US: American Psychological Association. doi: 10.1037 /0096-
1523.9.1.86
Russell, S. J. (1997). Rationality and intelligence. Artiﬁcial Intelli-
gence ,94(1-2), 57–77. doi: 10.1016 /S0004-3702(97)00026-X
Sacerdoti, E. D. (1974). Planning in a hierarchy of abstrac-
tion spaces. Artiﬁcial Intelligence ,5(2), 115 - 135. doi:
https: //doi.org /10.1016 /0004-3702(74)90026-5
Sanborn, S., Bourgin, D. D., Chang, M., & Gri ths, T. L. (2018).
Representational e ciency outweighs action e ciency in hu-
man program induction. In T. Rogers, M. Rau, X. Zhu, &
C. Kalish (Eds.), Proceedings of the 40th annual meeting of the
cognitive science society (pp. 2400–2405). Austin, TX: Cogni-
tive Science Society.
Schmidhuber, J. (1991). A possibility for implementing curiosity
and boredom in model-building neural controllers. In Proc. of
the international conference on simulation of adaptive behavior:
From animals to animats (pp. 222–227).
Silver, D., van Hasselt, H., Hessel, M., Schaul, T., Guez, A., Harley,
T., . . . Degris, T. (2017). The predictron: end-to-end learning
and planning. JMLR.
¸ Sim¸ sek, Ö., & Barto, A. G. (2004). Using relative novelty to
identify useful temporal abstractions in reinforcement learning.
InTwenty-ﬁrst international conference on machine learning -
icml ’04 (p. 95). New York, New York, USA: ACM Press. doi:
10.1145 /1015330.1015353
Solway, A., Diuk, C., Córdova, N., Yee, D., Barto, A. G., Niv, Y ., &
Botvinick, M. M. (2014). Optimal Behavioral Hierarchy. PLoS
Computational Biology ,10(8), e1003779. doi: 10.1371 /jour-
nal.pcbi.1003779
Sutton, R. S., Precup, D., & Singh, S. (1999). Between MDPs and
semi-MDPs: A framework for temporal abstraction in reinforce-
ment learning. Artiﬁcial Intelligence ,112(1), 181–211.
Taiga, A. A., Courville, A., & Bellemare, M. (2018). Approxi-
8 MARK K. HO, DA VID ABEL, THOMAS L. GRIFFITHS, MICHAEL L. LITTMAN
mate exploration through state abstraction. ICML Workshop on
Exploration in RL .
Topin, N., Haltmeyer, N., Squire, S., Winder, J., desJardins, M., &
MacGlashan, J. (2015). Portable option discovery for automatedlearning transfer in object-oriented Markov decision processes.
InProceedings of the International Joint Conference on Artiﬁ-
cial Intelligence (pp. 3856–3864).

Mark K
Ho
David Abel
Tom Griffiths
Michael L
Littman


Doing more with less: meta-reasoning and meta-learning in humans and machines.

Doing more with less: meta-reasoning and meta-learning in humans and machines[☆](#aep-article-footnote-id6)
===========================================================================================================

Artificial intelligence systems use an increasing amount of computation and data to solve very specific problems. By contrast, human minds solve a wide range of problems using a fixed amount of computation and limited experience. We identify two abilities that we see as crucial to this kind of general intelligence: meta-reasoning (deciding how to allocate computational resources) and meta-learning (modeling the learning environment to make better use of limited data). We summarize the relevant AI literature and relate the resulting ideas to recent work in psychology.

Section snippets
----------------

Meta-reasoning
--------------

The term ‘meta-reasoning’ contains within it the solution to the problem of how to efficiently deploy computational resources: meta-reasoning is reasoning about reasoning, which means making intelligent decisions about how to think [5]. In research on artificial intelligence, meta-reasoning plays a central role in the definition and design of rational agents that can operate on performance-limited hardware and interact with their environment in real-time [2,6, 7, 8,4]. Considering this problem

Meta-learning
-------------

While meta-reasoning focuses on the efficient use of cognitive resources, a related challenge is the efficient use of data. One of the most striking aspects of human learning is the ability to learn new words or concepts from limited numbers of examples [33]. This ability contrasts strongly with traditional machine learning methods, which typically require many labeled examples in order to even begin to classify stimuli accurately. Recent work in machine learning has aimed to narrow this gap,

Towards general intelligence
----------------------------

Meta-reasoning and meta-learning individually capture key components of human intelligence, but it may be by combining them that we come closest to identifying what is necessary to achieve the kind of general intelligence that currently eludes AI systems. While meta-learning has begun to explore problems that involve learning to perform many different tasks in parallel, those tasks remain relatively similar to one another. By contrast, what characterizes human environments is the need to

Conflict of interest statement
------------------------------

Nothing declared.

References and recommended reading
----------------------------------

Papers of particular interest, published within the period of review, have been highlighted as:* • of special interest
* •• of outstanding interest

References (50)
---------------

* S.J. Russell *et al.*### [Principles of metareasoning](/science/article/pii/000437029190015C)

### Artif Intell

(1991)
* S.J. Russell### [Rationality and intelligence](/science/article/pii/S000437029700026X)

### Artif Intell

(1997)
* E.J. Horvitz *et al.*### [Computational tradeoffs under bounded resources](/science/article/pii/S0004370201000510)

### Artif Intell

(2001)
* G. Gigerenzer *et al.*### [Heuristic decision making](/science/article/pii/S1364661315002041)

### Annu Rev Psychol

(2011)
* D. Amodei *et al.*### AI and Compute

(2018)
* S.J. Russell *et al.*### Do the Right Thing: Studies in Limited Rationality

(1991)
* T.L. Griffiths *et al.*### Rational use of cognitive resources: levels of analysis between the computational and the algorithmic

### Top Cognit Sci

(2015)
* S.J. Gershman *et al.*### Computational rationality: a converging paradigm for intelligence in brains, minds, and machines

### Science

(2015)
* M.T. Cox *et al.*### Metareasoning: Thinking about Thinking

(2011)
* S.J. Russell *et al.*### Provably bounded-optimal agents

### J Artif Intell Res

(1994)

- N. Hay *et al.*### Selecting computations: theory and applications
- F. Lieder *et al.*### Algorithm selection by rational metareasoning as a model of human strategy selection

### Advances in Neural Information Processing Systems

(2014)
- S. Milli *et al.*### When does bounded-optimal metareasoning favor few cognitive systems?
- F. Callaway *et al.*### Learning to select computations

### Uncertainty in Artificial Intelligence: Proceedings of the Thirty-Fourth Conference

(2018)
- C.H. Lin *et al.*### Metareasoning for Planning Under Uncertainty

### Proceedings of the 24th International Conference on Artificial Intelligence

(2015)
- F. Lieder *et al.*### An automatic method for discovering rational heuristics for risky choice

### Proceedings of the 39th Annual Meeting of the Cognitive Science Society (CogSci)

(2017)
- E. Vul *et al.*### One and done? Optimal decisions from very few samples

### Cognit Sci

(2014)
- S.J. Russell### Rationality and intelligence: a brief update
- L. Kocsis *et al.*### Bandit based Monte-Carlo planning

### European Conference on Machine Learning

(2006)
- F. Lieder *et al.*### Burn-in, bias, and the rationality of anchoring

### Advances in Neural Information Processing Systems

(2012)
- J. Dunlosky *et al.*### Metacognition

(2008)
- M. Oaksford *et al.*### A rational analysis of the selection task as optimal data selection

### Psychol Rev

(1994)
- T.M. Gureckis *et al.*### Self-directed learning: a cognitive and computational perspective

### Perspect Psychol Sci

(2012)
- J.W. Payne *et al.*### The Adaptive Decision Maker

(1993)
- Y.-L. Boureau *et al.*### Deciding how to decide: self-control and meta-decision making

### Trends Cognit Sci

(2015)
View more referencesCited by (74)
-------------

* ### [Experience-driven recalibration of learning from surprising events](/science/article/pii/S0010027722003328)

2023, CognitionCitation Excerpt :This conclusion is congruent with findings that statistical experience can shape learning and decision processes in a variety of domains (Constantino & Daw, 2015; McGuire & Kable, 2012; Ossmy et al., 2013; Schweighofer et al., 2006). Our findings support a class of models in which patterns of belief updating are guided by higher-order beliefs about environmental structure (Razmi & Nassar, 2022; Yu et al., 2021) and highlight the need to extend such models to incorporate experiential structure-learning or metalearning processes (Griffiths et al., 2019; Wang, 2021). Findings from the present work will provide useful constraints for further model development.

Show abstractDifferent environments favor different patterns of adaptive learning. A surprising event that in one context would accelerate belief updating might, in another context, be downweighted as a meaningless outlier. Here, we investigated whether people would spontaneously regulate the influence of surprise on learning in response to event-by-event experiential feedback. Across two experiments, we examined whether participants performing a perceptual judgment task under spatial uncertainty (*n* = 29, *n* = 63) adapted their patterns of predictive gaze according to the informativeness or uninformativeness of surprising events in their current environment. Uninstructed predictive eye movements exhibited a form of *metalearning* in which surprise came to modulate event-by-event learning rates in opposite directions across contexts. Participants later appropriately readjusted their patterns of adaptive learning when the statistics of the environment underwent an unsignaled reversal. Although significant adjustments occurred in both directions, performance was consistently superior in environments in which surprising events reflected meaningful change, potentially reflecting a bias towards interpreting surprise as informative and/or difficulty ignoring salient outliers. Our results provide evidence for spontaneous, context-appropriate recalibration of the role of surprise in adaptive learning.
* ### [Learning From Major Accidents: A Meta-Learning Perspective](/science/article/pii/S092575352200323X)

2023, Safety ScienceShow abstractLearning from the past is essential to improve safety and reliability in the chemical industry. In the context of Industry 4.0 and Industry 5.0, where Artificial Intelligence and IoT are expanding throughout every industrial sector, it is essential to determine if an artificial learner may exploit historical accident data to support a more efficient and sustainable learning framework. One important limitation of Machine Learning algorithms is their difficulty in generalizing over multiple tasks. In this context, the present study aims to investigate the issue of meta-learning and transfer learning, evaluating whether the knowledge extracted from a generic accident database could be used to predict the consequence of new, technology-specific accidents. To this end, a classification algorithm is trained on a large and generic accident database to learn the relationship between accident features and consequence severity from a diverse pool of examples. Later, the acquired knowledge is transferred to another domain to predict the number of fatalities and injuries in new accidents. The methodology is evaluated on a test case, where two classification algorithms are trained on a generic accident database (i.e., the Major Hazard Incident Data Service) and evaluated on a technology-specific, lower-quality database. The results suggest that automated algorithms can learn from historical data and transfer knowledge to predict the severity of different types of accidents. The findings indicate that the knowledge gained from previous tasks might be used to address new tasks. Therefore, the proposed approach reduces the need for new data and the cost of the analyses.
* ### [A meta-learning approach to improving radiation response prediction in cancers](/science/article/pii/S001048252200871X)

2022, Computers in Biology and MedicineCitation Excerpt :Many statistical methods have been proposed to solve this dilemma, “the curse of dimensionality”, including taking prior knowledge into account and borrowing information from other similar diseases [25–27]. Meta-learning, which has emerged in recent years, is a framework that relies on previously learned knowledge from similar tasks and quickly adapt to a new task with a small amount of training data [28,29]. Unlike traditional machine learning methods, the learning algorithm of which are usually predefined and handcrafted, meta-learning performs the optimization activity autonomously and explores the diversity of multiple learning algorithms [30].

Show abstractPredicting the efficacy of radiotherapy in individual patients has drawn widespread attention, but the limited sample size remains a bottleneck for utilizing high-dimensional multi-omics data to guide personalized radiotherapy. We hypothesize the recently developed meta-learning framework could address this limitation.

By combining gene expression, DNA methylation, and clinical data of 806 patients who had received radiotherapy from The Cancer Genome Atlas (TCGA), we applied the Model-Agnostic Meta-Learning (MAML) framework to tasks consisting of pan-cancer data, to obtain the best initial parameters of a neural network for a specific cancer with smaller number of samples. The performance of meta-learning framework was compared with four traditional machine learning methods based on two training schemes, and tested on Cancer Cell Line Encyclopedia (CCLE) and Chinese Glioma Genome Atlas (CGGA) datasets. Moreover, biological significance of the models was investigated by survival analysis and feature interpretation.

The mean AUC (Area under the ROC Curve) [95% confidence interval] of our models across nine cancer types was 0.702 [0.691–0.713], which improved by 0.166 on average over other the four machine learning methods on two training schemes. Our models performed significantly better (*p* < 0.05) in seven cancer types and performed comparable to the other predictors in the rest of two cancer types. The more pan-cancer samples were used to transfer meta-knowledge, the greater the performance improved (*p* < 0.05). The predicted response scores that our models generated were negatively correlated with cell radiosensitivity index in four cancer types (*p* < 0.05), while not statistically significant in the other three cancer types. Moreover, the predicted response scores were shown to be prognostic factors in seven cancer types and eight potential radiosensitivity-related genes were identified.

For the first time, we established the meta-learning approach to improving individual radiation response prediction by transferring common knowledge from pan-cancer data with MAML framework. The results demonstrated the superiority, generalizability, and biological significance of our approach.
* ### [Thermodynamic fluctuation theorems govern human sensorimotor learning](https://doi.org/10.1038/s41598-023-27736-8)

2023, Scientific Reports
* ### [Shaped-Charge Learning Architecture for the Human–Machine Teams](https://doi.org/10.3390/e25060924)

2023, Entropy
* ### [A computational process-tracing method for measuring people’s planning strategies and how they change over time](https://doi.org/10.3758/s13428-022-01789-5)

2023, Behavior Research Methods

[View all citing articles on Scopus](http://www.scopus.com/scopus/inward/citedby.url?partnerID=10&rel=3.0.0&eid=2-s2.0-85062041121&md5=eaa134c3594586f3b3e7bdb1f7bdc1)Recommended articles (6)
------------------------

* Research article### [MicroRNAs change the games in central nervous system pharmacology](/science/article/pii/S0006295219302473)

Biochemical Pharmacology, Volume 168, 2019, pp. 162-172Show abstractMicroRNAs (miRNAs) represent a class of important post-transcriptional regulators of gene expression, enabling cells to follow their intrinsic developmental program. By directly binding to their targets, miRNAs can both promote transcriptional patterns in crucial steps of cell growth, and act as powerful buffering system that titrate protein content in case of aberrant gene expression. The literature of the last decade showed that the presence of tissue-enriched miRNAs in body fluids could be reminiscent of disease state. This is particularly relevant in neurodegenerative disorders, in which peripheral biomarkers could be helpful means to detect disease onset. However, dysregulation of miRNAs is not merely a consequence of disease, but directly contributes to pathological outcomes. On this basis, increasing interest is growing in the development of pharmacological agents targeting specific miRNAs. Actually, this apparently futuristic approach is already part of the current therapies. In fact, several drugs approved for CNS disorders, such as L-Dopa or valproic acid, were also demonstrated to restore some miRNAs. Moreover, ongoing clinical trials demonstrated that miRNA-based drugs are effective against tumors, suggesting that miRNAs also represent a promising class of therapeutic molecules. However, several issues still need to be addressed, particularly in case of CNS diseases, in which stability and delivery are crucial aspects of the therapy. In this commentary, we highlighted potential advantages and limitations of miRNAs as next generation targets in CNS pharmacology, focusing on multiple sclerosis, a chronic demyelinating disease lacking specific therapeutic targets and bona-fide biomarkers.
* Research article### [Distributed semantic representations for modeling human judgment](/science/article/pii/S2352154618301712)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 31-36Show abstractPeople make judgments about thousands of different objects and concepts on a day-to-day basis; however, capturing the knowledge that subserves these judgments has been difficult. Recent advances in computational linguistics are filling this gap, as the statistics of language use yield rich, distributed semantic representations for natural objects and concepts. These representations have been shown to predict semantic and linguistic judgments, such as judgments of meaning and relatedness, and more recently, high-level judgments, including probability judgment and forecasting, stereotyping and various types of social judgment, consumer choice, and perceptions of risk. Distributed semantic representations are now a key component of computational models that represent knowledge, make evaluations and attributions, and give responses, in a human-like manner.
* Research article### [Linking neural responses to behavior with information-preserving population vectors](/science/article/pii/S2352154618302092)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 37-44Show abstractAll systems for processing signals, both artificial and within animals, must obey fundamental statistical laws for how information can be processed. We discuss here recent results using information theory that provide a blueprint for building circuits where signals can be read-out without information loss. Many properties that are necessary to build information-preserving circuits are actually observed in real neurons, at least approximately. One such property is the use of logistic nonlinearity for relating inputs to neural response probability. Such nonlinearities are common in neural and intracellular networks. With this nonlinearity type, there is a linear combination of neural responses that is guaranteed to preserve Shannon information contained in the response of a neural population, no matter how many neurons it contains. This read-out measure is related to a classic quantity known as the population vector that has been quite successful in relating neural responses to animal behavior in a wide variety of cases. Nevertheless, the population vector did not withstand the scrutiny of detailed information-theoretical analyses that showed that it discards substantial amounts of information contained in the responses of a neural population. We discuss recent theoretical results showing how to modify the population vector expression to make it ‘information-preserving’, and what is necessary in terms of neural circuit organization to allow for lossless information transfer. Implementing these strategies within artificial systems is likely to increase their efficiency, especially for brain–machine interfaces.
* Research article### [Planning at decision time and in the background during spatial navigation](/science/article/pii/S2352154618301682)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 69-76Show abstractPlanning is the model-based approach to solving control problems. The hallmark of planning is the endogenous generation of dynamical representations of future states, like goal locations, or state sequences, like trajectories to the goal location, using an internal model of the task. We review recent evidence of model-based planning processes and the representation of future goal states in the brain of rodents and humans engaged in spatial navigation tasks. We highlight two distinct but complementary usages of planning as identified in artificial intelligence: ‘at decision time’, to support goal-directed choices and sequential memory encoding, and ‘in the background’, to learn behavioral policies and to optimize internal models. We discuss how two kinds of internally generated sequences in the hippocampus – theta and SWR sequences – might participate in the neuronal implementation of these two planning modes, thus supporting a flexible model-based system for adaptive cognition and action.
* Research article### [Analogues of mental simulation and imagination in deep learning](/science/article/pii/S2352154618301670)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 8-16Show abstractMental simulation — the capacity to imagine what will or what could be — is a salient feature of human cognition, playing a key role in a wide range of cognitive abilities. In artificial intelligence, the last few years have seen the development of methods which are analogous to mental models and mental simulation. This paper outlines recent methods in deep learning for constructing such models from data and learning to use them via reinforcement learning, and compares such approaches to human mental simulation. Model-based methods in deep learning can serve as powerful tools for building and scaling cognitive models. However, a number of challenges remain in matching the capacity of human mental simulation for efficiency, compositionality, generalization, and creativity.
* Research article### [The value of abstraction](/science/article/pii/S2352154619300026)

Current Opinion in Behavioral Sciences, Volume 29, 2019, pp. 111-116
[☆](#baep-article-footnote-id6)This research was supported by contract number FA9550-18-1-0077 from the Air Force Office of Scientific Research and by contract number FA8650-18-2-7832 from the Defence Advanced Research Projects Agency.

[View full text](/science/article/pii/S2352154618302122)

Thomas L
Griffiths
Frederick Callaway
Michael B
Chang
Erin Grant
Paul M
Krueger
Falk Lieder


Empirical evidence for resource-rational anchoring and adjustment.

Psychon Bull Rev (2018) 25:775–784
DOI 10.3758/s13423-017-1288-6
BRIEF REPORT
Empirical evidence for resource-rational anchoring
and adjustment
Falk Lieder1,2·Thomas L. Griffiths1,5·Q u e n t i nJ .M .H u y s2,4·Noah D. Goodman3
Published online: 8 May 2017
© Psychonomic Society, Inc. 2017
Abstract People’s estimates of numerical quantities are
systematically biased towards their initial guess. This
anchoring bias is usually interpreted as sign of human irra-
tionality, but it has recently been suggested that the anchor-
ing bias instead results from people’s rational use of their
finite time and limited cognitive resources. If this were true,
then adjustment should decrease with the relative cost of
time. To test this hypothesis, we designed a new numerical
estimation paradigm that controls people’s knowledge and
varies the cost of time and error independently while allow-
ing people to invest as much or as little time and effort intorefining their estimate as they wish. Two experiments con-
firmed the prediction that adjustment decreases with time
cost but increases with error cost regardless of whether the
anchor was self-generated or provided. These results sup-
port the hypothesis that people rationally adapt their number
/envelopebackFalk Lieder
falk.lieder@berkeley.edu
1Helen Wills Neuroscience Institute, University of California,Berkeley USA
2Translational Neuromodeling Unit, Institute for BiomedicalEngineering, University of Z ¨urich and Swiss Federal Institute
of Technology (ETH) Z ¨urich, Z ¨urich, Switzerland
3Department of Psychology, Stanford University,Stanford, USA
4Department of Psychiatry, Psychotherapy and Psychosomatics,Hospital of Psychiatry, University of Z ¨urich, Z ¨urich,
Switzerland
5Department of Psychology, University of California,Berkeley, USAof adjustments to achieve a near-optimal speed-accuracy
tradeoff. This suggests that the anchoring bias might bea signature of the rational use of finite time and limitedcognitive resources rather than a sign of human irrationality.
Keywords Bounded rationality ·Heuristics ·Cognitive
biases ·Probabilistic reasoning ·
Anchoring-and-adjustment
Tversky and Kahneman’s demonstrations that human judg-
ment systematically violates the normative principles ofprobability theory fundamentally challenged the belief that
people are rational (Tversky & Kahneman, 1974). In one of
these demonstrations, Tversky and Kahneman (1974) spun
a wheel of fortune in front of their participants. Participants
were asked to judge whether the number of African coun-
tries in the United Nations was larger or smaller than thenumber the wheel had stopped on. Afterwards, participants
estimated exactly how many African countries there were
in the United Nations. Curiously, participants’ estimates ofthis quantity were significantly biased towards the irrelevantnumber they had just compared it to.. According to Tversky
and Kahneman ( 1974), this violation occurs because people
use a two-stage process called anchoring-and-adjustment.
In the first stage, people generate an initial guess called theiranchor . In the second stage, they adjust their estimate away
from this anchor to incorporate additional information, but
the adjustment is usually insufficient. In the experiment
described above, people appear to have anchored on the
random number provided by the experimenter and adjustedtheir estimate insufficiently. Consequently, when the anchorwas low, people’s judgments were too low; and when the
anchor was high, then their judgments were too high.
776 Psychon Bull Rev (2018) 25:775–784
At first sight, anchoring appears to be irrational, because
it deviates from the standards of logic and probability which
are typically used to assess rationality. But it could also arise
from an optimal tradeoff between the cost of error in judg-
ment and the cost of the time it takes to reduce this error.This hypothesis has been formalized by a computational
model that chooses the number of adjustments that mini-
mizes the expected sum of time cost and error cost (Lieder,Griffiths, & Goodman, 2012; Lieder, Griffiths, Huys, &
Goodman, 2017a , under review). This model predicts that
adjustment should decrease with time cost but increase witherror cost regardless of whether the anchor was providedor self-generated. Here, we experimentally test these pre-
dictions by varying both the cost of error and the cost of
time without imposing a deadline. This allows us to eval-uate how much time and effort people choose to invest
into adjusting their estimate away from the anchor against
the normative prescriptions of resource-rational anchor-ing and adjustment (Lieder, et al., 2012, 2017a , under
review).
After presenting the predictions of the resource-rational
anchoring and adjustment model, we first report an exper-iment in which we evaluated the effect of time cost and
error cost on adjustment from self-generated anchors, and
then investigate the same effects in an experiment with pro-vided anchors. We close by discussing the implications of
our findings for theories of anchoring and the debate about
human rationality (Stanovich, 2009).
Empirical predictions of resource-rational
anchoring and adjustment
Resource-rational anchoring and adjustment postulates that
the number of adjustments people perform achieves a near-
optimal tradeoff between the cost of error and the costof time (Lieder et al., 2012, 2013, 2017a , under review).
In brief, optimal resource allocation implies that relativeadjustment decreases with the relative cost of time. There-fore, the slope of the anchoring bias as a function of thedistance from the anchor to the correct value should be high-
est when time cost is high and error cost is low. Conversely,
the slope of the anchoring bias should be the shallowestwhen error cost is high and time cost is low. Lastly, when
time cost and error cost are both high or both low, then
the slope should be intermediate. Figure 1illustrates these
predictions.
Furthermore, resource-rational anchoring and adjustment
also predicts that (an upper bound on) the anchoring biasincreases linearly with the distance of the anchor fromthe correct value. This prediction leads to a linear regres-
sion model that allows us to estimate people’s anchor
Fig. 1 According to resource-rational anchoring-and-adjustment the
negative anchoring bias increases linearly with the distance from the
anchor to the true value. Importantly, this model predicts that the slope
of this function increase with time cost (TC) and decreases with errorcost (EC)
a, their relative adjustments (E[ˆX|x]−a
x−a), and the resulting
anchoring bias Bias t(x, a) by regressing their estimates ˆX
on the correct value x:
ˆX=α+β·x+ε, ε∼N(0,σ2
ε) (1)
E[ˆX|x]−a
x−a=β, a=α
1−β(2)
Biast(x, a) =α−(1−β)·x. (3)
Contrary to Epley and Gilovich (2006), resource-rational
anchoring and adjustment assumes that people adjust not
only from self-generated anchors but also from provided
anchors. If this assumption is correct, then error cost shouldincrease adjustment and time cost should decrease adjust-
ment regardless of whether the anchor was self-generated
(Experiment 1) or provided (Experiment 2).
Experiment 1: Self-generated anchors
In most previous anchoring experiments the biases in peo-
ple’s judgments resulted not only from anchoring but also
from the discrepancy between the truth and what peopleactually know (Lieder et al., 2017a ). To avoid this confound,
we designed a prediction task that controls people’s knowl-
edge about the quantity to be estimated. To test if people
adapt their number of adjustments rationally, we manip-ulated both the cost of time and the cost of error within
subjects. In this first experiment, no anchor was provided
to the participant. Instead, we estimated our participants’self-generated anchor from how the bias of their responses
changes with the correct value ( 1–2).
Psychon Bull Rev (2018) 25:775–784 777
Method
Participants We recruited 30 participants (14 male, 15
female, 1 unreported) on Amazon Mechanical Turk. Our
participants were between 19 and 65 years old, and theirlevel of education ranged from high school to graduate
degrees. Participants were paid $1 .05 for their time and
could earn a performance-dependent bonus of up to $0.80.
Six participants were excluded because they incorrectly
answered questions designed to test their understanding of
the task (see Procedure).
Materials The experiment was presented as a website pro-
grammed in HTML and JavaScript. Participants predicted
when a person would get on a bus given when he had arrived
at the bus stop based on the bus’s timetable and examplesof previous departure times. Figure 2shows a screenshot
from one of the trials. The timeline at the top of the screenwas used to present the relevant information and record pre-dictions. At the beginning of each trial the bus’s timetable
(orange bars) and the person’s arrival at the bus stop (blue
bars) were highlighted on the timeline. Participants indi-cated their prediction by clicking on the corresponding pointon the timeline. When participants were incentivized to
respond quickly, a falling red bar indicated the passage of
time and its cost, and the costs of time and error wereconveyed in the bottom left corner; see Fig. 2. Feedback
was provided by a pop-up window informing participantsabout how many points they had earned and a green barhighlighting the actual departure time on the number line.
Procedure After completing the consent form, each person
participated in four scenarios corresponding to the four con-
ditions of a 2 ×2 within-subject design with the independent
variables time cost (0 vs. 30 points/sec) and error cost (0 vs.
10 points/unit error). The order of the four conditions was
randomized between subjects. At the end of the experimentparticipants received a bonus payment proportional to the
number of points they had earned in the experiment. The
conversion rate was 1 cent per 100 points, and participantscould earn up to 100 points per trial.
Each scenario comprised a cover story, instructions, 10
examples, 5 practice trials, 5 attention check questions, 20prediction trials, 3 test questions, and one demographicquestion. Each cover story was about a person repeatedly
taking the same bus route in the morning, for example
“Jacob commutes to work with bus #22. On average, thefirst bus departs at 8:01 AM, and the second bus departs
at 8:21 AM but departure times vary. On some days Jacob
misses the first bus and takes the second bus.” In each sce-nario both the person and the bus route were different. The
task instructions informed participants about the cost oftime and error and encouraged them to attentively study the
examples and practice trials so that they would learn to makeaccurate predictions. After the cover story, participants were
shown when the bus had arrived on the ten workdays of the
two preceding weeks (10 examples); see Fig. 3.
Next, participants made 5 practice predictions with feed-
back. The ensuing attention check questions verified theparticipants’ understanding of the time line and the costsof time and error. Participants were allowed to go back
and look up this information if necessary. Participants who
made at least one error were required to retake this test untilthey got all questions correct. Once they had answered allquestions correctly, participants proceeded to 20 predictions
trials with feedback. In both the practice trials and the pre-
diction trials the feedback comprised the correct departuretime, the incurred error cost, the incurred time cost, and
the resulting number of points for the trial. The times at
which the fictitious person arrived at the bus stop were cho-sen such that the probability that he had missed the first bus
approximately covered the full range from 0 to 1 in equal
increments. In the 1
st,3rd,···,2nd-last prediction trial the
person arrived early and the bus was on time. The purposeof these odd-numbered trials was to set the anchor on the
even-numbered trials to a low value. After each scenario’s
prediction trials we tested participants’ understanding of thenumber line, the cost of time, and the cost of error once
again. We excluded six participants, because their answers
to these questions revealed that they had misunderstood thenumber line, the cost of time, or the cost of error in at least
one condition. After this they reported one piece of demo-
graphic information: age, gender, level of education, andemployment status respectively. On the last page of eachblock, participants were informed about the bonus they had
earned in the scenario.
To pose a different prediction problem on every trial of
each block despite the limited number of meaningfully dif-ferent arrival times, we varied the distribution of the bus’s
delays between blocks. There were four delay distributionsin total. All of them were Pearson distributions that differed
only in their variance. Their mean, skewness, and kurto-
sis were based on the bus lateness statistics from GreatBritain.
1The order of the delay distributions was random-
ized between participants independently of the incentives.
The 10 examples of bus departure times were chosen such
that their mean, variance, and skewness reflected the block’sdelay distribution as accurately as possible. For each trial,
a “correct” departure time xwas sampled from the con-
ditional distribution of departure times given that the fic-
titious person departs after his arrival at the bus stop. The
1Bus Punctuality Statistics GB 2007 report; http://estebanmoro.org/
2009/01/waiting-for-the-bus/
778 Psychon Bull Rev (2018) 25:775–784
Fig. 2 Screenshot of a prediction trial from Experiment 1with time cost and error cost. The number line on the top conveys the bus schedule and
when the person arrived at the bus stop. The cost of error and time are shown in the bottom left corner, and the red bar in the bottom right corner
shows the passage of time and the cost associated with it. The complete experiment can be inspected online at http://cocosci.berkeley.edu/mturk/
falk/PredictionExperiment1/experiment.html
condition’s cost of time ctand cost of error cedetermined the
number of points a participant would receive according to
points =max{0,100−ce·PE−ct·RT}, (4)
PE=|ˆx−x|, (5)
where PE is the absolute prediction error between the esti-
mateˆxand the true value x, and RT is the response time.
The bottom part of Fig. 2shows how time cost and error
cost were conveyed to the participants during the trials. Thered bar on the right moved downward and its position indi-cates how much time has passed and how many points haveconsequently been lost.
Results and discussion
To assess whether our participants’ predictions were sys-
tematically biased, we inspected their average prediction
for a range of true bus delays. The true bus delays weresampled from a distribution, of which subjects had seen 10
samples. We binned participants’ average predictions when
t h et r u eb u sd e l a yw a s0 .5±2.5m i n ,5.5±2.5m i n ,...,
or 35.5 ±2.5 min. Participants showed a systematic bias,
overestimating the delay when its true value was less than 3minutes (t( 815)=16.0,p < 10
−15), but underestimatingit when its true value was larger than 7 minutes (all p≤
0.0011; see Fig. 4). Visual inspection suggested that the
bias was approximately proportional to the correct value.
Applying the linear regression model of anchoring (1–3)
confirmed that the linear correlation between correct valueand bias was significantly different from zero (P( slope ∈
[−0.6148, −0.5596]) =0.95). As shown in Fig. 4, the bias
was positive when the delay was greater than 7.5min andnegative for greater delays. Our participants thus appearedto anchor around 7.5min and adjust their initial estimate by
about 41.3% of the total distance to the true value (95%-CI:
[38.52%, 44.04%]).
Since the data showed standard anchoring effects, we
can now proceed to testing its novel predictions. Con-sistent with our theory’s prediction, we found that time
cost made participants faster and less accurate whereas
error cost made them slower and more accurate (Fig. 5).
To determine whether the anchoring bias increased withtime cost and decreased with error cost we performed a
repeated-measures ANOV A of participants’ relative adjust-
ments as a function of time cost and error cost. To beprecise, we first estimated each participant’s relative adjust-
ment separately for each of the four conditions using our
linear regression model of anchoring and adjustment ( 1).
We then performed an ANOV A on the estimated relative
Psychon Bull Rev (2018) 25:775–784 779
Fig. 3 Screenshot of the first examples screen of Experiment 1
adjustments with the factors time cost and error cost
(fixed-effects) as well as participant number (random effect)
and the interaction effect of time cost and error cost. We
found that time cost significantly reduced relative adjust-ment from 50.7% to 31.0% (F( 1,69)=21.86,p <
0.0001) whereas error cost significantly increased it from
31.6% to 50.1% (F( 1,69)=19.49,p < 0.0001) and the
interaction was non-significant. The mean relative adjust-
ments of each condition are shown in Table 1. Consequently,
as predicted by our theory (Fig. 1), the anchoring bias
increased more rapidly with the true delay when time costw a sh i g ho re r r o rc o s tw a sl o w( F i g .6). This is consistent
with the hypothesis that people rationally adapt the number
of adjustments to the relative cost of time.
2
Quantitative comparisons of our resource-rational model
against alternative theories, including the anchoring and
adjustment heuristic proposed by Epley and Gilovich
(2006), also provided strong evidence for rational adjust-ment, and we provide a detailed summary of these results
in a technical report (Lieder et al., 2017b). While the results
presented here demonstrate that people adaptively trade off
being biased for being fast, our analysis had to postulate
and estimate people’s self-generated anchors. Therefore, we
2Estimating relative adjustment under the assumption that people
anchor on their previous estimate led to the same conclusions.cannot be sure whether people really self-generated anchors
and adjusted from them, or whether their responses merely
look as if they did so. If people really use anchoring and
adjustment in this task, then we should be able to shift thebiases shown in Fig. 4by providing different anchors; we
tested this prediction in Experiment 2.
-5 0 5 10 15 20 25 30
Actual Departure Time (min)-20-15-10-50510Anchoring Bias95%-confidence band on bias
bias estimated by regression
bias estimated by binning
no bias
Fig. 4 In Experiment 1the magnitude of the anchoring bias grew lin-
early with the correct value. The error bars indicate 95% confidence
intervals on the average bias, that is ±1.96 standard errors of the mean
780 Psychon Bull Rev (2018) 25:775–784
02468Mean Absolute ErrorNo Time Cost
High Time Cost
No Error Cost High Error Cost
No Error Cost High Error Cost0500100015002000Median RT (ms)No Time Cost
High Time Cost
Fig. 5 Mean absolute errors and reaction times as a function of time
cost and error cost indicate an adaptive speed-accuracy tradeoff
Experiment 2: Provided Anchors
To test whether the biases observed in Experiment 1resulted
from anchoring and to evaluate whether the effects of time
cost and error cost also hold for provided anchors, we ran a
second experiment in which anchors were provided by ask-
ing participants to compare the to-be-predicted delay to alow versus a high number before every prediction.
Method
The materials, procedures, models, and data analysis tools
used in Experiment 2were identical to those used in Exper-
iment 1unless stated otherwise.
Participants We recruited 60 participants (31 male, 29
female) on Amazon Mechanical Turk. They were between18 and 60 years old, and their level of education ranged from
high school diploma to PhD. Participants were paid $1.25for participation and could earn a bonus of up to $2 .20 for
the points they earned in the experiment.
Materials Experiment 2was presented as a website pro-
grammed in HTML and JavaScript. Experiment 2was
mostly identical to Experiment 1. The relevant changes
are summarized below. The complete experiment can be
inspected online at http://cocosci.berkeley.edu/mturk/falk/
PredictionExperiment2/experiment.html .
Procedure Experiment 2proceeded like Experiment 1ex-
cept for three changes: First, each prediction was precededby the question “Do you think he will depart before or after
Xam?”, where Xis the anchor. This question was pre-
sented between the sentence reporting the time the person
reached the bus stop and the number line. Participants wereTa b l e 1 Relative size of our participants’ adjustments of their initial
guesses towards the correct answer by incentive condition with 95%
confidence intervals
No error cost High error cost
No time cost 43.6±11.2% 57.8±4.8%
High time cost 19.6±9.0% 42.5±9.8%
required to answer this question by selecting “before” or
“after”. This is the standard procedure for providing anchors(Jacowitz & Kahneman, 1995; Russo & Schoemaker, 1989;
Tversky & Kahneman, 1974). In the two conditions with
time cost, participants were given 3 seconds to answerthis question before the timer started. Participants were not
allowed to make a prediction until they had answered. We
incentivized them to take this question serious by awarding+10 points for correct answers and −100 points for incor-
rect ones. For each participant the anchor was high in half ofthe trials of each condition and low in the other half. The lowanchor was 3 minutes past the scheduled departure of thefirst bus, and the high anchor was 3 minutes past the sched-
uled departure of the second bus. The list of anchors was
shuffled separately for each block and participant. Second,the 1
st,3rd,5th,···,2nd-last trial were no longer needed,
because they merely served to set the anchor on the evennumbered trials of Experiment 1to a small value. We there-
fore replaced those trials by 10 trials whose query times
tighten the grid of those in the even-numbered trials. Thus
for each participant, each block includes ten prediction trialswith low anchors and ten prediction trials with high anchors.Third, the conversion of points into bonuses remained lin-
ear but was scaled up accordingly. The instructions were
updated to reflect the changes.
We excluded one participant due to incomplete data, and
16 participants whose answers to our test questions indi-cated they had misunderstood the time line used to presentinformation and record predictions, or the cost of time or
error in at least one condition.
3
Results and Discussion
Our participants answered the anchoring questions cor-
rectly in 74 .8% of the trials. As in Experiment 1, people’s
predictions were systematically biased: Our participantssignificantly overestimated delays smaller than 8 min (all
p< 10
−11) and significantly underestimated delays larger
than 13 min (all p< 10−4); see Fig. 7. Furthermore,
the biases were shifted upwards when the anchor was
3This exclusion rate would be high in a laboratory experiment, but it
is not unusual for long online experiments run on Amazon Mechanical
Turk.
Psychon Bull Rev (2018) 25:775–784 781
Fig. 6 Anchoring bias in Experiment 1by time cost and error cost confirms our theoretical prediction; compare Fig. 1
high compared to when the anchor was low ( z=7.26,p<
10−12;s e eF i g . 7). This effect was also evident in our
participants’ average predictions: when the anchor was
0 5 10 15 20 25
True Delay [min]-15-10-50510Anchoring Bias with 95% CI [min]No Bias
Low Anchor
High Anchor
Fig. 7 Biases when the provided anchor was high versus low. Solid lines
show the results of linear regression. Shaded areas are 95% confidence
bands, the diamonds with error bars are the average biases within a fiveminute window and their 95% confidence intervals; that is ±1.96 SEMhigh, then participants predicted significantly later depar-
tures than when the anchor was low: 12.06 ±0.29 min
versus 10.03 ±0.15 min (t( 3438) =6.16,p < 10−15). To
estimate our participants’ anchors and quantify their adjust-ments, we applied the linear regression model describedabove ( 1). Overall, the estimated anchor was significantly
higher in the high anchor condition (12.69 min) than inthe low anchor condition (9.74 min, p< 10
−15). Adjust-
ments tended to be small: on average, participants adjusted
their estimate only 29 .86% of the distance from the anchor
to the correct value when the anchor was low (95% CI:
[26.38%; 30.85%]) and 27.25% of this distance when the
anchor was high (95% CI: [24.00%; 30.50%]). Thus, the
relative adjustments were significantly smaller than inExperiment 1(95% CI: [38.52%, 44.04%]) and they did
not differ between the high and low anchor condition (z =
1.16;p=0.12). Thus the linear relationship between the
bias and the true delay and difference between the biasesfor the high versus the low anchor (Fig. 7)m a yr e s u l t
from insufficient adjustment away from different anchors.This also explains why the average predictions were higherin the high anchor condition than in the low anchor
condition.
782 Psychon Bull Rev (2018) 25:775–784
Ta b l e 2 ANOV A of relative adjustment as a function of time cost and error cost
Source d.f. Sum Sq. Mean Sq. F p
error cost 1 1.0318 1.03178 15.5 0.0001
time cost 1 0.6912 0.69115 10.39 0.0014
subject 42 11.3544 0.27034 4.06 10−12
anchor (high vs. low) 1 0.0774 0.07739 1.16 0.2817
error cost ×time cost 1 0.1066 0.10659 1.6 0.2066
Error 297 19.7643 0.06655
Total 343 33.0256
Next, we assessed whether the amount by which par-
ticipants adjusted their initial estimate increased with error
cost and decreased with time cost. To answer this ques-tion we performed a repeated-measures ANOV A of relativeadjustment as a function of time cost and error cost.
To be precise, we first estimated each participant’s rela-
tive adjustment separately for each of the four conditionsand the two anchors using our linear regression model of
anchoring and adjustment (1). We then performed an
ANOV A on the estimated relative adjustments with thefactors time cost, error cost, and high vs. low anchor (fixed-
effects) as well as participant number (random effect) and
the interaction effect of time cost and error cost; see Table 2.
We found that time cost significantly reduced relativeadjustment from 37 .2% to 28.2% (F( 1,297)=15.5,p =
0.0001) whereas error cost significantly increased it from31.2% to 34.2% (F( 1,297)=10.39,p =0.0014), and the
interaction was non-significant. The mean relative adjust-
ments of each condition are shown in Table 3. Figure 8
shows the effects of incentives for speed and accuracy on
the anchoring bias in the provided anchors experiment; note
that the slope of each line is 1 minus the relative size of the
adjustments as estimated with the linear regression model ofanchoring (1–3). As predicted by our theory (cf. Fig. 1)a n d
observed for self-generated anchors (cf. Fig. 6), the slope of
the anchoring bias was largest when time cost was high anderrors were not penalized. Table 3summarizes the relative
adjustments sizes in the four incentive conditions. Figure 8
suggests that the effects of time cost and error cost wereweaker in the high anchor condition than in the low anchorcondition.As for Experiment 1, quantitative model comparisons
confirmed that Experiment 2provides strong evidence for
rational adjustment (Lieder et al., 2017b).
In summary, our participants’ predictions were signif-
icantly biased towards the provided anchors. This bias
increased linearly with the distance from the anchor to the
correct value. Critically, as predicted by resource-rationalanchoring and adjustment, the magnitude of this effect
decreased with error cost and increased with time cost (com-
pare Figs. 1and 8). Thus the bias towards the provided
anchors and the effects of time cost and error cost were
qualitatively the same as for self-generated anchors (Fig. 6).
Hence, consistent with Simmons et al. ( 2010) but contra-
ry
to Epley and Gilovich (2006), our results suggest that
incentives increase adjustment from both provided and self-generated anchors. Incentives can thus be more effective at re-
ducing the anchoring bias than initially assumed (Tversky &Kahneman, 1974; Epley & Gilovich, 2005), and anchoring
and adjustment may be sufficient to explain the effects ofboth provided and self-generated anchors. For a detaileddiscussion of how the resource-rational anchoring-and-
adjustment model evaluated in this paper goes beyond
previous proposals, please see Lieder et al. ( 2017a ).
Despite the qualitative commonalities between the results
for self-generated versus provided anchors, there were
quantitative differences: In three of the four conditions, the
adjustments were significantly smaller for provided anchorsthan for self-generated anchors. One reason could be that
in Experiment 1the anchoring biases towards high versus
low self-generated anchors cancelled each other out. A sec-
ond reason could be that people treat a provided anchors as
Ta b l e 3 Relative size of our participants’ adjustments of their initial guess towards the correct answer by incentive condition in the experiment
with provided anchors with 95% confidence intervals
No Error Cost High error cost
No time cost 30.0±7.4% 44.4±8.4%
High time cost 24.5±6.5% 32.0±9.1%
Psychon Bull Rev (2018) 25:775–784 783
Fig. 8 Effect of incentives for speed and accuracy when a high anchor was provided confirm our theory’s prediction; cf. Figure 1.A) Anchoring
bias of our participants’ judgments as a function of the true delay as estimated with the linear regression model of anchoring ( 1–3)f o rt h el o w -
anchor condition (left ) and the high-anchor condition (right ).B) Average anchoring bias of our participants’ judgments binned by the true delay
along with the fit of the linear regression model of anchoring ( 1–3) for each of the four incentive conditions
a conversational hint that the correct value is close to that
value (Zhang and Schwarz, 2013).
Conclusion
Our experiments confirmed the predictions of resource-rationalanchoring and adjustment. Most importantly, people appearto rationally adapt their number of adjustments to thecost of time and error. When errors were costly, people
invested more time and were more accurate, their adjust-
ments were larger, and their anchoring bias was smaller.By contrast, when time was costly then our participants
were faster and less accurate, their adjustments appeared
to be smaller, and their anchoring bias was larger. Thisis consistent with the rational allocation of mental effort(Shenhav et al., 2017) and our hypothesis that the number
of adjustments is chosen to achieve an optimal speed-accuracy tradeoff. However, since our experiment used onlytwo levels of time cost and error cost, it remains to beinvestigated whether the number of adjustments changes
gradually with its costs and benefits–as predicted by the
resource-rational model–or whether people can only choosebetween a fast versus a slow mode of numerical estima-
tion as postulated by dual-systems theories (Evans, 2008;
Kahneman, 2011). Hence, while people’s estimates are
biased in the statistical sense of the word, this bias might beconsistent with how people ought to reason. In this sense,
the anchoring “bias” might not be a cognitive bias afterall.
784 Psychon Bull Rev (2018) 25:775–784
Acknowledgements This research was supported by grant number
ONR MURI N00014-13-1-0341 from the Office of Naval Research
(TLG and NDG), grant number FA-9550-10-1-0232 from the Air
Force Office of Scientific Research (TLG), and a John S. McDonnell
Scholar Award (NDG).
References
Epley, N., & Gilovich, T. (2005). When effortful thinking influ-
ences judgmental anchoring: differential effects of forewarn-ing and incentives on self-generated and externally providedanchors. Journal of Behavioral Decision Making, 18(3), 199–
212.
Epley, N., & Gilovich, T. (2006). The anchoring-and-adjustment
heuristic. Psychological Science, 17(4), 311–318.
Evans, J. S. B. (2008). Dual-processing accounts of reasoning, judg-
ment, and social cognition. Annual Review of Psychology, 59,
255–278.
Jacowitz, K. E., & Kahneman, D. (1995). Measures of anchoring
in estimation tasks. Personality and Social Psychology Bulletin ,
21(11), 1161–1166.
Kahneman, D. (2011). Thinking, fast and slow. New York: Farrar,
Strauss and Giroux.
Lieder, F., Goodman, N. D., & Griffiths, T. L. (2013). Reverse-
engineering resource-efficient algorithms .L a k eT a h o e ,U S A :
Paper presented at NIPS-2013 Workshop Resource-Efficient ML.Lieder, F., Griffiths, T., Huys, Q., & Goodman, N. (2017b). Testing
models of anchoring and adjustment. PsyArXiv Preprint. Retrievedfrom https://osf.io/preprints/psyarxiv/94yvz (Version 3).
Lieder, F., Griffiths, T. L., & Goodman, N. D. (2012). Burn-in, bias,
and the rationality of anchoring. In Bartlett, P., Pereira, F.C.N.,Bottou, L., Burges, C.J.C., & Weinberger, K.Q. (Eds.) Advances
in Neural Information Processing Systems 26. Red Hook: CurranAssociates, Inc.
Lieder, F., Griffiths, T. L., Huys, Q. J. M., & Goodman, N. D. (2017a).
The anchoring bias reflects rational use of cognitive resources.Psychonomic Bulletin and Review.
Russo, J. E., & Schoemaker, P. J. H. (1989). Decision traps: Ten barri-
ers to brilliant decision-making and how to overcome them. Simonand Schuster.
Shenhav, A., Musslick, S., Lieder, F., Kool, W., Griffiths, T., Cohen,
J., & Botvinick, M. (2017). Toward a rational and mechanisticaccount of mental effort. Annual Review of Neuroscience, 40.
Simmons, J. P., LeBoeuf, R. A., & Nelson, L. D. (2010). The effect
of accuracy motivation on anchoring and adjustment: Do peopleadjust from provided anchors?. Journal of Personality and Social
Psychology, 99(6), 917–932.
Stanovich, K. E. (2009). Decision making and rationality in the
modern world.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty:
Heuristics and biases. Science, 185(4157), 1124–1131.
Zhang, Y . C., & Schwarz, N. (2013). The power of precise numbers:
A conversational logic analysis. Journal of Experimental Social
Psychology, 49(5), 944–946.

Falk Lieder
Thomas L
Griffiths
Quentin J
M
Huys
Noah D
Goodman