A computational process-tracing method for measuring people’s planning strategies and how they change over time.

BehaviorResearchMethods
https://doi.org/10.3758/s13428-022-01789-5
Acomputationalprocess-tracingmethodformeasuringpeople’s
planningstrategiesandhowtheychangeovertime
YashRajJain1,2·FrederickCallaway3·ThomasL.Griﬃths3·PeterDayan4·RuiqiHe1·PaulM.Krueger5·
FalkLieder1
Accepted:3January2022
©TheAuthor(s)2022
Abstract
One of the most unique and impressive feats of the human mind is its ability to discover and continuously refine its own
cognitive strategies. Elucidating the underlying learning and adaptation mechanisms is very difficult because changes in
cognitive strategies are not directly observable. One important domain in which strategies and mechanisms are studied is
planning. To enable researchers to uncover how people learn how to plan, we offer a tutorial introduction to a recently
developed process-tracing paradigm along with a new computational method for measuring the nature and development of
a person’s planning strategies from the resulting process-tracing data. Our method allows researchers to reveal experience-
driven changes in people’s choice of individual planning operations, planning strategies, strategy types, and the relative
contributions of different decision systems. We validate our method on simulated and empirical data. On simulated data, its
inferences about the strategies and the relative influence of different decision systems are accurate. When evaluated on human
data generated using our process-tracing paradigm, our computational method correctly detects the plasticity-enhancing
effect of feedback and the effect of the structure of the environment on people’s planning strategies. Together, these methods
can be used to investigate the mechanisms of cognitive plasticity and to elucidate how people acquire complex cognitive
skills such as planning and problem-solving. Importantly, our methods can also be used to measure individual differences in
cognitive plasticity and examine how different types (pedagogical) interventions affect the acquisition of cognitive skills.
Keywords Process-tracing ·Cognitive plasticity ·Panning ·Decision-making ·Individual differences ·Learning ·
Computational methods
Introduction
A remarkable feature of the human mind is its ability to
improve itself continually. As helpless babies develop into
mature adults, they not only acquire impressive perceptual
/envelopebackYash Raj Jain
yasshjain@gmail.com
1Max Planck Institute for Intelligent Systems,
T¨ubingen, Germany
2Birla Institute of Technology and Science, Pilani,
Hyderabad, India
3Department of Psychology, Princeton University,
Princeton, NJ, USA
4Max Planck Institute for Biological Cybernetics,
T¨ubingen, Germany
5Department of Computer Science, Princeton University,
Princeton, NJ, USAand sensory-motor skills and knowledge about the world.
They also acquire cognitive skills such as the abilities to
perform mental arithmetic, plan, and problem-solve (van
Lehn, 1996 ; Shrager & Siegler, 1998 ; Lieder & Griffiths,
2017 ;H ee ta l . , 2021 ;J a i ne ta l . , 2019 ). These abilities
can be understood in terms of computational procedures
that people perform on their mental representations of the
external environment. Such computational procedures are
known as cognitive strategies . Here, we focus on cognitive
strategies for planning and refer to them as planning
strategies . There are many different types of planning
strategies that people can use. And as a person gains more
experience they might switch from a less effective strategy
to a more effective one. For instance, the first time a person
plans a road trip they might start by thinking about which
nearby location they might visit first, mentally simulating
how good it would be to visit that location, then think
about where they might go next, mentally simulating what
it would be like to be there, and so on. By the time that

BehaviorResearchMethods
this person plans their tenth road trip, she might start by
mentally simulating especially attractive distant locationsthat the road should be designed to lead to. These two
examples illustrate that people’s planning strategies draw on
a shared set of elementary planning operations that mentally
simulate states and actions but differ in what planning
operation they perform under which conditions.
Developmental and learning-induced changes in how
people think and decide are collectively known as cognitive
plasticity . Just like the acquisition of perceptual skills
(Hubel & Wiesel, 1970 ), the acquisition of cognitive skills
requires specific experiences and practice (van Lehn, 1996 ;
Ericsson et al., 1993 ). Despite initial research on how people
acquire cognitive skills such as the abilities to perform
mental arithmetic, plan, and problem-solve (van Lehn,1996 ; Shrager & Siegler, 1998 ; Lieder & Griffiths, 2017 ;
He et al., 2021 ;J a i ne ta l . , 2019 ), the underlying learning
mechanisms are still largely unknown. Reverse-engineeringhow people discover effective cognitive strategies is very
challenging. This is chiefly because it is impossible
to observe directly people’s cognitive strategies or howpeople’s strategies and strategy choices change withexperience – let alone the underlying learning mechanisms.
Instead, cognitive plasticity has to be inferred from
observable changes in behavior. This is difficult becauseany observed behavior could have been generated by many
different cognitive mechanisms. This problem is pertinent
to all areas of cognition.
We assume that each planning strategy performs a sequence
of internal information gathering operations (Callaway
et al., 2022b ). Concretely, we assume that each of these
planning operations mentally simulates what might happenif one took a particular action in a particular situation. We
assume that the outcome of each simulation is the reward
that the person expects the action to generate. Furthermore,we treat the mental simulation of each state-action pair as
a separate planning operation. These assumptions make it
possible to measure planning by externalizing the process ofinformation gathering that would otherwise occur through
memory recall and mental simulation (Callaway et al., 2017 ;
Callaway et al., 2018 ; Callaway et al., 2022b ). Building on
this theory and a previous method for studying how peoplechoose between alternatives with multiple attributes (Payne
et al., 1993 ), we introduce a process-tracing paradigm for
revealing the sequence of information gathering operationspeople perform during planning (see Fig. 1) and a com-
putational method for inferring the underlying planning
strategies (see Fig. 2). We will refer to these methods as
theMouselab MDP paradigm and our computational micro-
scope .
Our process-tracing method renders people’s behavior in
a route planning task highly diagnostic of their planningstrategies by
Fig.1 Illustration of the Mouselab-MDP paradigm. This figure shows
a three-step planning task that can be created within the Mouselab-
MDP paradigm. Here, the participant has to choose a series of threemoves. Starting from the central location, the first decision is whetherto move left, up, or right (Step 1); in each case there is only one optionfor the second move (Step 2), and then the spider can turn either leftor right in the third step. Rewards are revealed by clicking, prior toselecting a path with the arrow keys. At each node each of the fourpossible rewards is equally likely to occur
requiring them to click on locations they consider visiting
to find out how costly or rewarding it would be to do so
(see Figure 1). That is, when a person clicks on the state
that they would get to by taking a certain action in a certainstate, we treat it as an indication that they just performed
the corresponding planning operation. The Mouselab-MDP
paradigm poses people a series of planning problems (onein each trial). For each trial, it records the sequence of
clicks (planning operations) that the participant performed,
which information each click revealed, and the plan thatthe participant selected based on the resulting information(see Fig. 3). As Fig. 3illustrates, this makes it possible
to observe how the type of planning operations a person
performs and the order in which she performs them changefrom each trial to the next. Our computational microscope
uses the resulting process-tracing data to perform model-
based inference on the trial-by-trial sequence of planningstrategies the participant used to make his or her decisions.
Together, these two methods allow researchers to specify a
planning task and directly measure how people’s planningstrategies change from one trial to the next (see Fig. 2). To
facilitate adoption of the toolbox, we provide JavaScript and
Python libraries for both components and a tutorial on how
to use them. We hope that this toolbox will help researchersmeasure how people’s planning strategies change depending
on their experience.
People changing their planning strategies in response
to how well they worked is a prime example of what we
BehaviorResearchMethods
Fig. 2 Illustration of the basic idea of measuring people’s planning
strategies. The Mouselab MDP paradigm is a process-tracing method
that utilizes mouse tracking to measure which pieces of information
people inspect during planning and in which order they inspect them.The computational microscope is a model-based inference method that
determines which of 79 different planning strategies the participant is
m o s tl i k e l yt oh a v eu s e do nag i v e nt r i a l
call metacognitive reinforcement learning (Krueger et al.,
2017 ; Lieder & Griffiths, 2017 ; Lieder et al., 2018c ;J a i n
et al., 2019 ;H ee ta l . , 2021 ). Metacognitive reinforcement
learning is set of mechanisms through which people learn
when to perform which cognitive operations through trial
and error. These mechanisms might play an important rolein how people discover new cognitive strategies, adapt their
strategies to the structure of their environment, and acquire
cognitive skills (Lieder & Griffiths, 2017 ; Krueger et al.,
2017 ;J a i ne ta l . , 2019 ;H ee ta l . , 2021 ).
Metacognitive learning is difficult to study because its
effects and mechanisms cannot be observed directly. Thr ough-
out this article we will present a series of case studies toillustrate that our new computational method is useful for
characterizing how people learn how to plan and elucidating
metacognitive reinforcement learning more generally.
The plan for this paper is as follows: First, we summarize
and illustrate the functionality offered by our toolbox for
measuring how people learn how to plan and explain howit works. Next, we provide a practical step-by-step user’sguide on how to apply it. We then demonstrate the reliability
and validity of the inferences of our computational micro-
scope.
In closing, we discuss directions for future work enabled
by the methodology introduced in this article.
Newmethodsformeasuringhowpeople
learnhowtoplan
Planning, like all cognitive processes, cannot be observed
directly but has to be inferred from observable behavior.This is generally an ill-posed problem. In previous work,
researchers have inferred properties of human planning
from the decisions participants ultimately made or askedparticipants to verbalize their planning process. However,
many different planning strategies can lead to the same final
decision, and introspective reports can be incomplete or inac-curate. In the 1970s researchers studying how people choose
between multiple alternatives (e.g., apartments) based on
several attributes (e.g., rent, size, location, etc.) faced asimilar problem (Payne, 1976 ). To overcome this problem,
Johnson et al. ( 1989 ) developed a process-tracing paradigm
that elicits and records behavioral signatures of people’s
decision strategies. Concretely, in the Mouselab paradigm(Payne et al., 1993 ), the alternatives’ attribute values are ini-
tially concealed and the participant can make clicks with
their computer mouse to reveal one attribute value at a time.The Mouselab paradigm allows researchers to trace peo-
ple’s decision strategies by recording which attributes of
which alternatives people inspect in which order (Payneet al., 1993 ). While these behavioral signatures are still
indirect measures of cognitive processes, and the means of
observation might disturb the normal processes of decision-
making, they do at least provide additional informationabout potential underlying decision strategies.
The Mouselab paradigm has enabled an extremely pro-
ductive stream of research on the processes of multi-attributedecision-making (Payne et al., 1988 ;F o r de ta l . , 1989 ;
Payne et al., 1993 ; Schulte-Mecklenbeck et al., 2011 ;
Schulte-Mecklenbeck et al., 2019 ). Here, we introduce two
new methods that extend the process-tracing methodol-ogy from the domain of multi-attribute decision-making
to the domain of planning. We start by describing a new
BehaviorResearchMethods
Fig. 3 Illustration of the process-tracing data that can be collected
with the Mouselab-MDP paradigm. The recorded interactions (clicksand moves) the participant made and the information the participantobserved are enumerated in the order in which they occurred. In thisexample, the first participant started out with a short-sighted planningstrategy and gradually discovered a more far-sighted one. On the firsttrial she made two clicks on immediate outcomes on their first trial andthen selected a path. In the last trial the first participant inspected three
final outcomes. The process-tracing data from the intermediate trials
documents the participant’s transition between these two very differentways of planning
process-tracing paradigm for measuring individual plan-
ning operations (Section 5). Measuring planning operations
can yield valuable insights into how people plan (Callaway
et al., 2017 ; Callaway et al., 2022b ). But most research ques-
tions, such as how human planning compares to planning
algorithms used in artificial intelligence, are not formulatedat the level of individual planning operations but instead at
the level of planning strategies .
Analyzing the data collected with our process-tracing
paradigm suggested that people use a wide range of
different planning strategies. We found that which strategypeople use does not only depend on the structure of the
environment (Callaway et al., 2018 ; Callaway et al., 2022b )
but also on the participant’s learning history and individualdifferences. Concretely, we found that people may use as
many as 79 different planning strategies across different
environments and different points in time. These strategiesprioritize different types of information, such immediateoutcomes versus long-term consequences, highly uncertain
outcomes, or outcomes following gains rather than losses,
and they also differ in when they stop collecting moreinformation (e.g., upon uncovering a path yielding a reward
of at least $48). The resulting set of strategies includes
variants of classic planning algorithms, such as breadth-firstsearch, depth-first search, and best-first search, as well as
several novel strategies, such as first identifying the best
possible final outcome and then planning backward from it.The 79 planning strategies can be grouped into 13 differenttypes, including goal-setting strategies with exhaustive
backward planning, forward-planning strategies similar to
breadth-first search, and forward planning strategies similarto best-first search (see Section 5for a list of all strategies
grouped by strategy type).
To make it possible for researchers to measure which
strategies were used, we developed a computational method
that leverages each participant’s process-tracing data to
infer which strategy he or she used on the first trial, thesecond trial, the third trial, etc. We introduce this method inSection 5. The basic idea is to invert a probabilistic model of
how the participant’s process-tracing data was generated by
a series of planning strategies through Bayesian inference.This is a challenging methodological problem because
people rarely execute any given strategy perfectly. We solve
this problem by explicitly modeling the variability in thestrategy that people use, in their execution of the strategy,
and in the way the execution of the strategy manifests
in their process-tracing data. In addition, we also modelthat there might be trials on which people don’t use anyparticular strategy or a strategy that is still unknown.
Our computational microscope can be applied to reveal
people’s planning strategies in a wide range of differenttask environments. Used in combination, our two methods
can be used to characterize the cognitive mechanisms
of human planning, investigate how a person’s planningstrategies evolve across trials, and uncover how planning
strategies are affected by contextual factors and differ
between individuals. Our methods support this researchby providing trial-by-trial measurements of four aspectsof human planning: the series of planning operations they
BehaviorResearchMethods
performed, which of the 79 different planning strategies
was the most likely source of those planning operations,which type of strategy it was, and how different types
of previously postulated mechanisms (e.g., habits vs.
Pavlovian mechanisms vs. reasoning) might have shaped aperson’s planning on a given trial.
Figure 4summarizes the information that our computa-
tional microscope provides the user about how a given par-ticipant planned in a given Mouselab-MDP experiment. The
following sections illustrate each of these functionalities in
turn.
In this section we give a brief high-level overview of the
functionality offered by our methods. The technical details
are presented in the following section.
Measuringindividualplanningoperations
withtheMouselab-MDPparadigm
To make individual planning operations measurable, we devel-
oped a process-tracing paradigm that externalizes people’sbeliefs and planning operations as observable states and
actions (Callaway et al., 2017 ). We refer to this paradigm
as the Mouselab-MDP paradigm because it extends theapproach of the Mouselab paradigm (Payne et al., 1993 )t o
a general class of planning tasks known as Markov Deci-
sion Processes (MDPs) (Sutton & Barto, 2018 ). A Markov
Decision Process comprises a series of decisions. Given
the current state (e.g., location) the agent has to choose an
action that, together with the current state, determines bothan immediate reward and the next state. The task is to max-imize the sum of all rewards over time. Inspired by the
Mouselab paradigm (Payne et al., 1993 ), the Mouselab-MDP
paradigm uses people’s mouse-clicking as a window intotheir planning. As illustrated in Fig. 1, this paradigm
presents participants with a series of route planning prob-lems. Each route planning problem is presented as a map
where each location (the gray circles), harbors a gain or
loss. These potential gains and losses are initially occluded,corresponding to a highly uncertain belief state. The par-
ticipant can (expensively) reveal each location’s reward by
clicking on it and paying a fee. This is similar to looking ata map to plan a road trip. Clicking on a circle corresponds
to thinking about a potential destination, evaluating how
enjoyable it would be to go there, or perhaps how costly itwould be to go through there on the way to somewhere else,and then adjusting one’s assessment of candidate routes
accordingly. The set of revealed rewards constitutes the state
of the participant’s knowledge which we will refer to asthebelief state . The tasks in this paradigm are designed
such that each planning operation requires the participant to
make a specific click and each click is the output of a spe-cific planning operation. Participants can make as few or
as many clicks as they like. After that the participant has
to select a route through the environment using the arrowkeys. For each location they visit, the corresponding rewardis added to their score. The task is to maximize the money
earned by traversing the environment minus the fees paid
for collecting information.
The Mouselab-MDP paradigm can be used to create
a wide range of environments that vary in size, layout
(structure), and reward distribution. Figures 1,7a-c, and 9
illustrate the variety of task environments that can be created
with this paradigm. Several of the illustrative examples
below and the experiments used to validate our methodsare based on the simple three-step planning task shown inFig. 1. Here, the participant can earn money by navigating
Fig. 4 Illustration of the hierarchically nested information that our
method provides about a participants planning throughout the ntrials
of a Mouselab-MDP experiment. The participant’s learning trajec-tory is characterized by the sequence of planning strategies that
the participant used on trial 1, trial 2, ···,t r i a l n, respectively.
The strategy the participant used on a given trial is characterizedby a procedural description, the general type of planning strategyit instantiates, the sequences of clicks it performed on that trial,
the plan that they selected on that trial, and how the influencesof different decision systems and other factors combine to gener-ate that strategy. Each click sequence comprises a series of clicks.
Each click is characterized by where the participant clicked and
which information (reward) their click unveiled. Timing data is alsoavailable
BehaviorResearchMethods
a money-loving spider through a “web of cash”. There are
six possible paths the participant can choose between. Eachpath comprises three steps, starts from the gray node in
the center of the web, and proceeds along the arrows. In
the first step, the spider can go left, up, or right. In thesecond step, it has to continue in that direction. In the third
step, it can choose to either turn left or right. Each node
that the spider might visit along the chosen path harbors again of up to $48 or loss of up to $-48. The player earns a
monetary bonus proportional to the sum of the three rewards
along the chosen path minus the fees they paid for clicking.In the beginning all gains and losses are concealed. Theparticipant can uncover them for a fee of $1 per click.
The participant can make as many or as few clicks as they
like. Once they are done collecting information (planning),they start acting by moving the spider with the arrow keys.
The participant receives the gain or loss at a given location
if and only if they move the spider there. Clicking on anode only reveals the information which gain or loss they
would receive if they moved to the inspected location but
does not collect that reward. Furthermore, whether or nota node has been inspected has no effect on the reward theparticipant receives when the spider enters that location.
Critically, in this particular three-step planning task, the
variance of the potential rewards is smallest for the nodesthat can be reached within one step, larger for the nodes
that can be reached within two steps, and largest for the
potential final destinations that are three steps away fromthe spider’s starting position at the center of the web (see
Figure 1). This captures a common feature of real-world
planning problems, namely that long-term outcomes aremore important than short-term rewards.
The Mouselab-MDP paradigm can be used to elicit infor-
mation about people’s planning operations at a level of
detail which was inaccessible with previous behavioralparadigms. It makes it possible to measure which informa-
tion people’s planning strategies consider in which order
and how this depends on the information revealed by pre-vious planning operations. Figure 3illustrates the kind
of process-tracing data that can be obtained with the
Mouselab-MDP paradigm. The data from any given trialtraces the strategy that an individual participant used toreach their decision on that trial. Taken together, the data
from a series of trials traces how the participant’s decision
strategy changed along with the observations and experi-enced rewards that preceded each change. Concretely, the
example illustrated in Fig. 3what the data might look
for a participant who starts out with a myopic planningstrategy and gradually discovers the optimal far-sighted
goal-setting strategy.Acomputationalmicroscopeforinferringpeople’s
planningstrategies
The fine-grained information about the planning operations
obtained from the Mouselab-MDP paradigm can be usedto draw much richer inferences about how people plan andhow the way they plan changes over time. However, the
raw click sequences are difficult to analyze directly without
sophisticated and typically theory-laden modeling tools.The computational microscope is a computational method
that makes it possible to characterize how the participants of
your experiment planned at the level of planning strategies,strategy types, and the contributions of different decision
systems and other factors. In this section, we first give an
overview of the computational microscope’s functionality.We then give a detailed account of how this functionality isimplemented and close with an illustrative example of how
the computational microscope can be used.
Overviewofthecomputationalmicroscope’sfunctionality
The computational microscope makes use of the informa-
tion about people’s planning operations collected with theMouselab-MDP process-tracing paradigm to help us better
understand how people plan and how their planning changes
over time. It makes it possible to infer which of 79 knownplanning strategies a participant used on a given trial from
their clicks in the Mouselab-MDP paradigm. The set of 79
planning strategies includes the strategy that does not planat all, a strategy that only inspects the immediate rewards,a strategy that inspects only the potential final outcomes
and terminates planning once it discovers a large positive
value, a variant of this strategy that plans backward fromthe preferred final outcome, search-based planning strate-
gies (Russell & Norvig, 2016 ), such as breadth-first search
(i.e. first explore nodes that are one step away, then explorenodes at are two steps away, and so on) and best-first search
(i.e., explore nodes in decreasing order of the values of the
paths they lie on), a strategy that explores all final nodesthat are farthest away from the start node, and many others.For the hypothetical data set illustrated in Fig. 3, our com-
putational microscope would likely infer that the participant
started with the myopic planning strategy that terminatesupon uncovering a positive value (Strategy 53 described in
Section 5) and eventually discover the optimal goal-setting
strategy (Strategy 6 described in Section 5).
In addition to fine-grained information about concrete
planning strategies, the computational microscope also pro-
vides high-level information about which kind of planningstrategy the person is using. Concretely, the microscope
BehaviorResearchMethods
distinguishes between 13 types of planning strategies: four
types of goal-setting strategies that explore potential finaloutcomes first, a strategy that explores immediate outcomes
on the paths to the best final outcomes, a satisficing version
of that strategy, forward-planning strategies (i.e strategiesthat start planning from nodes that are one step away from
the start node) similar to Breadth First Search, middle-
out planning (i.e the strategies that click the nodes in themiddle of a path, then click the nodes that are nearest
to the start node and then click nodes that are the far-
thest away), forward-planning strategies similar to BestFirst Search, local search strategies that focus on informa-tion about subtrees and next or previous steps along the
paths that have received the most consideration so far, fru-
gal planning strategies (i.e strategies that explore very littleor not at all), myopic planning strategies (i.e. strategies
that only explore nodes that are one step away from the
start node) and a few other strategies that do not fit anyof these categories. The four types of goal-setting strate-
gies differ in how many potential goals they inspect (all
vs. some), in how many and which earlier outcomes theyinspect (all vs. some), and in when and how often theytransition between inspecting goals versus earlier outcomes.
For instance, goal-setting with exhaustive backward plan-
ning inspects all potential goals and all earlier outcomes.By contrast, frugal goal-setting strategies only explore some
of the potential goals and none or only a small number of
the earlier outcomes. Maximizing goal-setting with limitedbackward planning first identifies an optimal final outcome
and then either terminates planning or inspects only the
nodes on the path leading to the best final outcome. Bycontrast, maximizing goal-setting with exhaustive backwardplanning inspects the paths to all potential goals in the order
of the goals’ rewards after having inspected all potential
goals.
For the hypothetical data set illustrated in Fig. 3our
computational microscope would likely infer that the partic-
ipant started with a frugal planning strategy and eventuallydiscovered a maximizing goal-setting strategy with limited
backward planning. The definitions of these strategy types
are presented in Section 5.
The computational microscope’s functionality is realized
through model-based probabilistic inference. The model
comprises three components: probabilistic models of 79
planning strategies, a probabilistic model of how planningstrategies generate click sequences ( observation model )
and a probabilistic model of the sequence of planning
strategies ( prior on strategy sequences ). As shown in Fig. 5,
our method assumes that which planning strategy ( S
t)a
participant uses can change from each trial ( t) to the next
but remains constant within each individual trial. In otherwords, we assume that exactly one planning strategy isused in each trial and that this strategy may be differentfrom the one that was used in the previous trial and the
one that will be used in the following trial. Furthermore,our method assumes that the strategies themselves do not
change. Therefore, the computational microscope infers
the trial-by-trial sequence of planning strategies that theparticipant used in the experiment (i.e., which strategy her
or she used in the first trial of the experiment, which
potentially different strategy he or she used in the secondtrial of the experiment, etc.). This sequence of planning
strategies is inferred from the corresponding sequence of
trial-by-trial click sequences (i.e., one click sequence foreach trial). The basic idea is to find the sequence of planningstrategies that is most likely to have generated the observed
sequence of click sequences. The trial-by-trial changes in
the relative influences of different decision systems andother factors can then be read off from the inferred strategy
sequence because we make the simplifying assumption
that way in which those factors interact to generate thebehavior of a given strategy does not change over time.
The computational microscope requires access to a set of
planning strategies which generate the planning operationsin a trial and models transitions among these strategiesusing a prior. We first describe how we formally model the
planning strategies. We then describe the generative model
of clicks (planning operations) given a strategy and thendiscuss how the computational microscope performs model
inversion by taking into consideration information about
participants’ clicks obtained from the Mouselab-MDP andthe prior on strategy sequences to make inferences about the
most likely sequence of strategies that might have generated
the data. Obtaining the most likely sequence of strategiesalso gives us information about the strategy types and thetemporal evolution of relative influence of decision systems
(see Section 5).
Modelingplanningstrategies
To make it possible to extract interpretable strategies from
the raw click sequences, we formulated a set of 79 planningstrategies (
S) through a data driven methodology. Con-
cretely, we manually inspecting the process-tracing data
from an experiment in which participants completed 31 tri-als of the 3-step planning task illustrated in Fig. 1(for
description, see Appendix A.1). We visually inspected this
data one click sequence at a time. Each time, we checkedwhether the current click sequence could be an instance ofan already identified strategy. When this was not the case,
we manually added an additional strategy to account for this
new pattern. We then proceeded to the next click sequenceand repeated the same procedure. If there was no appar-
ent pattern, we identified it as an instance of a strategy that
clicks randomly. We continued this process until our strate-gies were able to account for all click sequences of every
BehaviorResearchMethods
Fig. 5 Overview of the computational microscope describing the Hidden Markov model that generates the observed process-tracing data as a
graphical model
participant who participated in the experiment described in
Appendix A.1.
We modelled each of these planning strategies as a
stochastic procedure that generates a sequence of planning
operations (clicks). That is, a planning strategy specifiesa probability distribution over what the first click might
be and conditional probability distributions over what each
subsequent click might be depending on which clickswere made previously and which rewards they revealed.For instance, the best-first search strategy distributes the
probability of the first click evenly among the immediate
outcomes and concentrates the probability of subsequentclicks on proximal outcomes that follow the best immediate
reward(s). Furthermore, the planning strategy also specifies
the conditional probability to terminate planning and selectan action based on the information that has been revealed
so far. For instance, for many of our planning strategies, the
probability of terminating planning increases with the sumof the rewards of the best path that has been identified sofar. As detailed in the next section, each planning strategy
(s) entails a probability distribution ( P) over which process
tracing data ( d) might be observed if a participant used
that strategy ( P(d|s)). Different strategies differ in which
planning operations they perform first, in how they use
the revealed information to select the subsequent planningoperations, and in when they terminate planning. We
model each sequence of planning operations a participant
performed from the beginning of a trial to the end of thattrial as the manifestation of a single strategy.
1
According to our model, all strategies are probabilistic in
the sense that they randomly select between all functionally
equivalent planning operations that are consistent with whatthe strategy does in the current step. For instance, when the
1Those strategies may be composed of building blocks that are reused
in multiple strategies but we do not explicitly model this hierarchicalstructure.first step of a strategy is to inspect immediate outcomes
until it uncovers a positive value, then our model assumesthat the strategy chooses uniformly at random between all
planning operations that inspect an uninspected immediate
outcome. For more details about the strategies, please seeAppendix A.4.
We found that, collectively, the 79 planning strategies can
capture people’s click sequences much better than the ran-dom strategy. Concretely, we found that, on average, eachclick made by a participant is 3 to 6 times as likely under the
best fitting strategy than under the random strategy. That is,
for the environment with increasing variance, the maximumlikelihood estimate of people’s strategies achieve an aver-
age click likelihood of 0.38 whereas the random strategy
achieves an average click likelihood of only 0.10. For theenvironment with constant variance (Fig. 7b), the average
per click likelihood is 0.50 whereas it is 0.09 for the ran-
dom strategy. For the environment with decreasing variance(Fig. 7a), the average per click likelihood is 0.37 whereas it
is 0.08 for the random strategy. And finally, for the environ-
ment used in the transfer task (Fig. 7c), the average per click
likelihood is 0.19 whereas it is 0.03 for the random strategy.
Modelinghowstrategysequencesgenerate
process-tracingdata
To develop an efficient computational method for infer-
ring the temporal evolution of people’s planning strategies,
we make the simplifying assumption that the trial-by-trial
sequence of peoples’ cognitive strategies ( S
1,S2,···,ST)
forms a Markov chain whose hidden states emit the observed
process tracing data collected on each trial ( d1,···,dT).
This hidden Markov model requires additional methodolog-ical assumptions about i) how cognitive strategies manifest
in process-tracing data, ii) the set of cognitive mechanisms
that can be learned (defined in Section 5), and iii) the
nature and amount of cognitive plasticity that might occur.
BehaviorResearchMethods
The following paragraphs detail our assumptions about the
components i) and iii) in turn.
Observationmodel To plan in the Mouselab-MDP paradigm
participants have to gather information by making a sequence
of clicks. Our observation model thus specifies the proba-
bility of observing a sequence of clicks dton trial tif the
strategy was St(i.e.,P(dt|St)).
To achieve this, we quantify each planning strategy’s
propensity to generate a click c(or stop collecting informa-
tion) given the already observed rewards encoded in beliefstatebby a weighted sum of 51 features ( f
1(b, c), ···,
f51(b, c) ). The features describe the click crelative to this
information (e.g., by the value of the largest reward that canbe collected from the inspected location) and in terms of the
action it gathers information about (e.g., whether it pertains
to the first, second, or third step). A detailed description ofthe features and strategies is available in Appendix A.6.
The depth feature, for instance, describes each click by
how many steps into the future it looks. The features andweights jointly determine the strategy’s propensity to makeclickcin belief state baccording to
P(d
t|St)=|dt|/productdisplay
i=1exp/parenleftBig
1
τ/summationtext|w(S)|
k=1w(S)
kf(S)
k(ct,i,bt,i)/parenrightBig
/summationtext
c∈Cbtexp/parenleftBig
1
τ/summationtext|w(S)|
k=1w(S)
kf(S)
k(c, bt,i)/parenrightBig,
(1)
where dt,iis the ithclick the participant made on trial t(or
the decision to stop clicking and take action), the decisiontemperature τwas considered as a hyperparameter which
was set by the inference procedure, and w
(S)is the weight
vector of strategy S. According to this probabilistic soft-
max model, all clicks are possible under each strategy in
each situation but their probability is higher the better they
are aligned with the strategy.
The strategies differ in how much information they con-
sider (ranging from none to all to exploring all the nodes),
which information they focus on, and in the order in
which they collect it. Building on the observation modelin Eq. 1, we represent each strategy by a weight vector
w=(w
1,···,w51)that specifies the strategy’s preference
for features such as more vs. less planning, exploring nodeswith more uncertainty vs. less, considering immediate vs.
long-term consequences, satisficing vs. maximizing, avoid-
ing losses (cf. Huys et al., 2012 ), exploring paths that have
a larger number of explored nodes, exploring nodes thatare related to already observed nodes such as the ancestor
nodes, successor nodes and siblings, and other desiderata.
These weights are computed by generating data by simulat-ing which clicks each strategy would make and then fitting
the weights in Eq. 1using Maximum Likelihood Estimation
(MLE). These weights span a high-dimensional continuousspace with many intermediate strategies and mixtures of
strategies. Cognitive plasticity could be measured by track-ing how those weights change over time. But this would be a
very difficult ill-defined inference problem whose solution
would depend on our somewhat arbitrary choice of features.As a first approximation, our method therefore simplifies
the problem of measuring cognitive plasticity to inferring a
time-series of discrete strategies. A detailed description ofthe features used in the observation model can be found in
Appendix 5
Prioronstrategysequences Inferring a strategy from a sin-
gle click sequence could be unreliable. To smooth out its
inferences, our method therefore exploits temporal depen-dencies between subsequent strategies by using a probabilis-
tic model of strategy sequences.
Transitions from one strategy to the next can be grouped
into three types: repetitions, gradual changes, and abruptchanges. While most neuroscientific and reinforcement-
learning perspectives emphasize gradual learning (e.g.,
Hebb, 1949 ; Mercado, 2008 ; Lieder et al., 2018c ), others
suggest that animals change their strategy abruptly when
they detect a change in the environment (Gershman et al.,
2010 ). Symbolic models and stage theories of cognitive
development also assume abrupt changes (e.g., Piaget,
1971 ; Shrager & Siegler, 1998 ), and it seems plausible that
both types of mechanisms might coexist.
We considered three kinds of priors on the strategy tran-
sitions: gradual, abrupt and a combination of gradual and
abrupt transitions. We did not find any significant relation-
ship between the probability of transition from one strategyto the next and the distance between the strategies (see
Appendix A.2.1 ). We found that the frequency of a transi-
tion from a strategy to itself was more likely than a transitionfrom a strategy to some other strategy ( t(975)=7.55,
p< 0.0001 ,BF>1000). Model selection using either
AIC (Akaike, 1974 ) or BIC (Schwarz et al. 1978 )v a l u e s
computed using the likelihood values of the maximum like-lihood estimate of the strategy sequence also revealed the
abrupt prior to be the best performing. Therefore, we use
the abrupt prior for all our inferences. The gradual and themixed priors are described in Section 5.
The abrupt changes prior assumes that transitions are
either repetitions or jumps.
P(S
t+1=s|St,mabrupt)=
pstayI(St+1=St)+(1−pstay)I(s/negationslash=St)
|S|−1, (2)
whereSis the set of strategies, |S|is the number of
strategies and pstayis the probability of strategy repetitions.
We model the probability of the first strategy as a
uniform distribution over the space of decision strategies
(i.e.,P(S 1)=1
|S|).
BehaviorResearchMethods
Together with the observation model and the strategy
space described above, the prior defines a generative modelof a participant’s process tracing data d; this model has the
following form:
P(d,S
1,···,ST)=1
|S|T/productdisplay
t=2P(St|St−1|mabrupt)P(dt|St).( 3 )
Inverting this model gives rise to a computational method
for measuring an important aspect of cognitive plasticity.
Inferringstrategysequencebymodelinversion
Our model describes how the sequences of planning
strategies a participant uses across the different trials ofthe experiment manifests in their process-tracing data. To
measure this sequence of planning strategies, we have to
reason backwards from the process tracing data dto the
unobservable cognitive strategies S
1,···,STthat generated
it. To achieve this, we first model the generation of process-
tracing data using a Hidden Markov Model with the 79planning strategies as the possible values of its latent
states and the prior m+
abrupt as its transition prior. Having
modelled how likely alternative strategies are to generatea given sequence of clicks, we can apply Bayes theoremto compute how likely a person is to have used different
planning strategies given the clicks that they have made.
More concretely, the computational microscope computesthe sequence of strategies s
1,s2,···,sTthat is most likely
to have given rise to the process-tracing data observed
on the corresponding Ttrials ( d1,d2,···,dT). This is
achieved by applying the Viterbi algorithm (Forney, 1973 )
to compute the maximum a posteriori (MAP) estimate
arg max s1,s2,···,sTP(s 1,s2,···,sT|d1,d2,···,dT)of the
hidden sequence of planning strategies S1,···,STgiven
the observed process tracing data d, the measurement
model mabrupt , and the parameter ( pstayof Eq. 2and the
strategy temperature parameter τof the observation model.
This inference combines the likelihood that a possible
strategy would generate an observed click sequence with
how probable potential sequences of planning strategiesare a priori. The prior probability of strategy sequences
is assigned based on the knowledge that people are often
somewhat more likely to repeat the strategy they used on theprevious trial than to switch an arbitrary other strategy.
To estimate the model parameter p
staywe perform grid
search with a resolution of 0.02 over pstay∈[0,1].T h e
value of τis set using 50 iterations of Bayesian Opti-
mization, with the likelihood of MAP estimate of the click
sequence as the objective it maximizes. We use the Tree-
structured Parzen estimator approach to Bayesian Optimiza-tion implemented in the hyperopt Python package (Bergstra
et al., 2013 ) for optimizing the parameter τ.Inferring the hidden sequence of cognitive strategies in
this way lets us see otherwise unobservable aspects ofcognitive plasticity through the lens of a computational
microscope.
Inferenceonstrategytypesandmeta-control
To understand what types of strategies people use, we
grouped our 79 strategies using hierarchical clustering onthe distances between the strategies. Since the strategies
are probabilistic, we defined the distance metric /Delta1(s
1,s2)
between strategy s1ands2as the Symmetrised Kullback-
Leibler divergence
between the distributions of click sequences and belief
states induced by strategies s1ands2respectively, that is
/Delta1(s 1,s2)=JD[p(d|s1), p( d|s2)]
=KL[p(d|s1), p( d|s2)]
+KL[p(d|s2), p( d|s1)], (4)
and approximated it using Monte-Carlo integration.
Applying Ward’s hierarchical clustering method (Ward,
1963 ) to the resulting distances suggested 13 types of
planning strategies described in Section 5.
As discussed in Section 5, we assume that people’s
choice of planning operations is shaped by the interactions
of multiple decision systems and other factors. To measure
the contribution of each factor in a strategy, we first assignedeach feature to one of the decision systems. Then, for each
decision system, we added the weights of the features which
belonged to that decision system if the feature representedan increase in that decision system and subtracted it if it
represented a decrease in that decision system to give us a
weight w
dsfor a decision system. The relative influence of
the decision system on a strategy is measured by:
RIds=|wds|/summationtext
ds∈D|wds|, (5)
where Dis the set of all decision systems.
Anexampleofapplyingthecomputationalmicroscope
To illustrate the functionality of our computational micro-
scope, we applied it to data from an experiment evaluatingintelligent tutors that teach people effective planning strate-
gies (i.e., the experiment described in Appendix A.1). In this
experiment participants practiced planning in the three-stepdecision task illustrated in Fig. 1(see Section 5) for 10 trials
(training block) and were then tested on 20 more trials of the
same task (test block). Participants in the experimental con-ditions received two different types of feedback during thetraining block. Participants in the control condition received
no feedback.
BehaviorResearchMethods
Table 1 Summary of the planning strategies that people used most frequently in the environment illustrated in Fig. 1
Strategy type Strategy Used on %
of trialsUsed by %
of peoplePeople who used this
strategy (type) used it on
%o ft r i a l s
Maximizing goal-setting withlimited backward planning50.4% 68.8% 69.7%
Random search for best possible
final outcome36.6% 55.1% 63.1%
Consecutive second maximum 6.3% 17.6% 33.8%
Extra planning after observing
second best outcome2.2% 10.2% 20.6%
Frugal planning 14.8% 34.1% 41.2%
No planning 13.2% 26.1% 47.8%
Miscellaneous strategies 11.3% 43.8% 24.6%
Explore immediate outcomes of
second best outcomes2.2% 11.4% 18.0%
Local Search 7.4% 27.3% 25.8%
Satisficing Depth First Search 3.6% 14.2% 24.3%
Priority to explored immediate
ancestors2.1% 8.0% 25.0%
Myopic planning 6.5% 29.0% 21.2%
Explore all immediate outcomes
with satisficing2.1% 9.7% 20.4%
Explore all immediate outcomes 2.1% 9.1% 22.3%
Frugal goal-setting 6.1% 29.0% 20.0%
Goal-setting with positive satisficing 2.1% 8.0% 24.5%
Table 1lists all strategies that people used on at least
2% of the trials ordered by strategy type and frequency.
As can be seen, the most common strategy types weremaximizing goal-setting with limited backward planning,
frugal planning, local search, myopic planning, frugal goal-
setting, and other miscellaneous strategies that don’t belongto any other strategy type. These 6 types of strategies jointly
accounted for 96.5% of all strategies that people used in
this environment. For more information about these strategytypes and the corresponding planning strategies, please seeAppendix A.4.
Measuringtherelativecontributionsofdifferent
decisionsystemsandotherfactors
How people plan is shaped by the interaction of multiple
different types of mechanisms throughout the decision-making process (van der Meer et al., 2012 ; Huys et al.,
2012 ,2015 ; Dolan & Dayan, 2013 ; Cushman & Morris,
2015 ; Keramati et al., 2016 ;D a w , 2018 ). In most real-life
decisions it is infeasible or unwise to consider all possiblesequences of actions, states, and outcomes. To decide
which alternatives to consider and which ones to ignore,
the model-based system relies on the recommendationsof simpler mechanisms such as Pavlovian impulses (Huyset al., 2012 ), value estimates learned through model-free
reinforcement learning (Cushman & Morris, 2015 ), and
simple heuristics (Huys et al., 2015 ). Furthermore, previous
findings indicate the existence of an additional decision
system that is specialized for deciding between continuing
to gather information (e.g., by foraging) versus acting onthe information that is already available (Rushworth et al.,
2012 ). Since deciding how to plan is like foraging for
information, the decision when to stop planning might alsobe made separately from the decision how to plan. Thisdecision can be made by determining whether the best plan
identified so far is already good enough (satisficing) or
other stopping criteria. In addition, people are also knownto engage in metareasoning (Ackerman & Thompson, 2017 ;
Griffiths et al., 2019 ) – that is reasoning about reasoning –
to figure out what is the best way to figure out what to do.Furthermore, all else being equal, the way in which people
decide seems to follow the law of least mental effort (Patzelt
et al., 2019 ; Balle, 2002 ; Kool et al., 2010 ), that is people
seek to avoid mental effort.
We assume that all of these factors simultaneously
influence how a person selects his or her individual planning
operations while making a single decision (Keramatiet al., 2016 ; Huys et al., 2012 ,2015 ;D a w , 2018 ). To
measure the relative contributions of these different types
BehaviorResearchMethods
of factors to each of the 79 planning strategies, we
divided the features whose weights determine the strategies’preferences for alternative planning operations into five
categories: Pavlovian ,model-free values and heuristics ,
model-based metareasoning ,mental effort avoidance ,a n d
satisficing and stopping criteria .
The Pavlovian features report how attractive or repelling
it is to think about a state based on the rewards and lossesthat precede or follow it. The category model-free values
and heuristics includes structural and relational features
of state-action pairs that people might come to associatewith rewarded versus unrewarded planning operations. Thefeatures in the category model-based metareasoning are
derived from a model of how alternative planning operations
reduce the decision maker’s uncertainty about which plan isbest. The category mental-effort avoidance includes a single
feature that distinguishes between performing a planning
operation (more mental effort) versus acting without furtherplanning (less mental effort). The features in the category
satisficing and stopping criteria describe conditions under
which specific stopping rules would terminate planning,such as whether there is a path whose expected returnexceeds $48 which is an instance of satisficing (Simon,
1955 ). For a detailed definition of these categories in terms
of the constituent features see Appendix A.6. To measure
the relative influence of these five types of factors on how
ap e r s o np l a n n e do nag i v e nt r i a l ,w ef i r s ts u mu pt h e
weights that the inferred strategy assigns to features of thistype to get a total weight for the type and then normalize
its absolute value by the sum of absolute values of total
weights of all types. Performing this calculation separatelyfor first, second, third, ···, last trial allows us to track how
the relative influence of different decision systems (i.e., the
model-based system, the Pavlovian system, and model-free
systems) and other factors (i.e., mental effort avoidance andstopping criteria) changes as people learn how to plan.
For the hypothetical data set illustrated in Fig. 3our
computational microscope would likely infer that the par-ticipant started out relying primarily on structural features
(a sub-category of model-free values and heuristics), satis-
ficing features, and mental effort avoidance. Furthermore, itwould most likely infer that the participant then transitionedto relying increasingly more on model-based metareasoning
features.
Measuringcognitiveplasticity
Our method makes it possible to measure how people’s
approach to planning changes at multiple levels of resolu-tion across time scales ranging from seconds to decades.
It can resolve changes in people’s planning at the level of
individual planning operations, planning strategies, strategytypes, and the contributions of different decision systemsand other factors. By default, our method’s temporal res-
olution is the amount of time that passes from one trialto the next. This makes it suitable for reverse-engineering
the learning mechanisms through which people discover
and continuously refine their planning strategies (Jain et al.,2019 ). It can also measure how people’s approach to plan-
ning evolves over longer time scales, such as blocks, ses-
sions, years, and decades. This makes the computationalmicroscope suitable for investigating how people learn how
to plan and how they adapt their planning strategies to new
environments. Figure 6illustrates the computational micro-
scope’s ability to reveal how people’s propensities towardsdifferent types of planning strategies evolve as they learn
how to plan in the task illustrated in Fig. 1; to obtain these
results we applied the computational microscope to the datafrom the control condition of the experiment described in
Appendix A.1. The output of the computational microscope
revealed that the strategies which explore the final outcomesfirst and terminate upon finding a high value became the
most frequent strategy type. During this transition people
shifted away from frugal planning strategies (i.e., strate-gies that explore only a few outcomes) which were themost common strategies at the start of the experiment along
with the myopic planning strategies (strategies that explore
immediate outcomes first). The miscellaneous strategiesalso decreased in frequency. The frequency of local search
(i.e., the strategies that focus on information about subtrees
or paths that have been explored the most so far) and fru-gal goal-setting strategies (i.e., strategies that start exploring
from the final outcomes and only explore a few out-
comes) initially became more frequent and then decreasedagain.
In addition, the computational microscope can also be
used to measure the transfer of learning from one task to
another. Traditionally, transfer effects are established bydemonstrating the training’s effect on people’s average per-
formance in an untrained task. The computational micro-
scope makes it possible to determine whether people trans-fer the specific strategies they learned in the training task
to untrained tasks. To illustrate this, we applied the com-
putational microscope to data from a transfer experimentin which participants practiced planning in a simple, smallenvironment and were then tested on a larger and more com-
plex environment. Concretely, the participants in the second
experiment from Lieder ( 2018b ) performed the five-step
planning task illustrated in Fig. 7c after having practiced
planning in the three-step planning task illustrated in Fig. 1
with optimal feedback (experimental condition) or withoutfeedback (control condition). As shown in Fig. 8,t h ec o m -
putational microscope revealed that participants from both
conditions transferred the near-optimal goal-setting strat-egy they had learned in the three-step planning task to thefive-step planning task.
BehaviorResearchMethods
Fig.6 Measured time course of frequencies of strategy types in the experiment described in Appendix A.1.
Furthermore, our approach can also be used to charac-
terize how people’s approach to planning changes acrossthe lifespan (Das et al., 2019 ). Finally, our method can also
be used to detect and compare the effects of (pedagogical)
interventions on how people learn how to plan and to elu-cidate inter-individual differences in metacognitive learning(e.g., in psychiatric disorders).
Astep-by-stepguidetomeasuringhow
peoplelearnhowtoplan
Experimenters can make use of our paradigm and our com-
putational microscope very easily. In this section, we pro-vide a tutorial like introduction for running experiments
with the Mouselab-MDP paradigm and applying the compu-
tational microscope on data generated using the Mouselab-MDP paradigm.
Astep-by-stepguidetocreatingandrunning
process-tracingexperimentswiththeMouselab-MDPparadigm
Having motivated the paradigm, we briefly describe both the
interface through which experimenters specify experiments,and the interface through which participants engage in the
task. Two screenshots of the paradigm are shown in Fig. 9,
and a live demo can be viewed at http://cocosci.princeton.
edu/webexpt/mouselab-demo/ . The code for Mouselab-
MDP and an example of how to use it are available at https://
github.com/RationalityEnhancement/Mouselab-MDP .On each trial, an environment is conveyed by an intuitive
visualization (see Fig. 9). Formally, each environment
corresponds to a directed graph with states as nodes and
actions as edges. The participant navigates through the
graph using the keyboard, attempting to collect the maximaltotal reward. States or edges are annotated with the rewardfor reaching the state or taking the action. Crucially, these
labels may not be visible when the trial begins. Rather, the
participant may need to click or hover their mouse over astate or edge to see the associated reward. The timecourse
of these information-gathering operations provides fine-
grained information about the person’s planning strategy.Furthermore, our paradigm allows researchers to investigate
how people negotiate the tradeoff between the cost
of thinking and its benefits. This can be done bymanipulating the cost of information gathering; for instanceby charging participants a certain number of points per
click.
With the Mouselab-MDP jsPsych plugin, experimenters
can create a planning experiment by specifying the follow-
ing critical components:
1.graph is a mapping s/mapsto→Afrom a state sto action
contingencies A. Each action contingency is a mapping
a/mapsto→(r, s
/prime)from an action to a reward rand the
next state s/prime. The graph structure thereby specifies the
actions aavailable in each state, as well as the reward r
and resultant state s/primeassociated with each action.
2.initial is the state in which the participant begins
the trial.
3.layout is a mapping s/mapsto→(x, y ) that specifies the
location of each node on the screen.
BehaviorResearchMethods
Fig.7 Illustration of the environment with decreasing variance (a), the environment with constant variance (b), and the five-step version of the
environment with increasing variance (c). In the environment with decreasing variance, the rewards at the first, second, and third step are sampled
uniformly at random from the sets {−48,−24,+24,+48},{−8,−4,+4,+8},a n d{−4,−2,+2,+4}, respectively. In the environment with
constant variance, the rewards at all locations are independently sampled from the same uniform distribution over the set {−10,−5,+5,+10}.
In the five-step planning task with increasing variance the rewards at steps 1 to 4 are drawn from normal distributions with mean 0 and standarddeviation σ
1=20,σ1=21,σ1=22,a n dσ1=23, respectively, and the reward at step 5 is drawn from a normal distribution with mean 0 and
standard deviation σ5=25
Specifying only these settings will result in a graph with
rewards shown on the edges between nodes and no labels on
the states.
To take advantage of additional Mouselab features, the user
must specify at least one of the following optional properties:
1.stateLabels is a mapping s/mapsto→/lscriptthat specifies the
labels to be shown on each state.
2.stateDisplay ∈{‘never’, ‘hover’, ‘click’, ‘always’
}specifies when state labels are displayed. When set to
‘click’, clicking on the state causes the label to appear
and remain visible until the end of the trial. The optionalparameter stateClickCost specifies the cost (a
negative number) for clicking on a single state. When
set to ‘hover’, the label appears only while the mouse ishovering over the associated edge. There is no cost for
this option because the participant’s mouse might pass
over an edge by accident.
3.edgeLabels is analagous to stateLabels , except
that it defaults to the rewards associated with each edge.
4.edgeDisplay is analagous to stateDisplay .
edgeClickCost specifies the cost.
Using this concise yet flexible plugin, various state-
transition and reward structures can be displayed auto-
matically. This allows experimenters to quickly create
a large number of highly variable stimuli. Our pluginthereby enables experimenters with only basic knowl-
edge of JavaScript to create a wide range of qualita-
tively novel experiments that can be run online with
BehaviorResearchMethods
Fig.8 Comparison of frequencies of strategy types between the environment with increasing variance and transfer task. For a detailed description
of the strategy types see Appendix A.4
crowd-sourcing services such as Amazon Mechanical
Turk.
Step-by-stepguideonusingthecomputational
microscope
Given a data set collected with the Mouselab-MDP paradigm
with uniform click costs and no edge rewards, our computa-
tional microscope can be used to obtain a detailed analysisof how the participants learned how to plan without any
additional programming . Here, we provide a step-by-stepguide to applying the computational microscope. To help
users get started with the computational microscope without
having to collect data first, the computational microscopecomes with data from four experiments using the tasks
illustrated in Figs. 1and 7a-c, respectively. The compu-
tational microscope provides information about the strat-egy sequence, the amount of noise in the application of the
a) State values revealed with clicks b) State values shown while hovering the mouse
Fig.9 Two example paradigms created with the Mouselab-MDP plu-
gin for JsPsych: a) Each state is labeled with the reward for reachingthat state; these rewards become visible after they are clicked, witha $0.10 fee per click. b) The reward for making a transition is
revealed only while the mouse is hovering over the correspondingarrow
BehaviorResearchMethods
strategy, the sequence of strategy types and the change in the
relative frequency of decision systems. The computationalmicroscope requires git and Python3 to be installed on the
user’s machine. The following steps describe how to apply
the computational microscope to a data set and the output itprovides.
1. Access data sets and the source code of the computa-
tional microscope by cloning the corresponding githubrepository using the command:
git clone https://github.com/Rationality
Enhancement/ComputationalMicroscope.git
The repository includes four data sets that are
contained in the folder data/human/ . For a detailed
description of these data sets, see Table 2
2. Navigate to src/ and install the package require-
ments running the following command in the cloned
repository’s root directory:
pip3 install -r requirements.txt
3. Apply the computational microscope on any of the
4 data sets described in Table 2using the following
command:
python3 infer_sequences.py <dataset>
<block> <condition>
The values that the parameters in the above command
take can be found out by using the command:
python3 infer_sequences.py help
Here, the parameters <dataset> ,<block> and
<condition> define the name of the dataset,
the block of the experiment which generated the
dataset, and the condition of the experiment, thecomputational microscope is to be run on. Uponsuccessful completion, a dictionary with the par-
ticipant IDs as keys and the strategy sequences asits values are stored as a pickle file in the path
"results/inferred
sequences/<dataset>
<block> <condition> strategies.pkl"
and the corresponding noise parameter val-
ues, in the same format, are stored in
"results/inferred sequences/<dataset>
<block> <condition> temperatures.pkl" .
For example, to run the computational microscope
on the test block of the dataset with increasing variancefor participants who belong to the condition withoutfeedback, run the following command:
python3 infer_sequences.py
increasing_variance train none
4. Analyze the generated sequences by running the com-
mand:
python3 analyze_sequences.py <dataset>
<block> <condition>
This command produces plots of the trial-by-trial
changes in the frequencies of the top-5 strategiesand strategy types, and in the influence of differentdecision systems and other factors. It integrates the
data from all participants into the plots in the
"results/<dataset>
plots" directory.
For example, the following command generates the
plots shown in Fig. 10.
python3 analyze_sequences.py
increasing_variance test none
The computational microscope, in its current imple-
mentation, can be applied to task structures that aresymmetric and do not have cycles. But the general approach
described in this article works for arbitrary environments.
Table 2 Datasets included in the computational microscope repository
Dataset increasing variance (v1.0) decreasing variance (c2.1) constant variance (c1.1) transfer task (T1.1)
Blocks training, test training, test training, test pre-training, training, test
Condition meta, action, none none none noneDescription This dataset contains process-
tracing data from the exper-
iment with the environment
s h o w ni nF i g . 1. The experi-
ment consisted of 10 trainingtrials and 20 test trials had threeconditions that determined thekind of feedback that was pro-vided to the participants.This dataset contains process-
tracing data from the exper-
iment with the environment
shown in Fig. 7a. The experi-
ment consisted of 30 trainingtrials and 30 test trials onlyhad a single condition.This dataset contains process-
tracing data from the exper-
iment with the environment
shown in Fig. 7b. The experi-
ment consisted of 30 trainingtrials and 30 test trials onlyhad a single condition.This dataset contains process-
tracing data from the exper-
iment with the environment
shown in Fig. 7c. The exper-
iment consisted of 1 pre-train-ing trial, 10 training trials 20test trials only had a singlecondition.
Reference Appendix A.1 Callaway et al. ( 2018 ) Callaway et al. ( 2018 )L i e d e r ( 2018a )
The value in brackets references the experiment number in the code
BehaviorResearchMethods
a) b)
c)
Fig. 10 Generated analysis plots for training block of the no feedback condition of the increasing variance data set. aInfluence of different
decision systems and other factors. bTrial-wise changes in strategy type frequencies. cTrial-wise changes in strategy frequencies
The implementation and a detailed tutorial on applying
the computational microscope to a custom dataset are
available at https://github.com/RationalityEnhancement/
ComputationalMicroscope .
Doesitwork?
To test whether using the computational microscope in
conjunction with the Mouselab-MDP paradigm is a reliableway to measure how people plan, we test this approach
using simulations and empirical data. First, we perform
simulations to test our hypothesis that the Mouselab-MDPparadigm yields so much information about how peopleplan that it becomes possible to accurately infer which
planning strategy they used on a single trial and how that
strategy differed from the strategies that the participant usedon the preceding trial and on the following trial. In follow-
up simulations we then assess whether this is also true
for the relative contributions of different decision systems.Following these simulation studies, we test whether the
inferences of our method are valid measures of planing andlearning by applying it to empirical data from studies where
planning and learning were experimentally manipulated.
Simulationstudies
To test if our experimental paradigm makes it possible to
infer people’s planning strategies on a trial-by-trial basis, wesimulated which process-tracing data we would obtain in a
Mouselab-MDP experiment depending on which strategies
people use and how those strategies change from each trialto the next. We then applied our computational microscope
to the simulated process-tracing data to test if that data
would be sufficiently informative about the underlyingplanning strategies that we would be able to infer themcorrectly. Concretely, we report two sets of simulations
suggesting that our method can accurately measure changes
in people’s planning strategies and the relative influence ofdifferent decision systems, respectively.
Is the process-tracing data from the Mouselab-MDP
paradigm suﬃciently informative about people’s planningstrategies? We simulated a Mouselab-MDP experiment
BehaviorResearchMethods
with 31 trials of the 3-step planning task illustrated in Fig. 1
and described in Section 5for various different sequences
of planning strategies. We derived six sets of sequences
of planning strategies from five different models of how
people might learn how to plan. To generate the first dataset, we applied the rational model of strategy selection
learning by Lieder and Griffiths ( 2017 ); the parameters
of this model were fit to the data from 57 participantsperforming 31 trials of the 3-step planning task illustrated
in Fig. 1(i.e., the control condition of the experiment
described in Appendix A.1). We created four additional
data sets by modeling the temporal evolution of people’splanning strategies as gradual learning, insight-like learn-
ing, a mixture of both gradual and insight-like learning, or
a random process that chooses the strategy on each trialindependently at random (random model). In all cases, the
generation of the strategy sequence and the generation of
each click sequence given the sampled strategy involveda considerable amount of randomness that matched or
exceeded the variability observed in human data. For a
more detailed description of how the data was generated,please see Section 5in the Appendix. To avoid bias towards
any one of the five models, we used each of them to gen-
erate a data set with 500 simulated participants completing
31 trials each. We then combined the resulting five data setsinto a single data set from 2500 simulated participants.
We then used our computational microscope to compute
the maximum a posteriori estimate of the sequence of strate-gies for each participant and compared it to the ground
truth sequence of strategies. We evaluated the informa-
tiveness of our process-tracing paradigm in terms of howaccurately the strategies and strategy types could be inferredfrom the simulated process-tracing data. We found that the
process-tracing data made it possible to infer the true strat-
egy for 80 ±0.01% of the trials and to infer the true strategy
type for 92 ±0.00% of them. These findings suggest that
our experimental paradigm yields so much information that
we can hope to be able to infer people’s planning strategieson a trial-by-trial basis. Furthermore, these results sug-
gest that we have implemented our computational method
correctly and that the 79 candidate strategies are differ-ent enough that it is possible to discern between them. Fora detailed description of model-wise strategy and strategy
type accuracies, please see Appendix A.3.
Validationofmeasuringthecontributionsofdiﬀerentdeci-
sionsystems andotherfactors We validated our method’s
ability to recover the trend in the relative influence of dif-
ferent decision systems and other factors across a series
of 79 trials. Each simulation assumed one of three pos-sible trends: increasing influence, decreasing influence, orconstant influence. For each factor, for the increasing and
decreasing trends, we created a sequence of 79 strategiesin which each strategy appears only once and the order
of the strategies in the sequence is the sorted order of
the contribution of the factor to the corresponding strat-egy. We then generated a dataset of 500 sequences of click
sequences. For the constant case, for each factor, we parti-
tioned the set of strategies into up to 3 groups based on the33
rd,6 7thand 100thpercentiles of the relative influence
of the factor across all strategies. We validated our micro-
scope on 500 simulated sequences. To generate a sequence,we randomly selected one of the three groups to generatesequences from and then sampled 79 strategies from that
group and arranged them in sequence. Figure 11shows that
our computational microscope recovered the trends in therelative influence of the decision systems and other factors
very accurately.
Validationonempiricaldata
We also validated our computational microscope on
empirical data, that is we tested whether it can detect theeffects of experimental manipulations and task structure on
people’s planning strategies and metacognitive learning.
Detectingtheeﬀectoffeedbackoncognitiveplasticity
To verify whether our computational microscope can detect
the effect of an experimental manipulation expected to pro-mote cognitive plasticity, namely feedback, we applied it tothe Mouselab-MDP process-tracing data from the experi-
ment described in Appendix A.1 where 164 participants
solved 30 different 3-step planning problems of the forms h o w ni nF i g . 1. Participants in the control condition
received no feedback whereas participants in the first exper-
imental condition received feedback on their actions (ActionFB) and participants in the second experimental condi-
tion received feedback on how they made their decisions
(Metacognitive FB). Action FB stated whether the chosenmove was sub-optimal and included a delay penalty whoseduration was proportional to the difference between the
expected returns of the optimal move versus the chose one.
In contrast to Action FB, Metacognitive FB pertains to howthe decisions are made rather than to the decisions them-
selves. Metacognitive FB is given after every information
gathering operation (click). It has two components that con-vey the informational value of the planning operation and
the planning operation that the optimal strategy would have
chosen, respectively.
This metacognitive feedback was designed to be more
effective than action feedback at teaching people the optimal
BehaviorResearchMethods
Gr ound Truth Measured Values
Fig. 11 Smoothed plots for comparison of the actual and inferred
trends in the relative influence of different decision systems and otherfactors. The computational microscope was applied to click sequencesgenerated from strategy sequences where the weight of one of thefive factors was systematically increasing (top row), decreasing (cen-
ter row), and constant (bottom row) respectively. Each line is based ona different strategy sequence
planning strategy for the task illustrated in Fig. 1.T h i s
strategy (Callaway et al., 2018 ) starts by searching the
potential final destinations for the best possible outcome and
terminates planning when it finds one of them.As Fig. 12shows, the computational microscope cor-
rectly detected that feedback boosted metacognitive learning.Concretely, the computational microscope revealed that
metacognitive feedback boosted the discovery of the optimal
BehaviorResearchMethods
Fig.12 Comparison of frequencies of forward-planning and near-optimal strategies across different types of feedback in the experiment described
in Appendix A.1. The green, orange and the blue lines represent the metacognitive feedback, action feedback and the no feedback conditions
respectively. The circles represent the forward planning strategies and the stars represent the near-optimal planning strategies
planning strategy (58% vs. 31% in the no feedback condition,
z=15.44,p< 0.0001, BF >1000)2and decreased people’s
propensity to start planning by considering immediate out-
comes, i.e. forward planning (2% vs. 14% in the no feed-
back condition, z=− 13.27,p < 0.0001 ,BF>1000)
whereas action feedback reduced the frequency of the near-
optimal planning strategy (24% vs. 31% in the no feedback
condition, z=− 4.74,p < 0.0001 ,BF>1000) and did
not change the frequency of the forward planning strategies(15% vs. 16% in the no feedback condition, z=1.00,p=
0.3193 ,BF=0.10 ).
The computational microscope allows us to gain addi-
tional insights into how those changes in people’s strategies
come about. Concretely, correcting for multiple compar-
isons ( α
sidak = 0.0034) and applying Wilcoxon-signed rank
test, Fig. 13shows that metacognitive feedback significantly
accelerated people’s transition to choosing their planning
operations increasingly more based on the model-basedmetareasoning system ( T=248,p=0.0004 ,BF=65.31),
the Pavlovian system ( T=276,p=0.0007 ,BF=38.15),
2Here and throughout the manuscript, we always report the results of
a Bayesian hypothesis testing along with their frequentist equivalents.We performed Bayesian hypothesis testing using the BayesFactorpackage in R (Morey & Rouder, 2018 ) for all tests except the Wilcoxon
test for which we used JASP (JASP Team, 2021 ). We report the results
of each Bayesian hypothesis test in terms of the Bayes factor in favor
of the alternative hypothesis. BF=xmeans that the alternative
hypothesis is xtimes as likely to be true as the null hypothesis.and the system for deciding when to stop planning ( T=
82,p< 0.0001 ,BF=23568.70). This makes sense because
the structure of the environment makes it beneficial to
inspect nodes that are most uncertain (a feat accomplished
by the metareasoning system), explore nodes that lie on thepath to the most valuable nodes (as recommended by the
Pavlovian system), and to stop as soon as a very good path
has been identified (a feat that accomplished by the systemfor deciding when to stop). Also, Metacognitive feedback,in general, drove people towards planning more by reduc-
ing the amount of mental effort avoidance ( T=1.0,p=
0.0001 ,BF=167.25). Action FB, by contrast,
drove people towards relying more on the Pavlovian
system ( T=183,p=0.0004 ,BF=1236.80), and
the decision system for deciding when to stop planning(T=134,p=0.0001 ,BF=685.42) and relying less
on the model-free values and heuristics ( T=229,p=
0.0004 ,BF=172.56) decision system. In the condition
without feedback, people relied increasingly more on thePavlovian system ( T=148,p < 0.0001 ,BF=1852.39),
the system for deciding when to stop planning ( T=
173,p < 0.0002 ,BF=647.56) and on the model-based
metareasoning system ( T=206,p=0.0012 ,BF=38.51)
but less significantly when compared to the metacognitive
feedback condition.
The computational microscope also provides insights
into which unique strategy types people go through during
learning (learning trajectories) and how this is affected by
BehaviorResearchMethods
)b )a
c)
Fig.13 Temporal evolution of the relative influence of different decisions systems and other factors in the control condition without feedback ( a),
the experimental condition with metacognitive feedback ( b), and the experimental condition with action feedback ( c), respectively
feedback. Overall, we found that 86% of people’s learning
trajectories were unique. However, when we zoom out tothe level of strategy types, the computational microscope
reveals several common learning trajectories (see Table 3).
We found that the number of strategy types people go
through from their initial strategy to the final strategy waslower when participants received metacognitive feedbackthan when they received action feedback ( t(107)=− 3.73,
p=0.0002 ,BF=161.30) or no feedback ( t(107)=
−2.65,p=0.0046 ,BF=8.77). We found no significant
difference between the Action FB and the No Feedbackconditions ( t(106)=1.46,p=0.0737 ,BF=0.09)
Measuringhowpeople’splanningstrategiesdiﬀer
dependingonthestructureoftheenvironment
Previous work has shown that people adapt their cogni-
tive strategies to the structure of the decision environment(Payne et al., 1993 ; Callaway et al., 2018 ; Lieder & Grif-
fiths, 2017 ; Gigerenzer & Selten, 2002 ). Here, we verify that
our method is able to detect differences in people’s strate-
gies across the four environments described in Section 5.To do so, we applied the computational microscope to the
process-tracing data participants generated in the test blocks
of the corresponding experiments after they had learned
about their respective environment in the training block(see Table 2). Because participants went through a suffi-
ciently large number of training trials, we observed that
participants’ planning strategies were stable. As shown inTable 4, the computational microscope revealed that peo-
ple adapted their planning strategy to the structure of their
environment. These differences are systematic in the sense
that how people’s strategy choices differ across environ-ments roughly corresponds to how the strategies’ perfor-
mance differs across those environments. To quantify this,
we report the relative performance ( r
rel)o ft h em o s tc o m -
mon strategies relative to the best-performing strategy of
each environment. The performance of each strategy ( ri)
was determined by running 100,000 simulations, and then
normalized according to rrel
i=ri−minjrj
maxjrj−minjrj.
For both environments with increasing variance, our
computational microscope detected that the most common
strategy was the near-optimal goal-setting strategy whichexploits that the most distant rewards are most variable.
BehaviorResearchMethods
Table3 Common trajectories of strategy types by the type of feedback
participants received
Condition Usage % Strategy type trajectory
No FB
7% FP
6% MP →LS→MGS
6% MGS
4% FP-BFS →MGS
4% MS →MGS
MCFB
22% MGS
16% MS →MGS
7% FGS →MGS
7% FP →FGS→MGS
5% FP →MGS
Action FB
11% FP
4% MP4% FP →MP→FP→MP
2% LS →MS→MGS→LS→MS→
MGS→MS→MGS
2% MP →FP→MP→FP→MS→MGS
The strategy types are: FP - Frugal planning, MP - Myopic planning,
MGS - Maximizing goal-setting with limited backward planning, LS -Local search, FP-BFS - Forward planning like Best-First search, FGS- Frugal goal-setting strategies, MS - Miscellaneous strategies
By contrast, people almost never used this strategy in any
of the other environments. For the environment with decreas-
ing variance, our computational microscope detected thatpeople primarily use strategies that exploit the structure of
this environment by prioritizing its immediate outcomes.
For the environment with constant variance, the com-
putational microscope detected that after inspecting allimmediate outcomes the second most frequent strategy per-
forms Best-First Search with Satisficing, which is adaptive
in this environment (Callaway et al., 2018 ), although the
most commonly used strategy was not particularly adaptive.
These results show that the computational microscope
can reliably reveal how the planning strategies peopleuse differ depending on the structure of the environment.
Furthermore, comparing the strategies the computational
microscope inferred for the 5-step version of the increasingvariance environment that was used as a transfer task tothe 3-step version of that environment that was used as a
training task suggests that the computational microscope
can reveal the transfer of learning across environments.
Equally, the strategy types inferred by our computational
microscopes were consistent with previous findings sug-
gesting that people adapt their decision strategies to thestructure of the environment (Payne et al., 1993 ; Callawayet al., 2018 ; Lieder & Griffiths, 2017 ; Gigerenzer & Sel-
ten,2002 ). Table 5shows the performance and frequency of
the inferred strategy types in decreasing order of their fre-
quency for each of the 4 environments. The performance
of a strategy type was determined by the weighted aver-age of the performances of the strategies belonging to that
strategy type where the weight of a strategy is the relative
frequency of the strategy among the strategies belonging tothe cluster. As expected, we find that in both increasing vari-
ance environments, people primarily rely on strategies that
prioritize the potential final outcomes. For the environmentwith decreasing variance, the computational microscopeinferred that most people used the strategy type that is best
adapted to this type of environment, namely myopic plan-
ning strategies. For the environment with constant variance,the computational microscope inferred that forward plan-
ning strategies similar to best first-search was the second
most frequently type of planning strategies. The most com-mon strategy type was “Myopic Planning” which includes
several strategies that are similar to Best First Search (see
Section 5).
Overall, the results in Tables 4and5illustrate that our
computational microscope makes it easy for researchers to
describe both the adaptiveness of human planning and its
limits.
Discussion
We have developed a computational process-tracing method
that allows us look at how people plan and how their plan-
ning strategies change over time. Our method extends theMouselab paradigm for tracing people’s decision strate-
gies (Payne et al., 1993 ) in three ways. First, it progresses
from one-shot decisions to sequential decision problems.Second, it introduces computational methods for analyzingprocess tracing data in terms cognitive strategies. Third, we
have extend the approach to measuring how people’s plan-
ning strategies change over time. Our method is easy touse and freely available. We have successfully evaluated
our methods using simulations and human data. The results
suggest that our computational microscope can measurecognitive plasticity in terms of the temporal evolution of
people’s cognitive strategies and also provide us with valu-
able information about the trends in changes of strategies,strategy types and also how people change their strategieswith changes in environments. We have applied our compu-
tational microscope to a number of data sets. The results of
these analyses contribute to a more detailed understandingof how people plan and revealed some interesting empirical
characteristics of metacognitive learning.
Our method can be used to study many different types of
cognitive change across a wide range of different timescales.
BehaviorResearchMethods
Table 4 Summary of the performance of the most frequent strategies across four different environments
Environment Most common strategies Relative score ( rrel) Frequency
Increasing Variance
(3-steps)
Search for the best possible final outcome 1.00 45.1%
Consecutive second maximum 0.93 11.6%No planning 0.13 10.6%
Increasing Variance
(5-steps)
Search for best possible final outcome 1.00 20.8%
No planning 0.28 16.2%
Explore immediate and final outcomes with satisficing on finding
al a r g ev a l u e0.97 12.4%
Explore final outcomes and their parents 0.92 11.6%
Explore final outcomes and their parents with satisficing 0.97 6.6%Explore immediate outcomes on the paths to the best final outcomes 0.99 4.5%
Decreasing Variance
(3-steps)
Explore immediate outcomes and final outcomes with satisficing
on a positive value0.97 32.7%
Satisficing Best First Search after exploring all immediate
outcomes0.94 26.7%
No planning 0.00 12.5%
Explore immediate outcomes and final outcomes with satisficing
on a positive value0.96 10.8%
Explore sub-trees of positive immediate outcomes 0.93 5.4%
Explore all immediate outcomes 1.00 5.3%
Constant Variance
(3-steps)
Explore all immediate outcomes with satisficing 0.81 17.9%
Satisficing Best First Search after exploring all immediate
outcomes0.91 14.4%
Explore the immediate children of the best immediate outcome 0.81 10.2%
Non-terminating Best First Search 0.74 6.4%
Exploring immediate and final outcomes with positive satisficing 1.00 5.5%
Best First Search after exploring all immediate outcomes 0.75 5.5%Pruning of nodes with immediate negative rewards and choosing
actions with best long-term consequences0.94 4.8%
Leave out one immediate outcome 0.79 4.4%
Consecutive second maximum 0.85 4.0%Goal-setting with positive satisficing 0.90 3.0%
The performance of each strategy ( ri) was determined by running 100,000 simulations, and then normalized according to rrel
i=ri−minjrj
maxjrj−minjrj.
To be included in this table, a strategy had to be used in at least 3% of all trials
This makes it suitable for investigating learning, cognitive
development, decision-making, individual differences, and
psychopathology.
We are optimistic that computational microscopes will
become useful tools for investigating the learning mecha-
nisms that enable people to acquire complex cognitive skillsand shape the way we think and decide. This will be animportant step towards reverse-engineering people’s ability
to discover and continuously refine their own algorithms.
From a psychological perspective, this line of work mightalso help us understand why we think the way we do and
lead us to rethink our assumptions about what people can
and cannot learn. Developmental psychologists could useour method to trace the development of cognitive strategies
BehaviorResearchMethods
Table 5 Summary of the performance of the most frequent strategy types for four different environments
Environment Most common strategy types Relative score ( rrel) Frequency
Increasing variance
(3-steps)
Maximizing goal-setting with limited backward planning 1.00 62.9%
Frugal planning strategies 0.00 11.0%Miscellaneous strategies 0.81 10.2%
Local search strategies 0.92 5.5%
Myopic planning strategies 0.41 4.7%Frugal goal-setting strategies 0.75 3.8%
Increasing variance
(5-steps)
Maximizing goal-setting with limited backward planning 0.86 38.9%
Miscellaneous strategies 0.63 24.6%
Frugal planning strategies 0.00 16.2%Frugal goal-setting strategies 0.58 6.8%
Immediate rewards on the path to best final outcomes
with satisficing1.00 4.5%
Local search strategies 0.64 4.1%
Myopic planning strategies 0.33 3.2%
Decreasing variance(3-steps)
Myopic planning 1.00 53.4%
Forward planning strategies similar to Best First Search 0.85 33.0%Frugal planning strategies 0.10 12.6%
Constant variance
(3-steps)
3
Myopic planning 0.76 46.8%
Forward planning strategies similar to Best First Search41.00 26.8%
Frugal goal-setting strategies 0.65 7.5%Maximizing goal-setting with limited backward planning 0.75 6.8%
Miscellaneous strategies 0.22 4.6%
Local search strategies 0.40 3.5%
3The strategy type with the highest average score was “Frugal planning strategies” ( rrel=1.00, frequency: 2.1%). Its score is so high because its
most frequent strategy is a high-performing strategy similar to Best First Search. This strategy type is not listed because its relative frequency is
less than 3%
4The average score of this adaptive strategy type is surprisingly low because it includes strategies that incur a very high planning cost by inspecting
all of the information available
The performance of each strategy was determined by running 100,000 simulations. The performance of a strategy type was determined by the
weighted average of the performances of the strategies belonging to that strategy type where the weight of a strategy is the relative frequency ofthe strategy among the strategies of the same type. To be included in this table, a strategy type had to be used in at least 3% of all trials
across the lifespan and elucidate how learning contributes
to those developmental changes. Similarly, clinical psy-
chologists and computational psychiatrists could apply it
to trace how person’s cognitive strategies changes as theydevelop and recover from different mental disorders. Impor-
tantly, our method can also be used to investigate how
cognitive plasticity depends on the learning environment,individual differences, age (Das et al., 2019 ), time pres-
sure, motivation, and interventions – including feedback,instructions, and reflection prompts. Using our method to
measure individual differences in cognitive plasticity might
reveal why equivalent experience can have fundamentally
different effects on the psychological development of dif-ferent people. This, in turn, can help us understand why
some people are predisposed to develop certain cognitive
styles, personalities, and mental disorders. Applications incomputational psychiatry might use this approach to under-stand the development of mental disorders and to create
BehaviorResearchMethods
computational assays for detecting whether a person is at
risk for developing specific forms of psychopathology longbefore its symptoms occur.
To facilitate these applications, future work might extend
the proposed measurement model to continuous strategyspaces, a wider range of tasks and strategies, and learn-
ing at the timescale of individual cognitive operations. In
addition, future work will also leverage our computationalmicroscope to elucidate individual differences in cogni-
tive plasticity within and across psychiatric conditions and
different age groups. We will also work on making ourinferences more precise by learning models of strategiesand strategy transitions from human data. To move towards
a more naturalistic planning task, future versions of our
method could present participants with fully-revealed envi-ronments and infer their planning strategies from eye-
tracking data. The computational approach could be anal-
ogous to the one presented here instead that clicks arereplaced by saccades.
The ideas of our approach are not entirely novel. Process-
tracing has already been extensively used to study peo-ple’s decision strategies (Payne et al., 1993 ; Schulte-
Mecklenbeck et al., 2011 ; Schulte-Mecklenbeck et al.,
2019 ) and Bayesian inference has been used to infer which
decision strategies are include in individual participants’repertoire (Scheibehenne et al., 2013 ), when people switch
between different decision strategies (Lee & Gluck, 2021 ),
and which strategies people use in economic games Costa-Gomes and Crawford ( 2006 ), Crawford ( 2008 ), and Costa-
Gomes et al. ( 2001 ). Our method has several advantages.
What differentiates our approach from the original
Mouselab paradigm (Payne et al., 1993 ) is that it measures
how people plan and that we infer people’s strategies from
the process-tracing data. On a high level, the Bayesian
Toolbox approach by Scheibehenne et al. ( 2013 )a l s oi n f e r s
people’s strategies. Their approach infers which strategies
are included in the person’s repertoire. However, it does
not attempt to resolve which strategy was used on whichtrial. Instead, it makes the simplifying assumption that
every decision is influenced by all strategies that are in
the person’s toolbox. By contrast, our method makes thedifferent assumption that on each trial each participantdraws a single strategy from the toolbox. Based on this
assumption, our method infers which individual strategy a
participant used on the first trial, which individual strategythey used on the second trial, and so on.
The methods developed by Lee and Gluck ( 2021 )a n d
Lee et al. ( 2019 ) are more similar to our method in that
they infer which strategy each participant used on each
trial of the experiment. The main difference is that these
methods were developed for studying multi-cue decision-making whereas our method was developed for studyingplanning. The method by Lee et al. ( 2019 )h a st h eadvantage that it uses process-tracing data, verbal reports,
and choices whereas our method exclusively relies on theprocess-tracing data. While our method and Lee et al.
(2019 ) analyze the data of each participant individually,
the method by Lee and Gluck ( 2021 ) additionally performs
inference at the group level and constrains inferences
about individual participants by the characteristics of the
group. Furthermore, the method by Lee and Gluck ( 2021 )
additionally infers two aspects of the generative model of
strategy sequences from the data, namely the probabilities
of possible initial strategies and the probabilities of possiblestrategy transitions. The main advance of our method is thatit differentiates between a much larger number of different
strategies (79 vs. 4). Furthermore, we examined multiple
alternative models of strategy transitions and validated ourmethod on data from multiple different experiments that
varied the decision environment and induced systematic
learning-induced changes in people’s strategies over time.
Finally, the approaches that have been developed to infer
which strategies people use in economic games (Costa-
Gomes & Crawford, 2006 ; Crawford, 2008 ; Costa-Gomes
et al., 2001 ) assume that each person always uses the
same strategy and cannot measure how a person’s strategy
changes over time. Furthermore, the strategies these meth-
ods measure are specific to strategic social interaction. Thestrategies people use in tasks such as planning a road trip or
project are very different. Therefore, studying them requires
a different methodology such as the one we have developedin this work.
In conclusion, the approach introduced in this article
complements these existing approaches in useful ways thatmake it possible to measure people’s planning strategies andhow they discover them.
Our methods are not without limitations. First and
foremost, the Mouselab-MDP paradigm inherits at least one
of the limitations of the Mouselab paradigm that it is based
on. Concretely, the Mouselab-MDP paradigm might change
how people plan by making information acquisition costlierthan it might otherwise be. Previous research comparing
Mouselab-based measures of people’s decision processes
against equivalent measures based on eye-tracking foundthat the increased cost of information acquisition in theMouselab paradigm led people to acquire less information
and, to some extent, it also changed the order in which
people acquire information (Lohse & Johnson, 1996 ). We
believe that it is likely that similar differences also exist
for the Mouselab-MDP paradigm. As Lohse and Johnson
(1996 ) pointed out, such differences are more important for
some research questions than for others. Following the logic
of their analysis, we believe that there are many important
questions about planning and metacognitive learning thatare unaffected by such differences. Concretely, our methodshould be well-suited to characterize the qualitative effects
BehaviorResearchMethods
of experimental manipulations on planning and learning as
long as it can be expected that the qualitative effects wouldbe the same if the cost of information acquisition was lower.
Regardless thereof, we believe that comparing the process-
tracing data collected with the Mouselab-MDP paradigm tocorresponding process-tracing data based on eye-tracking is
an interesting direction for future work.
A perhaps more provocative possibility is that the
planning environment that the Mouselab-MDP paradigm
seeks to emulate is one in which people cannot simply
look up what the outcomes of their actions would be buthave to estimate them through effortful mental simulations.In this sense, it is conceivable that the Mouselab-MDP
paradigm is closer to the real-world problem that is designed
to mimic than an equivalent eye-tracking paradigm wouldbe. This suggest that future work should compare the plans
that people arrive at when they have to rely on mental
simulations to the plans that they arrive at when thosemental simulations are externalized with the Mouselab-
MDP paradigm.
One limitation of our computational microscope is that
its current implementation requires that the task environ-ment is symmetric and has no circular paths in it. This is
because of the features defined in Eq. 1are computable
currently only for such structures. Generalizing the imple-mentation of the computational microscope so that it can be
applied to other kinds of environments may be a worthwhile
direction for future work.
In summary, our method makes it possible to more
directly observe the previously hidden phenomenon of cog-
nitive plasticity in many of its facets – ranging from skillacquisition, learning to think differently, cognitive decline,self-improvement, changes in cognitive dispositions, and
the onset, progression, and recovery from psychiatric symp-
toms and mental disorders. In conclusion, we believe thatthe method introduced in this paper can be used to advance
cognitive science, psychology, and psychiatry in many
promising ways.
Funding Open Access funding enabled and organized by Projekt
DEAL.
AuthorContributions FL, FC, YRJ, PD, and TLG developed the ideas
for the methods and article. YRJ and FC contributed software. YRJconducted simulations. FL, PMK, and FC collected the data. YRJ,and FC analyzed the data. FL, YRJ, and FC wrote the article. FL,YRJ, PD, FC, and TLG edited and revised the article. RH performeda computational reproducibility audit and ensured that all results are
reproducible.
Open Practices Statement None of the experiments were preregis-
tered. The code of the Mouselab-MDP paradigm is available at: https://
github.com/RationalityEnhancement/Mouselab-MDP .The code of our computational process-tracing method is avail-
able at: https://github.com/RationalityEnhancement/Computational
Microscope .
The data sets used in this article are available at: https://github.
com/RationalityEnhancement/ComputationalMicroscope/tree/master/data/human .
Appendix
A.1Experiment(3-steptaskwithincreasing
variance)
We evaluated our computational microscope on data that
was collect in a pilot experiment by Callaway et al.
(2022a ). The methods and results of that experiment were
as follows.
Participants We recruited 164 participants on Amazon
Mechanical Turk (average age 35 years, range: 18–72 years;75 female). Balanced condition assignment and repeat-partici-
pant exclusion was performed using Psiturk (Gureckis et al.,
2016 ). None of the participants who finished the experiment
were excluded for analysis.
Procedure The experiment comprised instructions, a train-
ing block, a test block, and an exit survey. The trainingblock comprised 10 trials, and the test block comprised
20 trials. Each participant was assigned to receive either
metacognitive feedback (55 participants), action feedback(55 participants), or no feedback (54 participants) dur-
ing the training block. The metacognitive feedback used
the optimal planning strategy for the environment to pro-vide feedback on participants’ clicks. The action feed-back condition provided feedback on the actions (moves)
of the participants. The exit survey asked participants
about what they had learned, their age, and their genderidentity.
Materials Each trial of the experiment presented partic-
ipants with an instance of the 3-step planning problem
described in Fig. 1. The key structure of this problem is that
the range of possible rewards is smallest in the first step,larger in the second step, and largest in the third step. Tooperationalize the cost of planning, we charged participants
one virtual dollar per click. To simplify the implementa-
tion of metacognitive feedback, we required that all clicksbe made before the first move. To eliminate the time cost
of engaging in planning compared to speeding through the
experiment, participants who spent less than 7 seconds onplanning (e.g., only 3 seconds) had to wait for the remaining
BehaviorResearchMethods
time after executing their moves (e.g., for 4 seconds). In the
test block, participants started with an endowment of 50 vir-tual dollars and earned a bonus of 1 cents for every $5 they
made in the game.
Results In the test block, the average score of partici-
pants receiving no feedback was 27.58 points/trial (95% CI:
[26.21, 28.95]). Critically, participants receiving metacogni-tive feedback scored significantly higher (34.86 points/trial;
95% CI: [33.83, 35.89]; t(3268)=8.3268 ,p < 0.001,BF
=5.39·10
13). By contrast, giving participants conventional
feedback on their actions appeared to be ineffective. That is,participants receiving action feedback did not score signifi-
cantly higher than participants in the no-feedback condition
(27.57 points/trial; 95% CI: [26.21, 28.93]; t(3268)=
0.0108 ,p=0.504,BF=0.039) and performed significantly
worse than participants who received metacognitive feed-
back (t(3298)=−8.3425 ,p< 0.001,BF=6.13·10
13).
A.2Modelingtransitionsbetweenplanning
strategies
The transitions between strategies from one trial to the next
define the nature and the type of cognitive plasticity. One pos-
sible way in which people might switch from one strategy
to the other is based on the similarity of strategies (mea-sured in terms of distance between them). In addition to
the similarity of strategies, we have to define how the
similarities manifest themselves into the actual transitions.Therefore, we define different measures of similarity andmechanisms of how the distances manifest into strategy
transitions.
A.2.1Distances
To find out if the frequency of transitions between strategies
was dependent on how close the strategies are, we con-sider 6 metrics of distances between the strategies. Using
these distance metrics, we did not find any correlation
between the probabilities of transition from one strategy tothe other and the distance between them. In this section, we
describe each of the three types of metrics and their con-
stituents.
1. Behavioral distances The behavioral distance between
two strategies is measured by the distance between the prob-
ability distribution of clicks made by the two strategies. Weconsider two behavioral distances to quantify the similarity
between the strategies.Jensen-ShannonDivergence
/Delta1(s
1,s2)=JSD [p(d|s1), p( d|s2)]
=KL/bracketleftbigg
p(d|s1),p(d|s1)+p(d|s2)
2/bracketrightbigg
+KL/bracketleftbigg
p(d|s2),p(d|s2)+p(d|s1)
2/bracketrightbigg
, (6)
JeﬀreysDivergence(Symmetric-KLDivergence)
/Delta1(s 1,s2)=JD[p(d|s1), p( d|s2)]
=KL[p(d|s1), p( d|s2)]
+KL[p(d|s2), p( d|s1)](7)
2. Distances in the strategy weight space This distance
metric measures similarity between strategies based on the
weight space representation of the strategy. For computing
the distance between strategies according to this distancemetric, we represent a strategy using its preference for
particular features which is quantified using the weights
that are applied to the values of the features. Therefore, wequantify similarities between strategies by representing eachstrategy using a weight vector ( w) and then measure the
similarity of the strategies as the similarity of the weight
vectors. We consider two notions of similarity for strategyweight vectors.
Manhattandistanceinthestrategyweightspace
/Delta1(s
1,s2)=/bardblw1−w2/bardbl1, (8)
Euclideandistanceinthestrategyweightspace
/Delta1(s 1,s2)=/bardblw1−w2/bardbl2, (9)
where /bardblx/bardblprepresents the p-norm of the vector x,i . e .
/bardblx/bardblp=(n/summationdisplay
i=1|xi|p)1/p(10)
3. Distancesin thedecision systemweight space This dis-
tance metric measures similarity between strategies basedon the contribution of various decision systems to the strate-
gies. To compute the distance between strategies according
to this distance metric, we represent a strategy using itspreference for one decision system over the other. This pref-erence is quantified using the total preference over all the
features of the decision system. Therefore, we measure sim-
ilarities between strategies by representing each strategyusing a weight vector ( w
ds) where each weight in the vec-
tor quantifies a preference for a decision system and then
BehaviorResearchMethods
measure the similarity of the strategies as the similarity of
the decision weight vectors. We consider two notions ofsimilarity for strategy decision system weight vectors.
Manhattandistanceinthedecisionsystemweightspace
/Delta1(s
1,s2)=/vextenddouble/vextenddouble/vextenddoublewds
1−wds
2/vextenddouble/vextenddouble/vextenddouble
1, (11)
Euclideandistanceinthedecisionsystemweightspace
/Delta1(s 1,s2)=/vextenddouble/vextenddouble/vextenddoublewds
1−wds
2/vextenddouble/vextenddouble/vextenddouble
2, (12)
A.2.2Priors
In this section, we define how the distances between strate-
gies manifest into transitions between them. For this purpose,
we consider two priors on the strategy transitions that usethe distances defined in Appendix A.2.1 . They are the grad-
ual learning prior and the mixed prior. We describe each of
them in turn.
Graduallearningprior The gradual learning prior ( m
gradual
in Eq. 13) assumes that strategies change gradually (i.e
transitions to strategies that are close-by in terms of thedistance metrics when compared to other strategies).
P(S
t+1=s|St,mgradual)=exp(−1
τ/Delta1(s, S t))
/summationtext
s/prime∈Sexp(−1
τ/Delta1(s/prime,St)),(13)
Mixed prior The mixed prior ( mmixed in Eq. 14) assumes
that both insight-like ( minsight in Eq. 2) and gradual changes
coexist.
P(St+1=s|St,mmixed)
=pgradualP(St+1=s|St,mgradual)
+(1−pgradual)P(S t+1=s|St,mabrupt). (14)
In Eqs. 13and14,τis the temperature parameter which
balances how much the distance between the strategies
effects the transition probabilities between them. We found
that fitting τdid not improve our inferences. Therefore, we
set its value to 1.
We evaluated all the distance metrics defined in the
Section 5to verify if the gradual and mixed priors with those
distance metrics improved our inferences on the strategies.Model selection revealed that the model without a prior
based on the strategy distances (i.e. insight-like transition
prior) performed better than the model with the gradual andmixed transition priors.A.3Simulateddata
To validate our computational microscope on simulated
data, we generated data from 5 models: the random model
(that selects one strategy at random in each of the trial), theinsight model which generates click sequences according
to Eq. 2, the gradual model described in Eq. 13,t h em i x e d
model described in Eq. 14and the RSSL model based on
Lieder and Griffiths (2017). The first four models repeat the
previous strategy with a probability of 50% and samples it
from its model of strategy change otherwise. According toa mixed model, there is a 50% chance that such strategychanges will follow the gradual model ( 13) and a 50%
chance that they will follow the insight model ( 2). The
RSSL model treats the problem of deciding how to planas a 79-armed bandit with one arm for each strategy. It
performs Bayesian inference on the expected return of each
strategy and selects strategies via Thompson sampling. Ithas 79 ×2=158 free parameters that specify the prior
mean and variance of each strategy’s expected return. These
parameters of the model were fit using data from Lieder(2018b ). In all cases, the simulation of how the simulated
strategies generate click sequences was also probabilistic.
Concretely, the click sequences were generated according
to the probabilistic soft-max model of the correspondingstrategy ( 1). For each strategy the temperature parameter ( τ)
that determines the amount of randomness in the generation
of the click sequences was estimated from human data.
We found that for data generated from the random model,
our computational microscope correctly inferred the true
strategy for 76 ±0.007% of the trials and correctly predicted
the strategy type for 91 ±0.005% of the trials. Similarly for
the gradual learning model, our computational microscope
could correctly predict the true strategy in 77 ±0.007% of
the trials and the correct strategy in 91 ±0.004% of the trials.
For data generated from the insight-like learning model,
our method correctly predicted the ground-truth strategy in
88±0.005% of the trials and the ground-truth strategy type
in 96±0.003% of the trials. For the data generated from the
model which is assumes a combination of gradual learning
and abrupt insights, our method correctly inferred the truestrategy in 82 ±0.006% of the trials and the true strategy
type in 94 ±0.004% of the trials. For data generated from
the RSSL model, our computational microscope was able to
correctly infer the true strategy in 76 ±0.007% of the trials
and the true strategy type in 90 ±0.005% of the trials.
A.4Planningstrategies
We have considered a total of 79 strategies clustered into
13 types of strategies. The clustering was done by applying
Ward’s hierarchical clustering technique to the strategies’
BehaviorResearchMethods
pairwise distances computed by the Symmetric Kullback
Leibler Divergence between the probability distributionsof clicks induced by the strategies. Since the output of
the clustering is a hierarchical partitioning of the set of
strategies, we chose the level of hierarchy that gave us themost interpretable clusters. This section describes the strat-
egy types and the strategies that belong to each type. Please
note that the strategies are stochastic and the descriptiononly corresponds to the actual behavior most of the time.
All of the strategies described below make the best use of
the available information to make the final move. That is, thestrategies take the path that has the highest expected value.
Goal-settingwithexhaustivebackwardplanning
These strategies explore all the outcomes. They start by
exploring the final outcomes and then plan backwards .
They differ in when they initiate backward planning. The
model-free values and heuristics decision system and thePavlovian decision system together contribute at least 75%
on average to the strategies in this cluster.
Strategy 1: Goal-setting This strategy starts by exploring
the final outcomes in a random order and if a positive finaloutcome is found, it looks at the outcomes randomly along
the path to the start node and this procedure is repeated until
all outcomes are explored.
Strategy2:Immediategoalsetting This strategy’s behavior
is similar to strategy 1 but differs in the way it explores the
outcomes on the path to the start node. Instead of exploring
randomly on the path to the start node, it explores them levelby level.
Strategy 3: Immediate goal setting with preference for
siblings This strategy’s behavior is similar to that of
strategy 2 except that after exploring the path until the startnode, it explores the sibling of the previously observed final
outcome.
Strategy 4: Immediate goal setting with preference for
siblings and immediate ancestors This strategy’s behavior
is similar to that of strategy 3 but more priority is given toimmediate ancestors when there are multiple ancestors.
A.4.2MaximizingGoal-setting
withexhaustivebackwardplanning
The only strategy in this category first explores all final
outcomes and then plans backwards from them in the orderof their value until it has explored all the outcomes. The
model-free values and heuristics decision system and thePavlovian decision system together contribute at least 75%
on average to the strategy in this cluster.
Strategy 5: Maximizing goal-setting with exhaustive back-
ward planning This strategy starts by exploring the final
outcomes and then plans backwards in the decreasing order
of values of the final outcomes. This strategy doesn’t termi-
nate until it has observed all the outcomes.
A.4.3Maximizinggoal-settingwithlimited
backwardplanning
These strategies focus their exploration on potential final
outcomes and their termination depends on whether or
not a high value has been observed. These strategies donot do backward planning, except for strategy 6, which isan approximation to the near-optimal goal-setting strategy
for the increasing variance environment for the three step
task with increasing variance defined in Section 5.T h e
strategies differ in when they terminate planning, especially
with respect to how much they continue exploring after
uncovering sufficiently good information. The model-freevalues and heuristics decision system, the model-based
metareasoning decision system and the Pavlovian decision
system in combination contribute at least 75% on average tothe strategies in this cluster.
Strategy6:Searchforthebestpossibleﬁnaloutcome This
strategy starts with exploring the final outcomes in a randomorder and terminates clicking upon finding an outcome with
value equal to the maximum observable value of the reward
distribution. If such a node is not found, it explores all thefinal outcomes. If there are multiple final outcomes with
the same highest observed value, the strategy might do
backward planning along the paths from those outcomes.This strategy approximates the optimal strategy for the
three-step task with increasing variance.
Strategy7:Excessivegoal-setting This strategy starts with
exploring the final outcomes in a random and explores
one extra outcome after exploring an outcome that withvalue equal to the maximum observable value of the reward
distribution. If such a node is not found, it explores all the
final outcomes and terminates.
Strategy8:Leaveoutoneﬁnaloutcome This strategy ran-
domly explores all final outcomes except for one randomly-selected final outcome.
Strategy 9: Extra planning after exploring the second best
value This strategy starts by exploring the final outcomes
and terminates after exploring one extra final outcome after
BehaviorResearchMethods
having found an outcome whose value is greater than the
second largest observable value of the reward distribution.
Strategy 10: Explore as many ﬁnal outcomes as there are
initial outcomes This strategy explores as many final out-
comes as there are immediate outcomes in the task structure.
Strategy 11: One outcome per sub-tree of the start node
This strategy explores one random outcome from each
sub-tree of the start node.
Strategy 12: Consecutive second maximum This strategy
starts by exploring the final outcomes in a random orderand terminates after exploring two outcomes consecutively
whose values are greater than the second largest value of the
reward distribution.
Strategy 13: Explore two extra outcomes after exploring a
positive outcome This strategy starts with exploring final
outcomes in a random order and terminates after exploring
two extra final outcomes after exploring a positive finaloutcome.
Strategy14:Immediateoutcomesafterﬁnaloutcomeswith
satisﬁcing This strategy explores all the final outcomes
first and then explores all the immediate outcomes. Whileexploring the final outcomes, if it finds an outcome whose
value is equal to the maximum observable value of the
reward distribution, it terminates.
Strategy15:Exploreparentsoflargestﬁnaloutcomes This
strategy explores all the final outcomes and then explores
the parents of the final outcomes with the largest value
among the explored outcomes.
A.4.4Frugalgoal-settingstrategies
These strategies focus their exploration on potential finaloutcomes but explore less overall. They differ in the way
they terminate planning. The model-free values and heuris-
tics decision system, the model-based metareasoning deci-sion system and the Pavlovian decision system in combina-
tion contribute at least 75% on average to the strategies in
this cluster.
Strategy 16: Goal-setting with backward planning This
strategy starts by exploring the final outcomes. It explores
the final outcomes until a final outcome with a value equal
to the maximum observable value of the reward distributionis explored, plans backwards to the corresponding immedi-
ate outcome and then terminates. If such an outcome is not
found, it terminates after exploring all final outcomes.Strategy 17: Goal-setting with positive satisﬁcing This
strategy starts with exploring final outcomes and terminatesafter exploring an outcome whose value is positive. If such
an outcome is not found, it explores all the final outcomes
a n dt h e nt e r m i n a t e s .
Strategy18:Oneﬁnaloutcome This strategy explores one
random final outcome and terminates.
Strategy19:Goalsettingwithforwardplanning: This strat-
egy starts by exploring the final outcomes and after findingan outcome with value equal to the maximum observablevalue of the reward distribution, it explores the path from
the corresponding immediate outcome to that final outcome
a n dt h e nt e r m i n a t e s .
Strategy20:Exploreonesub-tree This strategy explores all
the outcomes of one random sub-tree of the start node.
Strategy21:Exploreparentofthebestﬁnaloutcome This
strategy explores all the final outcomes until it finds a finaloutcomes whose value is equal to the maximum observablevalue of the reward distribution and then explores the parent
of that outcome.
Strategy 22: Explore one path This strategy explores one
random path from an immediate outcome to a final outcomea n dt h e nt e r m i n a t e s .
Strategy23:Twoﬁnaloutcomes This strategy explores two
randomly chosen final outcomes and then terminates.
Strategy 24: Explore the parent of a positive ﬁnal outcome
This strategy starts by exploring the final outcomes in a
random order and upon finding a final outcome with a
positive value, it explores the parent of that outcome andterminates. If no final outcome with a positive value is
found, it explores all final outcomes and then terminates.
Strategy25:Exploreallﬁnaloutcomesofarandomlychosen
sub-treeandtheparentofarandomlychosenobservedﬁnaloutcome This strategy explores all the final outcomes of a
randomly chosen sub-tree of the start node and then explores
the parent of a randomly chosen final outcome from the setof observed final outcomes.
A.4.5Strategythatexploresimmediateoutcomes
onthepathstothebestﬁnaloutcomes
The only strategy in this category explores all the final out-
comes and then explores the immediate outcomes of the best
BehaviorResearchMethods
among them. The model-free values and heuristics deci-
sion system and the satisficing and stopping decision systemtogether contribute at least 75% on average to the strategies
in this cluster.
Strategy 26: Explore immediate outcomes on the paths to
the best ﬁnal outcomes This strategy starts by exploring
all the final outcomes and then explores the immediateoutcomes of paths that lead to the best final outcomes.
A.4.6Strategythatexploresimmediaterewardson
thepathstothebestﬁnaloutcomeswithsatisﬁcing
The only strategy in this cluster behaves similarly to the
strategy in the previous category but differs in the fact that it
stops exploring the final outcomes after finding an outcomewhose value is equal to the maximum observable value ofthe reward distribution and then explores the immediate
outcome of that node. The model-free values and heuristics
decision system, the model-based metareasoning decisionsystem and the Pavlovian decision system in combination
contribute at least 75% on average to the strategy in this
cluster.
Strategy 27: Explore immediate outcomes on the paths
to the best ﬁnal outcomes with satisﬁcing This strategy
explores all the final outcomes randomly until it finds a finaloutcome whose value is equal to the maximum observable
value of the reward distribution and then explores the
corresponding immediate outcome of that final outcome.
A.4.7Forwardplanningstrategiessimilar
toBreadthFirstSearch
These strategies perform planning similar to Breadth First
Search, i.e., they first inspect all outcomes at the first
level, before inspecting all outcomes at the second level,
and so on. These strategies differ in the order in whichoutcomes at the same level are explored. The strategy 30 in
this category is a satisficing version of breadth-first search
which terminates upon finding a high value. The model-based metareasoning decision system alone contributesmore than 75% on average to the strategies in this cluster.
Strategy28:RandomizedBreadthFirstSearch This strategy
explores outcomes level by level, that is, it randomly
explores the outcomes that are one step away, then randomly
exploring the outcomes that are two steps away and so onuntil all nodes are clicked.Strategy 29: Breadth First Search This strategy behaves
similar to strategy 5 except that sibling outcomes areobserved consecutively.
Strategy 30: Satisﬁcing Breadth First Search This strategy
explores outcome in the breadth-first search order and
terminates upon finding the maximum value of the reward
distribution.
A.4.8Middle-outplanning
The only strategy in this category explores the center out-comes first, then inspects immediate outcomes and finally
inspects final outcomes. The model-free values and heuris-
tics decision system and the model-based metareasoningdecision system in combination contribute at least 75% onaverage to the strategy in this cluster.
Strategy31:Middle-outplanning This strategy explores the
center nodes first, then explores the immediate outcomes
and then the final outcomes.
A.4.9ForwardplanningstrategiessimilartoBest
FirstSearch
These strategies are similar to the Best First Search plan-
ning strategy. They differ in how they start and how theyterminate. The model-free values and heuristics decision sys-
tem and the Pavlovian decision system together contribute
at least 75% on average to the strategies in this cluster.
Strategy 32: Non-terminating Best First Search This strat-
egy starts by exploring the immediate outcomes andexplores an unobserved child of the observed outcome withthe highest value. If no outcome is observed, it chooses the
immediate outcome randomly.
Strategy 33: Best First search after exploring all immediate
outcomes This strategy explores all immediate outcomes
first and then follows the best-first strategy (Strategy 32) toexplore outcomes further.
Strategy 34: Satisﬁcing Best First Search after exploring all
immediateoutcomes This strategy is similar to strategy 33
but it stops exploring when an outcome with value equal to
the maximum observable value of the rewards distribution
is observed.
Strategy 35: Explore immediate outcomes and then sub-
trees This strategy explores all immediate outcomes in a
random order and then explores all the outcomes of their
sub-trees, exploring each sub-tree in a random order.
BehaviorResearchMethods
Strategy 36: Explore sub-trees of positive immediate out-
comes This strategy explores all the immediate outcomes
and then explores complete sub-trees of the immediate
outcomes with a positive value in a random order.
A.4.10Localsearchstrategies
These strategies focus on information about the sub-trees
and next/previous steps along the paths that have receivedthe most consideration so far. These strategies differ in
whether they prioritize sub-trees or paths and whether
earlier versus later outcomes have already been observed.The model-free values and heuristics decision system
and the satisficing and stopping decision system together
contribute at least 75% on average to the strategies in thiscluster.
Strategy37:ProgressiveDeepening This strategy is similar
to Depth First Search (i.e., it starts with exploring the nodeslevel by level, first observing the node and then its children)but instead of choosing a sibling of a final outcome, it
chooses to explore a path starting from the immediate
outcome again.
Strategy 38: Priority to explored ancestors This strategy
randomly selects the first outcome to explore. Based on
the outcomes explored, this strategy prioritizes exploringoutcomes that have a larger number of observed ancestors
than the number of explored successors.
Strategy 39: Priority to explored successors This strategy
randomly selects the first outcome to explore. Based on
the outcomes explored, this strategy prioritizes exploring
outcomes that have larger number of observed successorsthan the number of explored ancestors.
Strategy40:Prioritytoexploredimmediateancestors This
strategy is similar in behavior to strategy 38 but an outcomeis given higher priority if it has higher number of immediateancestors than immediate successors. If the number of
immediate ancestors and ancestors and successors is equal,
then it prioritizes total number of ancestors over totalnumber of successors.
Strategy41:Prioritytoexploredimmediatesuccessors This
strategy is similar in behavior to strategy 39 but an outcomeis given higher priority if it has higher number of immediate
successors than immediate ancestors. If the number of
immediate ancestors and ancestors and successors is equal,then it prioritizes total number of ancestors over totalnumber of successors.Strategy 42: Satisﬁcing Depth First Search This strategy’s
behavior is similar to that of Depth First Search but itterminates upon finding an outcome with value equal to the
maximum observable value of the reward distribution.
Strategy43:Leaveoutonesub-tree This strategy explores
all sub-trees of the start node except one in a random
order, exploring all the outcomes of a sub-tree in a randomorder and then moving on to the next, while exploring the
outcomes in each sub-tree also in a random order.
Strategy44:Exploreallsub-trees This strategy explores all
sub-trees of the start node in a random order, exploring
all the outcomes of a sub-tree in a random order and thenmoving on to the next, while exploring the outcomes in each
sub-tree also in a random order.
Strategy 45: Explore all sub-trees with satisﬁcing This
strategy’s behavior is similar to that of strategy 44 but
it terminates planning upon finding an outcome that has
value equal to the maximum observable value of the rewarddistribution.
Strategy 46: One complete sub-tree and ﬁnal outcomes of
othersub-trees This strategy explores one random sub-tree
of the start node and then explores final outcomes of the
other sub-trees.
Strategy47:Twocompletesub-treesandﬁnaloutcomesof
thelastsub-tree This strategy explores all the outcomes of
two sub-trees randomly, exploring one after the other and
then explores the final outcomes of the other sub-trees.
Strategy48:Exploreallsub-treesuntilthemaximumvalue
of the reward distribution is found and then explore the
center outcome of an unobserved immediate outcomeThis strategy explores the outcomes of sub-trees of the startnode in a random order and if it finds a final outcome with a
value equal to the maximum observable value of the reward
distribution, it explores the center outcome on the path fromthe final outcome to the corresponding immediate outcome
a n dt h e nt e r m i n a t e s .
A.4.11Frugalplanningstrategies
These strategies explore very little or not at all. They differ
in which outcomes they inspect and when they terminate.The model-free values and heuristics decision system and
the mental effort avoidance decision system together con-
tribute at least 75% on average to the strategies in this clus-ter.
BehaviorResearchMethods
Strategy 49: Myopic Impulsive This strategy explores one
randomly chosen immediate outcome and then terminates.
Strategy50:Noplanning This strategy does not plan at all
(i.e does not explore any outcomes).
Strategy 51: Explore immediate outcomes and ﬁnal out-
comes with satisﬁcing on a positive value This strategy
explores all the immediate outcomes until an outcome witha positive value is found and then explores the final out-
comes reachable from that immediate outcome and explores
them until a final outcome with positive value is found. If itdoesn’t find an immediate outcome with a positive value, it
terminates.
Strategy 52: Explore one center outcome This strategy
chooses a random path and then explores the center outcomeof that path.
A.4.12Myopicplanningstrategies
These strategies start with exploring immediate outcomes
and then explore the sub-trees of the best immediate out-
comes. They differ in how many immediate outcomes theyexplore, which nodes in the sub-tree they explore and
when they terminate. The model-free values and heuristics
decision system, the model-based metareasoning decisionsystem and the Pavlovian decision system in combinationcontribute at least 75% on average to the strategies in this
cluster.
Strategy53:Exploreallimmediateoutcomeswithsatisﬁcing
This strategy starts with exploring the immediate outcomesand terminates upon finding an immediate outcome which
has a positive value. If an immediate outcome with a
positive value is not found, it terminates after exploring allimmediate outcomes.
Strategy54:Exploreallimmediateoutcomes This strategy
explores all immediate outcomes and then terminates.
Strategy 55: Pruning of nodes with immediate negative
rewards and choosing actions with best long-term conse-quences This strategy first explores all the immediate out-
comes and then for immediate outcomes with a positive
value, it explores the corresponding final outcomes such thatsibling outcomes are explored consecutively.
Strategy 56: Explore positive immediate outcomes and
ﬁnal outcomes with satisﬁcing This strategy’s behavior is
similar to strategy 55 but instead of observing all the finaloutcomes, the strategy terminates after finding the outcome
with a value equal to the maximum observable value of thereward distribution.
Strategy57:Leaveoutoneimmediateoutcome This strat-
egy explores all immediate outcomes except for one. Theleft-out immediate outcome is randomly selected.
Strategy 58: Explore immediate and ﬁnal outcomes with
satisﬁcing on ﬁnding a large value This strategy starts
by exploring immediate outcomes. It first explores an
immediate outcome and then explores the final outcomes
of the corresponding immediate outcome. If it finds anoutcome with a value that is equal the maximum observable
value of the reward distribution while exploring the final
outcomes, it terminates.
Strategy 59: Explore immediate and ﬁnal outcomes with
positivesatisﬁcing This strategy starts by exploring imme-
diate outcomes. It first explores an immediate outcome andif it has a positive value, it explores the final outcomes ofthe corresponding immediate outcome and this pattern is
repeated. If it finds an outcome with a positive value while
exploring the final outcomes, it terminates.
Strategy 60: Explore the sub-tree which contains largest
ﬁnaloutcome This strategy explores all the final outcomes
in a random order and then explores all the outcomes ofsub-trees which contain the final outcome with the largest
value.
Strategy 61: Explore the immediate children of the best
immediate outcome This strategy explores all the imme-
diate outcomes and then explores a single child of the
immediate outcome with the largest value.
Strategy 62: Explore ﬁnal outcomes with preference for
nodes in the same sub-tree of the root This strategy
explores the final outcomes, exploring all the final outcomesof one sub-tree before moving on to the next, and terminateswhen it finds an outcome whose value is equal to the maxi-
mum observed value of the reward distribution.
A.4.13Othermiscellaneousstrategies
These strategies do not fit the definition of any of the above
categories and appear to have little in common. The model-free values and heuristics decision system, the model-based
metareasoning decision system and the satisficing and
stopping decision system in combination contribute at least75% on average to the strategies in this cluster.
BehaviorResearchMethods
Strategy63:InverserandomizedBreadthFirstSearch This
strategy explores all the outcomes level by level, exploringthe farthest ones and moving on to the closer ones,
that is, exploring the outcomes that are three steps
away (the farthest nodes), then exploring outcomes thatare two steps away and so on until all outcomes are
explored.
Strategy64:Exploreimmediateoutcomesofﬁnaloutcomes
This strategy observes all the final outcomes first and then,
in the decreasing order of the outcomes values,observes the
immediate outcomes.
Strategy65:Aversionofgoal-settingthatchoosesbetween
equally-good goals based on the immediate reward This
strategy explores all the final outcomes first and then
compares paths of the final outcomes with the largest values
level by level from the final outcomes to the immediateoutcomes.
Strategy 66: Goal-setting with comparison of equivalent
goals This strategy’s behavior is similar to that of strategy
65 but the outcomes on the paths to the immediate outcome
are explored in a random order.
Strategy 67: Best Final Outcome This strategy explores all
final outcomes in a random order and then terminates.
Strategy 68: Random planning This strategy explores
outcomes in such a way that there is an equal probability of
exploring a given outcome and terminating planning.
Strategy 69: Explore immediate outcomes of second best
nodes This strategy first explores all the final outcomes and
then explores immediate outcomes of the final outcomes
with second-largest value.
Strategy 70: Explore immediate outcomes and ﬁnal out-
comes This strategy first explores all the immediate out-
comes in a random order and then explores all the final
outcomes in a random order.
Strategy 71: Explore immediate outcomes and ﬁnal out-
comes with termination This strategy first explores all
immediate outcomes in a random order and then explores
the final outcomes outcomes in a random order. Whileexploring the final outcomes, if an outcome whose valueis equal to the maximum observed value of the reward
distribution is found, it terminates.Strategy72:Allimmediateoutcomesafterallﬁnaloutcomes
This strategy explores all the final outcomes first and thenexplores all the immediate outcomes.
Strategy 73: Explore immmediate, ﬁnal and center out-
comes This strategy first explores all the immediate out-
comes, then explores all the final outcomes and thenexplores all the center outcomes.
Strategy 74: Explore all center outcomes This strategy
explores all center outcomes and then terminates.
Strategy 75: Explore the path to the ﬁnal outcome with
largest value and satisﬁcing This strategy explores final
outcomes until it finds a final outcome whose value isequal to the maximum observable value of the rewarddistribution and then explores the outcomes on the path from
that outcome to the corresponding immediate outcome in
a random order. After exploring the immediate outcome, itterminates.
Strategy 76: Explore center outcomes and then ﬁnal out-
comes This strategy explores all the center outcomes in a
random order and then explores all the final outcomes in arandom order.
Strategy 77: Explore center outcomes and one of their
children This strategy explores one center outcome and
then explores one of its randomly-chosen children and then
repeats this process until all center nodes are explored.
Strategy78:Exploreﬁnaloutcomesandtheirparents This
strategy first explores final outcomes of a sub-tree of thestart node and then explores the parent of the explored
final outcomes and then repeats this process for all of the
sub-trees.
Strategy 79: Explore ﬁnal outcomes and their parents with
satisﬁcing This strategy’s behavior is similar to that of
strategy 78 but it terminates when it finds a final outcome
whose value is equal to the maximum observable value ofthe reward distribution.
A.4.14Identiﬁabilityandconﬁdence
To estimate how accurately and how confidently individualstrategies can be distinguished based on a single click
sequence, we compared how probable the click sequence
generated by one strategy is under other strategies comparedto its likelihood under the true strategy. Our procedure was
as follows: For each strategy described in Section 5,w e
BehaviorResearchMethods
generated 1000 click sequences by applying the strategy
to 1000 different instances of a given environment. Then,for each click sequence d(one simulation), we evaluated
whether our method correctly inferred the strategy that
generated it and computed the likelihoods with which eachof the 79 strategies would generate that click sequence (i.e.,
P(d|s
1),P(d|s2),···,P(d|s79)). We then compute the
relative likelihood of the click sequence under each strategyby dividing the likelihood of the click sequence under that
strategy by the maximum likelihood for that click sequence
under all the strategies (i.e.,
P(d|s1)
maxiP(d|si),P(d|s2)
maxiP(d|si),e t c . ) .T o
get a representative statistic of how likely click sequencesgenerated by one strategy are under other strategies, we
compute the average of the relative likelihoods obtained for
the 1000 simulations (i.e., ρ
j,k=1
1000·/summationtext1000
t=1P(dj,t|sk)
maxiP(dj,t|si)
where jis the strategy that generated the click sequence
andkis the strategy whose average relative likelihood is
being evaluated). In addition, we estimated how confident
our method is in each of its inferences by computingthe ratio of likelihood of the inferred strategy over thelikelihood of the second most likely strategy (LR
1,2). We ran
this procedure for two environments: the 3-step increasing
variance environment and the 5-step transfer task.
The results of this evaluation for the three step envi-
ronment and the five step environment are summarized in
Tables 6and 7, respectively. The first column reports the
strategy that generated the data. The second column reports
our method’s typical confidence in its inferences in terms
of the median of the 1000 LR 1,2ratios. The third column
reports how often the strategy that our method inferredwas identical to the strategy that had generated the data.
The fourth column lists other strategies that our method
considers to be possible alternative explanations becausethey are at least 66% as likely as the true strategy at least half
of the time. The last five columns show the top 5 average
relative likelihood ratios for click sequences generated froma given strategy for the 3-step environment with increasing
variance and the transfer task respectively. That is, for the
strategy in row j, the entries in the columns labelled “1”,
“2”,···, “5” are the values of ρ
j,k1,ρj,k2,···,ρj,k5for the
strategy with the highest, second highest, ···, tenth highest
average likelihood ratio for the click sequences generated bystrategy j, respectively. In each row, the average likelihood
ratio of the true strategy is highlighted in bold.
As you can see from the position of the bolded average
likelihood ratios in Table 6, the true strategy was always
the most likely explanation, on average. Furthermore, forall strategies except for one, our method’s inferences were
correct most of the time. Concretely, as shown in the
third column, the proportion of correct inferences rangedfrom 64.3% to 100% with an average of 91.8% with theexception of Strategy 13, for which the proportion of correct
inferences was only 14.6%.Except for Strategy 13, all strategies also had acceptable
average likelihood ratios of at least 0.720. Strategy 13,which explores two extra final outcomes after uncovering
a positive final outcome, had an average likelihood ratio
of only 0.410. This strategy was most often confused withStrategy 9, which explores one more final outcome after
uncovering a final outcome with a value greater than or
equal to the second largest value of the reward distribution.To see if these strategies are distinguishable on other
environments, we performed the same analysis on the 5-
step transfer task. The results of this analysis are reportedin Table 7. We see that Strategy 13 is identifiable on the
transfer task while all other strategies remain identifiable in
terms of having high relative likelihoods on average.
The LR
1,2ratio in the second column of Tables 6and
7shows how much more likely the inferred strategy tends
to be compared the second most likely strategy. And the
fourth column (“Similar Strategies”) shows what the secondmost likely strategies tend to be. We set our criterion for
what it means for the computational microscope to be
highly confident about the inferred strategy to LR
1,2=1.5
meaning that the inferred strategy is at least 1.5 times aslikely as the second most likely explanation. We found that
on the 3-step increasing variance environment, this was
the case for 82% of the strategies and on the transfer taskthis proportion was 80%, and the proportion of strategies
for which this was the case for at least one of the two
environments was 92%. The only exceptions are Strategies6, 14, 17, 36, 67 and 69. Those strategies jointly accounted
for 44% of all human click sequences. This is primarily
because this set includes the optimal strategy and manystrategies that are very similar to it. Yet, even though thosestrategies are very similar, the third columns of Tables 6and
7show that our computational microscope can nevertheless
correctly identify them most of the time. This suggests thatour computational microscope is usually able to confidently
infer which strategy best explains a given click sequence.
We also investigated how many other alternative answers
there are when the inferred strategy is less than 1.5 times
as likely as the most likely alternative. In the 3-step
increasing variance environment, 86% of the inferences thatour method was not highly confident about (i.e., LR
1,2<
1.5) had only one alternative explanation, 7% had 2 possible
alternative explanations, and 7% had 3 possible alternative
explanations. Thus the median number of alternativeexplanations for inferences that our method is not highly
confident about was only 1. In 57% of the cases the
alternative explanations were other strategies of the sametype as the inferred strategy and in the other 43% of the
cases there was at most one alternative strategy type. For
the 5-step increasing variance environment, the mediannumber of alternative explanations for such inferences was3; in 21.5% of the cases there was only 1 alternative
BehaviorResearchMethods
Table 6 Summary of the likelihood of click sequences on the 3-step increasing variance environment
Strategy LR 1,2 True proportion Similar strategies 12345
11 . 0 ·1060.866 None 0.885 0.105 0.039 0.039 0.026
2 7.7 0.822 None 0.866 0.311 0.311 0.208 0.109
3 1.0 0.985 4 0.988 0.988 0.303 0.106 0.038
4 1.0 0.993 3 0.991 0.991 0.303 0.092 0.038
5 2.3 0.932 None 0.955 0.224 0.213 0.099 0.099
6 1.0 0.751 14,17,18,79 0.829 0.729 0.471 0.294 0.256
7 1.1 0.643 67 0.723 0.269 0.265 0.244 0.185
8 3.4 0.976 None 0.982 0.281 0.133 0.087 0.073
9 1.8 0.652 None 0.754 0.447 0.141 0.126 0.117
10 2.5 0.598 None 0.755 0.372 0.136 0.101 0.093
11 2.5 1.0 None 1.000 0.400 0.068 0.059 0.059
12 1.8 0.773 None 0.835 0.451 0.108 0.101 0.096
13 1.6 0.149 None 0.410 0.291 0.272 0.229 0.175
14 1.0 0.887 6,17,18,79 0.920 0.735 0.474 0.284 0.250
15 7.2 ·1041.0 None 1.000 0.070 0.046 0.013 0.000
16 2860.0 0.989 None 0.991 0.229 0.176 0.176 0.176
17 1.0 0.914 6,14,18,23,79 0.940 0.481 0.470 0.463 0.265
18 26 1.0 None 1.000 0.485 0.222 0.222 0.222
19 2860.0 0.991 None 0.992 0.177 0.177 0.177 0.177
20 4.3 0.949 None 0.958 0.176 0.150 0.101 0.072
21 1.0 0.933 24 0.946 0.451 0.174 0.174 0.174
22 2860.0 1.0 None 1.000 0.241 0.018 0.000 0.000
23 2.4 0.928 None 0.947 0.229 0.170 0.158 0.158
24 5.0 0.953 None 0.962 0.456 0.046 0.014 0.010
25 2860.0 1.0 None 1.000 0.050 0.035 0.000 0.000
26 7.2 ·1041.0 None 1.000 0.174 0.103 0.059 0.025
27 572.0 0.989 None 0.991 0.176 0.176 0.176 0.176
28 2.4 ·1050.935 None 0.939 0.236 0.062 0.003 0.001
29 15.0 0.976 None 0.982 0.065 0.028 0.016 0.009
30 2.4 ·1050.968 None 0.971 0.222 0.020 0.016 0.001
31 2.4 ·1051.0 None 1.000 0.000 0.000 0.000 0.000
32 3.2 ·1070.999 None 0.998 0.247 0.067 0.011 0.007
33 18.0 0.997 None 0.995 0.241 0.238 0.028 0.004
34 1.0 ·1051.0 None 0.996 0.239 0.057 0.035 0.013
35 8.0 ·1050.999 60 0.999 0.112 0.001 0.001 0.000
36 1.0 0.986 None 0.988 0.373 0.132 0.132 0.132
37 9.3 0.994 None 0.994 0.015 0.015 0.005 0.001
38 1.2 0.814 None 0.863 0.201 0.136 0.055 0.017
39 1.4 0.825 None 0.870 0.187 0.139 0.053 0.023
40 6.0 0.997 None 0.995 0.196 0.012 0.010 0.008
41 6.0 0.985 None 0.984 0.207 0.015 0.015 0.012
42 50.8 1.0 None 1.000 0.226 0.032 0.015 0.015
43 3.1 0.627 None 0.720 0.220 0.161 0.161 0.146
44 2.9 0.801 None 0.836 0.180 0.165 0.150 0.146
45 2.6 0.677 None 0.765 0.164 0.164 0.157 0.152
46 548.7 0.729 None 0.789 0.168 0.156 0.066 0.052
47 245.0 0.907 None 0.934 0.163 0.031 0.030 0.023
48 5.7 0.693 None 0.777 0.227 0.166 0.111 0.083
49 1.0 1.0 53 1.000 0.502 0.019 0.000 0.000
BehaviorResearchMethods
Table 6 (continued)
Strategy LR 1,2 True proportion Similar strategies 12345
50 13.0 1.0 None 1.000 0.077 0.000 0.000 0.000
51 2860.0 1.0 None 1.000 0.242 0.219 0.132 0.132
52 52.0 1.0 None 1.000 0.019 0.000 0.000 0.000
53 1.0 1.0 49, 54, 57 1.000 0.488 0.262 0.250 0.132
54 2860.0 1.0 None 1.000 0.254 0.132 0.132 0.132
55 1.0 1.0 51, 56 1.000 0.614 0.255 0.132 0.132
56 1.0 1.0 51, 55 1.000 0.610 0.226 0.143 0.132
57 286.0 1.0 None 1.000 0.239 0.003 0.000 0.000
58 2.4 0.899 None 0.912 0.425 0.103 0.056 0.044
59 4.0 0.898 None 0.931 0.433 0.066 0.057 0.022
60 1.2 ·1050.964 None 0.975 0.374 0.039 0.000 0.000
61 2.6 ·1041.0 None 1.000 0.000 0.000 0.000 0.000
62 2.9 1.0 None 1.000 0.087 0.042 0.036 0.014
63 2.4 ·1050.994 None 0.990 0.093 0.004 0.002 0.000
64 1.5 ·1040.963 None 0.984 0.218 0.018 0.018 0.018
65 2.0 0.989 None 0.987 0.302 0.088 0.073 0.073
66 275.3 0.651 7, 69 0.795 0.328 0.032 0.032 0.019
67 1.0 0.965 None 0.970 0.806 0.274 0.238 0.174
68 6.8 ·10290.786 None 0.795 0.084 0.037 0.029 0.026
69 1.0 0.965 67 0.973 0.799 0.174 0.160 0.091
70 2.4 ·1050.985 None 0.986 0.255 0.020 0.007 0.003
71 20.6 0.859 None 0.912 0.244 0.143 0.102 0.048
72 2.4 ·1051.0 None 1.000 0.185 0.103 0.023 0.000
73 2.4 ·1051.0 None 1.000 0.000 0.000 0.000 0.000
74 2860.0 1.0 None 1.000 0.000 0.000 0.000 0.000
75 3872.8 0.984 None 0.988 0.237 0.177 0.177 0.177
76 2.4 ·1051.0 None 1.000 0.000 0.000 0.000 0.000
77 1.8 ·1051.0 None 1.000 0.000 0.000 0.000 0.000
78 2.2 ·1071.0 None 1.000 0.185 0.000 0.000 0.000
79 2.9 1.0 None 1.000 0.283 0.283 0.271 0.245
This table summarizes likelihood of click sequences generated using a given strategy and the average relative likelihoods of the generated click
sequences under the top 5 strategies on the 3-step increasing variance environment. Each row describes the results for a given strategy. The columnLR describes the median ratio of the likelihoods of the first-best strategy to the second-best strategy computed for each simulation separately.
The columns 1-5 describe the decreasing order of the average relative likelihoods under the top 5 strategies The numbers in bold represent the
average likelihood ratios of the strategy that the simulations were generated from. The column “True proportion” describes the proportion of clicksequences for which the true strategy was the most likely strategy. The column “Similar strategies” describes the strategies that the strategy wasconfused with on more than 20% of the click sequences
explanation, in 21.5% of the cases there were 2 alternative
explanations, and in 57% of the cases there were 3
alternative explanations. In 29% of the cases all alternative
strategies were of the same type as the inferred strategy;in 42% of the cases they included one additional strategytype, and in 29% of the cases they included two or more
alternative strategy types.
Overall, our findings suggest that most of the strategies
are highly identifiable in at least one of the environments
and that even when our method is uncertain about the strat-
egy, there is only a small number of alternative explanationsand those alternative explanations often instantiate the same
strategy type.
A.6Features
The strategies described in section 1 were implementedusing the 51 features described below. The features aregrouped using the decision-making factor they represent.These features are defined in terms of the nodes in the
Mouselab-MDP paradigm. All the features take a belief
BehaviorResearchMethods
Table 7 Summary of the likelihood of click sequences on the 5-step transfer task
Strategy LR 1,2 True proportion Similar strategies 12345
14 . 0 ·10311.0 None 1.000 0.000 0.000 0.000 0.000
24 . 6 ·10221.0 None 1.000 0.000 0.000 0.000 0.000
39 . 8 ·10300.994 None 0.994 0.019 0.000 0.000 0.000
41 . 2 ·10291.0 None 1.000 0.017 0.000 0.000 0.000
51 . 1 ·10221.0 None 1.000 0.000 0.000 0.000 0.000
6 1.0 0.987 14,17,18,19,27 0.989 0.848 0.621 0.530 0.333
76 . 7 ·1060.982 None 0.982 0.088 0.085 0.064 0.064
81 . 7 ·10131.0 None 1.000 0.000 0.000 0.000 0.000
91 . 7 ·1080.996 None 0.997 0.002 0.002 0.001 0.000
10 249.3 0.999 None 0.999 0.047 0.047 0.043 0.042
11 1.9 ·1041.0 None 1.000 0.003 0.003 0.002 0.002
12 91.0 0.997 None 0.997 0.048 0.011 0.005 0.000
13 1.9 ·1040.936 None 0.939 0.024 0.014 0.013 0.012
14 1.0 0.982 6,17,19,27 0.986 0.664 0.646 0.509 0.225
15 1.0 0.991 66,67,69,75 0.994 0.523 0.521 0.512 0.363
16 1.1 ·10150.996 None 0.996 0.007 0.000 0.000 0.000
17 1.0 0.993 6,14,18,19,23,24 0.994 0.801 0.727 0.320 0.295
18 1.0 0.992 6,17,24 0.994 0.728 0.613 0.209 0.196
19 1.0 0.995 6,14,17,27 0.995 0.857 0.551 0.500 0.269
20 753.6 0.999 None 0.998 0.046 0.005 0.001 0.000
21 9.6 ·10101.0 None 1.000 0.003 0.000 0.000 0.000
22 1.7 ·10131.0 None 1.000 0.000 0.000 0.000 0.000
23 105.1 0.99 None 0.991 0.276 0.222 0.201 0.164
24 1.0 0.985 17,18,23 0.987 0.807 0.585 0.217 0.129
25 2.0 ·10101.0 None 1.000 0.001 0.000 0.000 0.000
26 1.6 ·10101.0 None 1.000 0.106 0.093 0.013 0.004
27 1.0 0.994 6,14,19 0.995 0.662 0.531 0.531 0.155
28 9.4 ·10151.0 None 1.000 0.000 0.000 0.000 0.000
29 5.6 ·10101.0 None 1.000 0.000 0.000 0.000 0.000
30 6.3 ·1061.0 None 1.000 0.000 0.000 0.000 0.000
31 3.0 ·10171.0 None 1.000 0.000 0.000 0.000 0.000
32 1.1 ·10311.0 None 1.000 0.000 0.000 0.000 0.000
33 1.4 0.979 None 0.936 0.578 0.000 0.000 0.000
34 1.4 0.949 None 0.958 0.575 0.000 0.000 0.000
35 9.6 ·10281.0 None 1.000 0.000 0.000 0.000 0.000
36 1.0 1.0 54 1.000 1.000 0.128 0.128 0.000
37 4.1 ·10221.0 None 1.000 0.000 0.000 0.000 0.000
38 1.8 ·10191.0 None 1.000 0.000 0.000 0.000 0.000
39 2.1 ·10251.0 None 1.000 0.000 0.000 0.000 0.000
40 3.4 ·10241.0 None 1.000 0.000 0.000 0.000 0.000
41 4.4 ·10290.997 None 0.997 0.003 0.000 0.000 0.000
42 8.4 ·1061.0 None 1.000 0.001 0.000 0.000 0.000
43 1047.9 0.971 None 0.978 0.050 0.010 0.006 0.005
44 379.7 0.859 None 0.899 0.303 0.008 0.002 0.000
45 6.1 0.906 None 0.938 0.289 0.022 0.022 0.019
46 2.3 ·10110.934 None 0.949 0.055 0.047 0.007 0.002
47 2.9 ·1060.989 None 0.990 0.016 0.011 0.009 0.008
48 3.5 ·10241.0 None 1.000 0.000 0.000 0.000 0.000
49 376.0 1.0 None 0.982 0.251 0.000 0.000 0.000
BehaviorResearchMethods
Table 7 (continued)
Strategy LR 1,2 True proportion Similar strategies 12345
50 34.0 1.0 None 1.000 0.029 0.000 0.000 0.000
51 68.5 0.997 None 0.997 0.422 0.026 0.013 0.008
52 2.2 ·1071.0 None 1.000 0.000 0.000 0.000 0.000
53 2.9 ·1060.985 None 0.963 0.246 0.015 0.000 0.000
54 1.0 1.0 36 1.000 1.000 0.128 0.128 0.000
55 1.8 ·10211.0 None 1.000 0.128 0.128 0.128 0.000
56 3.4 ·1090.995 None 0.997 0.128 0.128 0.128 0.004
57 597.8 1.0 None 1.000 0.000 0.000 0.000 0.000
58 1.0 ·1040.896 None 0.918 0.166 0.031 0.018 0.004
59 25.7 0.974 None 0.980 0.158 0.006 0.005 0.002
60 8.6 ·10121.0 None 1.000 0.002 0.000 0.000 0.000
61 4.6 ·10101.0 None 1.000 0.000 0.000 0.000 0.000
62 43.0 0.994 None 0.995 0.425 0.027 0.005 0.001
63 3.0 ·10171.0 None 1.000 0.000 0.000 0.000 0.000
64 1.3 ·10221.0 None 1.000 0.000 0.000 0.000 0.000
65 1.6 ·10140.999 None 0.999 0.013 0.004 0.001 0.001
66 1.0 0.966 15,67,69,75 0.973 0.528 0.518 0.513 0.353
67 1.0 0.996 15,66,69,75 0.997 0.697 0.652 0.522 0.512
68 9.3 ·10920.943 None 0.946 0.025 0.015 0.014 0.012
69 1.0 0.991 15,66,67,75 0.993 0.699 0.361 0.352 0.352
70 5.3 ·1060.999 None 0.999 0.001 0.000 0.000 0.000
71 1.4 ·10100.995 None 0.996 0.004 0.002 0.002 0.002
72 1.1 ·10161.0 None 1.000 0.001 0.000 0.000 0.000
73 3.0 ·10171.0 None 1.000 0.000 0.000 0.000 0.000
74 1.1 ·10111.0 None 1.000 0.000 0.000 0.000 0.000
75 1.0 0.997 15,66,67,69 0.997 0.651 0.516 0.511 0.351
76 6.7 ·10161.0 None 1.000 0.000 0.000 0.000 0.000
77 5.6 ·10151.0 None 1.000 0.000 0.000 0.000 0.000
78 5.3 ·10271.0 None 1.000 0.000 0.000 0.000 0.000
79 5.2 ·10141.0 None 1.000 0.076 0.076 0.072 0.072
This table summarizes the likelihood of click sequences generated using a given strategy and the average relative likelihoods of the generated
click sequences under the top 5 strategies on the 5-step transfer task
state, computation pair as input. We define the considered
node to be the entity whose value is found out after clicking.The value of all the features is 0 for termination unlessotherwise specified.
A.6.1Mentaleﬀortavoidance
Feature 1: “Termination Constant”: The value of this
feature is 1 for all clicks and 0 for the termination operationin all belief states.A.6.2Model-basedmetareasoningfeatures
These features capture uncertainty about the values of theunobserved nodes. Uncertainty is defined as the standard
deviation of the values of the distribution. The following
features capture uncertainty:Feature 2: “Uncertainty” : The value of this feature for a
click in a given belief state is the uncertainty in the value of
the considered node.Feature 3: “Max Uncertainty” : The value of this feature
for a click in a given belief state is the is the maximum
BehaviorResearchMethods
uncertainty in return for the current trial from all the paths
that the considered node lies on.Feature 4: “Successor Uncertainty” :T h ev a l u eo f
this feature for a click in a given belief state is the
total uncertainty in the values of all the successors ofthe considered node on the current trial.
Feature 5: “Trial level standard deviation” :T h ev a l u eo f
this feature for a click is the uncertainty in the value of theconsidered node as estimated across all trials attempted so
far by the agent.
Feature 6: “Current trial level standard deviation” :T h e
value of this feature for a click in a given belief state is theuncertainty in the value of nodes at the same depth as the
considered node as estimated for the current trial.
Feature 7: “Does the node lie on the second mostpromising path?” : The value of this feature for a click in a
given belief state is 1 if the considered node lies on the path
which has the second highest expected return for the currenttrial, and 0 otherwise.
A.6.3PavlovianFeatures
These features are based on greedy maximization. Pavlovianbehavior is captured by the following features:Feature 8: “Best expected value” : The value of this feature
for a click in a given belief state is the best expected return
for a path in the current trial among all the paths that the
considered node lies on.F e a t u r e9 :“ B e s tl a r g e s tv a l u e ” : The value of this feature
for a click in a given belief state is the maximum value
observed among all the paths that the considered node lieson.
Feature 10: “Does the node lie on the most promising
path?” : The value of this feature for a click in a given belief
state is 1 if the considered node lies on the path with thehighest expected return for the current trial, and 0 otherwise.
Feature 11: “Value of the max expected return” :T h e
value of this feature for all clicks in a given belief state is themaximum expected return from all paths in the current trial.
Feature 12: “Does a successor node have a maximum
value?” : The value of this feature for a click in a given
belief state is 1 if any of the considered node’s observed
successors in the current trial has a value which is the
maximum possible value for the reward distribution, and 0otherwise.Feature 13: “Maximum value of a successor” :T h ev a l u e
of this feature for a click in a given belief state is the
maximum value that has been observed among all thesuccessors of the considered node in the current trial.
Feature 14: “Maximum value of an immediate succes-
sor” : The value of this feature for a click in a given belief
state is the maximum value that has been observed amongall the immediate successors of the considered node in the
current trial.Feature 15: “Value of the parent node” :T h ev a l u eo f
this feature for a click in a given belief state is the value
of the considered node’s parent if the parent node has beenobserved, and 0 otherwise.
Pruningfeatures These features are designed to capture the
idea of pruning branches (Huys et al., 2012 ). The value for
these features for all clicks is -1 if the maximum expected
loss that can be incurred in the current belief state is worse
than the pruning threshold and 0 otherwise. We considerfeatures with different pruning thresholds: -48, -24, -8 and0 (features 16-19). In addition, we consider the following
features:
Feature 20: “Soft Pruning” : The value of this feature for a
clicks is the maximum expected loss that can be incurred in
a given belief state from all paths that the considered node
lies on.Feature 21: “Is the previous observed node a successor
and has negative value” : The value of this feature for a
click in a given belief state is 1 if the last observed node inthe current trial is a child of the considered node and has anegative value, and 0 otherwise.
A.6.4Satisﬁcingandstopppingfeatures
Satisﬁcing features These features determine when the
planning satisfices (Simon, 1956 ). The value for these
features is -1 for termination if the maximum expectedreturn for the current trial is greater than the satisficing
threshold. We consider features with different satisficing
thresholds: 0, 8, 16, 24, 32, 40 and 48 (features 22-28). Inaddition, we consider the following 2 features:
Feature 29: “Soft Satisficing” : The value of this feature for
all clicks in a given belief state is the maximum return thatcan be expected on the current trial from all paths that theconsidered node lies on.
StoppingCriteria These features have same value for all the
clicks and a different value for termination.Feature 30: “Are all max paths observed?” :T h ev a l u e
of this feature is -1 for all clicks and 0 for termination
action in all belief states if all the paths path leading to afinal outcome, which has the maximum value among the
observed final outcomes, has been observed in the current
trial and 0 otherwise.Feature 31: “Is a max path observed?” : The value of this
feature is -1 for all clicks in all belief states if any path
leading to the node, which has the maximum value possible
for the reward distribution, has been observed in the currenttrial and 0 otherwise.
BehaviorResearchMethods
Feature 32: “Is a positive node observed?” :T h ev a l u eo f
this feature is -1 for all clicks in all belief states if a nodewith a positive value has been observed in the current trial
and 0 otherwise.
Feature 33: “Is the previous observed node maximal?” :
The value of this feature is -1 for all clicks if the last
observed node in the current trial has the maximum value
possible for the reward distribution and 0 otherwise.Feature 34: “Is a complete path observed?” :T h ev a l u eo f
this feature is -1 for all nodes in all belief states if at least
one path has been completely observed from immediateoutcome to final outcome, and 0 otherwise.Feature 35: “All final outcomes observed?” :T h ev a l u e
of this feature is -1 for all clicks in all belief states if all
final outcomes have been observed in the current trial and 0otherwise.
Feature 36: “Are all immediate outcomes observed?” :
The value of this feature is -1 for all clicks in all belief statesif all immediate outcomes have been observed in the current
trial and 0 otherwise.
Feature 37: “Are final outcomes of positive immediateoutcomes observed?” : The value of this feature is -1 for all
clicks in all belief states if all the final outcomes that can
be reached from the positive observed immediate outcomes
have been observed, and 0 otherwise.
A.6.5Model-freevaluesandheuristicsfeatures
Relational features The values of these features for a
considered node are dependent on its neighboring nodes.
Feature 38: “Ancestor count” : The value of this feature for
a click in a given belief state is the number of ancestors ofthe considered node that have been observed in the current
trial.
Feature 39: “Depth Count” : The value of this feature for
a click in a given belief state is the number of times thatany node at the same depth as the considered node has been
observed in the current trial.
Feature 40: “Is the node a final outcome and has apositive ancestor?” : The value of this feature for a click
in a given belief state is 1 if the considered node is a final
outcome and it has an observed ancestor with a positivevalue in the current trial and 0 otherwise.
Feature 41: “Immediate successor count” :T h ev a l u eo f
this feature for a click in a given belief state is the numberof children of the considered node that have been observedin the current trial.
Feature 42: “Is parent observed?” : The value of this
feature for a click in a given belief state is 1 if the parentnode of the considered node has been observed, and 0
otherwise.Feature 43: “Successor Count” : The value of this feature
for a click in a given belief state is the number of observedsuccessors of the considered node for the current trial.
Feature 44: “Squared Successor Count” :T h ev a l u eo f
this feature for a click in a given belief state is the square ofthe number of observed successors of the considered node
for the current trial.
Feature 45: “Siblings Count” : The value of this feature for
a click in a given belief state is the number of siblings of the
considered node that have been observed in the current trial.
Feature 46: “Minimum number of observed nodes onbranch” : The value of this feature for a click in a given
belief state is the minimum number of nodes observed on
all the branches containing the considered node.
Feature 47: ”Is the previous observed node a succes-sor?” : The value of this feature for a click in a given belief
state is 1 if the last observed node in the current trial is one
of the successors of the considered node, and 0 otherwise.
Structuralfeatures The values of these features are depen-
dent no the task structure.Feature 48: “Depth” : The value of this feature for a click
in a given belief state is the distance of the considered node
from the starting position.Feature 49: “Is the node an immediate outcome?” :T h e
value of this feature for a click in a given belief state is
1 if the considered node in an immediate outcome and 0
otherwise.Feature 50: “Is the node a final outcome?” :The value of
this feature for a click in a given belief state is 1 if the
considered node is a final outcome and 0 otherwise.Feature 51: “Observed height” : The value of this feature
for a click in a given belief state is the length of the
maximum observed path to a final outcome starting fromthe considered node.
Open Access This article is licensed under a Creative Commons
Attribution 4.0 International License, which permits use, sharing,adaptation, distribution and reproduction in any medium or format, aslong as you give appropriate credit to the original author(s) and the
source, provide a link to the Creative Commons licence, and indicate
if changes were made. The images or other third party material in thisarticle are included in the article’s Creative Commons licence, unlessindicated otherwise in a credit line to the material. If material is notincluded in the article’s Creative Commons licence and your intendeduse is not permitted by statutory regulation or exceeds the permitteduse, you will need to obtain permission directly from the copyrightholder. To view a copy of this licence, visit http://creativecommons.
org/licenses/by/4.0/ .
BehaviorResearchMethods
References
Ackerman, R., & Thompson, V . A. (2017). Meta-reasoning: Monitor-
ing and control of thinking and reasoning. Trends in Cognitive
Sciences ,21(8), 607–617.
Akaike, H. (1974). A new look at the statistical model identification.
IEEE Transactions on Automatic Control ,19(6), 716–723.
Balle, M. (2002). La loi du moindre effort mental: Les representations
mentales. Sciences Humaines (Auxerre) , (128), 36–39.
Bergstra, J., Yamins, D., & Cox, D. D. (2013). Making a science
of model search: Hyperparameter optimization in hundreds of
dimensions for vision architectures. In Proceedings of the 30th
International Conference on International Conference on MachineLearning - Volume 28, ICML’13 , (pp. I-115-I-123): JMLR.org.
Callaway, F., Jain, Y . R., van Opheusden, B., Krueger, P. M., Das,
P . , I w a m a , G . , ..., L i e d e r , F . ( 2022a). Leveraging artificial
intelligence to improve people’s planning strategies. Proceedings
of the National Academy of Sciences ,119(12), e2117432119.
https://doi.org/10.1073/pnas.2117432119
Callaway, F., van Opheusden, B., Gul, S., Das, P., Krueger,
P. M., Griffiths, T. L., & Lieder, F. (2022b). Rational use ofcognitive resources in human planning. Nature Human Behaviour .
https://doi.org/10.1038/s41562-022-01332-8 .
Callaway, F., Lieder, F., Das, P., Gul, S., Krueger, P. M., & Griffiths,
T. L. (2018). A resource-rational analysis of human planning.InProceedings of the 40th Annual Conference of the Cognitive
Science Society .
Callaway, F., Lieder, F., Krueger, P. M., & Griffiths, T. L. (2017).
Mouselab-MDP: A new paradigm for tracing how people plan. In
The 3rd multidisciplinary conference on reinforcement learningand decision making .
Costa-Gomes, M. A., Crawford, V . P., & Broseta, B. (2001). Cognition
and behavior in normal-form games: An experimental study.Econometrica ,69(5), 1193–1235.
Costa-Gomes, M. A., & Crawford, V . P. (2006). Cognition and
behavior in two-person guessing games: An experimental study.American Economic Review ,96(5), 1737–1768.
Crawford, V . P. (2008). Look-ups as the Windows of the Strategic Soul.
In Caplin, A., & Schotter, A. (Eds.) The foundations of Positive
and normative Economics: A handbook , (pp. 249–80): Oxford
University Press Oxford.
Cushman, F., & Morris, A. (2015). Habitual control of goal selection
in humans. Proceedings of the National Academy of Sciences ,
112(45), 13817–13822.
Das, P., Callaway, F., Griffiths, T. L., & Lieder, F. (2019). Remediating
cognitive decline with cognitive tutors.
Daw, N. D. (2018). Are we of two minds?. Nature Neuroscience ,
21(11), 1497–1499.
Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain. Neuron ,
80(2), 312–325.
Ericsson, K. A., Krampe, R. T., & Tesch-R ¨omer, C. (1993). The role
of deliberate practice in the acquisition of expert performance.Psychological Review ,100(3), 363.
Ford, J. K., Schmitt, N., Schechtman, S. L., Hults, B. M., & Doherty,
M. L. (1989). Process tracing methods: Contributions, problems,
and neglected research questions. Organizational Behavior and
Human Decision Processes ,43(1), 75–117.
Forney, G. D. (1973). The Viterbi algorithm. Proceedings of the IEEE ,
61(3), 268–278.
Gigerenzer, G., & Selten, R. (2002). Bounded Rationality: The
Adaptive Toolbox . Cambridge: MIT Press.
Gershman, S. J., Blei, D. M., & Niv, Y . (2010). Context, learning, and
extinction. Psychological Review ,117(1), 197.
Griffiths, T. L., Callaway, F., Chang, M. B., Grant, E., Krueger,
P. M., & Lieder, F. (2019). Doing more with less: meta-reasoningand meta-learning in humans and machines. Current Opinion in
Behavioral Sciences ,29, 24–30.
Gureckis, T. M., Martin, J., McDonnell, J., Alexander S., Markant,
D . , C o e n e n , A . , ..., C h a n , P . ( 2016). psiTurk: An open-source
framework for conducting replicable behavioral experimentsonline. Behavior research methods ,48(3), 829–842.
Measuring and modelling how people learn how to plan and how
people adapt their planning strategies the to structure of the
environment (2021)
Hebb, D. O. (1949). The organization of behavior: a neuropsycholog-
ical theory . Wiley.
Hubel, D. H., & Wiesel, T. N. (1970). The period of susceptibility to
the physiological effects of unilateral eye closure in kittens. The
Journal of Physiology ,206(2), 419–436.
Huys, Q. J. M., Eshel, N., O’Nions, E., Sheridan, L., Dayan, P., &
Roiser, J. P. (2012). Bonsai trees in your head: how the pavloviansystem sculpts goal-directed choices by pruning decision trees.PLoS Computational Biology ,8(3), e1002410.
Huys, Q. J. M., Lally, N., Faulkner, P., Eshel, N., Seifritz, E.,
G e r s h m a n ,S .J . ,...,R o i s e r ,J .P .( 2015). Interplay of approximate
planning strategies. Proceedings of the National Academy of
Sciences ,112(10), 3098–3103.
JASP Team (2021). JASP (Version 0.16)[Computer software]. https://
jasp-stats.org/ .
Jain, Y . R., Gupta, S., Rakesh, V ., Dayan, P., Callaway, F., & Lieder,
F. (2019). How do people learn how to plan? In Conference
on Cognitive Computational Neuroscience (CCN 2019) , (pp. 826–
829).
Johnson, E. J., Payne, J. W., Bettman, J. R., & Schkade, D. A. (1989).
Monitoring information processing and decisions: The Mouselabsystem . Technical Report. Duke Univ Durham NC Center for
Decision Studies.
Keramati, M., Smittenaar, P., Dolan, R. J., & Dayan, P. (2016).
Adaptive integration of habits into depth-limited planning defines
a habitual-goal–directed spectrum. Proceedings of the National
Academy of Sciences ,113(45), 12868–12873.
Kool, W., McGuire, J. T., Rosen, Z. B., & Botvinick, M. M. (2010).
Decision making and the avoidance of cognitive demand. Journal
of Experimental Psychology: General ,139(4), 665.
Krueger, P. M., Lieder, F., & Griffiths, T. L. (2017). Enhancing
metacognitive reinforcement learning using reward structures andfeedback. In Proceedings of the 39th Annual Conference of the
Cognitive Science Society : Cognitive Science Society.
Lieder, F., & Griffiths, T. L. (2017). Strategy selection as rational metar-
easoning. Psychological Review ,124(6), 762–794. https://doi.org/
10.1037/rev0000075
Lee, M. D., Gluck, K. A., & Walsh, M. M. (2019). Understanding the
complexity of simple decisions: Modeling multiple behaviors andswitching strategies. Decision ,
6(4), 335.
Lee, M. D., & Gluck, K. A. (2021). Modeling strategy switches
in multi-attribute decision making. Computational Brain &
Behavior ,4(2), 148–163.
Lieder, F. (2018a). Beyond bounded rationality: Reverse-engineering
and enhancing human intelligence . Berkeley: University of
California.
Lieder, F. (2018b). Developing an intelligent system that teaches
people optimal cognitive strategies. In Lieder, F. (Ed.) Beyond
bounded rationality: Reverse-engineering and enhancing humanintelligence . Berkeley: University of California.
Lieder, F., Shenhav, A., Musslick, S., & Griffiths, T. L. (2018c).
Rational metareasoning and the plasticity of cognitive control.PLoS Computational Biology ,14(4), e1006043.
Lohse, G. L., & Johnson, E. J. (1996). A comparison of two process
tracing methods for choice tasks. Organizational Behavior and
Human Decision Processes ,68(1), 28–43.
BehaviorResearchMethods
Mercado, I., E. (2008). Neural and cognitive plasticity: From maps to
minds. Psychological Bulletin ,134(1), 109.
Morey, R. D., & Rouder, J. N. (2018). BayesFactor: computation of
bayes factors for common designs. R package version 0.9.12-4.2.
https://CRAN.R-project.org/package=BayesFactor
Patzelt, E. H., Kool, W., Millner, A. J., & Gershman, S. J. (2019).
The transdiagnostic structure of mental effort avoidance. Scientific
Reports ,9(1), 1–10.
Payne, J. W. (1976). Task complexity and contingent processing in
decision making: An information search and protocol analysis.Organizational Behavior and Human Performance ,16(2), 366–
387.
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988). Adaptive strategy
selection in decision making. Journal of Experimental Psychol-
ogy: Learning, Memory, and Cognition ,14(3), 534.
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive
decision maker . Cambridge university press.
Piaget, J. (1971). The theory of stages in cognitive development .
McGraw-Hill.
Rushworth, M. F. S., Kolling, N., Sallet, J., & Mars, R. B. (2012).
Valuation and decision-making in frontal cortex: one or manyserial or parallel systems?. Current Opinion in Neurobiology ,
22(6), 946–955.
Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern
approach , (3rd ed.). Harlow: Pearson Education Limited.
Schulte-Mecklenbeck, M., Kuehberger, A., & Johnson, J. G. (2011).
Visiting the decision factory: Observing cognition with mouse-labweb and other information acquisition methods. In Schulte-Mecklenbeck, M., K¨ uhberger, A., & Johnson, J. G. (Eds.) Ahandbook of process tracing methods for decision research ,
(pp. 37–58): Psychology Press.
Schulte-Mecklenbeck, M., K¨ uhberger, A., & Johnson, J. G. (2019). A
handbook of process tracing methods . Routledge.
Schwarz, G. et al. (1978). Estimating the dimension of a model. The
Annals of Statistics ,6(2), 461–464.
Scheibehenne, B., Rieskamp, J., & Wagenmakers, E.-J. (2013). Testing
adaptive toolbox models: A Bayesian hierarchical approach.Psychological review ,120(1), 39.
Shrager, J., & Siegler, R. S. (1998). SCADS: A model of children’s
strategy choices and strategy discoveries. Psychological Science ,
9(5), 405–410. https://doi.org/10.1111/1467-9280.00076
Simon, H. A. (1956). Rational choice and the structure of the environ-
ment. Psychological review ,63(2), 129.
Simon, H. A. (1955). A behavioral model of rational choice. The
Quarterly Journal of Economics ,69(1), 99–118.
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An
introduction . MIT press.
van der Meer, M., Kurth-Nelson, Z., & Redish, A. D. (2012). Informa-
tion processing in decision-making systems. The Neuroscientist ,
18(4), 342–359.
van Lehn, K. (1996). Cognitive skill acquisition.
Annual Review of
Psychology ,47(1), 513–539.
Ward, Jr., J. H. (1963). Hierarchical grouping to optimize an objective
function. Journal of the American statistical association ,58(301),
236–244.
Publisher’s note Springer Nature remains neutral with regard to
jurisdictional claims in published maps and institutional affiliations.

Jain
Y
R
Callaway
F
Griffiths
T
L
Dayan
P
He
R
Krueger
P
M
& Lieder
F