Is state-dependent valuation more adaptive than simpler rules?.

Is state-dependent valuation more adaptive than simpler rules?
==============================================================

Abstract
--------

[McNamara et al. (2012)](#bib0030) claim to provide an explanation of certain systematic deviations from rational behavior using a mechanism that could arise through natural selection. We provide an arguably much simpler mechanism in terms of computational limitations, that performs better in the environment described by [McNamara et al. (2012)](#bib0030). To argue convincingly that animals’ use of state-dependent valuation is adaptive and is likely to be selected for by natural selection, one must argue that, in some sense, it is a better approach than the simple strategies that we propose.

Introduction
------------

Although much animal behavior can be understood as rational, in the sense of making a best response in all situations, some systematic deviations from rationality have been observed. For example, Marsh et al. (2004) presented starlings with two potential food sources, one which had provided food during “tough times”, when the birds had been kept at low weight, while other had provided food during “good times”, when the birds were well fed. They showed that the starlings preferred the food source that had fed them during the tough times, even when that source had a longer delay to food than the other source. Similar behavior was also observed in fish and desert locusts (Aw et al., 2009, Pompilio et al., 2006).

McNamara et al. (2012) claim to provide an explanation of this behavior using a mechanism that could arise through natural selection. They provide an abstract model of the bird-feeding setting where a decision maker can choose either a “risky” action or a “safe” action. They also provide a mechanism that takes internal state into account and can lead to good results (where, in the example above, the internal state could include the fitness of each source). However, as we observe, for the particular parameters used in their model, there is a *much* better (in the sense of getting a higher survival probability) and much simpler approach than their mechanism that does not take the internal state into account: simply playing safe all the time. It is hard to see how the mechanism proposed by McNamara et al. could arise in the model that they use by natural selection; the simpler mechanism would almost surely arise instead.

The fact that always playing safe does well depends on the particular parameter settings used by McNamara et al. Playing safe would not be a good idea for other parameter settings. However, we show that a simple 2-state automaton that more or less plays according to what it last got also does quite well. It does significantly better than the McNamara et al. mechanism, and does well in a wide variety of settings. Although our automaton also takes internal state into account (the internal state keeps track of the payoff at the last step), it does so in a minimal way, which does not suffice to explain the irrational behavior observed.

It seems to us that to argue convincingly that the type of mechanism proposed by McNamara et al. is adaptive and is likely to be selected for by natural selection, and thus explains animals’ use of state-dependent valuation, then one must argue that, in some sense, it is a better approach than the simple strategies that we propose. Now it could be that the simple strategies we consider do not work so well in a somewhat more complicated setting, and in that setting, taking the McNamara et al.'s approach does indeed do better. However, such a setting should be demonstrated; it does not seem easy to do so. In any case, at a minimum, these observations suggest that McNamara et al.'s explanation for the use of state-dependent strategies is incomplete.

We should add that we are very sympathetic to the general approach taken by McNamara et al., although our motivation has come more from the work of Wilson (2015) and Halpern et al., 2012, Halpern et al., 2014, which tries to explain seemingly irrational behavior, this time on the part of humans, in an appropriate model. That work assumes that people are resource-bounded, which is captured by modeling people as finite-state automata, and argues that an optimal (or close to optimal) finite-state automaton will exhibit some of the “irrational” behavior that we observe in people. (The 2-state automaton that we mentioned above is in fact a special case of a more general family of automata considered in Halpern et al. (2012); see Section 3.3.) We believe that taking computational limitations seriously might be a useful approach in understanding animal behavior, and may explain at least some apparently irrational behavior.

The rest of this paper is organized as follows. In Section 2, we review the model used by McNamara et al. (2012) and compare it to that of Halpern et al. (2012). In Section 3, we describe four strategies that an agent can use in the McNamara et al. model, under the assumption that the agent knows which action is the risky action and which is the safe action. One is the strategy used by McNamara et al.; another is a simplification of the strategy that we considered in our work; the remaining two are baseline strategies. In Section 4, we evaluate the strategies under various settings of the model parameters. In Section 5, we consider what happens if the agent does not know which action is risky and which is safe and, more generally, the issue of learning. We conclude in Section 6.

Section snippets
----------------

The model
---------

McNamara et al. (2012) assume that agents live at most one year, and that each year is divided into two periods, winter and summer. Animals can starve to death during a winter if they do not find enough food. If an agent survives the winter, then it reproduces over the summer, and reproductive success is independent of the winter behavior.

A “winter” is a series of *T* discrete time steps. At any given time, the environment is in one of two states: *G* (good) or *S* (sparse); the state of the

Four strategies
---------------

In this section, we describe four strategies that an agent can use in the McNamara et al. model. We will be interested in the probability that an agent survives a “winter” period using each of these strategies. Note that the higher this probability is, the greater the probability that this strategy will emerge as the dominant strategy in an evolutionary process.

Evaluating the strategies
-------------------------

In this section we evaluate the four strategies discussed in the previous section under various settings of the model parameters. We calculate the survival probability of an agent using the strategy over a winter of length *T* = 500 steps by simulating 100,000 winters and looking at the ratio of runs in which the agent survived. We initialize the environment to the sparse state and the resource level to the maximum of 10.

Discussion
----------

While we show that the automaton strategy is better than the value strategy in many scenarios, as we discussed before, the value strategy (or, more generally, state-dependent strategies), seem to be what animals actually use in the scenarios studied in previous papers. We now discuss some possible explanations for this.

Conclusion
----------

Our results show that some very simple strategies seem to consistently outperform the value strategy. This gap grows as the task of surviving becomes more challenging (either because “winter” is longer or the rewards are not as high). This at least suggests that the model considered by McNamara et al. (2012) is not sufficient to explain the evolution of the value strategy in animals. McNamara et al. claim that “[w]hen an animal lacks knowledge of the environment it faces, it may be adaptive for

Acknowledgements
----------------

This work was supported in part by NSF grants IIS-0911036 and CCF-1214844, by ARO grant W911NF-14-1-0017, by Simons Foundation grant #315783, and by the Multidisciplinary University Research Initiative (MURI) program administered by the AFOSR under grant FA9550-12-1-0040. Thanks to Arnon Lotem, Alex Kacelnik, and Pete Trimmer for their useful comments. Most of this work was carried out while the second author was at Harvard’s Center for Research on Computation and Society; Harvard's support is

References (8)
--------------

* J.M. Aw *et al.*### [State-dependent valuation learning in fish: banded tetras prefer stimuli associated with greater past deprivation](/science/article/pii/S0376635708002131)

### Behav. Process.

(2009)
* I. Erev *et al.*### A choice prediction competition for market entry games: an introduction

### Games

(2010)
* J.Y. Halpern *et al.*### I’m doing as well as I can: modeling people as rational finite automata

### Proc. Twenty-Sixth National Conference on Artificial Intelligence (AAAI’12)

(2012)
* J.Y. Halpern *et al.*### Decision theory with resource-bounded agents

### Top. Cogn. Sci.

(2014)

There are more references available in the full text version of this article.Cited by (0)
------------

Recommended articles (6)
------------------------

* Research article### [Measuring Olfactory Processes in *Mus musculus*](/science/article/pii/S0376635717301183)

Behavioural Processes, Volume 155, 2018, pp. 19-25Show abstractThis paper briefly reviews the literature that describes olfactory acuity and odour discrimination learning. The results of current studies that examined the role of the neurotransmitters noradrenalin and acetylcholine in odour discrimination learning are discussed as are those that investigated pattern recognition and models of human disease. The methodology associated with such work is also described and its role in creating disparate results assessed. Recommendations for increasing the reliability and validity of experiments so as to further our understanding of olfactory processes in both healthy mice and those modelling human disease are made throughout the paper.
* Research article### [Tuberculosis detection by pouched rats: Opportunities for reinforcement under low-prevalence conditions](/science/article/pii/S0376635717300347)

Behavioural Processes, Volume 155, 2018, pp. 2-7Show abstractGiant African pouched rats (*Cricetomys ansorgei*) have been employed successfully in two operational tuberculosis-detection projects in which they sniff sputum samples from symptomatic individuals who have visited tuberculosis clinics. The prevalence of pulmonary tuberculosis in this population is high, approximately 20% in the regions where the rats have been used. If the rats are to be used to screen individuals from lower-prevalence populations, their performance under such conditions must first be evaluated. In this study, the prevalence of tuberculosis-positive samples presented to eight pouched rats was reduced to approximately 5%, and the percentage of known-positive samples included as opportunities for reinforcement was varied in sequence from 10 to 8, 6, 4, 2, 4, and 2. Liquid food reinforcers were delivered for identification responses to known-positive samples and at no other time. The rats’ accuracy was clinically and statistically significantly lower at 2% than at the other values. These results indicate that the rats can perform well in low-prevalence scenarios but, if they are used under the conditions of the present study, at least 4% of the samples presented to them must be opportunities for reinforcement.
* Research article### [Grouping promotes risk-taking in unfamiliar settings](/science/article/pii/S0376635717302954)

Behavioural Processes, Volume 148, 2018, pp. 41-45Show abstractActing collectively in a group provides risk-reducing benefits. Yet individuals differ in how they take risks, with some being more willing than others to approach dangerous or unfamiliar settings. Therefore, individuals may need to adjust their behaviour when in groups, either as a result of perceiving greater safety or to coordinate collective responses, the latter of which may rely on within-group dynamics biased by group composition. In zebrafish we explored how these aspects of grouping affect risk-taking behaviour by comparing solitary to group conditions and testing the ability of group-member solitary responses to predict collective responses. We focused on approach-latency towards a novel object and an unusual food to test this, for shoals of five fish. There was no indication that collective latencies are predicted by how each fish responded when alone in terms of the extremes, the variance or the mean of group-member latency towards the unusual food and the novel-object. However, fish were overall faster and less variable in their approach when shoaling. This indicates lower risk aversion by individuals in groups, presumably as a result of group safety. An interesting consequence of the overall low risk-aversion in shoals is that more risk-aversive fish adjust their behaviour more than less risk averse fish.
* Research article### [An investigation of the probability of reciprocation in a risk-reduction model of sharing](/science/article/pii/S0376635717304552)

Behavioural Processes, Volume 157, 2018, pp. 583-589Show abstractA laboratory study investigated whether reductions in the probability of reciprocation would influence sharing in situations of shortfall risk. Choice in twelve adults was evaluated against the predictions of a risk-reduction model of sharing derived from a risk-sensitive foraging theory (the energy-budget rule). Participants responded on a computer task to earn hypothetical money which could be later exchanged for real money. If participants selected the sharing option, their earnings were pooled and split with a (fictitious) partner. To model shortfall risk, the task was arranged so that participants lost their accumulated earnings if it fell below an earnings requirement. Across conditions the probability that the partner would contribute to the pool was .95, .65, and 0. Choosing the sharing option was optimal under the .95 and .65, but not 0 condition. Although levels of preference for the sharing option were below optimal, participants chose it significantly more in the .95 and .65 conditions than in the 0 condition. Sharing was lower in the .65 condition than .95 condition but the difference was not statically significant. The results are consistent with prior cooperation research and demonstrate that under shortfall risk, reductions in the probability of reciprocation by partners may decrease sharing.
* Research article### [Effects of age on the courtship, copulation, and fecundity of *Pardosa pseudoannulata* (Araneae: Lycosidae)](/science/article/pii/S0376635717302474)

Behavioural Processes, Volume 146, 2018, pp. 10-15Show abstractAccording to sexual selection theory, age affects the preference of mate choice, and this preference ultimately influences the fecundity of the female. *Pardosa pseudoannulata* (Araneae: Lycosidae) is a valued predator in many cropping systems. By determining oviposition rate, egg hatching rate, and also the number and carapace width of the 2nd instar spiderlings of the F1 generation, we explored the effects of age on fecundity of the female spider. There were no significant effects of age on courtship duration, sexual cannibalism rate, mating rate, oviposition rate, egg hatching rate, or the number and carapace width of 2nd instar spiderings of *P. pseudoannulata.* However, age had a significant effect on courtship latency, courtship intensity, and mating duration of the spider. Courtship latency decreased significantly with an increase in the age of the male, and courtship intensity of the low-age male increased with increasing female age. Increasing age of male and female spiders was associated with significantly prolonged mating duration. The results indicated that low-age male spiders were more inclined to mate with high-age females, and age had no significant effect on sexual cannibalism rate or the fecundity of the female.
* Research article### [The influence of lameness and individuality on movement patterns in sheep](/science/article/pii/S0376635717302449)

Behavioural Processes, Volume 151, 2018, pp. 34-38Show abstractWe investigated how individuality and lameness altered social organisation by assessing food-directed movement patterns in sheep. One hundred and ninety-six mature Merino ewes were walked in 16 different runs around a 1.1 km track following a food source. Flock position and lameness were measured and temperament was assessed using an Isolation Box Test. The mean value for the correlations of position between a run and the run preceding it was *r* = 0.55 ± SEM 0.03. All correlations between runs were positive (*r* = 0.08–0.76) and all but two were statistically significant (*P* < 0.05). The weakest and least statistically significant correlations were for run 14: where all 16 runs were conducted approximately 3 times a week, except with an interval of 20 weeks between runs 13 and 14. Additionally, there were differences in overall positions for a lame versus a non-lame individual (all *P* < 0.05) with lame sheep being further back in position when compared to their non-lame mean positions. These results indicate the movement patterns, as measured by flock position during a food-directed forced movement order are relatively stable provided tests occur frequently, possibly on a bi-weekly basis. However, further work will be required to better account for individual animal variation.
[View full text](/science/article/pii/S0376635717302048)


An automatic method for discovering rational heuristics for risky choice.

An automatic method for discovering rational heuristics for risky choice
Falk Lieder1(falk.lieder@berkeley.edu)
Paul M. Krueger1(pmk@berkeley.edu)
Thomas L. Grifﬁths (tom grifﬁths@berkeley.edu)
Department of Psychology,University of California Berkeley, Berkeley, CA 94720 USA
1These authors contributed equally.
Abstract
What is the optimal way to make a decision given that your
time is limited and your cognitive resources are bounded? To
answer this question, we formalized the bounded optimal de-
cision process as the solution to a meta-level Markov deci-
sion process whose actions are costly computations. We ap-
proximated the optimal solution and evaluated its predictions
against human choice behavior in the Mouselab paradigm,
which is widely used to study decision strategies. Our compu-
tational method rediscovered well-known heuristic strategies
and the conditions under which they are used, as well as novel
heuristics. A Mouselab experiment conﬁrmed our model’s
main predictions. These ﬁndings are a proof-of-concept that
optimal cognitive strategies can be automatically derived as the
rational use of ﬁnite time and bounded cognitive resources.
Keywords: Decision-Making; Heuristics; Bounded Rational-
ity; Strategy Selection; Rational Metareasoning
Introduction
Some situations require us to decide quickly whereas others
call for careful consideration of all available options and po-
tential consequences. People seem to master this challenge
by choosing adaptively from a toolbox of diverse decision
strategies (Payne, Bettman, & Johnson, 1988; Gigerenzer &
Selten, 2002). This toolbox is assumed to include fast-and-
frugal heuristics (Gigerenzer & Goldstein, 1996) as well as
slower and more effortful strategies. Fast-and-frugal heuris-
tics include Take-The-Best (TTB), which chooses the alterna-
tive that is favored by the most predictive attribute and ignores
all other attributes, satisﬁcing (SAT) (Simon, 1956), which
chooses the ﬁrst alternative whose expected value exceeds
some threshold, and random choice; slower strategies include
the Weighted-Additive Strategy (WADD), which computes
all gambles’ expected values based on all possible payoffs.
Except for WADD, all of these strategies are heuristics: they
solve some problems very efﬁciently but err on others.
The systematic errors that result from people’s use of
heuristics are inconsistent with classic notions of rationality
such as logic, probability theory, and expected utility theory
(Tversky & Kahneman, 1974). Making good decisions is re-
markably constrained: decisions have to be made in a ﬁnite
amount of time, people’s cognitive resources are limited, and
maximizing expected utility entails intractable computational
problems. This makes expected utility theory an unrealisti-
cally high bar for human rationality. According to a more
realistic normative standard, people should decide in a way
that makes the best possible use of their limited cognitive re-
sources (Grifﬁths, Lieder, & Goodman, 2015). Previous re-
search has applied this resource-rational approach to numer-ical estimation (Lieder, Grifﬁths, & Goodman, 2012), avail-
ability biases (Lieder, Hsu, & Grifﬁths, 2014), and strategy
selection (Lieder, Plunkett, et al., 2014). However, this ap-
proach has not been applied to the domain in which heuristics
have perhaps been studied in greatest detail: multi-alternative
risky choice. Work on risky choice suggests that people adap-
tively switch between multiple different strategies depend-
ing on how much time is available and whether one of the
outcomes is much more likely than the others (Payne et al.,
1988). Yet, it remains unclear how people’s decision pro-
cesses compare to resource-rational behavior.
To answer these questions, we model the decision process
as a sequence of costly computations and formalize the opti-
mal decision process as the solution to a meta-level Markov
decision process. We combine this theory with an algorithm
for approximating the optimal solution to create a computa-
tional method that can automatically derive optimal cogni-
tive strategies. These rational heuristics can be interpreted
as a fair normative standard for human decision making that
takes into account that people’s time is costly and that their
cognitive resources are bounded. We are optimistic that this
novel approach will lead to new insights about how decision-
makers cope with limited time and bounded computational
resources, and advance the debate about human rationality.
We illustrate our approach in multi-alternative risky choice
and test its predictions using the Mouselab paradigm that
is widely used to study decision strategies (Johnson, Payne,
Bettman, & Schkade, 1989). Two known heuristics, TTB
and random choice, emerged from our theory as resource-
rational strategies for low-stakes decisions with high and low
dispersion of their outcome probabilities, respectively. In ad-
dition, our computational method discovered a novel heuristic
that combines TTB with satisﬁcing. Our experiment demon-
strated that people do indeed use the newly discovered heuris-
tic and conﬁrmed our rational model’s predictions of when
people use which strategy: people used simple heuristics
more frequently when the stakes were low, employed fast-
and-frugal heuristics less frequently when all outcomes were
almost equally likely (low dispersion), and invested more
time and effort when the stakes were high. This is the ﬁrst
demonstration that rational meta-reasoning can be used to au-
tomatically discover decision strategies used by people.
Background
We will formulate our theory using the mathematical frame-
works of Markov decision processes, bounded optimality, and
rational metareasoning, introduced in this section.
Markov Decision Processes
Each sequential decision problem can be modeled as a
Markov Decision Process (MDP)
M= (S;A;T;g;r;P0); (1)
where Sis the set of states, Ais the set of actions, T(s;a;s0)
is the probability that the agent will transition from state sto
state s0if it takes action a, 0g1 is the discount factor
on future rewards, r(s;a;s0)is the reward generated by this
transition, and P0is the probability distribution of the initial
state S0(Sutton & Barto, 1998). A policy p:S7!Aspeci-
ﬁes which action to take in each of the states. The expected
sum of discounted rewards that a policy pwill generate in the
MDP Mstarting from a state sis known as its value function
Vp
M(s) =E"
¥
å
t=0gtr(St;p(St);St+1)#
: (2)
The optimal policy p?
Mmaximizes the expected sum of dis-
counted rewards, that is
p?
M=argmax
pE"
¥
å
t=0gtr(St;p(St);St+1)#
: (3)
Bounded optimality and rational metareasoning
People and robots have to make decisions in a limited amount
of time and with bounded cognitive resources. Given that
these resources are scarce, which strategy should a decision-
maker employ to use its resources most effectively? The
theory of bounded optimality and rational metareasoning
(Russell & Wefald, 1991; Russell & Subramanian, 1995) was
developed to answer this question for rational agents with
limited performance hardware. It frames this problem as se-
lecting computations so as to maximize the sum of the re-
wards of resulting decisions minus the costs of the computa-
tions involved.
Concretely, the problem of choosing computations opti-
mally can be formulated as a meta-level MDP (Hay, Russell,
Tolpin, & Shimony, 2012). A meta-level MDP
Mmeta= (B;C;Tmeta;rmeta) (4)
is a Markov decision process whose actions Care cognitive
operations, its states Brepresent the agent’s probabilistic be-
liefs, and the transition function Tmetamodels how cognitive
operations change the agent’s beliefs. In addition to a set
of computations Cthat update the agent’s belief, the cog-
nitive operations also include the meta-level action ?that
terminates deliberation and translates the current belief into
action. The meta-level state btencodes the agent’s proba-
bilistic beliefs about the domain it is reasoning about.The
meta-level reward function rmetacaptures the cost of think-
ing (Shugan, 1980) and the reward rthe agent expects to re-
ceive from the environment when it stops deliberating andtakes action. The computations Cdo not yield any external
reward. Their only effect is to update the agent’s beliefs.
Hence, the meta-level reward for performing a computation
c2Cisrmeta(bt;c) =cost(c). By contrast, terminating de-
liberation and taking action ( ?) does not update the agent’s
belief. Instead, its value lies in the anticipated reward for tak-
ing action, that is
rmeta(bt;?) =argmaxab(µ)
t(a); (5)
where b(µ)
t(a)is the expected reward of taking action aac-
cording to the belief bt.
Adaptive strategy selection in risky choice
Consistent with rational metareasoning, people ﬂexibly adapt
their decision processes to the structure of the problem they
face. Concretely, Payne et al. (1988) found that people use
fast-and-frugal heuristics, like TTB, more frequently when
they are under time pressure and when one outcome is much
more likely than the others. In this research, participants were
given the choice between gambles g1;;gn. Each gamble
was deﬁned by the payoffs it assigns to each of four possible
outcome whose probabilities are known ( P(O)). Participants
could inspect a payoff matrix Vo;gwith one row for each out-
come oand one column for each gamble g. Critically, each
payoff is only revealed when the participant clicks on the cor-
responding cell of the payoff matrix using a mouse; this task
is hence referred to as the Mouselab paradigm (see Figure 1).
The adaptiveness of people’s strategy choices in the
Mouselab paradigm suggests that their decision processes are
efﬁcient and effective. But it is difﬁcult to test whether they
are optimal, because it is unclear what it means to decide op-
timally when one’s time is valuable and one’s cognitive re-
sources are limited. To clarify this, the following section de-
velops a normative theory of resource-bounded decision mak-
ing in the Mouselab paradigm.
Boundedly-optimal decision-making
To model the meta-decision problem posed by the Mouse-
lab task, we characterize the decision-maker’s belief state
btby probability distributions on the expected values
e1=E[vO;g1];en=E[vO;gn]of the navailable gambles
g1;;gn. Furthermore, we assume that for each element vo;g
of the payoff matrix Vthere is one computation co;gthat in-
spects the payoff vo;gand updates the agent’s belief about the
expected value of the inspected gamble according to Bayesian
inference. Since the entries of the payoff matrix are drawn
from the normal distribution N(¯v;s2
v), the resulting posterior
distributions are also Gaussian. Hence, the decision-maker’s
belief state btcan be represented by bt= (bt;1;;bt;n)with
bt;g=
b(µ)
t;g;b(s2)
t;g
; (6)
where b(µ)
t;gandb(s2)
t;gare the mean and the variance of the
probability distribution on the expected value of gamble gof
the belief state bt.
Given the set Otof the indices (k(1)
o;k(1)
g);;(k(t)
o;k(t)
g)of
thetobservations made so far, the means and variances char-
acterizing the decision-maker’s beliefs are given by
b(µ)
t;g=å
(o;g)2Op(o)vo;g+å
(o;g)=2Op(o)¯v (7)
b(s2)
t;g=å
(o;g)=2Op(o)2s2
v: (8)
The meta-level transition function T(bt;co;g;bt+1)encodes
the probability distribution on what the updated means and
variances will be given the observation of a payoff value Vo;g
sampled from N(¯v;s2
v). The meta-level reward for perform-
ing the computation co;g2Cencodes that acquiring and pro-
cessing an additional piece of information is costly. We as-
sume that the cost of all such computations is an unknown
constant l. The meta-level reward for terminating delibera-
tion and taking action is rmeta(bt;?) =max gb(µ)
t(g).
Approximating the optimal meta-level policy:
Bayesian value function approximation
Unfortunately, computing the optimal policy for the meta-
level MDP deﬁned above is intractable. However, it can be
approximated using methods from reinforcement learning.
We initially used the semi-gradient SARSA algorithm (Sutton
& Barto, 1998) with limited success. We therefore developed
a new algorithm that replaces the gradient descent component
of that algorithm by Bayesian linear regression.
Our algorithm learns a linear approximation to the meta-
level Q-function
Qmeta(b;c)å
kwkfk(b;c); (9)
whose features finclude a constant, features of the belief
state bt, and features of the computation ct. The features
of the belief state were the expected value of the maximum
of the gambles’ expected values ( E[max gEgjbt]) and the
decision-maker’s uncertainty about it (p
Var[max gEgjbt]).
The largest posterior mean (max gb(µ)
t;g) and its associated
uncertainty (q
µ(s2)
t;g?where g?=argmax gb(µ)
t;g), the second
largest posterior mean and the decision-maker’s uncertainty
about it, and the expected regret E[regret (g)jbt]that the
decision-maker would experience if they chose based on
their current belief (where regret (g) =max gEgmax gb(µ)
t;g
forEiN(b(µ)
t;i;b(s)
t;i)for all gambles i). The features of
the computation co;gwere its myopic value of computation
(VOC (bt;co;g); see Russell & Wefald, 1991), the current un-
certainty about the expected value of the inspected gamble
(b(s)
t;g), the probability of the inspected outcome, the differ-
ence between the largest posterior mean and the posterior
mean of the inspected outcome, a binary variable indicat-
ing whether the computation acquired new information, and
the expected reduction in the expected regret ER (b)minus its
cost (i.e. E[ER(Bt+1)jbt;c]ER(bt)l, where Bt+1is theunknown belief state resulting from performing computation
cin belief state btand ER (bt) =E[regret (argmax gb(µ)
t;g)jbt]).
The weights ware learned by Bayesian linear regression of
the bootstrap estimate ˆQ(b;c)of the meta-level value function
onto the features f. The bootstrap estimator is
ˆQ(bt;ct) =rmeta(bt;ct)+ˆw0
tf(bt+1;ct+1); (10)
where ˆ wtis the posterior mean on the weights wgiven the ob-
servations from the ﬁrst ttrials, and f(bt+1;ct+1)is the feature
vector characterizing the subsequent belief state bt+1and the
computation ct+1that will be selected in it.
Given the learned posterior distribution on the feature
weights w, the next computation cis selected by contextual
Thompson sampling (Agrawal & Goyal, 2013). Speciﬁcally,
to make the tthmeta-decision, a weight vector ˜ wis sampled
from the posterior distribution of the weights given the series
of meta-level states, selected computations, and the resulting
value estimates experienced so far, that is
˜wP(wj(b1;c1;ˆQ(b1;c1));;(bk1;ck1;ˆQ(bk1;ck1))):
The sampled weight vector ˜ wis then used to predict the
Q-values of each available computation c2Caccording to
Equation 9. Finally, the computation with the highest pre-
dicted Q-value is selected.
Application to Mouselab experiment
As a proof of concept, we applied our approach to the Mouse-
lab experiment described below. The experiment comprises
50% high-stakes problems and 50% low-stakes problems.
Since participants are informed about the stakes, we learned
two separate policies for high-stakes and low-stakes prob-
lems, respectively. Half of each of those problems had nearly
uniform outcome probabilities (“low dispersion”) and for the
other half one outcome was much more likely than all others
combined (“high dispersion”). The parameters of the simu-
lated environment were exactly equal to those of the experi-
ment described below. Our model assumed that people play
each game as if they receive the payoff of the selected gam-
ble. We estimated the cost per click to be about l=3 cents.
This value was selected to roughly match the average number
of acquisitions observed in the experiment.
To approximate the optimal meta-decision policy for this
task, we ran our feature-based value function approximation
method for 4000 low-stakes training trials and 4000 high-
stakes training trials, respectively.
Model predictions
The meta-level MDP described above formalizes the costs
and beneﬁts of acquiring and processing additional pieces of
information: acquiring additional information can improve
the decision that will be taken later on but also incurs an
immediate cost. Hence, the optimal solution approximated
by our computational method executes a cognitive operation
or sequence of operations if and only if the resulting im-
provement in decision quality is larger than cost of those
Figure 1: The Mouselab paradigm, showing an example se-
quence of clicks generated by the SAT-TTB strategy, which
was discovered through approximate rational metareasoning.
operations. Intuitively, this means that the decision process
prescribed by our model achieves the optimal tradeoff be-
tween decision quality versus decision time and mental ef-
fort. This tradeoff depends on the stakes of the decision such
that higher stakes usually warrant more deliberation. Like-
wise, since processing probable outcomes is more likely to
improve the quality of the resulting decision than process-
ing improbable outcomes, we expect our model to prioritize
probable outcomes over less probable outcomes—especially
in high-dispersion trials.
Our computational method automatically discovered
strategies that people are known to use in the Mouselab
paradigm as well as a novel strategy that has not been re-
ported yet. Our method rediscovered TTB, WADD, and the
random choice strategy. In addition, it discovered a new hy-
brid strategy that combines TTB with satisﬁcing (SAT-TTB).
Like TTB, SAT-TTB inspects only the payoffs for the most
probable outcome. But unlike TTB and like SAT, SAT-TTB
terminates as soon as it ﬁnds a gamble whose payoff for the
most probable outcome is high enough. On average, this
value was about $0 :15 when the payoffs ranged from $0 :01 to
$0:25 (i.e., low-stakes trials). Figure 1 illustrates this strategy.
Furthermore, our model makes intuitive predictions about
the contingency of people’s choice processes on stakes and
outcome probabilities. First, our model predicts that people
should use fast-and-frugal heuristics more frequently in high-
dispersion trials. This is intuitively rational because high dis-
persion means that one outcome is much more likely than
all others and fast-and-frugal heuristics ignore all outcomes
except for the most probable one(s). Concretely, our model
generated TTB as the strategy of choice for 100% of the high-
dispersion problems with low-stakes, but for low-dispersion
problems with low-stakes the model considered the random
choice strategy to be optimal in the majority (56%) of cases;
it used the SAT-TTB hybrid strategy for 24% of such trials,
and it indicated the TTB strategy only for the remaining 20%.
Second, our model predicts that people should use simple
heuristics, like TTB, SAT-TTB, and random choice, primar-
ily when the stakes are low. This, too, is intuitively rational
because fast and frugal heuristics tend to be faster but less ac-curate than more effortful strategies. Our model used these
heuristics for 100% of the low-stakes problems. But for high-
stakes problems, the model never used any of these or other
frugal strategies. Instead, the model typically inspected the
vast majority of all cells (24 :8=28 for low-dispersion prob-
lems and 23 :7=28 for high-dispersion problems). The few
cells that it did not inspect were mostly the payoffs of less-
likely outcomes of the best gamble when its inspected payoffs
for the most likely outcome(s) were high enough to guarantee
that it would be optimal.
Third, our model predicts that when the stakes are high
people should invest more time and effort ( F(1;396) =
9886 :8;p<0:0001) to reap a higher fraction of the highest
possible expected payoff ( F(1;339) =135:24;p<0:0001).
This, too, is consistent with the rational speed-accuracy trade-
off inherent in our theory. When the stakes were low the
model inspected only 4 :3 payoffs on average and reaped only
87% of the possible reward; but when the stakes were high the
model inspected 24 :3 of the 28 possible payoffs and reaped
99% of the best expected payoff on average. In 97% of these
trials, the model achieved this near-maximal performance
while being more efﬁcient and more frugal than the WADD
strategy which it employed for only 3% of these problems.
Experimental test of novel predictions
To test the predictions of our model, we conducted a new
Mouselab experiment that manipulated the stakes and disper-
sion of outcome probabilities within subjects in an identical
manner to the model simulations.
Methods
Participants We recruited 200 participants on Amazon
Mechanical Turk. The experiment took about 30min. Partic-
ipants received a base pay of $1 :50, and one of their twenty
winnings was selected at random and awarded as a bonus to
motivate them to take each trial seriously (avg. bonus $3 :53).
Procedure Participants performed a variation of the
Mouselab task (Payne et al., 1988). Participants played a se-
ries of 20 games divided into two blocks. Figure 1 shows
a screenshot of one game. Every game began with a 4 7
grid of occluded payoffs: there were seven gambles to choose
from (columns) and four possible outcomes (rows). The oc-
cluded value in each cell speciﬁed how much the gamble indi-
cated by its column would pay if the outcome indicated by its
row occurred. The outcome probabilities were described by
the number of balls of a given color in a bin of 100 balls, from
which the outcome would be drawn. For each trial, partici-
pants were free to inspect any number of cells before select-
ing a gamble, with no time limit. The value of each inspected
cell remained visible onscreen for the duration of the trial.
Upon selecting a gamble, the resulting reward was displayed.
Experimental design The experiment used a 2 2 within
subjects design. Each block of ten trials was either low-stakes
or high-stakes, with block order randomly counterbalanced
across participants. In games with low-stakes, the possible
outcomes ranged from $0 :01 to $0 :25, while in high-stakes
games, outcomes ranged from $0 :01 to $9 :99. The payoffs
were drawn from a truncated normal distribution with mean
rmax+rmin
2and standard deviation 0 :3(rmaxrmin). Within
each block, there were ﬁve low-dispersion trials and ﬁve high-
dispersion, ordered randomly. In low-dispersion trials, the
probability of each of the four outcomes ranged from 0 :1 to
0:4, whereas in high-dispersion trials, the probability of the
most likely outcome ranged from 0 :85 to 0 :97.
Strategy identiﬁcation We identiﬁed six different decision
strategies, in humans and in simulations, using the follow-
ing deﬁnitions: TTB was deﬁned as inspecting all cells in the
row corresponding to the most probable outcome and nothing
else. SAT occurs when one gamble’s payoffs are inspected
for all four outcomes, potentially followed by the inspection
of all outcomes of another gamble, and so on, but leaving
at least one gamble unexamined. The hybrid strategy, SAT-
TTB, was deﬁned as inspecting the payoffs of 1 to 6 gambles
for the most probable outcome and not inspecting payoffs for
any other outcome. TTB2 was deﬁned as inspecting all four-
teen cells of the two most probable outcomes, and nothing
else. WADD was deﬁned as inspecting all 28 cells column by
column.Random decisions mean zero samples were taken.
Results
Our process tracing data conﬁrmed that people do indeed
use the SAT-TTB strategy discovered by our model. Table
1 shows the frequency of various decision strategies, for each
of the four different types of trials. Out of 4000 trials across
all participants, TTB was the most common strategy overall,
accounting for 25 :3% of all trials. SAT-TTB was the sec-
ond most common strategy among those we examined: par-
ticipants employed this strategy on 10 :7% of all trials. In
8:0% of trials participants chose randomly without making
any observations—mostly during low-stakes games. Interest-
ingly, we also observed a second novel strategy that we call
Take-The-Best-Two (TTB2). This strategy inspects all gam-
bles’ payoffs for the twomost probable outcomes, and was
used in 6 :3% of trials. The WADD strategy occurred in 4 :5%
of trials. Finally, the SAT strategy was used in 3 :1% of games.
Consistent with our model’s ﬁrst prediction, people used
TTB more frequently when the dispersion was high ( c2(1) =
897:9;p<0:0001). Consistent with our model’s second pre-
diction, participants used simple heuristics more frequently
when the stakes were low: the frequency of the random
choice—the simplest heuristic—increased signiﬁcantly from
4:2% on high-stakes problems to 19 :9% on low-stakes prob-
lems ( c2(1) =88:2;p<0:0001), and so did the frequency of
the second simplest heuristic, SAT-TTB ( c2(1) =86:3;p<
0:0001), and the third simplest heuristic, TTB ( c2(1) =
20:0;p<0:0001). The frequency of SAT also increased
from high- to low-stakes games ( c2(1) =3:4;p<0:05, one-Frequency
Strategy Total HS-HD HS-LD LS-HD LS-LD
TTB 1012 392 64 449 107
SAT-TTB 412 68 54 140 150
Random 320 41 42 111 126
TTB2 251 34 94 25 98
WADD 178 33 84 19 42
SAT 89 14 22 23 30
HS-HD = High-stakes, high-dispersion HS-LD = High-stakes, low-dispersion
LS-HD = Low-stakes, high-dispersion LS-LD = Low-stakes, low-dispersion
Table 1: Frequency of strategy types for each type of trial.
tailed). Finally, consistent with our model’s third prediction,
the frequency of the most effortful and most accurate strategy,
WADD, increased with the stakes ( c2(1) =19:3;p<0:0001).
Together, the strategies reported in Table 1 account for only
about half (48 :6%) of all trials.To test our model’s predictions
on all of the trials, we quantiﬁed people’s decision style by
four metrics introduced by Payne et al. (1988): the number of
inspected cells ( acquisitions ), the proportion of those inspec-
tions that pertained to the most probable outcome ( prioritiza-
tion), the degree to which subsequent acquisitions inspected
the payoffs of different gambles for the same outcome ver-
sus the payoffs of the same gamble for different outcomes
(outcome-based processing :nsame outcomensame gamble
nsame outcome +nsame gamble), and
the average ratio of the expected value of the chosen gamble
over the expected value of the optimal choice ( relative per-
formance ). To further test our model’s predictions, we ran a
2-way mixed-effects ANOV A for each of these four metrics.
As shown in Figure 2, the effects of the stakes and out-
come probabilities on the four metrics conﬁrmed the model’s
predictions. Our model’s ﬁrst prediction that high dispersion
promotes the use of fast-and-frugal heuristics was conﬁrmed
by a decrease in the number of acquisitions ( F(1;3798) =
78:24;p<0:0001) in conjunction with an increases in
outcome-based processing ( F(1;3432) =68:31;p<0:0001)
and prioritization ( (F(1;3478) =280:1;p<0:0001)). The
increase in prioritization was especially striking: while only
40:4% of participants’ clicks inspected the most probable
outcome when dispersion was low, they focused 70 :6% of
their acquisitions on the most probable outcome when dis-
persion was high. Our model’s second prediction that the
higher stakes should decrease people’s reliance on fast-and-
frugal heuristics was conﬁrmed by a signiﬁcant increases
in the number of acquisitions ( F(1;3798) = 281:47;p<
0:0001) which was accompanied by a decrease in prioritiza-
tion ( F(1;3478) =62:42;p<0:0001) and an increase in rel-
ative performance ( F(1;3798) =47:62;p<0:0001). Consis-
tent with the model’s third prediction, the average outcome-
based processing metric was lower for high stakes but this ef-
fect was not statistically signiﬁcant ( F(1;3432) =2:45;p=
0:06, one-tailed). Our model’s third prediction that high-
stakes increases time, effort, and performance, was con-
ﬁrmed by a signiﬁcant increases in the number of acquisitions
(F(1;3798) =281:47;p<0:0001) and relative performance
(F(1;3798) =47:62;p<0:0001) with high stakes.
Figure 2: People’s decision style by stakes and dispersion of
the outcome probabilities.
Despite these qualitative agreements, there were quantita-
tive differences. Most notably, our model predicted a more
pronounced effect of the stakes on the number of acquisitions
than we observed in people ( +19:6 vs. +3:4); the smaller
effect in people might reﬂect their concave utility function.
Discussion
In summary, our resource-rational theory of multi-alternative
risky choice predicted some of the main strategies people use
in the Mouselab paradigm and the conditions under which
they are selected. In addition to automatically discovering
known strategies and contingencies, our computational ap-
proach also discovered a novel, previously unknown heuris-
tic that integrates TTB with satisﬁcing (SAT-TTB), and our
experiment conﬁrmed that people do indeed use SAT-TTB on
a non-negligible fraction of problems—especially when the
stakes are low.
Tajima, Drugowitsch, and Pouget (2016) solved meta-level
MDPs to derive boundedly optimal drift-diffusion models.
The strategy discovery method presented here generalizes this
approach to more complex decision mechanisms that can pro-
cess and generate evidence in many different ways.
One limitation of the current work is that we do not know
how closely our algorithm approximated the optimal pol-
icy, and it is possible that a more accurate approximation
would yield somewhat different predictions. Future work
will systematically evaluate the accuracy of our approxima-
tion method on smaller problems for which the optimal meta-
level policy can be computed exactly. Another limitation of
the present work is that the cost of computation had to be ﬁt to
the participants’ responses. Future work will control the cost
per click and measure it independently. This will enable a di-
rect comparison of the time and effort people invest against
the optimal amount of deliberation. However, a thorough an-
swer to this question will require a more detailed model of
people’s cognitive architecture including a model of working
memory. Another direction for future work is to characterizethe decision strategies the model employed on the vast major-
ity of high-stakes problems where it did not use WADD.
Our proof-of-concept study suggests that formulating the
problem of making optimal use of ﬁnite time and limited
cognitive resources as a meta-level MDP is a promising ap-
proach to discovering cognitive strategies. This approach can
be leveraged to develop more realistic normative standards
of human rationality. This might enable future work to sys-
tematically evaluate the extent to which people are resource-
rational. In the long term, our approach could be used to im-
prove human reasoning and decision-making by discovering
rational heuristics and teaching them to people.
Acknowledgments. This work was supported by grant number
ONR MURI N00014-13-1-0341 and a grant from the Templeton
World Charity Foundation.
References
Agrawal, S., & Goyal, N. (2013). Thompson sampling for contex-
tual bandits with linear payoffs. In Proceedings of the 30th in-
ternational conference on machine learning (pp. 127–135).
Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning the fast and
frugal way: models of bounded rationality. Psychological
review ,103(4), 650.
Gigerenzer, G., & Selten, R. (2002). Bounded rationality: The
adaptive toolbox . MIT Press.
Grifﬁths, T. L., Lieder, F., & Goodman, N. D. (2015). Rational use
of cognitive resources: Levels of analysis between the com-
putational and the algorithmic. Topics in Cognitive Science ,
7, 217–229.
Hay, N., Russell, S., Tolpin, D., & Shimony, S. (2012). Select-
ing computations: Theory and applications. In N. de Freitas
& K. Murphy (Eds.), Uncertainty in artiﬁcial intelligence:
Proceedings of the twenty-eighth conference. P.O. Box 866
Corvallis, Oregon 97339 USA: AUAI Press.
Johnson, E. J., Payne, J. W., Bettman, J. R., & Schkade, D. A.
(1989). Monitoring information processing and decisions:
The mouselab system (Tech. Rep.). DTIC Document.
Lieder, F., Grifﬁths, T. L., & Goodman, N. D. (2012). Burn-in,
bias, and the rationality of anchoring. Advances in Neural
Information Processing Systems , 2699–2707.
Lieder, F., Hsu, M., & Grifﬁths, T. L. (2014). The high availability
of extreme events serves resource-rational decision-making.
InProceedings of the 36th annual conference of the cognitive
science society.
Lieder, F., Plunkett, D., Hamrick, J. B., Russell, S. J., Hay, N., &
Grifﬁths, T. (2014). Algorithm selection by rational metarea-
soning as a model of human strategy selection. In Z. Ghahra-
mani, M. Welling, C. Cortes, N. Lawrence, & K. Weinberger
(Eds.), Advances in neural information processing systems 27
(pp. 2870–2878). Curran Associates, Inc.
Payne, J. W., Bettman, J. R., & Johnson, E. J. (1988). Adaptive strat-
egy selection in decision making. Journal of Experimental
Psychology: Learning, Memory, and Cognition ,14(3), 534.
Russell, S. J., & Subramanian, D. (1995). Provably bounded-
optimal agents. Journal of Artiﬁcial Intelligence Research ,
2, 575–609.
Russell, S. J., & Wefald, E. (1991). Principles of metareasoning.
Artiﬁcial Intelligence ,49(1-3), 361–395.
Shugan, S. M. (1980). The cost of thinking. Journal of consumer
Research ,7(2), 99–111.
Simon, H. A. (1956). Rational choice and the structure of the envi-
ronment. Psychological review ,63(2), 129.
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An
introduction . Cambridge, MA, USA: MIT press.
Tajima, S., Drugowitsch, J., & Pouget, A. (2016). Optimal policy for
value-based decision-making. Nature communications ,7.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty:
Heuristics and biases. Science ,185(4157), 1124–1131.


When Does Bounded-Optimal Metareasoning Favor Few Cognitive Systems?.

When Does Bounded-Optimal Metareasoning Favor Few Cognitive Systems?
Smitha Milli
EECS Department
University of California
Berkeley, CA 94720
smilli@berkeley.eduFalk Lieder
Helen Wills Neuroscience Institute
University of California
Berkeley, CA 94720
falk.lieder@berkeley.eduThomas L. Grifﬁths
Department of Psychology
University of California
Berkeley, CA 94720
tomgriffiths@berkeley.edu
Abstract
While optimal metareasoning is notoriously intractable,
humans are nonetheless able to adaptively allocate their
computational resources. A possible approximation that
humans may use to do this is to only metareason over
a ﬁnite set of cognitive systems that perform variable
amounts of computation. The highly inﬂuential “dual-
process” accounts of human cognition, which postulate
the coexistence of a slow accurate system with a fast
error-prone system, can be seen as a special case of this
approximation. This raises two questions: how many
cognitive systems should a bounded optimal agent be
equipped with and what characteristics should those
systems have? We investigate these questions in two set-
tings: a one-shot decision between two alternatives, and
planning under uncertainty in a Markov decision pro-
cess. We ﬁnd that the optimal number of systems de-
pends on the variability of the environment and the cost-
liness of metareasoning. Consistent with dual-process
theories, we also ﬁnd that when having two systems is
optimal, then the ﬁrst system is fast but error-prone and
the second system is slow but accurate.
Introduction
An agent should not think long about what to do when it is
about to get hit by a car, but should deﬁnitely do so before
declaring war. The optimal amount to think before making
a decision is dependent on the setting, involving a trade-off
between the cost of thinking and the cost of making an error.
Thus, in order for an agent to be optimal across different
settings it must metareason to adaptively choose the correct
amount to think (Russell and Wefald 1991a).
Optimal metareasoning is generally computationally in-
tractable (Russell and Wefald 1991a; Lin et al. 2015). De-
spite this, humans are still able to adaptively allocate their
computational resources in inference and decision-making
(Vul et al. 2014; Grifﬁths, Lieder, and Goodman 2015;
Gershman, Horvitz, and Tenenbaum 2015). One method
by which humans may perform approximate metareason-
ing is to choose between a discrete set of cognitive sys-
tems that perform variable amounts of computation (Lieder
et al. 2014; Lieder and Grifﬁths 2015), rather than directly
Copyright c
2017, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.determining the exact amount of computation that should
be done. This kind of approximation is a generalization
of dual-process accounts of human cognition (Evans 2008;
Kahneman 2011), which posit the existence of two types of
cognitive systems, one that is slow and accurate and another
that is fast but error-prone.
The postulated existence of distinct cognitive systems
raises two interesting questions. First, how many systems
should an agent have? Second, what characteristics should
those systems have? These questions are important for un-
derstanding what the existence of multiple cognitive systems
entails for human rationality (Evans 2008; Stanovich 2011)
as well as for designing bounded-optimal AI systems (Rus-
sell and Subramanian 1995) that can metareason efﬁciently.
We investigate these questions in two settings. The ﬁrst
setting is a simple two-alternative forced choice task where
the agent samples from a posterior distribution to infer
which of two options is better. In this case, metareason-
ing selects how many samples to draw before taking an
action (Vul et al. 2014). The second setting involves plan-
ning under uncertainty in a Markov decision process (MDP)
(Lin et al. 2015). In this case, metareasoning determines the
number of rollouts to perform to approximate the solution
to the MDP with bounded real-time dynamic programming
(BRTDP; McMahan, Likhachev, and Gordan, 2005).
Our experiments show that the optimal number of sys-
tems increases with the variability of the environment but
decreases with the costliness of metareasoning. In addition,
when it is optimal to have two systems, then the difference
in their speed-accuracy tradeoffs increases with the variabil-
ity of the environment. In variable environments, this results
in one system that is accurate but costly to use and another
system that is fast but error-prone. These predictions mirror
the assertions of dual-process accounts of cognition (Evans
2008; Kahneman 2011) suggesting that bounded optimality
might provide a rational reinterpretation of those theories.
Background
We draw on the idea that the mind is composed of multi-
ple cognitive systems, as instantiated in dual-process theo-
ries of human cognition, and the ideas of bounded optimal-
ity and rational metareasoning developed in artiﬁcial intelli-
gence research. We review these ideas in turn.
Dual-process theories
The idea that human minds are composed of multiple in-
teracting cognitive systems ﬁrst came to prominence in the
literature on reasoning (Evans 2008; Stanovich 2011). While
people are capable of reasoning in ways that are consis-
tent with the prescriptions of logic, they often do not. Dual-
process theories suggested that this is because people em-
ploy two types of cognitive strategies: fast but fallible heuris-
tics that are triggered automatically and deliberate strategies
that are slow but accurate. This idea has subsequently been
applied to explain a wide range of mental phenonomena, in-
cluding judgment and decision-making, where it has been
popularized by the distinction between System 1 and System
2 (Kahneman 2011), and moral reasoning where the distinc-
tion is made between a fast deontological system and a slow
utilitarian system (Greene 2015).
In parallel with this literature in cognitive psychology, re-
search on human reinforcement learning has led to similar
conclusions. Behavioral and neural data suggest that the hu-
man brain is equipped with two distinct decision systems: a
fast, reﬂexive, system based on habits and a slow, deliberate
system based on goals (Dolan and Dayan 2013). The mech-
anisms employed by these systems have been mapped onto
model-based versus model-free reinforcement learning algo-
rithms that are used to solve Markov decision processes. A
model-free and model-based distinction has also been sug-
gested to account for the nature of the two systems posited
to underlie moral reasoning (Cushman 2013).
The empirical support for the idea that the human mind is
composed of two types of cognitive systems raises the ques-
tion of why such a composition would evolve from natural
selection. Given that people outperform AI systems in most
complex real-world tasks despite their very limited cogni-
tive resources (Gershman, Horvitz, and Tenenbaum 2015;
Grifﬁths, Lieder, and Goodman 2015), we ask whether there
are any principled reasons why bounded agents should be
equipped with two types of reasoning systems rather than
just one system, or three or more systems.
Metareasoning
Intelligent agents with a large repertoire of possible compu-
tations can solve problems in many different ways. On the
one hand, this ﬂexibility enables highly efﬁcient solutions
that exploit the structure of each problem. On the other hand,
this ﬂexibility poses the challenging problem of deciding
when to perform which computation. This problem is known
as rational metareasoning (Russell and Wefald 1991a). In-
spired by psychology, many AI architectures solve this prob-
lem through metacognition (Anderson and Oates 2007; Cox
and Raja 2011) which includes the monitoring and control
of computation as well as metalearning (Smith-Miles 2009;
Schaul and Schmidhuber 2010; Thornton et al. 2013).
The adaptive control of computation is critical for in-
telligent systems to be able to solve complex and poten-
tially time-critical problems on performance-limited hard-
ware (Horvitz, Cooper, and Heckerman 1989; Russell and
Wefald 1991a). For instance, it is necessary for a patient-
monitoring system used in emergency medicine to metar-eason in order to decide when to terminate diagnostic rea-
soning and recommend treatment (Horvitz and Rutledge
1991). Selecting computations can be formalized as a meta-
level Markov decision process (Hay et al. 2012). This for-
mulation highlights that selecting computations optimally
is a computation-intensive problem because the value of
each computation depends on the potentially long sequence
of computations that can be performed afterwards. Con-
sequently, in most cases, solving the metareasoning prob-
lem optimally would defeat the purpose of trying to save
computation (Lin et al. 2015; Hay et al. 2012; Russell
and Wefald 1991b). Yet, people appear capable of allocat-
ing their ﬁnite computational resources near-optimally with
very little effort (Gershman, Horvitz, and Tenenbaum 2015;
Keramati, Dezfouli, and Piray 2011).
To make optimal use of their ﬁnite computational re-
sources bounded-optimal agents (Russell and Subramanian
1995) must optimally distribute their resources between
metareasoning and reasoning about the world. This gener-
ally requires an efﬁcient approximation to optimal metar-
easoning. But exactly how rational metareasoning should be
approximated to maximize the agent’s overall computational
efﬁciency is an open problem. One common approximation
to optimal metareasoning is the meta-myopic approximation
(Russell and Wefald 1991b; Hay et al. 2012) which assumes
that the agent will act immediately after executing the ﬁrst
computation. This approximation fails when no single com-
putation can improve the agent’s decision but a longer se-
quence of computations would improve it signiﬁcantly.
Another strategy to approximate rational metareasoning is
to exploit computational properties of the speciﬁc computa-
tional process being controlled (Russell and Wefald 1989;
Lin et al. 2015; Vul et al. 2014). For instance, Vul et
al. exploited the statistical properties of sampling from a
Bernoulli distribution to solve the problem of when to termi-
nate deliberation using the sequential probability ratio test.
Lin et al. were also able to make metareasoning about plan-
ning tractable by making a meta-myopic assumption and
then exploiting speciﬁc convergence properties of BRTDP.
A third approximation strategy is to metareason only over
a limited number of computational mechanisms that gener-
ate decisions (Lieder et al. 2014). This approximation can
drastically reduce the computational complexity of metarea-
soning while achieving human-level performance (Lieder et
al. 2014; Lieder and Grifﬁths 2015). However, reducing the
space of computational mechanisms the agent can choose
from entails that there may be problems for which the opti-
mal computational mechanisms will be no longer available.
This induces a tradeoff between the efﬁciency of metarea-
soning and the performance of the best available computa-
tional mechanism. Although expanding the computational
repertoire increases the expected performance of the se-
lected mechanism, it also makes metareasoning more costly.
This raises the question of how many and which computa-
tional mechanisms a bounded-optimal metareasoning agent
should be equipped with, which we proceed to explore in the
following sections.
Optimal Choice of Cognitive Systems
At an abstract level, each cognitive system can be charac-
terized by how it trades off speed versus accuracy. In the re-
mainder of this paper we characterize the optimal set of cog-
nitive systems an agent should be equipped with in terms of
the systems’ speed-accuracy tradeoffs. To address this ques-
tion as straight-forwardly as possible, we study sets of sim-
ple decision-making algorithms that abstract away the com-
plexity of real cognitive systems but capture the diversity
of their speed-accuracy tradeoffs. Speciﬁcally, we investi-
gate what determines the optimal set of cognitive systems
M?for an agent acting in an environment characterized
by a distribution Pover a set of decision problems D. An
unboundedly rational agent would always pick the utility-
maximizing action, a= arg maxa2AUd(a), whereUd(a)
is the utility of taking action ain decision problem dandA
is the set of possible actions. However, any physical agent
is necessarily bounded. This means when the agent executes
an algorithm t2T to decide which action to take it incurs
some execution cost ce(t)corresponding to the required time
and computational resources. Hence, bounded agents have
to balance the cost of thinking with the expected value of
the subsequent action A. In this case, the optimal thinking
algorithm,t, for the agent to use on decision problem dis:
t= arg max
t2TE[Ud(A)jt]ce(t): (1)
To be optimal across all decision problems in its environ-
ment, the agent must metareason to adapt its algorithm t
to each individual problem d2D. However,Tis a poten-
tially inﬁnite set. Hence, metareasoning over all algorithms
inTcan be intractable or prohibitively expensive. For this
reason, instead of forcing an agent to metareason over the
entirety ofT, the designer of the agent may opt to equip the
agent only with a small set of cognitive systems MT to
make metareasoning tractable.
However as the number of systems increases, so does the
difﬁculty of choosing between them. To capture this, we in-
troduce the metareasoning cost cm:P(T)!R. The opti-
malMfor the agent designer to choose is
M= arg max
MTE[UD(A)ce(M(D))]cm(M);(2)
whereM(D) :D!M is the agent’s metareasoning pol-
icy, the way the agent chooses which system to pick from the
setM, andAis the action selected after using that system.
Next, we evaluate Mthrough computational experiments
in two settings: a simple two-alternative forced choice task
and planning under uncertainty.
Two-Alternative Forced Choice
Our ﬁrst experiment focuses on a two-alternative forced
choice (2AFC) task, a one-shot decision problem where the
agent must choose between two actions. Here, the agent
gets a reward of +1for picking the correct action and 0
for picking the incorrect action. An unboundedly rational
agent would always pick the action with a higher probability
of being correct. Yet, although simple in set-up, computing
the probability of an action being correct generally requires
Figure 1: The expected utility per time of the optimal choice
of systems,M, asjMj increases. As the costliness of metar-
easoning,1
rmdecreases, the optimal number of systems in-
creases. In this example E[re] = 100 and(re) = 100 .
jMjVar(re)
103104105
1 3 3 1
2 3, 5 1, 5 1, 7
3 3, 5, 7 1, 3, 7 1, 3, 9
4 1, 3, 5, 7* 1, 3, 5, 7 1, 3, 7, 13
(*) Any set of four systems that included 3, 5, 7 was optimal.
Table 1: The optimal set of algorithms, M, for different val-
ues ofjMj as Var (re)increases. This example is the 2AFC
setting with E[re] = 100 andrm= 1000 .
complex inferences over many interconnected variables. To
approximate these often intractable inferences people appear
to perform probabilistic simulations of the outcomes, and
the variability and biases of their predictions (Grifﬁths and
Tenenbaum 2006; Lieder, Grifﬁths, and Goodman 2012) and
choices (Vul et al. 2014; Lieder, Hsu, and Grifﬁths 2014)
match those of efﬁcient sampling algorithms. Previous work
has therefore modeled people as bounded-optimal sample-
based agents (Vul et al. 2014; Grifﬁths, Lieder, and Good-
man 2015). A sample-based agent draws a number of sam-
ples from the distribution over correct actions and then picks
the action that was sampled most frequently.
For the 2AFC setting, let a0anda1be the actions avail-
able to the agent. We assume that the correct action aigiven
all of the information available to the agent was sampled
from a Bernoulli distribution, that is iBern(). The prob-
abilitythata1is correct varies across different problems.
We model this by sampling from a uniform distribution.
Without loss of generality, we assume that P=
Unif(0:5;1)because we can always rename the actions so
thata1is more likely to be correct than a0.
Under these assumptions, the expected value of the utility
Uof a decision made based on ksamples is given by
E[Ujk] =Z
((1CDF(k=2;;k))
+ (1) ( CDF(k=2;;k)))P(d);(3)
where CDFis the binomial cumulative density function.
If there were no cost for samples, then the agent could
take an inﬁnite number of samples to ensure choosing the
correct action. But this is, of course, impractical in the real
world because drawing a sample takes time and time is lim-
ited. Vul et al. (2014) show how the optimal number of sam-
ples changes based on the cost of sampling in various 2AFC
problems. They parameterize the cost of sampling as the ra-
tio,re, between the time for acting and the execution time of
taking 1sample. In this setting, the optimal number of sam-
ples an agent should draw to maximize its expected utility
per unit time is
k= arg max
k2N0E[Ujk]
1 +k
re: (4)
When the time it takes to generate a sample is at least one
tenth of the time it takes to execute the action ( re10),
then the optimal number of samples is either zero or one. In
general, the ﬁrst sample provides the largest gain in decision
quality and the returns diminish with every subsequent sam-
ple.The point where the gain in decision quality falls below
the cost of sampling depends on the value of re. Since this
value can differ drastically across environments, achieving a
near-optimal tradeoff in all environments requires adjusting
the number of samples. Even a simple heuristic-based metar-
easoner that adapts the number of samples it takes based on
a few thresholds on redoes better than one which always
draws the same number of samples (Icard 2014).
Here, we study an agent that chooses how many samples
to draw by metareasoning over a ﬁnite subset Mof all pos-
sible numbers of samples. Furthermore, we assume that the
time spent metareasoning increases linearly with the num-
ber of systems. By analogy to Vul et al. (2014), we formal-
ize the metareasoning cost in terms of the ratio rmof the
time it takes to act over the time it takes to predict the per-
formance of a single system. This allows us to formalize the
metareasoning problem as computing a metacognitive pol-
icyM:R!M[f 0gthat decides which of the agent’s
sampling systems ( M) to use given the relative cost of sam-
pling (re). Given this formulation of the problem, the opti-
mal set of systems that a bounded sampling agent should be
equipped with is
M= arg max
MNEre"
E[UjM(re)]
1 +M(re)
re+jMj
rm#
; (5)
whereM(re)can be an arbitrary metareasoning policy. In
the remainder of this paper, we will focus on bounded agents
that metareason optimally according Equation (4). For these
agents the optimal set of systems deﬁned in Equation 5 be-
comes
M= arg max
MNEre"
max
k2M[f 0gE[Ujk]
1 +k
re+jMj
rm#
:(6)
Figure 2: The optimal number of systems for the 2AFC
problem as a function of the standard deviation of reand
1=rm. In this example E[re] = 10 .
Note that the optimal set of systems depends on the distri-
bution of the sampling cost reacross different environments.
In the following we assume that re1(;)was sam-
pled from a Gamma distribution so that acting is always as
least as costly as sampling ( re1).
Figure 1 shows a representative example1of the expected
utility per time as a function of the number of systems for
different metareasoning costs. Note that under a large range
of metareasoning costs the optimal number of systems is just
one, but as the costliness of metareasoning decreases, the op-
timal number of systems increases. However even when the
optimal number of systems is more than one, each additional
system tends to only result in a marginal increase in utility,
suggesting that one reason for few cognitive systems may be
that the beneﬁt of additional systems is very low.
Figure 2 shows that the optimal number of systems in-
creases with the variance of reand decreases with the cost-
liness of metareasoning (i.e.,1
rm). Interestingly, there is a
large set of plausible combinations of variability and metar-
easoning cost for which the bounded-optimal agent has two
cognitive systems. In addition, when the optimal number of
systems is two, then the gap between the values of the two
systems picked increases with the variance of re(see Table
1), resulting in one system that has high accuracy but high
cost and another system that has low accuracy and low cost,
which matches the characteristics of the systems posited by
dual-process accounts. Thus, the conditions under which we
would most expect to see two cognitive systems like the ones
suggested by dual-process theories are when the environ-
ment has high variability but there is also a correspondingly
1For all experiments reported in this paper, we found that al-
ternative values for E[re]or Var (re)did not change the qualitative
conclusions, unless otherwise indicated.
Figure 3: The number of actions it takes an agent to reach a
goal as a function of the number of BRTDP rollouts executed
before each action. For 0rollouts the expected number of
actions was 500(the maximum allowed).
high enough cost of metareasoning that the optimal number
of cognitive systems is only two.
Planning under Uncertainty
When an agent is unable to precompute an optimal policy
for an environment it must interleave planning with acting.
However, planning in the real world comes with a situation-
dependent cost, so an optimal agent must metareason to
adaptively choose how much to plan. The problem of choos-
ing how to choose the optimal sequence of computations
to plan which actions to take in the MDP corresponds to
a meta-level MDP (Hay et al. 2012).
Here, we study an agent solving a stochastic shortest path
problem in a ﬁnite-horizon MDP with states S, actionsA,
cost function c:S!R, transition probability t:SA
S![0;1], and horizon h. In the meta-level decision process
each time step consists of a thinking stage followed by an
acting stage. The agent is equipped with a set of planning
algorithmsT, and the cost of running an algorithm t2T
is given byf(t). The optimal policy in this metareasoning
problem can be expressed in terms of the trajectory of the
states and chosen planning algorithms at each time step:
= arg max
E2
4X
(s;t)2c(s) +f(t)3
5; (7)
wheredepends on the agent’s metalevel policy, .
Our agent’s thinking algorithms are based on bounded
real-time dynamic programming (BRTDP; McMahan,
Likhachev, and Gordan, 2005), an anytime online planning
algorithm for solving an MDP, that was previously used
by Lin et al. to metareason for planning under uncertainty.
BRTDP starts with an initial heuristic for a lower and up-
per bound on the value function of the MDP and updates
its estimate of the bounds via rollouts. As more rollouts aredone, the gap between the lower and upper bound decreases,
eventually converging to the optimal value function.
Because optimally solving a meta-level decision problem
is generally computationally intractable (Lin et al. 2015), we
focus on a special case where the agent’s thinking algorithms
differ only in the number of rollouts ( k) they perform before
each action. During the acting stage, the agent acts greedily
with respect to the upper bound. Thus the agent’s policy is
deﬁned entirely by k. This type of policy corresponds to the
Think*Act policy from Lin et al..
We consider environments with a constant cost for each
action (ca) and assume a constant execution cost ( ce) for
each rollout of BRTDP. We can therefore reparameterize the
costs by the ratio of the cost of acting over the cost of think-
ing,re=ca
ce. This simpliﬁes Equation 7 to
k= arg min
k2N0
1 +k
re
E[Njk]; (8)
whereNis the number of actions taken in the MDP until
reaching the goal state or time horizon.
We assume that there is a distribution of potential MDPs
that our agent may act in, and while reis constant within
each MDP, it varies across different MDPs. Therefore, op-
timally allocating ﬁnite computational resources requires
metareasoning. Metareasoning incurs a cost, cm, that can be
similarly reparameterized as rm=ca=cm. Assuming that
the agent chooses optimally from its set of planning systems,
the optimal set of systems that it should be equipped with is
M= arg min
MNEre
min
k2M[f 0g(re+k)E[Njk]
re
+jMj
rm:(9)
We investigated the size and composition of the optimal
set of planning systems for a simple 2020grid world
where the agent’s goal is to get from the lower left corner
to the upper right corner with as little cost as possible. The
horizon was set to 500, the maximum number of rollouts at
any thinking stage to 10, and the depth of each BRTDP roll-
out to 10. BRTDP was initialized with a constant value func-
tion of 0for the lower bound and a constant value function
of106for the upper bound. This means that the agent’s ini-
tial policy was to act randomly–which is highly suboptimal.
For each environment, the ratio of the cost of action over
the cost of planning ( re) was again drawn from a Gamma
distribution and shifted by one, that is re1(;).
The expected number of steps required to achieve the goal
E[Njk]was estimated via simulation (see Figure 3).
We ﬁnd that all our results match the 2AFC setting ex-
tremely closely. Because the agent rarely reached the goal
without planning ( E[Njk= 0] = 500 ) one system pro-
vided the largest reduction in expected cost with each addi-
tional system providing at most marginal reductions (Figure
4). The optimal number of systems increased with the vari-
ance ofreand decreased with the metareasoning cost (1
rm).
This resulted in the optimal number of cognitive systems be-
ing two for a wide range of plausible combinations of vari-
ability and metareasoning cost (Figure 5). In addition, when
the number of systems was two, the difference between the
amount of planning performed by the two optimal systems
Figure 4: The expected cost incurred is a U-shaped func-
tion of the number of planning systems. As the costliness of
metareasoning,1
rm, decreases, the optimal number of sys-
tems increases. The expected cost of 0 systems was 500,
thus 1 system provided the greatest reduction in cost. In this
example E[re] = 100 , Var(re) = 105, andca= 1.
jMjVar(re)
103104105
1 9 7 7
2 7, 9 4, 7 2, 7
3 1, 7, 9 4, 7, 9 1, 4, 9
4 1, 2, 7, 9 2, 4, 7, 9 1, 4, 7, 9
Table 2: The optimal set of algorithms, M, for different val-
ues ofjMj as Var (re)increases while E[re] = 100 .
increased with the variance of re.2This resulted in one sys-
tem that does a high amount of planning but is costly and
another system that plans very little but is computationally
inexpensive, matching the characteristics of the two types of
systems postulated by dual-process theories.
Conclusion
We found that across two different domains the optimal
number and diversity of cognitive systems increases with the
variability of the environment but decreases with the cost of
metareasoning. Each additional system tends to provide at
most marginal improvements, so the solutions tend to favor
small numbers of cognitive systems, with two systems being
optimal across a wide range of plausible values for metarea-
soning cost and variability. Furthermore, when the optimal
number of cognitive systems was two, then one of them was
typically faster but more error-prone whereas the second one
was slower but more accurate, matching the systems posited
by dual-process theories.
2This observation holds until the variance becomes extremely
high (107for Table 2), in which case both systems move towards
lower values (Table 2). However, this is not a general problem but
merely a quirk of the skewed distribution we used for re.
Figure 5: The optimal number planning systems as a func-
tion of the standard deviation of reandrmforE[re] = 100 .
One limitation of our analysis is that the cognitive sys-
tems we studied are simple algorithms that abstract away
most of the complexity and sophistication of intelligent cog-
nitive systems. A second limitation is that there is no guar-
antee that the results we obtained in the domain of decision-
making will necessary transfer to all other domains of cog-
nition. However, it seems plausible that for most metarea-
soning systems the cost of metareasoning increases with the
number of systems. As long as this is the case, the opti-
mal number of cognitive systems should still depend on the
tradeoff between metareasoning cost and cognitive ﬂexibil-
ity studied above, even though its exact value may be differ-
ent. Thus, our key ﬁnding that the optimal number of sys-
tems increases with the variability of the environment and
decreases with the cost of metareasoning is likely to gener-
alize to other tasks and more complex architectures.
Future work in AI might apply our approach to de-
sign bounded-optimal metareasoning agents whose perfor-
mance comes closer to the impressive resource-efﬁciency
and ﬂexibility of the human mind. According to our analy-
sis, bounded agents should be equipped with a small number
of cognitive systems dependent on the variability agent’s en-
vironment and the cost of metareasoning rather than just one
or dozens of reasoning systems.
Our ﬁndings also have implications for the debate about
human rationality. In this debate, some researchers have
interpreted the existence of a fast, error-prone cognitive
system that violated the rules of logic, probability theory,
and expected utility theory as a sign of human irrational-
ity (Ariely 2009; Marcus 2009). By contrast, our analysis
suggests that having a fast but fallible cognitive system in
additional to a slow but accurate system can be boundedly
optimal.
A conclusive answer to the question whether it is bound-
edly optimal for humans to have two types of cognitive sys-
tems will require more rigorous estimates of the variability
of decision problems that people experience in their daily
lives and the cost of metareasoning. Regardless thereof, our
analysis suggests that the incoherence in human reasoning
and decision-making might be a signature of the rational use
of a bounded-optimal set of cognitive systems rather than a
sign of irrationality.
References
Anderson, M. L., and Oates, T. 2007. A review of recent
research in metareasoning and metalearning. AI Magazine
28(1):12.
Ariely, D. 2009. Predictably Irrational . New York: Harper
Collins.
Cox, M. T., and Raja, A. 2011. Metareasoning: Thinking about
thinking . Cambridge, MA: MIT Press.
Cushman, F. 2013. Action, outcome, and value a dual-system
framework for morality. Personality and social psychology re-
view 17(3):273–292.
Dolan, R. J., and Dayan, P. 2013. Goals and habits in the brain.
Neuron 80(2):312 – 325.
Evans, J. S. B. 2008. Dual-processing accounts of reasoning,
judgment, and social cognition. Annual Review of Psychology
59:255–278.
Gershman, S. J.; Horvitz, E. J.; and Tenenbaum, J. B. 2015.
Computational rationality: A converging paradigm for intelli-
gence in brains, minds, and machines. Science 349(6245):273–
278.
Greene, J. D. 2015. Beyond point-and-shoot morality: Why
cognitive (neuro) science matters for ethics. The Law & Ethics
of Human Rights 9(2):141–172.
Grifﬁths, T. L., and Tenenbaum, J. B. 2006. Optimal predictions
in everyday cognition. Psychological science 17(9):767–773.
Grifﬁths, T. L.; Lieder, F.; and Goodman, N. D. 2015. Ratio-
nal use of cognitive resources: Levels of analysis between the
computational and the algorithmic. Topics in cognitive science
7(2):217–229.
Hay, N.; Russell, S.; Tolpin, D.; and Shimony, S. 2012. Select-
ing Computations: Theory and Applications. In de Freitas, N.,
and Murphy, K., eds., Proceedings of the 28th conference on
Uncertainty in Artiﬁcial Intelligence . Corvallis: AUAI Press.
Horvitz, E., and Rutledge, G. 1991. Time-dependent utility
and action under uncertainty. In Proceedings of the Seventh
Conference on Uncertainty in Artiﬁcial Intelligence , 151–158.
Morgan Kaufmann Publishers Inc.
Horvitz, E. J.; Cooper, G. F.; and Heckerman, D. E. 1989. Re-
ﬂection and action under scarce resources: Theoretical princi-
ples and empirical study. In Proceedings of the Eleventh In-
ternational Joint Conference on Artiﬁcial Intelligence , 1121–
1127. San Mateo, CA: Morgan Kaufmann.
Icard, T. 2014. Toward boundedly rational analysis. In Pro-
ceedings of the 36th Annual Conference of the Cognitive Sci-
ence Society , 637–642.
Kahneman, D. 2011. Thinking, fast and slow . New York: Farrar,
Strauss and Giroux.
Keramati, M.; Dezfouli, A.; and Piray, P. 2011. Speed/accuracy
trade-off between the habitual and the goal-directed processes.
PLoS Computational Biology 7(5):e1002055.Lieder, F., and Grifﬁths, T. L. 2015. When to use which heuris-
tic: A rational solution to the strategy selection problem. In
Proceedings of the 37th annual conference of the cognitive sci-
ence society , 1362–1367.
Lieder, F.; Plunkett, D.; Hamrick, J. B.; Russell, S. J.; Hay, N.;
and Grifﬁths, T. 2014. Algorithm selection by rational metar-
easoning as a model of human strategy selection. In Advances
in Neural Information Processing Systems 27 , 2870–2878.
Lieder, F.; Grifﬁths, T.; and Goodman, N. 2012. Burn-in, bias,
and the rationality of anchoring. In Advances in neural infor-
mation processing systems 25 , 2690–2798.
Lieder, F.; Hsu, M.; and Grifﬁths, T. L. 2014. The high
availability of extreme events serves resource-rational decision-
making. In Proceedings of the 36th Annual Conference of the
Cognitive Science Society . Cognitive Science Society.
Lin, C. H.; Kolobov, A.; Kamar, E.; and Horvitz, E. 2015.
Metareasoning for planning under uncertainty. In Proceedings
of the 24th International Conference on Artiﬁcial Intelligence ,
1601–1609. AAAI Press.
Marcus, G. 2009. Kluge: The haphazard evolution of the human
mind . Boston: Houghton Mifﬂin Harcourt.
McMahan, H. B.; Likhachev, M.; and Gordon, G. J. 2005.
Bounded real-time dynamic programming: Rtdp with mono-
tone upper bounds and performance guarantees. In Proceed-
ings of the 22nd international conference on Machine learning ,
569–576. ACM.
Russell, S. J., and Subramanian, D. 1995. Provably bounded-
optimal agents. Journal of Artiﬁcial Intelligence Research
2:575–609.
Russell, S., and Wefald, E. 1989. On optimal game-tree search
using rational meta-reasoning. In Proceedings of the 11th in-
ternational joint conference on Artiﬁcial intelligence-Volume 1 ,
334–340. Morgan Kaufmann Publishers Inc.
Russell, S., and Wefald, E. 1991a. Principles of metareasoning.
Artiﬁcial intelligence 49(1-3):361–395.
Russell, S. J., and Wefald, E. 1991b. Do the right thing: studies
in limited rationality . Cambridge, MA: MIT press.
Schaul, T., and Schmidhuber, J. 2010. Metalearning. 5(6):4650.
revision 91489.
Smith-Miles, K. A. 2009. Cross-disciplinary perspectives on
meta-learning for algorithm selection. ACM computing surveys
41(1).
Stanovich, K. 2011. Rationality and the reﬂective mind . Ox-
ford: Oxford University Press.
Thornton, C.; Hutter, F.; Hoos, H. H.; and Leyton-Brown, K.
2013. Auto-WEKA: Combined selection and hyperparameter
optimization of classiﬁcation algorithms. In Proceedings of the
19th ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining , 847–855. New York: ACM.
Vul, E.; Goodman, N.; Grifﬁths, T. L.; and Tenenbaum, J. B.
2014. One and done? Optimal decisions from very few samples.
Cognitive science 38(4):599–637.