Chapter 2
Rationality and Intelligence: A Brief Update
Stuart Russell
Abstract The long-term goal of AI is the creation and understanding of intelli-
gence. This requires a notion of intelligence that is precise enough to allow the
cumulative development of robust systems and general results. The concept of
rational agency has long been considered a leading candidate to fulﬁll this role.
This paper, which updates a much earlier version (Russell, Artif Intell 94:57–77,
1997), reviews the sequence of conceptual shifts leading to a different candidate,
bounded optimality , that is closer to our informal conception of intelligence and
reduces the gap between theory and practice. Some promising recent developments
are also described.
Keywords Rationality • Intelligence • Bounded rationality • Metareasoning
2.1 Artiﬁcial Intelligence
AI is a ﬁeld whose ultimate goal has often been somewhat ill-deﬁned and subject
to dispute. Some researchers aim to emulate human cognition, others aim at
the creation of intelligence without concern for human characteristics, and still
others aim to create useful artifacts without concern for abstract notions of
intelligence.
My own motivation for studying AI is to create and understand intelligence
as a general property of systems, rather than as a speciﬁc attribute of humans.
I believe this to be an appropriate goal for the ﬁeld as a whole, and it certainly
includes the creation of useful artifacts—both as a spin-off from and a driving force
for technological development. The difﬁculty with this “creation of intelligence”
view, however, is that it presupposes that we have some productive notion of what
intelligence is. Cognitive scientists can say “Look, my model correctly predicted
S. Russell (/envelopeback)
Computer Science Division, University of California, Berkeley, CA 94720, USA
e-mail: russell@cs.berkeley.edu
© Springer International Publishing Switzerland 2016
V .C. Müller (ed.), Fundamental Issues of Artiﬁcial Intelligence ,
Synthese Library 376, DOI 10.1007/978-3-319-26485-1\_27
8 S. Russell
this experimental observation of human cognition,” and artifact developers can say
“Look, my system is worth billions of euros,” but few of us are happy with papers
saying “Look, my system is intelligent.”
A deﬁnition of intelligence needs to be formal —a property of the system’s
input, structure, and output—so that it can support analysis and synthesis. The
Turing test does not meet this requirement, because it references an informal (and
parochial) human standard. A deﬁnition also needs to be general , rather than a
list of specialized faculties—planning, learning, game-playing, and so on—with a
deﬁnition for each. Deﬁning each faculty separately presupposes that the faculty is
necessary for intelligence; moreover, the deﬁnitions are typically not composable
into a general deﬁnition for intelligence.
The notion of rationality as a property of agents —entities that perceive and act—
is a plausible candidate that may provide a suitable formal deﬁnition of intelligence.
Section 2.2provides background on the concept of agents. The subsequent sections,
following the development in Russell ( 1997 ), examine a sequence of deﬁnitions
of rationality from the history of AI and related disciplines, considering each as a
predicate Pthat might be applied to characterize systems that are intelligent:
•P1:Perfect rationality , or the capacity to generate maximally successful
behaviour given the available information.
•P2:Calculative rationality , or the in-principle capacity to compute the perfectly
rational decision given the initially available information.
•P3:Metalevel rationality , or the capacity to select the optimal combination of
computation-sequence-plus-action, under the constraint that the action must be
selected by the computation.
•P4:Bounded optimality , or the capacity to generate maximally successful
behaviour given the available information and computational resources.
For each P, I shall consider three simple questions. First, are P-systems interesting,
in the sense that their behaviour is plausibly describable as intelligent? Second,
could P-systems ever exist? Third, to what kind of research and technological
development does the study of P-systems lead?
Of the four candidates, P4, bounded optimality, comes closest to meeting the
needs of AI research. It is more suitable than P1through P3because it is a
real problem with real and desirable solutions, and also because it satisﬁes some
essential intuitions about the nature of intelligence. Some important questions about
intelligence can only be formulated and answered within the framework of bounded
optimality or some relative thereof.
2.2 Agents
In the early decades of AI’s history, researchers tended to deﬁne intelligence with
respect to speciﬁc tasks and the internal processes those tasks were thought to
require in humans. Intelligence was believed to involve (among other things) the
2 Rationality and Intelligence 9
ability to understand language, the ability to reason logically, and the ability to
solve problems and construct plans to satisfy goals. At the core of such capabilities
was a store of knowledge. The standard conception of an AI system was as a
sort of consultant : something that could be fed information and could then answer
questions. The output of answers was not thought of as an action about which the AI
system had a choice, any more than a calculator has a choice about what numbers
to display on its screen given the sequence of keys pressed.
The view that AI is about building intelligent agents —entities that sense their
environment and act upon it—became the mainstream approach of the ﬁeld only
in the 1990s (Russell and Norvig 1995 ; Dean et al. 1995 ), having previously been
the province of specialized workshops on “situatedness” and “embeddedness”. The
“consultant” view is a special case in which answering questions is a form of
acting—a change of viewpoint that occurred much earlier in the philosophy of
language with the development of speech act theory. Now, instead of simply giving
answers, a consulting agent could refuse to do so on the grounds of privacy or
promise to do so in return for some consideration. The agent view also naturally
encompasses the full variety of tasks and platforms—from robots and factories
to game-playing systems and ﬁnancial trading systems—in a single theoretical
framework.
What matters about an agent is what it does, not how it does it. An agent can
be deﬁned mathematically by an agent function that speciﬁes how an agent behaves
under all circumstances. More speciﬁcally, let Obe the set of percepts that the agent
can observe at any instant (with O/ETXbeing the set of observation sequences of any
length) and Abe the set of possible actions the agent can carry out in the external
world (including the action of doing nothing). The agent function is a mapping
fWO/ETX!A. This deﬁnition is depicted in the upper half of Fig. 2.1.
As we will see in Sect. 2.3, rationality provides a normative prescription for agent
functions and does not specify—although it does constrain—the process by which
the actions are selected. Rather than assume that a rational agent must, for example,
reason logically or calculate expected utilities, the arguments for (Nilsson 1991 )o r
against (Agre and Chapman 1987 ; Brooks 1989 ) the inclusion of such cognitive
Fig. 2.1 The agent receives
percepts from the
environment and generates a
behaviour which in turn
causes the environment to
generate a state history. The
performance measure
evaluates the state history to
arrive at the value of the agentPercept history
Behaviour
State historyAgent function
Environment
Performance measure Value
10 S. Russell
faculties must justify their position on the grounds of efﬁcacy in representing a
desirable agent function. A designer of agents has, a priori, complete freedom in
choosing the speciﬁcations, boundaries, and interconnections of subsystems, as long
they they compose to form a complete agent. In this way one is more likely to avoid
the “hallucination” problem that arises when the fragility of a subsystem is masked
by having an intelligent human providing input to it and interpreting its outputs.
Another important beneﬁt of the agent view of AI is that it connects the
ﬁeld directly to others that have traditionally looked on the embedded agent as a
natural topic of study, including economics, operations research, control theory, and
even evolutionary biology. These connections have facilitated the importation of
technical ideas (Nash equilibria, Markov decision processes, and so on) into AI,
where they have taken root and ﬂourished.
2.3 Perfect Rationality
So which agent functions are intelligent? Clearly, doing the right thing is more
intelligent that doing the wrong thing. The rightness of actions is captured by the
notion of rationality: informally, an action is rational to the extent that is consistent
with the agent’s goals (or the task for which it was designed), from the point of view
of the information possessed by the agent.
Rationality is, therefore, always understood relative to the agent’s ultimate goals.
These are expressed mathematically by a performance measure Uon sequences
of environment states. Let V.f;E;U/denote the expected value according to U
obtained by an agent function fin environment class E, where (for now) we will
assume a probability distribution over elements of E. Then a perfectly rational agent
is deﬁned by an agent function foptsuch that
foptDargmaxfV.f;E;U/ (2.1)
This is just a fancy way of saying that the best agent does the best it can. The point is
that perfectly rational behaviour is a well-deﬁned function of the task environment
ﬁxed by EandU.
Turning to the three questions listed in Sect. 2.1: Are perfectly rational agents
interesting things to have? Yes, certainly—if you have one handy, you prefer it
to any other agent. A perfectly rational agent is, in a sense, perfectly intelligent.
Do they exist? Alas no, except for very simple task environments, such as those
in which every behavior is optimal (Simon 1958 ). Physical mechanisms take
time to perform computations, while real-world decisions generally correspond to
intractable problem classes; imperfection is inevitable.
Despite their lack of existence, perfectly rational agents have, like imaginary
numbers, engendered a great deal of interesting research. For example, economists
prove nice results about economies populated by them and game-theoretic mecha-
nism designers much prefer to assume perfect rationality on the part of each agent.
2 Rationality and Intelligence 11
Far more important for AI, however, was the reduction from a global optimization
problem (Eq. 2.1) to a local one: from the perfect rationality of agents to the
perfect rationality of individual actions. That is, a perfectly rational agent is one
that repeatedly picks an action that maximizes the expected utility of the next
state. This reduction involved three separate and largely unconnected results: the
axiomatic utility theory of von Neumann and Morgenstern ( 1944 ) (which actually
takes for granted the agent’s ability to express preferences between distributions
over immediate outcomes), Bellman’s 1957 theory of sequential decisions, and
Koopmans’ 1972 analysis of preferences over time in the framework of multiat-
tribute utility theory (Keeney and Raiffa 1976 ).
While utility is central to the decision-theoretic notion of perfect rationality,
goals are usually considered to deﬁne the task for a logic-based agent: according
to Newell ( 1982 ), such an agent is perfectly rational if each action is part of a plan
that will achieve one of the agent’s goals. There have been attempts to deﬁne goals in
terms of utilities, beginning with Wellman and Doyle ( 1991 ), but difﬁculties remain
because goals are essentially incomplete as task speciﬁcations. They do not specify
what to do when goal achievement cannot be guaranteed, or when goals conﬂict,
or when several plans are available for achieving a goal, or when the agent has
achieved all its goals. It may be better to interpret goals not as primary deﬁnitions of
the agent’s task but as subsidiary devices for focusing computational effort with an
overall decision-theoretic context. For example, someone moving to a new city may,
after weighing many alternatives and tradeoffs under uncertainty, settle on the goal
of buying a particular apartment and thereafter focus their deliberations on ﬁnding
a plan to achieve that goal, to the exclusion of other possibilities. At the moment we
do not have a good understanding of goal formation by a decision-theoretic agent,
but it is clear that such behavior cannot be analyzed within the framework of perfect
rationality.
As discussed so far, the framework does not say where the beliefs and the
performance measure reside—they could be in the head of the designer or of the
agent itself. If they are in the designer’s head, the designer has to do all the work
to build the agent function, anticipating all possible percept sequences. If they are
in the agent’s head, the designer can delegate the work to the agent; for example,
in the setting of reinforcement learning, it is common to equip the agent with
a ﬁxed capacity to extract a distinguished reward signal from the environment,
leaving the agent to learn the corresponding utility function on states. The designer
may also equip the agent with a prior over environments (Carnap 1950 ), leaving
the agent to perform Bayesian updating as it observes the particular environment
it inhabits. Solomonoff ( 1964 ) and Kolmogorov ( 1965 ) explored the question of
universal priors over computable environments; universality, unfortunately, leads to
undecidability of the learning problem. Hutter ( 2005 ) makes an ambitious attempt to
deﬁne a universal yet computable version of perfect rationality, but does not pretend
to provide the instantaneous decisions required for an actual P1-system; instead, this
work belongs in the realm of P2-systems, or calculatively rational agents.
Perhaps the biggest open question for the theory of perfect rationality lies
in its extension from single-agent to multi-agent environments. Game theorists
12 S. Russell
have proposed many solution concepts —essentially, deﬁnitions of admissible
strategies—but have not identiﬁed one that yields a unique recommendation (up to
tie-breaking) for what to do (Shoham and Leyton-Brown 2009 ).
2.4 Calculative Rationality
The theory of P1, perfect rationality, says nothing about implementation; P2,
calculative rationality, on the other hand, is concerned with programs for computing
the choices that perfect rationality stipulates.
To discuss calculative rationality, then, we need to discuss programs. The agent’s
decision-making system can be divided into the machine M , which is considered
ﬁxed, and the agent program l , which the designer chooses from the space LMof
all programs that the machine supports. ( Mneed not be a raw physical computer, of
course; it can be a software “virtual machine” at any level of abstraction.) Together,
the machine Mand the agent program ldeﬁne an agent function fDAgent .l;M/,
which, as noted above, is subject to evaluation. Conversely, lis an implementation
of the agent function fonM; there may, of course, be many such implementations,
but also, crucially, there may be none (see Sect. 2.6).
It is important to understand the distinction between an agent program and the
agent function it implements. An agent program may receive as input the current
percept, but also has internal state that reﬂects, in some form, the previous percepts.
It outputs actions when they have been selected. From the outside, the behaviour
of the agent consists of the selected actions interspersed with inaction (or whatever
default actions the machine generates). Depending on how long the action selection
takes, many percepts may go by unnoticed by the program.
Calculative rationality is displayed by programs that, if executed inﬁnitely fast ,
would result in perfectly rational behaviour. That is, at time t, assuming it is
not already busy computing its choice for some previous time step, the program
computes the value fopt.Œo1;:::; ot/c141/.
Whereas perfect rationality is highly desirable but does not exist, calculative
rationality often exists—its requirements can be fulﬁlled by real programs for many
settings—but it is not necessarily a desirable property. For example, a calculatively
rational chess program will choose the “right” move, but may take 1050times too
long to do so.
The pursuit of calculative rationality has nonetheless been the main activity of
theoretically well-founded research in AI; the ﬁeld has been ﬁlling in a table whose
dimensions are the various environment properties (deterministic or stochastic, fully
or partially observable, discrete or continuous, dynamic or static, single-agent or
multi-agent, known or unknown) for various classes of representational formalisms
(atomic, propositional, or relational). In the logical tradition, planning systems and
situation-calculus theorem-provers satisfy the conditions of calculative rationality
for discrete, fully observable environments; moreover, the power of ﬁrst-order logic
renders the required knowledge practically expressible for a wide range of problems.
2 Rationality and Intelligence 13
In the decision-theoretic tradition, there are calculatively rational agents based on
algorithms for solving fully or partially observable Markov decision processes,
deﬁned initially atomic by atomic formalisms (e.g., transition matrices), later by
propositional representations (e.g., dynamic Bayesian networks), and now by ﬁrst-
order probabilistic languages (Srivastava et al. 2014 ). For continuous domains,
stochastic optimal control theory (Kumar and Varaiya 1986 ) has solved some
restricted classes of problems, while many others remain open.
In practice, neither the logical nor the decision-theoretic traditions can avoid
the intractability of the decision problems posed by the requirement of calculative
rationality. One response, championed by Levesque ( 1986 ), is to rule out sources
of exponential complexity in the representations and reasoning tasks addressed, so
that calculative and perfect rationality coincide—at least, if we ignore the little
matter of polynomial-time computation. The accompanying research results on
tractable sublanguages are perhaps best seen as indications of where complexity
may be an issue rather than as a solution to the problem of complexity, since real-
world problems usually require exponentially large representations under the input
restrictions stipulated for tractable inference (Doyle and Patil 1991 ).
The most common response to complexity has been to use various speedup
techniques and approximations in the hope of getting reasonable behaviour. AI has
developed a very powerful armoury of methods for reducing the computational cost
of decision making, including heuristic evaluation functions, pruning techniques,
sampling methods, problem decomposition, hierarchical abstraction, compilation,
and the application of metalevel control. Although some of these methods can retain
guarantees of optimality and are effective for moderately large problems that are
well structured, it is inevitable that intelligent agents will be unable to act rationally
in all circumstances. This observation has been a commonplace since the very
beginning of AI. Yet systems that select suboptimal actions fall outside calculative
rationality per se, and we need a better theory to understand them.
2.5 Metalevel Rationality
Metalevel rationality, also called Type II rationality by Good ( 1971 ), is based
on the idea of ﬁnding an optimal tradeoff between computational costs and
decision quality. Although Good never made his concept of Type II rationality very
precise—he deﬁnes it as “the maximization of expected utility taking into account
deliberation costs ”—it is clear that the aim was to take advantage of some sort
ofmetalevel architecture to implement this tradeoff. Metalevel architecture is a
design philosophy for intelligent agents that divides the agent program into two
(or more) notional parts. The object level carries out computations concerned with
the application domain—for example, projecting the results of physical actions,
computing the utility of certain states, and so on. The metalevel is a second decision-
making process whose application domain consists of the object-level computations
themselves and the computational objects and states that they affect. Metareasoning
14 S. Russell
has a long history in AI, going back at least to the early 1970s (see Russell and
Wefald 1991a , for historical details). One can also view selective search methods
and pruning strategies as embodying metalevel expertise concerning the desirability
of pursuing particular object-level search operations.
The theory of rational metareasoning formalizes Good’s intuition that the
metalevel can “do the right thinking.” The basic idea is that object-level compu-
tations are actions with costs (the passage of time) and beneﬁts (improvements
in decision quality). A rational metalevel selects computations according to their
expected utility. Rational metareasoning has as a precursor the theory of information
value (Howard 1966 )—the notion that one can calculate the decision-theoretic
value of acquiring an additional piece of information by simulating the decision
process that would be followed given each possible outcome of the information
request, thereby estimating the expected improvement in decision quality averaged
over those outcomes. The application to computational processes, by analogy to
information-gathering, seems to have originated with Matheson ( 1968 ). In AI,
Horvitz ( 1987 ,1989 ), Breese and Fehling ( 1990 ), and Russell and Wefald ( 1989 ,
1991a ,b) all showed how the idea of value of computation could solve the basic
problems of real-time decision making.
Perhaps the simplest form of metareasoning occurs when the object level is
viewed by the metalevel as a black-box anytime (Dean and Boddy 1988 )o rﬂex-
ible(Horvitz 1987 ) algorithm, i.e., an algorithm whose decision quality depends on
the amount of time allocated to computation. This dependency can be represented by
aperformance proﬁle and the metalevel simply ﬁnds the optimal tradeoff between
decision quality and the cost of time (Simon 1955 ). More complex problems arise
if one wishes to build complex real-time systems from anytime components. First,
one has to ensure the interruptibility of the composed system—that is, to ensure
that the system as a whole can respond robustly to immediate demands for output.
The solution is to interleave the execution of all the components, allocating time
to each component so that the total time for each complete iterative improvement
cycle of the system doubles at each iteration. In this way, we can construct a
complex system that can handle arbitrary and unexpected real-time demands just
as if it knew the exact time available in advance, with just a small ( /DC44) constant
factor penalty in speed (Russell and Zilberstein 1991 ). Second, one has to allocate
the available computation optimally among the components to maximize the total
output quality. Although this is NP-hard for the general case, it can be solved
in time linear in program size when the call graph of the components is tree-
structured (Zilberstein and Russell 1996 ). Although these results are derived in the
simple context of anytime algorithms with well-deﬁned performance proﬁles, they
point to the possibility of more general schemes for allocation of computational
resources in complex systems.
The situation gets more interesting when the metalevel can go inside the object
level and direct its activities, rather than just switching it on and off. The work
done with Eric Wefald looked in particular at search algorithms, in which the
object-level computations extend projections of the results of various courses of
actions further into the future. For example, in chess programs, each object-level
2 Rationality and Intelligence 15
computation expands a leaf node of the game tree and advances the clock; it is
an action in the so-called joint-state Markov decision process , whose state space
is the Cartesian product of the object-level state space (which includes time) and
the metalevel state space of computational states—in this case, partially generated
game trees. The actions available are to expand a leaf of the game tree or to terminate
search and make a move on the board. It is possible to derive a greedy or myopic
approximation to the value of each possible computation and thereby to control
search effectively. This method was implemented for two-player games, two-player
games with chance nodes, and single-agent search. In each case, the same general
metareasoning scheme resulted in efﬁciency improvements of roughly an order
of magnitude over traditional, highly-engineered algorithms (Russell and Wefald
1991a ).
An independent thread of research on metalevel control began with work by Koc-
sis and Szepesvari ( 2006 ) on the UCT algorithm, which operates in the context of
Monte Carlo tree search (MCTS) algorithms. In MCTS, each computation takes the
form of a simulation of a randomized sequence of actions leading from a leaf of the
current tree to a terminal state. UCT is a metalevel heuristic for selecting a leaf from
which to conduct the next simulation, and has contributed to dramatic improvements
in Go-playing algorithms over the last few years. It views the metalevel decision
problem as a multi-armed bandit problem (Berry and Fristedt 1985 ) and applies
an asymptotically near-optimal bandit decision rule recursively to make a choice
of which computation to do next. The application of bandit methods to metalevel
control seems quite natural, because a bandit problem involves deciding where to
do the next “experiment” to ﬁnd out how good each bandit arm is. Are bandit
algorithms such as UCT approximate solutions to some particular case of the
metalevel decision problem deﬁned by Russell and Wefald? The answer, perhaps
surprisingly, is no. The essential difference is that, in bandit problems, every
trial involves executing a real object-level action with real costs, whereas in the
metareasoning problem the trials are simulations whose cost is usually independent
of the utility of the action being simulated. Hence UCT applies bandit algorithms to
problems that are not bandit problems. A careful analysis (Hay et al. 2012 )s h o w s
that metalevel problems in their simplest form are isomorphic to selection problems ,
a class of statistical decision problems studied since the 1950s in quality control and
other areas. Hay et al. develop a rigorous mathematical framework for metalevel
problems, showing that, for some cases, hard upper bounds can be established for
the number of computations undertaken by an optimal metalevel policy, while, for
other cases, the optimal policy may (with vanishingly small probability) continue
computing long past the point where the cost of computation exceeds the value of
the object-level problem.
Achieving accurate metalevel control remains a difﬁcult open problem in the
general case. Myopic strategies—considering just one computation at a time—can
fail in cases where multiple computations are required to have any chance of altering
the agent’s current preferred action. Obviously, the problem of optimal selection
of computation sequences is at least as intractable as the underlying object-level
problem. One possible approach could be to apply metalevel reinforcement learning,
16 S. Russell
especially as the “reward function” for computation—that is, the improvement in
decision quality—is easily available to the metalevel post hoc. It seems plausible
that the human brain has such a capacity, since its hardware is unlikely to have a
method of deriving clever new algorithms for new classes of decision problems.
Indeed, there is a sense in which algorithms are not a necessary part of AI
systems . Instead, one can imagine a general, adaptive process of rationally guided
computation interacting with properties of the environment to produce more and
more efﬁcient decision making.
Although rational metareasoning seems to be a useful tool in coping with
complexity, the concept of metalevel rationality as a formal framework for resource-
bounded agents does not seem to hold water. The reason is that, since metareasoning
is expensive, it cannot be carried out optimally. Thus, while a metalevel-rational
agent would be highly desirable (although not quite as desirable as a perfectly
rational agent), it does not usually exist. The history of object-level rationality has
repeated itself at the metalevel: perfect rationality at the metalevel is unattainable
and calculative rationality at the metalevel is useless. Therefore, a time/optimality
tradeoff has to be made for metalevel computations, as for example with the myopic
approximation mentioned above. Within the framework of metalevel rationality,
however, there is no way to identify the appropriate tradeoff of time for metalevel
decision quality. Any attempt to do so via a metametalevel simply results in a
conceptual regress. Furthermore, it is entirely possible that in some environments,
the most effective agent design will do no metareasoning at all, but will simply
respond to circumstances. These considerations suggest that the right approach is
to step outside the agent, as it were; to refrain from micromanaging the individual
decisions made by the agent. This is the approach taken in bounded optimality.
2.6 Bounded Optimality
The difﬁculties with perfect rationality and metalevel rationality arise from the
imposition of optimality constraints on actions orcomputations , neither of which
the agent designer directly controls. The basic problem is that not all agent functions
arefeasible (Russell and Subramanian 1995 ) on a given machine M; the feasible
functions are those implemented by some program for M. Thus, the optimization
over functions in Eq. ( 2.1) is meaningless. It may be pointed out that not all agent
functions are computable, but feasibility is in fact much stricter than computability,
because it relates the operation of a program on a formal machine model with ﬁnite
speed to the actual temporal behaviour generated by the agent.
Given this view, one is led immediately to the idea that optimal feasible behaviour
is an interesting notion, and to the idea of ﬁnding the program that generates it. P4,
bounded optimality, is exhibited by a program loptthat satisﬁes
loptDargmaxl2LMV.Agent .l;M/;E;U/: (2.2)
2 Rationality and Intelligence 17
Certainly, one would be happy to have lopt, which is as intelligent as possible given
the computational resources and structural constraints of the machine M. Certainly,
bounded optimal programs exist, by deﬁnition. And the research agenda appears to
be very interesting, even though it is difﬁcult.
In AI, the idea of bounded optimality ﬂoated around among several discussion
groups interested in resource-bounded rationality in the late 1980s, particularly
those at Rockwell (organized by Michael Fehling) and Stanford (organized by
Michael Bratman). The term itself seems to have been originated by Horvitz ( 1989 ),
who deﬁned it informally as “the optimization of computational utility given a set
of assumptions about expected problems and constraints on resources.”
Similar ideas also emerged in game theory, where there has been a shift from
consideration of optimal decisions in games to a consideration of optimal decision-
making programs. This leads to different results because it limits the ability of each
agent to do unlimited simulation of the other, who is also doing unlimited simulation
of the ﬁrst, and so on. Depending on the precise machine limitations chosen, it is
possible to prove, for example, that the iterated Prisoner’s Dilemma has cooperative
equilibria (Megiddo and Wigderson 1986 ; Papadimitriou and Yannakakis 1994 ;
Tennenholtz 2004 ), which is not the case for arbitrary strategies.
Philosophy has also seen a gradual evolution in the deﬁnition of rationality.
There has been a shift from consideration of act utilitarianism —the rationality
of individual acts—to rule utilitarianism , or the rationality of general policies for
acting. The requirement that policies be feasible for limited agents was discussed
extensively by Cherniak ( 1986 ) and Harman ( 1983 ). A philosophical proposal
generally consistent with the notion of bounded optimality can be found in the
“Moral First Aid Manual” (Dennett 1988 ). Dennett explicitly discusses the idea
of reaching an optimum within the space of feasible decision procedures, using as
an example the Ph.D. admissions procedure of a philosophy department. He points
out that the bounded optimal admissions procedure may be somewhat messy and
may have no obvious hallmark of “optimality”—in fact, the admissions committee
may continue to tinker with it since bounded optimal systems may have no way to
recognize their own bounded optimality.
My work with Devika Subramanian placed the general idea of bounded optimal-
ity in a formal setting and derived the ﬁrst rigorous results on bounded optimal
programs (Russell and Subramanian 1995 ). This required setting up completely
speciﬁed relationships among agents, programs, machines, environments, and time.
We found this to be a very valuable exercise in itself. For example, the informal
notions of “real-time environments” and “deadlines” ended up with deﬁnitions
rather different than those we had initially imagined. From this foundation, a very
simple machine architecture was investigated in which the program consists of a
collection of decision procedures with ﬁxed execution time and decision quality.
In a “stochastic deadline” environment, it turns out that the utility attained by
running several procedures in sequence until interrupted is often higher than that
attainable by any single decision procedure. That is, it is often better ﬁrst to prepare
a “quick and dirty” answer before embarking on more involved calculations in case
the latter do not ﬁnish in time. In an entirely separate line of inquiry, Livnat and
18 S. Russell
Pippenger ( 2006 ) show that, under a bound on the total number of gates in a circuit-
based agent, the bounded optimal conﬁguration may, for some task environments,
involve two or more separate circuits that compete for control of the effectors and,
in essence, pursue separate goals.
The interesting aspect of these results, beyond their value as a demonstration
of nontrivial proofs of bounded optimality, is that they exhibit in a simple way
what I believe to be a major feature of bounded optimal agents: the fact that
the pressure towards optimality within a ﬁnite machine results in more complex
program structures. Intuitively, efﬁcient decision-making in a complex environment
requires a software architecture that offers a wide variety of possible computational
options, so that in most situations the agent has at least some computations available
that provide a signiﬁcant increase in decision quality.
One objection to the basic model of bounded optimality outlined above is that
solutions are not robust with respect to small variations in the environment or
the machine. This in turn would lead to difﬁculties in analyzing complex system
designs. Theoretical computer science faced the same problem in describing the
running time of algorithms, because counting steps and describing instruction
sets exactly gives the same kind of fragile results on optimal algorithms. The
O./notation was developed to provide a way to describe complexity that is
independent of machine speeds and implementation details and that supports the
cumulative development of complexity results. The corresponding notion for agents
is asymptotic bounded optimality (ABO) (Russell and Subramanian 1995 ). As with
classical complexity, we can deﬁne both average-case and worst-case ABO, where
“case” here means the environment. For example, worst-case ABO is deﬁned as
follows:
Worst-case asymptotic bounded optimality
an agent program l is timewise (or spacewise) worst-case ABO in Eon M
iff
9k;n08l0;nn>n0)V/ETX.Agent .l;kM/;E;U;n//NAK
V/ETX.Agent .l0;M/;E;U;n/
where kM denotes a version of M speeded up by a factor k (or with
k times more memory) and V/ETX.f;E;U;n/is the minimum value of
V.f;E;U/for all E in Eof complexity n.
In English, this means that the program is basically along the right lines if it just
needs a faster (larger) machine to have worst-case behaviour as good as that of any
other program in all environments.
Another possible objection to the idea of bounded optimality is that it simply
shifts the intractable computational burden of metalevel rationality from the agent’s
metalevel to the designer’s object level. Surely, one might argue, the designer now
has to solve ofﬂine all the metalevel optimization problems that were intractable
when online. This argument is not without merit—indeed, it would be surprising
2 Rationality and Intelligence 19
if the agent design problem turns out to be easy. There is however, a signiﬁcant
difference between the two problems, in that the agent designer is presumably
creating an agent for an entire class of environments, whereas the putative metalevel
agent is working in a speciﬁc environment. That this can make the problem easier
for the designer can be seen by considering the example of sorting algorithms. It may
be very difﬁcult indeed to sort a list of a trillion elements, but it is relatively easy to
design an asymptotically optimal algorithm for sorting. In fact, the difﬁculties of the
two tasks are unrelated. The unrelatedness would still hold for BO as well as ABO
design, but the ABO deﬁnitions make it a good deal clearer.
It can be shown easily that worst-case ABO is a generalization of asymptot-
ically optimal algorithms, simply by constructing a “classical environment” in
which classical algorithms operate and in which the utility of the algorithm’s
behaviour is a decreasing positive function of runtime if the output is correct
and zero otherwise. Agents in more general environments may need to trade off
output quality for time, generate multiple outputs over time, and so on. As an
illustration of how ABO is a useful abstraction, one can show that under certain
restrictions one can construct universal ABO programs that are ABO for any time
variation in the utility function, using the doubling construction from Russell and
Zilberstein ( 1991 ). Further directions for bounded optimality research are discussed
below.
2.7 What Is to Be Done?
The 1997 version of this paper described two agendas for research: one agenda
extending the tradition of calculative rationality and another dealing with metarea-
soning and bounded optimality.
2.7.1 Improving the Calculative Toolbox
The traditional agenda took as its starting point the kind of agent could be built
using the components available at that time: a dynamic Bayesian network to model
a partially observable, stochastic environment; parametric learning algorithms to
improve the model; a particle ﬁltering algorithm to keep track of the environment
state; reinforcement learning to improve the decision function given the state
estimate. Such an architecture “breaks” in several ways when faced with the
complexity of real-world environments (Russell 1998 ):
1. Dynamic Bayesian networks are not expressive enough to handle environments
with many related objects and uncertainty about the existence and identity of
objects; a more expressive language—essentially a uniﬁcation of probability and
ﬁrst-order logic—is required.
20 S. Russell
2. A ﬂat space of primitive action choices, especially when coupled with a greedy
decision function based on reinforcement learning, cannot handle environments
where the relevant time scales are much longer than the duration of a single
primitive action. (For example, a human lifetime involves tens of trillions
of primitive muscle activation cycles.) The agent architecture must support
hierarchical representations of behaviour, including high-level actions over long
time scales.
3. Attempting to learn a value function accurate enough to support a greedy one-
step decision procedure is unlikely to work; the decision function must support
model-based lookahead over a hierarchical action model.
On this traditional agenda, a great deal of progress has occurred. For the ﬁrst item,
there are declarative (Milch et al. 2005 ) and procedural (Pfeffer 2001 ; Goodman
et al. 2008 )probabilistic programming languages that have the required expressive
power. For the second item, a theory of hierarchical reinforcement learning has been
developed (Sutton et al. 1999 ; Parr and Russell 1998 ). The theory can be applied to
agent architectures deﬁned by arbitrary partial programs —that is, agent programs
in which the choice of action at any point may be left unspeciﬁed (Andre and
Russell 2002 ;M a r t h ie ta l . 2005 ). The hierarchical reinforcement learning process
converges in the limit to the optimal completion of the agent program, allowing the
effective learning of complex behaviours that cover relatively long time scales. For
the third item, lookahead over long time scales, a satisfactory semantics has been
deﬁned for high-level actions, at least in the deterministic setting, enabling model-
based lookahead at multiple levels of abstraction (Marthi et al. 2008 ).
These are promising steps, but many problems remain unsolved. From a practical
point of view, inference algorithms for expressive probabilistic languages remain far
too slow, although this is the subject of intense study at present in many research
groups around the world. Furthermore, algorithms capable of learning new model
structures in such languages are in their infancy. The same is true for algorithms
that construct new hierarchical behaviours from more primitive actions: it seems
inevitable that intelligent systems will need high-level actions, but as yet we do not
know how to create new ones automatically. Finally, there have been few efforts at
integrating these new technologies into a single agent architecture. No doubt such
an attempt will reveal new places where our ideas break and need to be replaced
with better ones.
2.7.2 Optimizing Computational Behaviour
A pessimistic view of Eq. ( 2.2) is that it requires evaluating every possible program
in order to ﬁnd one that works best—hardly the most promising or original strategy
for AI research. But in fact the problem has a good deal of structure and it is possible
to prove bounded optimality results for reasonably general classes of machines and
task environments.
2 Rationality and Intelligence 21
Modular design using a hierarchy of components is commonly seen as the only
way to build reliable complex systems. The components fulﬁll certain behavioural
speciﬁcations and interact in well-deﬁned ways. To produce a composite bounded-
optimal design, the optimization problem involves allocating execution time to
components (Zilberstein and Russell 1996 ) or arranging the order of execution of
the components (Russell and Subramanian 1995 ) to maximize overall performance.
As illustrated earlier in the discussion of universal ABO algorithms, the techniques
for optimizing temporal behaviour are largely orthogonal to the content of the
system components, which can therefore be optimized separately. Consider, for
example, a composite system that uses an anytime inference algorithm over a
Bayesian network as one of its components. If a learning algorithm improves
the accuracy of the Bayesian network, the performance proﬁle of the inference
component will improve, which will result in a reallocation of execution time that
is guaranteed to improve overall system performance. Thus, techniques such as the
doubling construction and the time allocation algorithm of Zilberstein and Russell
(1996 ) can be seen as domain-independent tools for agent design. They enable
bounded optimality results that do not depend on the speciﬁc temporal aspects of
the environment class. As a simple example, we might prove that a certain chess
program design is ABO for all time controls ranging from blitz to full tournament
play.
The results obtained so far for optimal time allocation have assumed a static,
ofﬂine optimization process with predictable component performance proﬁles and
ﬁxed connections among components. One can imagine far more subtle designs in
which individual components must deal with unexpectedly slow or fast progress in
computations and with changing needs for information from other components. This
might involve exchanging computational resources among components, establishing
new interfaces, and so on. This is more reminiscent of a computational market,
as envisaged by Wellman ( 1994 ), than of the classical subroutine hierarchies, and
would offer a useful additional level of abstraction in system design.
2.7.3 Learning and Bounded Optimality
In addition to combinatorial optimization of the structure and temporal behaviour of
an agent, we can also use learning methods to improve the design:
•T h e content of an agent’s knowledge base can of course be improved by inductive
learning. Russell and Subramanian ( 1995 ) show that approximately bounded
optimal designs can be guaranteed with high probability if each component
is learned in such a way that its output quality is close to optimal among
all components of a given execution time. Results from statistical learning
theory, particularly in the agnostic learning and empirical risk minimization
models (Kearns et al. 1992 ; Vapnik 2000 ), can provide learning methods—such
as support vector machines—with the required properties. The key additional
22 S. Russell
step is to analyze the way in which slight imperfection in each component carries
through to slight imperfection in the whole agent.
•Reinforcement learning can be used to learn value information such as utility
functions, and several kinds of /SI-ıconvergence guarantees have been established
for such algorithms. Applied in the right way to the metalevel decision problem,
a reinforcement learning process can be shown to converge to a bounded-optimal
conﬁguration of the overall agent.
•Compilation methods such as explanation-based learning can be used to trans-
form an agent’s representations to allow faster decision making. Several agent
architectures including S OAR (Laird et al. 1986 ) use compilation to speed up
all forms of problem solving. Some nontrivial results on convergence have been
obtained by Tadepalli ( 1991 ), based on the observation that after a given amount
of experience, novel problems for which no solution has been stored should be
encountered only infrequently.
Presumably, an agent architecture can incorporate all these learning mechanisms.
One of the issues to be faced by bounded optimality research is how to prove
convergence results when several adaptation and optimization mechanisms are
operating simultaneously.
2.7.4 Ofﬂine and Online Mechanisms
One can distinguish between ofﬂine and online mechanisms for constructing
bounded-optimal agents. An ofﬂine construction mechanism is not itself part of the
agent and is not the subject of bounded optimality constraints. Let Cbe an ofﬂine
mechanism designed for a class of environments E. Then a typical theorem will say
thatCoperates in a speciﬁc environment E2Eand returns an agent design that is
ABO (say) for E—that is, an environment-speciﬁc agent.
In the online case, the mechanism Cis considered part of the agent. Then a typical
theorem will say that the agent is ABO for all E2E. If the performance measure
used is indifferent to the transient cost of the adaptation or optimization mechanism,
the two types of theorems are essentially the same. On the other hand, if the cost
cannot be ignored—for example, if an agent that learns quickly is to be preferred to
an agent that reaches the same level of performance but learns more slowly—then
the analysis becomes more difﬁcult. It may become necessary to deﬁne asymptotic
equivalence for “experience efﬁciency” in order to obtain robust results, as is done
in computational learning theory.
It is worth noting that one can easily prove the value of “lifelong learning” in
the ABO framework. An agent that devotes a constant fraction of its computational
resources to learning-while-doing cannot do worse, in the ABO sense, than an agent
that ceases learning after some point. If some improvement is still possible, the
lifelong learning agent will always be preferred.
2 Rationality and Intelligence 23
2.7.4.1 Fixed and Variable Computation Costs
Another dimension of design space emerges when one considers the computational
cost of the “variable part” of the agent design. The design problem is simpliﬁed
considerably when the cost is ﬁxed. Consider again the task of metalevel reinforce-
ment learning, and to make things concrete let the metalevel decision be made
by a Q function mapping from computational state and action to value. Suppose
further that the Q function is to be represented by a neural net. If the topology
of the neural net is ﬁxed, then all Q functions in the space have the same execution
time. Consequently, the optimality criterion used by the standard Q-learning process
coincides with bounded optimality, and the equilibrium reached will be a bounded-
optimal conﬁguration.1On the other hand, if the topology of the network is subject
to alteration as the design space is explored, then the execution time of the different
Q-functions varies. In this case, the standard Q-learning process will not necessarily
converge to a bounded-optimal conﬁguration; typically, it will tend to build larger
and larger (and therefore more and more computationally expensive) networks to
obtain a more accurate approximation to the true Q-function. A different adaptation
mechanism must be found that takes into account the passage of time and its effect
on utility.
Whatever the solution to this problem turns out to be, the important point is that
the notion of bounded optimality helps to distinguish adaptation mechanisms that
will result in good performance from those that will not. Adaptation mechanisms
derived from calculative rationality will fail in the more realistic setting where an
agent cannot afford to aim for perfection.
2.7.5 Looking Further Ahead
The discussion so far has been limited to fairly sedate forms of agent architecture
in which the scope for adaptation is circumscribed to particular functional aspects
such as metalevel Q functions. However, an agent must in general deal with an
environment that is far more complex than itself and that exhibits variation over time
at all levels of granularity. Limits on the size of the agent’s memory may imply that
almost complete revision of the agent’s mental structure is needed to achieve high
performance. (For example, songbirds grow their brains substantially during the
singing season and shrink them again when the season is over.) Such situations may
engender a rethinking of some of our notions of agent architecture and optimality,
and suggest a view of agent programs as dynamical systems with various amounts of
compiled and uncompiled knowledge and internal processes of inductive learning,
forgetting, and compilation.
1A similar observation was made by Horvitz and Breese ( 1990 ) for cases where the object level is
so restricted that the metalevel decision problem can be solved in constant time.
24 S. Russell
If a true science of intelligent agent design is to emerge, it will have to operate
in the framework of bounded optimality. One general approach—discernible in the
examples given earlier—is to divide up the space of agent designs into “architecturalclasses” such that in each class the structural variation is sufﬁciently limited. Then
ABO results can be obtained either by analytical optimization within the class or
by showing that an empirical adaptation process results in an approximately ABOdesign. Once this is done, it should be possible to compare architecture classes
directly, perhaps to establish asymptotic dominance of one class over another. For
example, it might be the case that the inclusion of an appropriate “macro-operatorformation” or “greedy metareasoning” capability in a given architecture will result
in an improvement in behaviour in the limit of very complex environments—that is,
one cannot compensate for the exclusion of the capability by increasing the machinespeed by a constant factor. Moreover, within any particular architectural class it is
clear that faster processors and larger memories lead to dominance. A central tool
in such work will be the use of “no-cost” results where, for example, the allocationof a constant fraction of computational resources to learning or metareasoning can
do no harm to an agent’s ABO prospects.
Getting all these architectural devices to work together smoothly is an important
unsolved problem in AI and must be addressed before we can make progress on
understanding bounded optimality within these more complex architectural classes.
If the notion of “architectural device” can be made sufﬁciently concrete, then AImay eventually develop a grammar for agent designs, describing the devices and
their interrelations. As the grammar develops, so should the accompanying ABO
dominance results.
2.8 Summary
I have outlined some directions for formally grounded AI research based onbounded optimality as the desired property of AI systems. This perspective on AI
seems to be a logical consequence of the inevitable philosophical “move” from
optimization over actions or computations to optimization over programs. I havesuggested that such an approach should allow synergy between theoretical and
practical AI research of a kind not afforded by other formal frameworks. In the
same vein, I believe it is a satisfactory formal counterpart of the informal goalof creating intelligence. In particular, it is entirely consistent with our intuitions
about the need for complex structure in real intelligent agents, the importance of the
resource limitations faced by relatively tiny minds in large worlds, and the operationof evolution as a design optimization process. One can also argue that bounded
optimality research is likely to satisfy better the needs of those who wish to emulate
human intelligence, because it takes into account the limitations on computationalresources that are presumably an important factor in the way human minds are
structured and in the behaviour that results.
2 Rationality and Intelligence 25
Bounded optimality and its asymptotic version are, of course, nothing but
formally deﬁned properties that one may want systems to satisfy. It is too early to tell
whether ABO will do the same kind of work for AI that asymptotic complexity hasdone for theoretical computer science. Creativity in design is still the prerogative
of AI researchers. It may, however be possible to systematize the design process
somewhat and to automate the process of adapting a system to its computationalresources and the demands of the environment. The concept of bounded optimality
provides a way to make sure the adaptation process is “correct.”
My hope is that with these kinds of investigations, it will eventually be possible
to develop the conceptual and mathematical tools to answer some basic questions
about intelligence. For example, why do complex intelligent systems (appear to)
have declarative knowledge structures over which they reason explicitly? Thishas been a fundamental assumption that distinguishes AI from other disciplines
for agent design, yet the answer is still unknown. Indeed, Rod Brooks, Hubert
Dreyfus, and others ﬂatly deny the assumption. What is clear is that it willneed something like a theory of bounded optimal agent design to answer this
question.
Most of the agent design features that I have discussed here, including the use
of declarative knowledge, have been conceived within the standard methodology
of “ﬁrst build calculatively rational agents and then speed them up.” Yet one
can legitimately doubt that this methodology will enable the AI community todiscover all the design features needed for general intelligence. The reason is that
no conceivable computer will ever be remotely close to approximating perfect
rationality for even moderately complex environments. It may well be the case,therefore, that agents based on approximations to calculatively rational designs are
not even close to achieving the level of performance that is potentially achievable
given the underlying computational resources. For this reason, I believe it isimperative not to dismiss ideas for agent designs that do not seem at ﬁrst glance
to ﬁt into the “classical” calculatively rational framework.
Acknowledgements An earlier version of this paper appeared in the journal Artiﬁcial Intelli-
gence , published by Elsevier. That paper drew on previous work with Eric Wefald and Devika
Subramanian. More recent results were obtained with Nick Hay. Thanks also to Michael Wellman,Michael Fehling, Michael Genesereth, Russ Greiner, Eric Horvitz, Henry Kautz, Daphne Koller,Bart Selman, and Daishi Harada for many stimulating discussions topic of bounded rationality.The research was supported by NSF grants IRI-8903146, IRI-9211512 and IRI-9058427, and by aUK SERC Visiting Fellowship. The author is supported by the Chaire Blaise Pascal , funded by the
l’État et la Région Île de France and administered by the Fondation de l’École Normale Supérieure.
References
Agre, P. E., & Chapman, D. (1987). Pengi: An implementation of a theory of activity. In
Proceedings of the Tenth International Joint Conference on Artiﬁcial Intelligence (IJCAI-87) ,
Milan (pp. 268–272). Morgan Kaufmann.
26 S. Russell
Andre, D., & Russell, S. J. (2002) State abstraction for programmable reinforcement learning
agents. In Proceedings of the Eighteenth National Conference on Artiﬁcial Intelligence (AAAI-
02), Edmonton (pp. 119–125). AAAI Press.
Bellman, R. E. (1957). Dynamic programming . Princeton: Princeton University Press.
Berry, D. A., & Fristedt, B. (1985). Bandit problems: Sequential allocation of experiments .
London: Chapman and Hall.
Breese, J. S., & Fehling, M. R. (1990). Control of problem-solving: Principles and architecture. In
R. D. Shachter, T. Levitt, L. Kanal, & J. Lemmer (Eds.), Uncertainty in artiﬁcial intelligence
4. Amsterdam/London/New York: Elsevier/North-Holland.
Brooks, R. A. (1989). Engineering approach to building complete, intelligent beings. Proceedings
of the SPIE—The International Society for Optical Engineering, 1002 , 618–625.
Carnap, R. (1950). Logical foundations of probability . Chicago: University of Chicago Press.
Cherniak, C. (1986). Minimal rationality . Cambridge: MIT.
Dean, T., & Boddy, M. (1988) An analysis of time-dependent planning. In Proceedings of the
Seventh National Conference on Artiﬁcial Intelligence (AAAI-88) , St. Paul (pp. 49–54). Morgan
Kaufmann.
Dean, T., Aloimonos, J., & Allen, J. F. (1995). Artiﬁcial intelligence: Theory and practice .
Redwood City: Benjamin/Cummings.
Dennett, D. C. (1988). The moral ﬁrst aid manual. In S. McMurrin (Ed.), Tanner lectures on human
values (V ol. 7, pp. 121–147). University of Utah Press and Cambridge University Press.
Doyle, J., & Patil, R. (1991). Two theses of knowledge representation: Language restrictions,
taxonomic classiﬁcation, and the utility of representation services. Artiﬁcial Intelligence, 48 (3),
261–297
Good, I. J. (1971) Twenty-seven principles of rationality. In V . P. Godambe & D. A. Sprott (Eds.),
Foundations of statistical inference (pp. 108–141). Toronto: Holt, Rinehart, Winston.
Goodman, N. D., Mansinghka, V . K., Roy, D. M., Bonawitz, K., & Tenenbaum, J. B. (2008).
Church: A language for generative models. In Proceedings of UAI-08 , Helsinki (pp. 220–229).
Harman, G. H. (1983). Change in view: Principles of reasoning . Cambridge: MIT.
Hay, N., Russell, S., Shimony, S. E., & Tolpin, D. (2012). Selecting computations: Theory and
applications. In Proceedings of UAI-12 , Catalina Island.
Horvitz, E. J. (1987). Problem-solving design: Reasoning about computational value, trade-offs,
and resources. In Proceedings of the Second Annual NASA Research Forum, NASA Ames
Research Center , Moffett Field, CA (pp. 26–43).
Horvitz, E. J. (1989). Reasoning about beliefs and actions under computational resource con-
s t r a i n t s .I nL .N .K a n a l ,T .S .L e v i t t ,&J .F .L e m m e r( E d s . ) , Uncertainty in artiﬁcial intelligence
3(pp. 301–324). Amsterdam/London/New York: Elsevier/North-Holland.
Horvitz, E. J., & Breese, J. S. (1990). Ideal partition of resources for metareasoning (Technical
report KSL-90-26), Knowledge Systems Laboratory, Stanford University, Stanford.
Howard, R. A. (1966). Information value theory. IEEE Transactions on Systems Science and
Cybernetics, SSC-2 , 22–26.
Hutter, M. (2005). Universal artiﬁcial intelligence: Sequential decisions based on algorithmic
probability . Berlin/New York: Springer.
Kearns, M., Schapire, R. E., & Sellie, L. (1992). Toward efﬁcient agnostic learning. In Proceedings
of the Fifth Annual ACM Workshop on Computational Learning Theory (COLT-92) , Pittsburgh.
ACM.
Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: Preferences and value
tradeoffs .N e wY o r k :W i l e y .
Kocsis, L., & Szepesvari, C. (2006). Bandit-based Monte-Carlo planning. In Proceedings of
ECML-06 ,B e r l i n .
Kolmogorov, A. N. (1965). Three approaches to the quantitative deﬁnition of information.
Problems in Information Transmission, 1 (1), 1–7.
Koopmans, T. C. (1972). Representation of preference orderings over time. In C.B. McGuire &
R. Radner (Eds.), Decision and organization . Amsterdam/London/New York: Elsevier/North-
Holland.
2 Rationality and Intelligence 27
Kumar, P. R., & Varaiya, P. (1986). Stochastic systems: Estimation, identiﬁcation, and adaptive
control . Upper Saddle River: Prentice-Hall.
Laird, J. E., Rosenbloom, P. S., & Newell, A. (1986). Chunking in Soar: The anatomy of a general
learning mechanism. Machine Learning, 1 , 11–46.
Levesque, H. J. (1986). Making believers out of computers. Artiﬁcial Intelligence, 30 (1), 81–108.
Livnat, A., & Pippenger, N. (2006). An optimal brain can be composed of conﬂicting agents.
Proceedings of the National Academy of Sciences of the United States of America 103 (9), 3198–
3202.
Marthi, B., Russell, S., Latham, D., & Guestrin, C. (2005). Concurrent hierarchical reinforcement
learning. In Proceedings of IJCAI-05 , Edinburgh.
Marthi, B., Russell, S. J., & Wolfe, J. (2008). Angelic hierarchical planning: Optimal and online
algorithms. In Proceedings of ICAPS-08 , Sydney.
Matheson, J. E. (1968). The economic value of analysis and computation. IEEE Transactions on
Systems Science and Cybernetics, SSC-4 (3), 325–332.
Megiddo, N., & Wigderson, A. (1986). On play by means of computing machines. In J. Y . Halpern
(Ed.), Theoretical Aspects of Reasoning About Knowledge: Proceedings of the 1986 Conference
(TARK-86), IBM and AAAI , Monterey (pp. 259–274). Morgan Kaufmann.
Milch, B., Marthi, B., Sontag, D., Russell, S. J., Ong, D., & Kolobov, A. (2005). BLOG:
Probabilistic models with unknown objects. In Proceedings of IJCAI-05 , Edinburgh.
Newell, A. (1982). The knowledge level. Artiﬁcial Intelligence, 18 (1), 82–127.
Nilsson, N. J. (1991). Logic and artiﬁcial intelligence. Artiﬁcial Intelligence, 47 (1–3), 31–56
Papadimitriou, C. H., & Yannakakis, M. (1994). On complexity as bounded rationality. In
Symposium on Theory of Computation (STOC-94) , Montreal.
Parr, R., & Russell, S. J. (1998). Reinforcement learning with hierarchies of machines. In M. I.
Jordan, M. Kearns, & S. A. Solla (Eds.), Advances in neural information processing systems
10. Cambridge: MIT.
Pfeffer, A. (2001). IBAL: A probabilistic rational programming language. In Proceedings of IJCAI-
01, Seattle (pp. 733–740).
Russell, S. J. (1997). Rationality and intelligence. Artiﬁcial Intelligence, 94 , 57–77.
Russell, S. J. (1998). Learning agents for uncertain environments (extended abstract). In Proceed-
ings of the Eleventh Annual ACM Workshop on Computational Learning Theory (COLT-98) ,
Madison (pp. 101–103). ACM.
Russell, S. J., & Norvig, P. (1995). Artiﬁcial intelligence: A modern approach . Upper Saddle River:
Prentice-Hall.
Russell, S. J., & Subramanian, D. (1995). Provably bounded-optimal agents. Journal of Artiﬁcial
Intelligence Research, 3 , 575–609.
Russell, S. J., & Wefald, E. H. (1989). On optimal game-tree search using rational meta-reasoning.
InProceedings of the Eleventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-
89), Detroit (pp. 334–340). Morgan Kaufmann.
Russell, S. J., & Wefald, E. H. (1991a). Do the right thing: Studies in limited rationality .
Cambridge: MIT.
Russell, S. J., & Wefald, E. H. (1991b). Principles of metareasoning. Artiﬁcial Intelligence 49 (1–3),
361–395.
Russell, S. J., & Zilberstein, S. (1991). Composing real-time systems. In Proceedings of the
Twelfth International Joint Conference on Artiﬁcial Intelligence (IJCAI-91) , Sydney. Morgan
Kaufmann.
Shoham, Y ., & Leyton-Brown, K. (2009). Multiagent systems: Algorithmic, game-theoretic, and
logical foundations . Cambridge/New York: Cambridge University Press.
Simon, H. A. (1955). A behavioral model of rational choice. Quarterly Journal of Economics, 69 ,
99–118.
Simon, H. A. (1958). Rational choice and the structure of the environment. In Models of bounded
rationality (V ol. 2). Cambridge: MIT.
Solomonoff, R. J. (1964). A formal theory of inductive inference. Information and Control, 7 ,
1–22, 224–254.
28 S. Russell
Srivastava, S., Russell, S., Ruan, P., & Cheng, X. (2014). First-order open-universe POMDPs. In
Proceedings of UAI-14 , Quebec City.
Sutton, R., Precup, D., & Singh, S. P. (1999). Between MDPs and semi-MDPs: A framework for
temporal abstraction in reinforcement learning. Artiﬁcial Intelligence, 112 , 181–211.
Tadepalli, P. (1991). A formalization of explanation-based macro-operator learning. In Proceedings
of the Twelfth International Joint Conference on Artiﬁcial Intelligence (IJCAI-91) , Sydney
(pp. 616–622). Morgan Kaufmann.
Tennenholtz, M. (2004). Program equilibrium. Games and Economic Behavior, 49 (2), 363–373.
Vapnik, V . (2000). The nature of statistical learning theory . Berlin/New York: Springer.
von Neumann, J., & Morgenstern, O. (1944). Theory of games and economic behavior (1st ed.).
Princeton: Princeton University Press.
Wellman, M. P. (1994). A market-oriented programming environment and its application to
distributed multicommodity ﬂow problems. Journal of Artiﬁcial Intelligence Research, 1 (1),
1–23.
Wellman, M. P., & Doyle, J. (1991). Preferential semantics for goals. In Proceedings of the Ninth
National Conference on Artiﬁcial Intelligence (AAAI-91) , Anaheim (V ol. 2, pp. 698–703).
AAAI Press.
Zilberstein, S., & Russell, S. J. (1996). Optimal composition of real-time systems. Artiﬁcial
Intelligence 83 , 181–213.