Negotiable Reinforcement Learning for Pareto Optimal Sequential Decision-Making.

Negotiable Reinforcement Learning for Pareto
Optimal Sequential Decision-Making
Nishant Desai
Center for Human-Compatible AI
University of California, Berkeley
nishantdesai@berkeley.eduAndrew Critch
Department of EECS
University of California, Berkeley
critch@berkeley.edu
Stuart Russell
Computer Science Division
University of California, Berkeley
russell@cs.berkeley.edu
Abstract
It is commonly believed that an agent making decisions on behalf of two or more
principals who have different utility functions should adopt a Pareto optimal policy,
i.e. a policy that cannot be improved upon for one principal without making
sacriﬁces for another. Harsanyi’s theorem shows that when the principals have a
common prior on the outcome distributions of all policies, a Pareto optimal policy
for the agent is one that maximizes a ﬁxed, weighted linear combination of the
principals’ utilities. In this paper, we derive a more precise generalization for the
sequential decision setting in the case of principals with different priors on the
dynamics of the environment. We refer to this generalization as the Negotiable
Reinforcement Learning (NRL) framework. In this more general case, the relative
weight given to each principal’s utility should evolve over time according to how
well the agent’s observations conform with that principal’s prior. To gain insight
into the dynamics of this new framework, we implement a simple NRL agent and
empirically examine its behavior in a simple environment.
1 Introduction
It has been argued that the ﬁrst AI systems with generally super-human cognitive abilities will play a
pivotal decision-making role in directing the future of civilization [Bostrom, 2014]. If that is the case,
an important question will arise: Whose values will the ﬁrst super-human AI systems serve? Since
safety is a crucial consideration in developing such systems, assuming the institutions building them
come to understand the risks and the time investments needed to address them [Baum, 2016], they
will have a large incentive to cooperate in their design rather than racing under time-pressure to build
competing systems [Armstrong et al., 2016].
Therefore, consider two nations—allies or adversaries—who must decide whether to cooperate in the
deployment of an extremely powerful AI system. Implicitly or explicitly, the resulting system would
have to strike compromises when conﬂicts arise between the wishes of those nations. How can they
specify the degree to which that system would be governed by the distinctly held principles of each
nation? More mundanely, suppose a couple purchases a domestic robot. How should the robot strike
compromises when conﬂicts arise between the commands of its owners?
It is already an interesting and difﬁcult problem to robustly align an AI system’s values with those
of asingle human (or a group of humans in close agreement). Inverse reinforcement learning (IRL)
32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.
[Russell, 1998] [Ng and Russell, 2000] [Abbeel and Ng, 2004] and cooperative inverse reinforcement
learning (CIRL) [Hadﬁeld-Menell et al., 2016] represent successively realistic early approaches to
this problem. But supposing some adequate solution eventually exists for aligning the values of a
machine intelligence with a single human decision-making unit, how should the values of a system
serving multiple decision-makers be “aligned”?
In the present work, we attempt to begin answering this question. We begin by observing some
deﬁciencies of optimizing a ﬁxed, linear weighted sum of principals’ utilities, as prescribed by
Harsanyi’s social aggregation theorem, in the case that those principals have differing beliefs about
the probability distributions dictating the agent’s observations. We show that building a decision
agent whose policy optimizes such an objective is not, in general, ex-ante Pareto optimal (i.e. "as
evaluated before the agent has taken any actions"). Intuitively, linear weighted aggregation fails
because, before the jointly owned agent has taken any actions, each principal evaluates its policy with
respect to their own beliefs, meaning that a policy that selectively prefers one principal over the other
conditioned on its observations can be desirable to both principals.
Section 3 addresses the shortcomings of Harsanyi-style preference aggregation by presenting the
Negotiable Reinforcement Learning (NRL) framework. In this domain, we model each principal’s
prior on the environment and utility function as a Partially Observable Markov Decision Process
(POMDP). We place necessary and sufﬁcient conditions on Pareto optimality for an agent acting over
these POMDPs with policy . We then construct a third POMDP and show that the optimal policy for
this single POMDP satisﬁes the conditions for Pareto optimality. We refer to an agent implementing
a policy that solves this reduced POMDP as a NRL agent.
Following directly from this reduction is the intriguing property that a Pareto optimal policy must,
over time, prefer the utility function of the principal whose beliefs are a better predictor of the agent’s
observations. This counter-intuitive result constitutes the main theorem of this paper. This can be
seen as settling a kind of bet between the two principals: whichever principal makes the correct
predictions gets to have their utility prioritized. In Section 4, we implement a simple NRL agent and
make empirical observations of this bet settling behavior.
2 Related work
Social choice theory. The entirety of social choice theory and voting theory may be viewed as
an attempt to specify an agreeable formal policy to enact on behalf of a group. Harsanyi’s utility
aggregation theorem [Harsanyi, 1980] suggests one form of solution: maximizing a linear combination
of group members’ utility functions. The present work shows that this solution is inappropriate when
principals have different beliefs, and Theorem 4 may be viewed as an extension of Harsanyi’s form
that accounts simultaneously for differing priors and the prospect of future observations. Indeed,
Harsanyi’s form follows as a direct corollary of Theorem 4 when principals do share the same beliefs.
Multi-agent systems. Zhang and Shah [2014] may be considered a sequential decision-making
approach to social choice: they use MDPs to represent the decisions of principals in a competitive
game, and exhibit an algorithm for the principals that, if followed, arrives at a Pareto optimal Nash
equilibrium satisfying a certain fairness criterion. Among the literature surveyed here, this paper
is the closest to the present work in terms of its intended application: roughly speaking, achieving
mutually desirable outcomes via sequential decision-making. However, the work is concerned with
an ongoing interaction between the principals, rather than selecting a policy for a single agent to
follow as in this paper.
Multi-objective sequential decision-making. There is also a good deal of work on Multi-
Objective Optimization (MOO) [Tzeng and Huang, 2011], including for sequential decision-making,
where solution methods have been called Multi-Objective Reinforcement Learning (MORL). For
instance, Gábor et al. [1998] introduce a MORL method called Pareto Q-learning for learning a set
of a Pareto optimal polices for a Multi-Objective MDP (MOMDP). Soh and Demiris [2011] deﬁne
Multi-Reward Partially Observable Markov Decision Processes (MR-POMDPs), and use genetic
algorithms to produce non-dominated sets of policies for them. Roijers et al. [2015] refer to the same
problems as Multi-objective POMDPS (MOPOMDPs), and provide a bounded approximation method
for the optimal solution set for all possible weightings of the objectives. Wang [2014] surveys MORL
2
methods, and contributes Multi-Objective Monte-Carlo Tree Search (MOMCTS) for discovering
multiple Pareto optimal solutions to a multi-objective optimization problem.
However, none of these or related works address scenarios where the objectives are derived from
principals with differing beliefs, from which the priority-shifting phenomenon of Theorem 4 arises.
Differing beliefs are likely to play a key role in negotiations, so for that purpose, the formulation of
multi-objective decision-making adopted here is preferable.
3 Negotiable Reinforcement Learning
Consider, informally, a scenario wherein two principals — perhaps individuals, companies, or states
— are considering cooperating to build or otherwise obtain a machine that will then interact with an
environment on their behalf.1In such a scenario, the principals will tend to bargain for “how much”
the machine will prioritize their separate interests, so to begin, we need some way to quantify “how
much” each principal is prioritized.
For instance, one might model the machine as maximizing the expected value, given its observations,
of some utility function Uof the environment that equals a weighted sum
w(1)U(1)+w(2)U(2)(1)
of the principals’ individual utility functions U(1)andU(2), as Harsanyi’s social aggregation theorem
recommends [Harsanyi, 1980]. Then the bargaining process could focus on choosing the values of
the weightsw(i).
However, this turns out to be a bad idea. As we shall see in the following example, this solution form
is not generally compatible with Pareto optimality when agents have different beliefs. Harsanyi’s
setting does not account for agents having different priors, nor for decisions being made sequentially,
after future observations. In such a setting, we need a new form of solution, exhibited here.
A cake-splitting scenario. Alice (principal 1) and Bob (principal 2) are about to be presented with
a cake which they can choose to split in half to share, or give entirely to one of them. They have
(built or purchased) a robot that will make the cake-splitting decision on their behalf. Alice’s utility
function returns 0if she gets no cake, 20if she gets half a cake, or 30if she gets a whole cake. Bob’s
utility function works similarly.
However, Alice and Bob have slightly different beliefs about how the environment works. They both
agree on the state of the environment that the robot will encounter at ﬁrst: a room with a cake in
it (s1=“cake” ). But Alice and Bob have different predictions about how the robot’s sensors will
perceive the cake: Alice thinks that when the robot perceives the cake, it is 90% likely to appear
with a red tint ( o1=“red" ), and 10% likely to appear with a green tint ( o1=“green" ), whereas Bob
believes the exact opposite. In either case, upon seeing the cake, the robot will either give Alice the
entire cake ( a1=(all, none) ), split the cake half-and-half ( a1=(half, half) ), or give Bob the entire
cake (a1=(none, all)). Moreover, Alice and Bob have common knowledge of all these facts.
Now, consider the following Pareto optimal policy that favors Alice (principal 1) when o1is red, and
Bob (principal 2) when o1is green:
^(jred) = 100% (all, none)
^(jgreen ) = 100% (none, all)
This policy can be viewed intuitively as a bet between Alice and Bob about the value of o1, and is
highly appealing to both principals:
E(1)[U(1); ^] = 90%(30) + 10%(0) = 27
E(2)[U(2); ^] = 10%(0) + 90%(30) = 27
In particular, ^is more appealing to both Alice and Bob than an agreement to deterministically split
the cake (half, half). We start to see that when principals evaluate their expected returns under a
policywith respect to differing beliefs about action outcomes, they may mutually agree to a policy
that favors one principal over the other, contingent on the action-observation history. We formalize
this intuition and explore its consequences in the following sections.
1The results here all generalize from two principals to nprincipals being combined successively in any order,
but for clarity of exposition, the two person case is prioritized.
3
3.1 A POMDP formulation
Let us formalize the machine’s decision-making situation using the structure of a Partially Observable
Markov Decision Process (POMDP) [Sondik, 1973]. It is assumed that the principals will have
common knowledge of the policy = (1;:::;n)they select for the machine to implement, but that
the principals may have different beliefs about how the environment works, and of course different
utility functions. We refer to this as the common knowledge assumption .
We encode each principal j’s outlook as a POMDP, D(j)= (S(j);A;T(j);U(j);O;
(j);n), which
simultaneously represents that principal’s beliefs about the environment, and the principal’s utility
function.
S(j)is a set of possible states sof the environment.
Ais the set of possible actions aavailable to the NRL agent.
T(j)is the conditional probabilities principal jbelieves will govern the environment state transitions,
i.e.,P(j)(si+1jsi;ai).
U(j)is principalj’s utility function from sequences of environmental states (s1;:::;sn)toR.2
Ois the set of possible observations oof the NRL agent.

(j)is the conditional probabilities principal jbelieves will govern the agent’s observations, i.e.,
P(j)(oijsi).
nis the time horizon.
Thus, principal j’s subjective probability of an outcome (s;o;a), for any s2(S(j))n, is given by a
probability distribution P(j)that takesas a parameter:
P(j)(s;o;a;) :=P(j)(s1)nY
i=1P(j)(oijsi)(aijoia<i)P(j)(si+1jsi;ai) (2)
We say that the POMDPs D(1)andD(2)arecompatible if any policy for one may be viewed as
a policy for the other, i.e., they have the same set of actions Aand observationsO, and the same
number of time steps n. By the common knowledge assumption, principals’ outlooks are assumed to
be encoded by a set of compatible POMDPs.
3.2 Pareto optimal policies
In this context, where a policy may be evaluated relative to more than one POMDP, we use
superscripts to represent which POMDP is governing the probabilities and expectations, e.g.,
E(j)
[U(j)] :=X
s2(S(j))nP(j)(s;)U(j)(s)
represents the expectation in D(j)of the utility function U(j), assuming policy is followed.
Deﬁnition 1 (Pareto optimal policies) .A policyisPareto optimal for a compatible pair of POMDPs
(D(1);D(2))if for any other policy 0, either
E(1)
[U(1)]E(1)
0[U(1)]orE(2)
[U(2)]E(2)
0[U(2)]:
It is assumed that, during negotiation, the principals will be seeking a Pareto optimal policy for the
agent to follow, relative to the POMDPs D(1)andD(2)describing each principal’s outlook.
If we allow the policy to come from the space of stochastic policies, mapping action-observation
histories to distributions over actions:
:
A;O
7!A,
then we can show that the following condition is necessary and sufﬁcient for to be Pareto optimal.
2For the sake of generality, U(j)isnot assumed to be stationary, as reward functions often are.
4
Lemma 2. A policyis Pareto optimal to principals 1and2if and only if there exist weights
w1;w20withw1+w2= 1such that
2argmax
2
w(1)E(1)
[U(1)] +w(2)E(2)
[U(2)]
(3)
Proof. See supplementary material.
3.3 Reduction to a single POMDP
We shall soon see that any Pareto optimal policy must favor, as time progresses, optimizing the utility
of whichever principal’s beliefs were a better predictor of the NRL agent’s inputs. This phenomenon
is most easily shown via a reduction of the POMDPs describing the outlooks of principals 1 and 2 to
a third POMDP, as follows.
For any weights, w(1);w(2)0withw(1)+w(2)= 1, we deﬁne a new POMDP that works by
ﬂipping a (w(1);w(2))-weighted coin, and then running D(1)orD(2)thereafter, according to the
coin ﬂip. Explicitly, we deﬁne a POMDP mixture as a POMDP D=w(1)D(1)+w(2)D(2). We give
Dthe same action space Aand observation space Oas the compatible POMDPs D(1)andD(2). In
the state space of D, we include a latent, stochastic, binary variable B2f1;2gthat is ﬁxed for all
time and initial belief given by P(B= 1) =w(1)andP(B= 2) =w(2). In the case that B= 1, we
draw states, transition probabilities, observation probabilities, and utilities from the parameters of D1.
Formally,
PD(s;o;ajB= 1;) =P(1)(s;o;a;),
and
E(D)
[UjB= 1] = E(1)
[U1].
Likewise for B= 2. This is a generalized version of a well-known POMDP reduction used in the
Bayesian Reinforcement Learning literature [Ghavamzadeh et al., 2016].
Given any policy , the expected payoff of inD=w1D1+w2D2is exactly
P(B= 1)E(UjB= 1) + P(B= 2)E(UjB= 2) (4)
=w(1)E(1)
[U(1)] +w(2)E(2)
[U(2)] (5)
Therefore, using the above deﬁnitions, Lemma 2 may be restated in the following equivalent form:
Lemma 3. Given a pair (D(1);D(2))of compatible POMDPs, a policy is Pareto optimal for that
pair if and only if there exist weights w(j)such thatis an optimal policy for the single POMDP
given byw(1)D(1)+w(2)D(2).
3.4 Structural properties of Pareto optimal POMDP reduction
Expressed in the form of Equation 3, it might not be clear how a Pareto optimal policy makes use of its
observations over time, aside from storing them in memory. For example, is there any sense in which
the NRL agent carries “beliefs" about the environment that it “updates" at each time step? Lemma 3
allows us to answer this and related questions by translating theorems about single POMDPs into
theorems about compatible pairs of POMDPs.
We introduce the notation hito represent the NRL agent’s action-observation history at time i, i.e.
hi= (oi;a<i). At timestep i, the NRL agent has a belief over the latent value B, conditioned on hi.
Note that given an action-observation history hi, the NRL agent’s posterior belief over the value of B
is proportional to the probabilities assigned by each principal’s outlook to the realized observation
sequence:
P(B=jjhi)/w(j)P(j)(oija<i).
This relation, combined with the expected value expression in Equation 4, reveals a pattern in how
the weights on the principals’ conditionally expected utilities must change over time, which is the
main result of this paper:
5
Theorem 4 (Pareto optimal policy recursion) .Given a pair (D(1);D(2))of compatible POMDPs of
lengthn, a policyis Pareto optimal if and only if its components iforinsatisfy the following
backward recursion for some pair of weights w(1);w(2)0withw(1)+w(2)= 1:
(hi)2argmax
2A
w(1)P(1)
oija<i
E(1)
[U(1)jhi;ai]
+w(2)P(2)
oija<i
E(2)
[U(2)jhi;ai]
In words, to achieve Pareto optimality, the machine must
1.use each principal’s own beliefs when estimating the degree to which a decision favors that
principal’s utility function, and
2.shift the relative priorities of the principals’ expected utilities in the machine’s decision
objective over time, by a factor proportional to how well the principals predict the machine’s
inputs.
Proof. By Lemma 3, the Pareto-optimality of for(D(1);D(2))is equivalent to its classical opti-
mality forD=w(1)D(1)+w(2)D(2)for some (w(1);w(2)). Writing Pfor probabilities in D, this is
equivalent to =(hi)maximizing the following expression F()for eachi2f0;:::;ng:
F() =E(D)
[Ujhi;ai]: (6)
The above property states, in words, that the optimal policy is the one that maximizes future expected
rewards given an observation history, without regard to any alternate histories and by recursively
assuming the same selection process for future timesteps. This is a standard formulation for POMDPs,
and is exactly Bellman’s Principle of Optimality [Bellman, 1957, Chap III, 3.]
The tower property of expectation allows us to write the above expectation factor as
E(D)[Ujhi;ai] =P(B= 1jhi)E(D)[Ujhi;ai;B = 1]
+P(B= 2jhi)E(D)[Ujhi;ai;B = 2]:(7)
Now, observe that, by Bayes’ rule, the posterior probability
P(B=jjhi)/w(j)P(j)(oija<i). (8)
By construction, E(D)[Ujhi;ai;B =j] =E(j)[U(j)jhi;ai]. Substituting this
expression and (8) into Equation 7 gives us the result.
An intuition about this property is gained by noting that as the NRL agent takes actions in the single
POMDPw(1)D(1)+w(2)D(2), its posterior belief about the value of the latent variable Bis exactly
equal to its belief about which utility function is “correct” for the POMDP it is acting in.
When the principals have the same beliefs, they always assign the same probability to the machine’s
inputs, so the weights on their respective expectations do not change over time. In this case, Harsanyi’s
utility aggregation formula is recovered as a special instance.
3.5 Interpretation as Bet Settling
Theorem 4 shows that a Pareto optimal policy must tend, over time, toward prioritizing the expected
utility of whichever principal’s beliefs best predict the machine’s inputs better. From some perspec-
tives, this is a little counterintuitive: not only must the machine gradually place more predictive weight
on whichever principal’s prior is a better predictor, but it must reward that principal by attending
more to their utility function as well.
Thus, a machine implementing a Pareto optimal policy can be viewed as a kind of bet-settling device.
If Alice is 90% sure the Four Horsemen will appear tomorrow and Bob is 80% sure they won’t, it
makes sense for Alice to ask—while bargaining with Bob for the machine’s policy—that the machine
6
Figure 1: The agent initially heads for Goal 2in the top-right corner. However, at Frame 9 it has
observed eight deterministic transitions in a row. Its conﬁdence in MDP 1 is high enough at that point
that it veers downward and heads to Goal 1Shown: Frames 0, 6, 12, and 15 of the trajectory.
prioritize her values more if the Four Horsemen arrive tomorrow, in exchange for prioritizing Bob’s
values more if they don’t. Both parties will be happy with this agreement in expectation.
4 Evaluation
4.1 Experiment Environment
Our experiments are run in a modiﬁed version of the FrozenLake environment in OpenAI Gym
[Brockman et al., 2016]. FrozenLake is a grid world environment that simulates a goal MDP. The
agent receives a reward upon reaching the speciﬁed goal position. The agent can choose to move
NORTH ,SOUTH ,EAST , orWEST , and the transition model can be chosen to be either stochastic or
deterministic. Under a stochastic transition model, simulating the eponymous frozen lake, the agent’s
action fails with probability 0.2, and the agent transitions into one of the unintended neighboring
positions. In the FrozenLake environment, state is fully observed by the agent. We modify the
environment to support multiple possible goal states, labeled 1and2, corresponding to the utilities of
Principal 1 and 2, respectively. For simplicity, we treat the state as fully-observable.
In these experiments Principal 1 assigns utility to the agent reaching each goal labeled 1and has the
belief that the environment has a deterministic transition model. Principal 2 assigns utility to the
agent reaching each goal labeled 2and has the belief that the environment has a stochastic transition
model. The agent is initialized with initial belief state w1, corresponding to a subjective belief that the
agent is in Principal 1’s MDP, M1, with probability w1and Principal 2’s MDP, M2, with probability
1w1=w2. We use point-based value iteration to learn a the belief-space policy. The agent is then
placed in either M1orM2, and we observe over time as the agent acts according to its belief state.
4.2 Experiments
Observed Behavior In this set of experiments, we run the NRL agent in order to verify that its
behavior resembles a type of bet-settling. The true environment is chosen to be deterministic for this
experiment. After running point-based value iteration [Pineau et al., 2003] with a belief set of 331
points, we execute the resulting policy in this environment. A portion of the agent’s trajectory is
seen in Figure 1. The purple arrows represent the agent’s choice of action at each physical position
under the current belief state. The color of each square represents the agent’s subjective value at each
position under the current belief.
Observing the trajectory, we see that it initially moves towards goal 2. However, each time an action
succeeds, the agent’s belief in the stochastic environment decreases. By the ninth frame, the agent’s
belief in the stochastic world, and as a result its belief that goal 2grants reward, is low enough that
the policy shifts to push the agent to goal 1. This is the type of behavior we would expect of an agent
that maximizes each principal’s utility based on the likelihood of their beliefs.
Next, we use the same policy and place the agent in a stochastic world. The very ﬁrst action results in
a stochastic transition, and the likelihood of Principal 1’s belief immediately falls to 0. At that point,
the agent knows it is in Principal 2’s MDP and believes that goal 2will give it reward. It heads to
goal 2accordingly.
7
Figure 2: Left: Principal 1’s predicted expected reward while in MDP 2. Right: Principal 2’s predicted
expected reward during trajectory depicted in MDP 1.
Full trajectories for both of these experiments are presented in the supplementary material.
Subjective State Value During Execution The promise of compromise behavior may bring parties
into agreement over the decision to build a NRL agent. However if, after the agent begins taking
actions, parties feel that the agent will not take actions that guarantee them high future rewards, those
parties may be tempted to end cooperation. The next experiment attempts to test how losing parties
assess the agent’s behavior during execution.
To address this concern, we turn back to the trajectories discussed in the previous section. In the
ﬁrst trajectory, the agent is, in fact, operating in Principal 1’s MDP. At each timestep, we ask what
Principal 2 believes their expected sum of future rewards to be. Recall that each principal believes
the agent is acting in their own MDP. Since both principals know the agent’s policy, physical state,
and belief state at each timestep, each principal can estimate their future rewards by simulating the
agent acting in their respective MDP with initial conﬁguration given by the current conﬁguration.
For the experiments here, we rollout 100 simulations from each conﬁguration encountered along
the two trajectories. For each simulation, we place the agent within the losing principal’s MDP and
measure the average reward attained by the agent from that starting conﬁguration. Principal 2’s
assessment of their expected future rewards during execution in MDP 1 is shown in the right frame of
Figure 2. Principal 1’s assessment of their expected future rewards during execution in MDP 2 is
shown in the left frame.
We see that until frame 9, Principal 2 believes that the agent has a high chance of reaching goal 2. It
is only when the agent’s behavior shifts and it veers towards goal 1that Principal 2 begins to assess
their expected future utility as decreasing sharply. In contrast, as soon as Principal 1 observes that
w1= 0:0, they realize that the agent will never reach goal 1. Both of these assessments are consistent
with the bet-settling behavior we expect.
5 Conclusion
Insofar as Theorem 4 is not particularly mathematically sophisticated—it employs only basic facts
about convexity and linear algebra—this suggests there may be more low-hanging fruit to be found in
the domain of “machine implementable social choice theory." To recapitulate, Theorem 4 represents
two deviations from the intuition of naïve utility aggregation: to achieve Pareto optimality for
principals with differing beliefs, an agent must (1) use each principal’s own beliefs in evaluating how
well an action will serve that principal’s utility function, and (2) shift the relative priority it assigns
to each principal’s expected utilities over time by a factor proportional to how well that principal’s
beliefs predict the machine’s inputs.
As a ﬁnal remark, consider that social choice theory and bargaining theory were both pioneered during
the Cold War, when it was particularly compelling to understand the potential for cooperation between
human institutions that might behave competitively. In the coming decades, machine intelligence
will likely bring many new challenges for cooperation, as well as new means to cooperate, and new
reasons to do so. As such, new technical aspects of social choice and bargaining, along the lines of
this paper, will likely continue to emerge.
8
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In
Proceedings of the twenty-ﬁrst international conference on Machine learning , page 1. ACM, 2004.
Stuart Armstrong, Nick Bostrom, and Carl Shulman. Racing to the Precipice: A Model of Artiﬁcial
Intelligence Development. AI & SOCIETY , 31(2):201–206, 2016.
Seth D Baum. On the Promotion of Safe and Socially Beneﬁcial Artiﬁcial Intelligence. AI &
SOCIETY , pages 1–9, 2016.
Richard Bellman. Dynamic Programming . Princeton University Press, 1957.
Nick Bostrom. Superintelligence: Paths, dangers, strategies . OUP Oxford, 2014.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and
Wojciech Zaremba. OpenAI Gym, 2016.
Zoltán Gábor, Zsolt Kalmár, and Csaba Szepesvári. Multi-Criteria Reinforcement Learning. In ICML ,
volume 98, pages 197–205, 1998.
Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Bayesian Reinforcement
Learning: A Survey. ArXiv e-prints , September 2016.
Dylan Hadﬁeld-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Cooperative Inverse
Reinforcement Learning, 2016.
John C Harsanyi. Cardinal Welfare, Individualistic Ethics, and Interpersonal Comparisons of Utility.
InEssays on Ethics, Social Behavior, and Scientiﬁc Explanation , pages 6–23. Springer, 1980.
Andrew Y Ng and Stuart J Russell. Algorithms for Inverse Reinforcement Learning. In ICML , pages
663–670, 2000.
Joelle Pineau, Geoff Gordon, and Sebastian Thrun. Point-based Value Iteration: An Anytime
Algorithm for POMDPs. IJCAI’03, pages 1025–1030, 2003.
Diederik M Roijers, Shimon Whiteson, and Frans A Oliehoek. Point-based planning for multi-
objective pomdps. In IJCAI 2015: Proceedings of the Twenty-Fourth International Joint Conference
on Artiﬁcial Intelligence , pages 1666–1672, 2015.
Stuart Russell. Learning Agents for Uncertain Environments. In Proceedings of the eleventh annual
conference on Computational learning theory , pages 101–103. ACM, 1998.
Harold Soh and Yiannis Demiris. Evolving Policies for Multi-Reward Partially Observable Markov
Decision Processes (MR-POMDPs). In Proceedings of the 13th annual conference on Genetic and
evolutionary computation , pages 713–720. ACM, 2011.
Edward J. Sondik. The Optimal Control of Partially Observable Markov Processes over a Finite
Horizon. Oper. Res. , 21(5):1071–1088, October 1973.
Gwo-Hshiung Tzeng and Jih-Jeng Huang. Multiple Attribute Decision Making: Methods and
Applications . CRC press, 2011.
Weijia Wang. Multi-Objective Sequential Decision Making . PhD thesis, Université Paris Sud-Paris
XI, 2014.
Chongjie Zhang and Julie A Shah. Fairness in Multi-Agent Sequential Decision-Making. In Advances
in Neural Information Processing Systems , pages 2636–2644, 2014.
9

Nishant Desai
Andrew Critch
Stuart J
Russell


Reasoning about Rationality.

Reasoning About Rationality
Adam Bjorndahl, Joseph Y. Halpern, Rafael Pass
September 22, 2017
Abstract
We provide a sound and complete axiomatization for a class of logics
appropriate for reasoning about the rationality of players in games, and
show that essentially the same axiomatization applies to a very wide class
of decision rules. We also consider epistemic representations of games
in which players may be uncertain as to what decision rules their op-
ponents are using, and dene in this context a new solution concept,
D-rationalizability.
1 Introduction
Decision-making in an uncertain environment is a fundamental component of
game theory: players must choose what to do without necessarily knowing what
their opponents will do. Under certainty, decision-making is straightforward:
one simply chooses the course of action that leads to the most preferred outcome.
Under uncertainty, however, a player must evaluate many possible outcomes
in a manner that somehow takes into account her relative degrees of belief.
There are many ways to do this. The maximin decision rule, for example,
focuses entirely on worst-case scenarios. An expected utility maximizer , on the
other hand, weights each outcome according to her (subjective) assessment of its
probability and chooses the course of action that maximizes the corresponding
expected value.
One can argue about which decision rules are reasonable and which are not, and
the word \rational" might be invoked to denote this very divide. However, this
is not the debate that concerns us here. Rather, assuming that we have xed
a decision rule for a given player, we can ask whether that player is, in fact,
making choices in accordance with it, and call her rational precisely when she
is. In classical game theory, for example, rationality is typically identied with
expected utility maximization: a player is rational if and only if she is acting to
maximize her expected utility.
Rationality in this sense plays a crucial role in the analysis of games; indeed,
many notions of equilibrium require not only that each player is rational, but
1
also that each player believes their opponents are rational, believes their oppo-
nents believe their opponents are rational, and so on. Epistemic game theory
concerns itself with models expressive enough to represent the complexities of
higher-order beliefs. Formal logic furnishes a powerful and versatile class of
such models; namely, modal logics of belief and the Kripke structures typically
used to give semantics to these logics. However, while the notion of rationality
has been incorporated into these models both syntactically and semantically, no
axiomatization of the resulting logical systems has been provided. This paper
lls this gap.
We take as our point of departure axioms for rationality in the sense of expected
utility maximization rst presented in [Bjorndahl, Halpern, and Pass 2011]. We
then extend these axioms to arbitrary decision rules; this allows us to reason
about other standard rules beyond expected utility maximization, such as max-
imin and minimax regret (see [Halpern 2003] for a discussion of all the decision
rules mentioned in this paper). We also consider the eect of representing un-
certainty by a setof probability measures rather than a single one; this allows
us to capture well-known decision rules such as maxmin expected utility and
minimax expected regret . Finally, we turn our attention to modeling situations
where players might be uncertain about which decision rules their opponents
are using. Endogenizing decision rules in this way broadens not only the notion
of rationality but also that of iterative rationality ; this, we argue, provides a
better epistemic foundation for a number of solution concepts.
The rest of this paper is organized as follows. In Section 2, we dene the core
concepts formally: games, modal logics of belief appropriate for reasoning about
games, and the incorporation of rationality into these logics. Section 3 presents
the main axiomatization together with a proof of soundness and completeness;
we also extend these core results to logics in which the players' uncertainty is
represented by sets of probabilities, and discuss the role of language. In Section
4, we consider logics in which players may be uncertain about the decision rules
used by their opponents, and provide a natural application of this framework in
the form of a new solution concept, D-rationalizability, as well as an axiomatiza-
tion. Section 5 concludes with a discussion of future work. Proofs are collected
in Appendix A.
2 Reasoning about games
Given a tuple ( Xi)i2Iover some nite index set I=f1;:::;ng, we adopt the
usual notational convention of writing
X:=Y
i2IXiandXi:=Y
j6=iXj:
We also write X0
iXifor
X1Xi1X0
iXi+1Xn
2
and similarly ( x0
i;xi) for (x1;:::;xi1;x0
i;xi+1;:::;xn).
A(normal-form) game is a tuple  = ( I;(i)i2I;(ui)i2I) whereI=f1;:::;ng
is the set of players , iis the (nite) set of strategies available to player i, and
ui:!Risplayeri's utility function , where  =Q
iidenotes the set of
strategy proles .
2.1 Syntax
One way of reasoning formally about a game is to build a logical language that
is expressive enough to capture the aspects of play that we are interested in
analyzing. To this end, given a game , we begin by dening a propositional
modal language of belief and then specializing the primitive propositions to
correspond to the strategies available to the players.
Given an arbitrary set  of primitive propositions, let LB() be the language
recursively generated by the grammar
'::=pj:'j'^ jBi';
wherep2 andi2I. Here:and^are the usual Boolean connectives
corresponding to logical negation and conjunction, while Bi'is read \player i
believes'". We dene other connectives such as _,!, and$as usual in terms
of^and:, and writebBi'for:Bi:'(\playericonsider it possible that '").
We also write
E1'B1'^^Bn'and
Ek'E1(Ek1')
for \everyone believes '" and itsk-fold iteration, respectively. Set
:=fplayi(i):i2I; i2ig;
where we read playi(i) as \player iis playing strategy i"; we write
play()play1(1)^^ playn(n) and
playi(i)^
j6=iplayj(j)
for \the players are playing according to the strategy prole " and the anal-
ogous statement regarding the players other than i. The languageLB() is
thus appropriate for reasoning about the beliefs of the players with respect to
the strategies they are playing.
2.2 Semantics
A language of belief can be interpreted using Kripke-style possible world seman-
tics, where associated to each world !and each player iis a probability measure
3
on the set of all worlds, used to interpret player i's beliefs at !. In the case
of a language like LB(), we also must take care to interpret the primitive
propositions appropriately.
We restrict our attention to nite models. A nite -structure is a tuple
M= (
;(Pri)i2I;s) satisfying the following conditions:
(C1)
 is a nonempty, nite set;
(C2) Priassociates with each !2
 a probability measure Pri(!) on
;
(C3) Pri(!)(f!02
:Pri(!0) =Pri(!)g) = 1;
(C4)s:
! satises Pri(!)(f!02
:si(!0) =si(!)g) = 1.
Conditions (C1) and (C2) set the stage to interpret player i's beliefs at !by
the measure Pri(!). Condition (C3) then ensures that at each world !, each
player is sure of (i.e. assigns probability 1 to) her own beliefs. Finally, condition
(C4) establishes that the strategy function sassigns to each world !a strategy
proles(!) in game |intuitively, the strategy that each player is playing at
!|and moreover, that each player is sure of her own strategy.
A nite -structure Minduces an interpretation [ [] ]M:LB()!2
dened
recursively as follows:
[ [playi(i)] ]M:=f!2
:si(!) =ig
[ ['^ ] ]M:= [ ['] ]M\[ [ ] ]M
[ [:'] ]M:=
 [ ['] ]M
[ [Bi'] ]M:=f!2
:Pri(!)([ ['] ]M) = 1g:
Thus, the primitive propositions are interpreted in the obvious way using the
strategy function ( sidenotes the ith component function of s), the Boolean
connectives are interpreted classically, and the formula Bi'holds at all and
only those worlds !at which Pri(!) assigns probability 1 to '. As is standard,
we often write ( M;! )j='or just!j='to indicate that !2[ ['] ]M. Similarly,
we writeMj='if [ ['] ]M=
, and when ( M;! )6j=', we say that Mrefutes'
(at!), or just that !refutes'.
2.3 Rationality
Informally, a player is rational if the strategy she is playing is a best response
to her beliefs about the outcome of the game, given her preferences. But there
is no single conception of what constitutes a \best response"; a wide variety of
principles of decision-making have been proposed and studied.
One in
uential notion, particularly in game theory, is that of expected utility
maximization. Given a game  and a probability measure on i(thought of
4
as representing player i's beliefs about the strategies her opponents will play),
theexpected utility of a strategy i2iis just the expected value of the
functionui(i;):i!Rwith respect to :
EUi(i;):=X
i2iui(i;i)(i):
A best response for an expected utility maximizer is a strategy that maximizes
this value. Abstractly, we might identify the mandate \maximize expected
utility" for player i(in the game ) with a function deu
ithat takes as input a
beliefon iand returns as output the set of strategies i2ithat maximize
playeri's expected utility given :
deu
i():=fi2i:(80
i2i)(EUi(i;)EUi(0
i;))g:
Let (i) denote the set of all probability measures on  i. Then, generalizing
the above, we dene a decision rule for player i(in)to be a function
di:(i)!2if;g. Intuitively, i2di() just in case iis a best
response for player ito the belief according to the decision rule di. For
another example, the \maximin" mandate for player i, which says to maximize
the worst-case outcome among those considered possible, corresponds to the
decision rule dened as follows: let
WCi(i;):= minfui(i;i):(i)>0g;
and set
dm
i():=fi2i:(80
i2i)(WCi(i;)WCi(0
i;))g:
Decision rules can be interpreted in -structures; roughly speaking, for each
world!, we can dene the set of di-best responses for player iat!, and thus
determine whether or not player iis being di-rational (i.e., acting in accordance
with the decision rule di) at!. Formally, given a -structure M, for each player i
and each world !, the probability measure Pri(!) induces a probability measure
i;!dened on ias follows:
i;!(i):=Pri(!)([ [playi(i)] ]M):
In fact,i;!is the pushforward ofPri(!) bysi. Since Pri(!) is interpreted
as representing player i's beliefs at !, it makes sense to apply her decision rule
toi;!. This leads us to dene the set of di-best responses for player i
at!to be di(i;!). We say that player iisdi-rational at !just in case
si(!)2di(i;!).
Since we wish to reason formally about games using the logic developed above,
and rationality is an important concept for such analyses, we expand the lan-
guage to include primitive propositions denoting rationality of the players.
Given a prole of decision rules d= (di)i2I, let
d
:=  [fRATdi
i:i2Ig;
5
where RATdi
iis read \player iisdi-rational". Henceforth, except where noted
otherwise, we work with an arbitrary but xed prole of decision rules d. For
notational convenience, we omit the diwhen possible, referring to \best re-
sponses" and \rationality" instead of \ di-best responses" and \ di-rationality",
and writing RATiinstead of RATdi
i. We also make use of the syntactic abbre-
viation
RATRAT 1^^ RATn
for \everyone is rational".
Given a -structure M, the interpretation [ [ ] ]Mis extended toLB(d
) in the
obvious way, by setting
[ [RATi] ]M:=f!2
:si(!)2di(i;!)g:
When d=deu, so each player's decision rule is given by expected utility max-
imization, we regain the traditional notion of rationality in game theory. Ra-
tionality so dened can be used to characterize several well-known solution
concepts in terms of -structures. For example, as shown by Tan and Werlang
[1988] and Brandenburger and Dekel [1987], given a game , a strategy iis
rationalizable if and only if there exists a -structure Mand a state !therein
such that!j=playi(i) and for every k2N,!j=Ek(RAT ) (i.e., it is common
belief that everyone is rational).
It is worth noting that decision rules are general enough to represent processes
that fall outside the traditional purview of \rationality". For example, suppose
that player iis a computer system and  iis a collection of actions it can execute.
Suppose also that the system maintains a database consisting of estimates of
the values of certain variables, which can be represented as beliefs about the
strategies of \opponents". In this context, a decision rule can be thought of
as a (nondeterministic) process that the computer system might use to choose
which action to execute on the basis of the information in its database. In
particular, RATdi
iasserts that the system has executed an action consistent
with the process di.1
Example 2.1: In order to help solidify the framework just introduced, we
present a simple example. Consider the standard Bach-or-Stravinsky game  BoS
[Osborne and Rubinstein 1994], in which each of two players must choose which
of two concerts to attend this evening: one featuring the music of Bach, and
one of Stravinsky. Player 1 prefers to attend the Bach concert, while player two
prefers the Stravinsky; moreover, each much prefers to attend the same concert
as the other. We can represent these preferences with the utility functions
summarized in Table 1.
1In this sense, we can think of a decision rule as a knowledge-based program [Fagin, Halpern,
Moses, and Vardi 1995]. Knowledge-based programs have previously been used to characterize
rationality and solution concepts [Halpern and Moses 2007].
6
Bach Stravinsky
Bach 3,2 0,0
Stravinsky 0,0 2,3
Table 1: Bach or Stravinsky?
We now describe a  BoS-structure in which we can reason about the beliefs and
rationality of the players. Of course, there are many such  BoS-structures, each
representing a dierent conguration of facts and beliefs; the one we consider
here is chosen simply to provide a concrete illustration of the connection between
the logical formalism and the game.
Let
 =f!0;!1;!2gand let Pr1(!0) be the uniform distribution on f!1;!2g,
so player 1 considers !1and!2equally likely. Consider a strategy function s
dened such that s2(!1) = Bach and s2(!2) = Stravinsky. It is easy to see that
in this case, the induced measure 1;!0assigns probability :5 to Bach and :5 to
Stravinsky. It follows easily that player 1's expected utility on choosing Bach is
1:5, while his expected utility on choosing Stravinsky is 1. Thus,
!0j=RATdeu
1
1$play1(Bach):
On the other hand, the worst-case outcome for player 1 on choosing either Bach
or Stravinsky yields a utility of 0, so we have
!0j=RATdm
1
1;
regardless of which strategy player 1 actually chooses at !0.
3 Axiomatization
Although reasoning formally about beliefs and rationality has long been recog-
nized as important to game theory (see, e.g., [Aumann 1999]), to the best of
our knowledge, rationality has not been axiomatized in a logic of belief. In this
section we provide an axiomatization and prove that it is sound and complete.
While most of the axioms are easy to state and understand, two require some
preliminary denitions. Fix a game  and a prole of decision rules d. Given a
subsetSi, let
i;S^
i2SbBiplayi(i)^^
i=2S:bBiplayi(i):
Intuitively, the formula i;Ssays that player iconsiders possible alland only
the strategy proles for his opponents that are elements of S. It is easy to see
that for each player iand any world !, exactly one of the formulas i;Sholds.
7
Given a measure on a nite set X, let
supp():=fx2X:(x)>0g;
thesupport of. For each player iand eachi2i, letS+
i(i) denote the
collection of all Sisuch that there exists a probability measure with
supp() =Sand with respect to which iis a best response; that is, i2di().
Similarly, deneS
i(i) to be the collection of all Sisuch that there exists
a probability measure with supp() =Sandi=2di().
Consider the following axiom schemes:
G1._
i2iplayi(i)
G2.:(playi(i)^playi(0
i)), fori6=0
i
G3. playi(i)$Biplayi(i)
G4. RATi$Bi(RATi)
G5. (playi(i)^RATi)!_
S2S+
i(i)i;S
G6. (playi(i)^:RATi)!_
S2S
i(i)i;S.
G1{G4are straightforward. G1andG2say that, at each state, each player
plays exactly one strategy. G3andG4say that each player is certain of her
strategy and of whether or not she is rational. The interesting axioms are G5
andG6. Intuitively, G5says that if RATiholds and player iis playingi, then
playerimust consider possible a collection of strategy proles2on which she
could place a probability that would justify her playing i.G6is interpreted
analogously. Notice that player i's actual (i.e. quantitative) beliefs are not fully
specied. This may be somewhat surprising, given that the semantic interpreta-
tion ofRATiis dened in terms of the quantitative probabilities that constitute
playeri's beliefs; we return to this point in Section 3.2.
LetGLd
be the axiom system that results from adding G1{G6to the standard
KD45 axioms and rules of inference of belief logic (see, e.g., [Fagin, Halpern,
Moses, and Vardi 1995]). These axioms completely characterize the logical
properties of rationality as expressible in the language LB(d
).
Theorem 3.1: GLd
is a sound and complete axiomatization of the language
LB(d
)with respect to the class of all nite -structures.
2Note that the sets S+
i(i) andS
i(i) could be empty; by convention, we dene the empty
disjunction to be ?(orfalse). If, for example, S+
i(i) =;, this means that iis not a best
response to any beliefs, in which case the corresponding axiom ( playi(i)^RAT i)! ? is
intuitively correct.
8
It bears emphasizing that GLd
is parametrized by two variables: the underlying
game  that determines the players, their strategies, and their preferences, and
the prole of decision rules dthat determines the meaning of \rationality" for
each player. Thus, what we are axiomatizing here is not a single logic but a class
of logics. For each xed  and d, the corresponding axiom system GLd
is trivially
decidable (i.e., we can eectively determine whether a formula is an instance of
an axiom scheme) because KD45 is decidable and there are only nitely-many
instances of the axiom schemes G1{G6(the implicit quantication in these
schemes ranges in some cases over players and in others over strategies, all of
which are nite sets).
3.1 Belief as lower probability
In the above we take for granted that each player's uncertainty is represented
by a probability measure. While this is a very standard assumption, it is by
no means the only framework that has been considered; see [Halpern 2003] for
an overview of dierent ways of modeling uncertainty. Here we show that, with
very minor modications, the axiomatization given above also works in the more
general context where beliefs are represented using setsof probability measures.
Given a set Pof probability measures, the lower probability of an event E,
denoted P(E), is dened to be the inmum of the probabilities assigned to E
by members of P:
P(E):= inff(E):2Pg:
Fix a game . A nite lower -structure is a tupleM= (
;(PRi)i2I;s)
satisfying the following conditions:
(L1)
 is a nonempty, nite set;
(L2)PRiassociates to each !2
 a set PRi(!) of probability measures on
;
(L3)PRi(!)(f!02
:PRi(!0) =PRi(!)g) = 1;
(L4)s:
! satises PRi(!)(f!02
:si(!0) =si(!)g) = 1.
These conditions are simply the analogues of conditions (C1) through (C4)
where uncertainty is represented by sets of probability measures and certainty
is identied with lower probability 1. Accordingly, we dene the interpretation
[ [] ]Mas before, except for the clause corresponding to the belief modalities,
which is replaced by the following:
[ [Bi'] ]M:=f!2
:PRi(!)([ ['] ]M) = 1g:
Finally, a decision rule for playeriin this context is a function di:2(i)!
2if;g, since player imust make her choice based on the uncertainty given
9
by a set of probability measures. For example, the \maximin expected utility"
decision rule for player iwould be given by the following:
dmeu
i(P):=
i2i:(80
i2i)
min
2PfEUi(i;)gmin
2PfEUi(0
i;)g
:
Other rules, such as minimax expected regret [Hayashi 2008], can also easily be
dened in this setting.
As before, such decision rules makes sense in a -structure M: for each player
iand each world !, the probability measures in the set PRi(!) can be pushed
forward by sito probability measures on  i. Let Pi;!denote the set of all
such pushforwards:
Pi;!:=fi;!:2PRi(!)g:
Then we can dene di-rationality for playeriat!in the obvious way, namely,
by the requirement that si(!)2di(Pi;!).
The axiomatization of Section 3 can be generalized as well. First observe that
the dual belief modality, bBi:Bi:, is interpreted as positive upper proba-
bility, where the upper probability of an event Ewith respect to a set Pof
probability measures, denoted P(E), is given by
P(E):= supf(E):2Pg:
Accordingly, given a set Pof probability measures on a nite space X, we dene
supp(P):=fx2X:(92P)((x)>0)g:
For eachi2i, letS+
i(i) denote the collection of all Sisuch that
there exists a set of probability measures Pwith supp(P) =Sandi2di(P).
Similarly, deneS
i(i) to be the collection of all Sisuch that there exists
a setPof probability measures with supp(P) =Sandi=2di(P). It is not hard
to see that all of these denitions generalize what was presented in Sections
2 and 3; indeed, by considering the special case where all sets of probability
measures are singletons, we recover that framework exactly. Moreover, the
axiom system GLd
, interpreted in this more general setting using the denitions
above, remains sound and complete.
Theorem 3.2: GLd
is a sound and complete axiomatization of the language
LB(d
)with respect to the class of all nite lower -structures.
3.2 The role of language
In this section we focus on the prole of decision rules d=deu, with respect
to which each player is rational precisely if they are playing a strategy that
maximizes their expected utility. As noted, it is somewhat surprising that G5
andG6are sucient to capture this notion of rationality. Whether or not a
10
player is maximizing their expected utility depends on their quantitative beliefs;
however, while G5andG6specify the possible supports for player i's beliefs,
they say nothing about the actual weights placed on the individual outcomes.
Nor could they|the language LB(d
) cannot express anything beyond such
qualitative properties of the measures Pri(!).
Expressivity, however, is a double-edged sword: when working with a less ex-
pressive language, though we are more limited in the possible axioms we have
available, there are also fewer validities to worry about proving. This, in essence,
is why GLd
can be a complete axiomatization: the properties of rationality it
fails to encode are precisely those properties that are not expressible in the
language at all.
A richer language|in particular, one with a ner-grained representation of
belief|may not be axiomatizable at all, or at least not using the techniques
in this paper. Consider, for example, a language with belief modalities B
ifor
each2[0;1], whereB
i'is interpreted as saying that player iassigns prob-
abilityto'. In this case, GLd
(replacingBibyB1
i) is sound but certainly
not complete. It can easily happen, for example, that in the game  it is only
rational for player ito playiif she assigns probability1
2toi; however, the
corresponding validity
(playi(i)^RATi)!B1
2
iplayi(i)
is clearly not a theorem of GLd
. Moreover, extending GLd
to this richer lan-
guage runs into diculties. The axiom schemes G5andG6essentially work by
insisting that the players' beliefs be compatible with rationality or its negation,
respectively. In the language LB(d
), this amounts to specifying the possible
supports for the players' beliefs, which can be written using nite formulas since
each iis nite and therefore has only nitely-many subsets. By contrast, in
the language with belief modalities B
ifor every2[0;1], the \formula" that
says that player i's beliefs are compatible with rationality may be innitely long.
A still richer language, however, can circumvent these diculties entirely. Fix
a game  and consider the language of linear likelihood inequalities dened by
the grammar
'::=pj:'j'^ ja1`i('1) ++ak`i('k)b;
wherep2,i2I,k2N, anda1;:::;ak;b2R. The likelihood terms
`i(') are meant to be read as \the probability of 'according to player i", and a
likelihood formula a1`i('1)++ak`i('k)bshould be thought of as asserting
the corresponding inequality. More precisely, we interpret such formulas in a
-structure Mas follows:
[ [a1`i('1) ++ak`i('k)b] ]M:=f!2
:kX
j=1ajPri(!)([ ['j] ]M)bg:
11
For example, the formula `i(')1 says that player iassigns probability at least
(and therefore exactly) 1 to ', while the formula ( `i(')1
2)^(`i(:')1
2)
says that player iassigns probability1
2to'. See [Halpern 2003] for a thorough
discussion of this and related logics; a sound and complete axiomatization is
given in [Fagin, Halpern, and Megiddo 1990].
In this language, rationality in the sense of expected utility maximization can
bedened , thus obviating the need for a separate axiomatization. Indeed, if we
leti0
ibe an abbreviation for the formula
X
i2iui(i;i)`i(playi(i))X
i2iui(0
i;i)`i(playi(i))0;
which says that the expected value according to player iof playingiis no less
than the expected value of playing 0
i, then it is easy to see that
RATdeu
i
i$_
i2i
playi(i)^^
0
i2ii0
i
is valid. Since rationality is expressible in this language, it is axiomatized as
well.
4 Endogenizing decision rules
In general, we may wish to reason about players who are uncertain about which
decision rules their opponents are using. For example, player imight believe
that if player jis maximizing her expected utility, then she will play j, but
if she plays 0
j, then she might instead be minimizing the worst-case outcome.
One way to try to model such uncertainty is to expand the logic so that there is
asetDiof decision rules associated with each player i. Consider the collection
of primitive propositions
D
:=  [fRATdi
i:i2I;di2Dig:
Interpret RATdi
ias before. Then LB(D
) is a language for reasoning about
the strategies and beliefs of the players i2Ias well as their adherence to the
various decision rules di2Di.
In Appendix A.3, we show how to modify G5andG6to obtain a sound and
complete axiomatization of LB(D
) with respect to the class of all nite -
structures. But there is something unsatisfying about using this language to
model players' uncertainty about decision rules: the propositions RATdi
isay
that player iis playing a di-best-response, but notthat player iis actually
using the rule dito decide her strategy. To see the dierence, consider a player
iwho is trying to maximize her expected utility (i.e. using deu
i), and happens
to also play a maximin strategy; contrast this with a scenario in which she is
12
actively seeking to maximize the worst-case outcome (i.e. using dm
i), and in so
doing happens to play a strategy that maximizes her expected utility. Although
playeriis following dierent decision rules in these two cases, the language
LB(D
) cannot express this dierence; the formula RATdeu
i
i^RATdm
i
iholds
either way. For instance, in Example 2.1, it is not hard to check that
!0j=play1(Bach)!
RATdeu
1
1^RATdm
1
1
:
In sum, the propositions RATdi
ido not say anything about how playeriis
making her decision, but simply record whether or not the decision she does
make is compatible with the rule di. What we want is a dierent kind of
proposition, say rulei(di), that says that player ireally is using the rule diin
deciding her strategy.
Decision rules interpreted in this sense are particularly relevant in a dynamic
setting. When an opponent does something unexpected and seemingly irra-
tional, there is the question of how to update your beliefs. One option is to
abandon the belief that your opponent is rational, but this is unsatisfying both
conceptually and methodologically. An alternative response is to update your
beliefs about your opponent's beliefs: what they did actually was rational with
respect to their beliefs, you had just misjudged what those beliefs were (see,
e.g., [Battigalli and Siniscalchi 2002]). But in some cases, this too is unsatisfy-
ing: for example, \continuing" at the second-last stage of the centipede game
(see Example 4.3) is only rational for a player who believes his opponent to be
irrational. When decision rules are present in the model as objects of belief,
however, a third option becomes available: abandon the belief that your oppo-
nent is di-rational, but not that they are behaving rationally with respect to
some other decision rule. Though an analysis of decision rules in extensive-form
games is beyond the scope of this paper, the groundwork for such a study can
be laid by formalizing them in a static context.
Fix a game  and a prole D= (Di)i2Iof sets of decision rules for each player
i2I. Expand the set  of primitive propositions that we considered earlier
by taking
;D:=  [frulei(di):i2I;di2Dig:
In order to interpret the primitive propositions rulei(di), we must extend the
semantic model so that it associates with each world !the decision rule that each
playeriis using at that world; furthermore, we must constrain the strategies
used at each world so that they are compatible with the corresponding decision
rules. Formally, a ( ;D)-structure is a tupleM= (
;(Pri)i2I;s;r) satisfying
(C1) through (C4) as well as the following additional conditions:
(C5)r:
!Dsatises Pri(!)(f!02
:ri(!0) =ri(!)g) = 1;
(C6)si(!)2ri(!)(i;!).
Condition (C5) says that the decision function rassigns to each world !a prole
of decision rules r(!)|intuitively, ri(!)2Diis the rule that player iis using at
13
!|and moreover, each player is sure of her own decision rule. Condition (C6)
requires that, at each world !, the strategy si(!) is anri(!)-best response for
playeri; in other words, player ireally is following the decision rule ri(!) at!.
The languageLB(;D) can be interpreted in a ( ;D)-structure Mas before,
with the additional clause
[ [rulei(di)] ]M:=f!2
:ri(!) =dig:
The resulting logic can be axiomatized using essentially the same technique as
in Section 3 (see Section 4.2). But perhaps more interesting than axiomatizing
this logic is the prospect of applying it to the analysis of games.
4.1 D-rationalizability
It is quite natural in certain strategic contexts for players to reason not only
about their opponents' strategies and beliefs, but also the decision-making pro-
cess that they might be using. A decision rule like minimax regret, for instance,
can lead to very dierent behaviour in games like the centipede game or the
traveler's dilemma [Halpern and Pass 2012]; as such, it is reasonable in such
contexts to wonder, for example, whether an opponent is motivated to maxi-
mize utility or to avoid regret.
Recall that strategies that are consistent with common belief of rationality are
called rationalizable . Common belief of rationality in games|the requirement
that every player is rational, believes their opponents are rational, believes their
opponents believe their opponents are rational, and so on|is often conceived of
as a kind of \minimal" condition for equilibrium. But games like the traveler's
dilemma, where the rationalizable strategies are far from optimal and quite dis-
tinct from the typical strategies employed by human players [Capra, Goeree,
Gomez, and Holt 1999], belie this intuition of minimality. However, by decou-
pling the meaning of rationality from expected utility maximization, the notion
of \rationalizability" can be expanded to other decision rules, thereby providing
what is arguably a better epistemic foundation for equilibrium theory.
More precisely, generalizing the traditional epistemic characterization of ratio-
nalizability, we dene a strategy ito be D-rationalizable (in )just in case
there exists a ( ;D)-structure in which iis played at some state. Of course,
the standard notion arises as the special case where each Di=fdeu
ig. It is
easy to see (via a straightforward iterated deletion argument3) that when the
strategy sets are nite, D-rationalizable strategies must exist. Moreover, if for
each player iwe have DiD0
i, then clearly every ( ;D)-structure is also a
(;D0)-structure; this immediately establishes the following:
3Set (0)
i= i, and inductively dene (k+1)
i=S
di2Difi:(92((k)
i))(i2di())g.
Then (0);(1);:::is a nested decreasing sequence that cannot include the empty set, so it
must stabilize if  is nite. From such a stable (K), it is easy to construct a ( ;D)-structure
in which each i2(K)
iis played at some state.
14
Proposition 4.1: For each player i, letDiD0
i. Then ifiisD-rationalizable,
it is also D0-rationalizable.
We illustrate these concepts with two examples. It will be useful rst to formally
dene the minimax regret decision rule in our setting. Given a game  and
probability measure on i, let
MRi(i;):= maxfmax
0
i2iui(0
i;i)ui(i;i):(i)>0g;
corresponding to the maximum \regret" player imight feel having played i,
where \regret" is interpreted as the dierence between the best-case payo and
the actual payo (with respect to the strategy proles ithat player iconsiders
possible). The minimax regret decision rule dr
iseeks to minimize this value:
dr
i():=fi2i:(80
i2i)(MRi(i;)MRi(0
i;))g:
Example 4.2: Consider the traveller's dilemma: each of two players must
name an amount in  1=  2=f2;3;:::; 100g, which is the reimbursement they
are requesting for luggage that was damaged by their airline. The airline will
reimburse them both by the minimum amount requested, with one catch: the
person who asks for less receives a $2 bonus, while the person who asks for more
receives a $2 penalty (if they ask for the same amount, no bonuses or penalties
are applied). Thus, payos are dened as follows:
ui() =8
<
:i ifi=i
i+ 2 ifi<i
i2 ifi>i:
Clearly the best payo is achieved by undercutting one's fellow traveler by 1 if
possible, and otherwise (if the other traveler plays 2), playing 2. It is easy to
see that playing 100 is never a deu
i-best response; an iterative deletion argument
then shows that the only rationalizable strategy is to play 2. By contrast, when
eachDi=fdr
ig, playing 100 is D-rationalizable. To prove this, by denition,
it suces to exhibit a ( ;D)-structure in which 100 is played at some state.
Consider the structure presented in Figure 1:
15
Figure 1: A ( ;D)-structure for the traveler's dilemma
Each of the four states of this structure is labeled with the strategy prole being
played at that state, while the edges labeled irepresent which states player i
considers possible (i.e., assigns positive probability to) from which other states
(numerical probabilities are irrelevant for this analysis and so are suppressed).
We must show that each player is playing according to minimax regret. Take
player 1's perspective (the argument for player 2 is analogous); observe rst that
she considers 96 and 100 to be the only possible plays her opponent might make.
Given this, player 1's maximum regret on playing 1>96 must be at least 3,
sinceu1(1;96) = 94, whereas u1(95;96) = 97. Similarly, player 1's maximum
regret on playing 196 must be at least 3, since u1(1;100) =1+ 298,
whereasu1(99;100) = 101. Moreover, it is straightforward to check that player
1's maximum regret on playing either 96 or 100 is exactly 3. It follows that each
of 96 and 100 constitutes a dr
1-best response.
Example 4.3 : Consider the normal-form version of the centipede game
[Rosenthal 1982] depicted in Figure 2: each player must choose whether to quit
at some stage or play to the end. Let  1=fQ1;Q3;Qgand  2=fQ2;Q4;Qg,
whereQkstands for quitting at stage kandQstands for playing to the end.
Payos are determined by the the minimal stage that some player quit at, as
shown in Figure 2. For instance: u(Q1;Q2) =u(Q1;Q4) = (1;0), since in either
case player 1 quits at the rst stage (making player 2's choice irrelevant); on
the other hand, u(Q;Q4) = (2;8), since player 1 never quits and player 2 quits
at the fourth stage.
16
Figure 2: The centipede game
It is well known that all pure strategies in this game are rationalizable; how-
ever, the only strategy that is rationalizable for player 1 under conservative
beliefs|namely, beliefs that ascribe positive probability to the actual state|is
Q1, quitting immediately [Halpern and Pass 2013]. By contrast, we now show
that when each Di=fdeu
i;dr
ig, all strategies are D-rationalizable even under
conservative beliefs.
It is easy to see that Q1andQ2areD-rationalizable with conservative beliefs:
indeed, the structure with exactly one state !wheres(!) = (Q1;Q2) is a (;D)-
structure because each player imust be sure of the actual state, and is easily seen
to be playing a deu
i-best response to this belief. This observation is an instance
of Proposition 4.1 applied to the fact that Q1andQ2are rationalizable (with
conservative beliefs) in the traditional sense of rationalizability.
To show that the remaining strategies are D-rationalizable under conservative
beliefs, it suces to construct a ( ;D)-structure at which each of these strategies
is played and all beliefs are conservative. Consider the structure presented in
Figure 3:
Figure 3: A ( ;D)-structure for the centipede game
As in Figure 1, each of the four states !1,!2,!3, and!4of this structure
is labeled with the strategy prole being played at that state, while the edges
17
labeledirepresent which states player iconsiders possible (i.e., assigns positive
probability to) from which other states. In addition, the fractions adjacent
to the arrowheads specify the numeric probability of each state; for example,
the fractions2
3and1
3indicate that Pr1(!1)(f!1g) = Pr1(!2)(f!1g) =2
3and
Pr1(!1)(f!2g) =Pr1(!2)(f!2g) =1
3, respectively.
We must show that at each state, each player iis playing according to either deu
i
ordr
i. First we show that player 1 is maximizing expected utility in states !1
and!2. In these states player 1 quits at stage 3, which yields an expected utility
of 4 (since player 1 is sure that player 2 will not quit beforehand). Playing Q1
has an expected utility of 1, so Q1is strictly dominated by Q3. Finally, playing
Qresults in a2
3chance of a utility of 2, and a1
3chance of a utility of 7, for an
expected utility of11
3, soQis dominated by Q3.
Next we show that player 1 is minimizing her maximum regret in states !3and
!4. In these states, player 1 plays Qand believes that player 2 will play either
Q4orQ. In the rst case, player 1's payo is 2, but it could have been as high
as 4 had she played Q3; in the second case, her payo is 7, but it could have
been as high as 8 had she played Q4. Thus her maximum regret is 2. How does
this compare to her maximum regret on choosing an alternative strategy? If she
playsQ3, her maximum regret is 3, which arises when player 2 plays Q: in this
case, her payo is 4 but it could have been 7 had she played Qinstead. Even
worse, her maximum regret on playing Q1is 7 (arising as above when player 2
playsQ). This shows that Qis indeed a dr
1-best response in states !3and!4.
Similar arguments show that Q4is adeu
2-best response in states !1and!3, and
Qis adr
2-best response in states !2and!4. Thus, we can set
r(!1) = ( deu
1;deu
2)
r(!2) = ( deu
1;dr
2)
r(!3) = ( dr
1;deu
2)
r(!4) = ( dr
1;dr
2)
to make Figure 3 into a ( ;D)-structure, which proves that each of the strategies
played therein is D-rationalizability under conservative beliefs.
4.2 Axiomatization
Consider the following axiom schemes:
P1._
i2iplayi(i)
P2.:(playi(i)^playi(0
i)), fori6=0
i
P3. playi(i)$Biplayi(i)
18
P4._
di2Dirulei(di)
P5.:(rulei(di)^rulei(d0
i)), for di6=d0
i
P6. rulei(di)$Birulei(di)
P7. (playi(i)^rulei(di))!_
S2S+
i(i)i;S.
Here, as before,S+
i(i) denotes the collection of all Sisuch that there
exists a probability measure with supp() =Sand such that i2di().
Note that there is no need for a symmetric axiom involving S
i(i) for this
logic, because the formula :rulei(di), unlike:RATi, does not say that iis
incompatible with player i's beliefs and the decision rule di; it simply says that
playeridid not usethe rule dito help choose her strategy i(though she may,
coincidentally, have beliefs with respect to which iis adi-best response).
LetGL;Dbe the axiom system that results from adding P1{P7to the KD45
axioms and rules of inference of belief logic. Then we have the following re-
sult, the proof of which proceeds analogously to that of Theorem A.1 given in
Appendix A.1.
Theorem 4.4: GL;Dis a sound and complete axiomatization of the language
LB(;D)with respect to the class of all nite (;D)-structures.
5 Discussion
Almost all solution concepts in game theory are grounded in the idea of rational-
ity and best responding. Thus, one natural application of a logic of rationality
is to the analysis of solution concepts. But doing so raises a number of research
issues.
One subtlety involves the use of mixed strategies. The language LB() has
formulas that represent pure strategy choices, but not mixed strategies. In the
context of Nash equilibrium, this dierence turns out to be (at least formally)
innocuous: one can view a mixed strategy `i2(i) either as a conscious
randomization on the part of player i, or as the common conjecture of the play-
ersj6=iabout what pure strategy iwill choose|either way, the set of mixed
strategy Nash equilibria stays the same. However, this insensitivity is, in part,
dependent on that fact that rationality in the sense of expected utility maxi-
mization \plays well" with mixing: `i2(i) maximizes player i's expected
utility (with respect to some xed beliefs) if and only if every pure strategy i
in the support of `imaximizes expected utility. But this correspondence breaks
down when \expected utility maximization" is replaced with the generalized
notion of rationality presented in Section 2.3: in the context of an arbitrary
19
decision rule di:(i)!2i;, there is no principled way to extend the
notion of \best response" from  ito (i). This suggests that further research
into the interaction between pure and mixed strategies under general decision
rules may be fruitful.
A second issue involves reconsidering what happens to various solution concepts
when we replace maximizing expected utility by another decision rule. Consider,
for example, Nash equilibrium. In principle, it makes sense to consider \ d-Nash
equilibria", dened by replacing deuwith an arbitrary prole of decision rules
din the denition of Nash equilibrium. It is certainly too much to hope that
Nash's famous existence theorem applies in full force to this wider concept;
however, properties of dthat suce to guarantee the existence of equilibria are
of interest, and potentially admit a logical characterization. Such questions are
the subject of ongoing research.
Yet another issue involves understanding the implications for computability of
using various decision rules. In Section 3, we observed that the axiom sys-
tems GLd
are nite extensions of the KD45 system and thus trivially decidable.
Thus, we can, for example, compute whether a formula is a logical consequence
of rationality in any given axiom system GLd
. But there is arguably a more
interesting question as far as decidability goes. Up to now we have considered
decision rules as functions dened with respect to some xed game. But rules
like expected utility maximization, maximin, or minimax regret can be applied
in all games in a uniform way. To capture this, dene a decision paradigm
to be a function that maps each game  to a decision rule in . Suppose that
we are given decision paradigms Difor each player associating with each game
 a decision rule Di() for that player in . We might want to know, given the
prole D= (Di)i2I, whether the mapping
7!GLD()

is decidable; in other words, given as input a game , can we eectively de-
termine whether a formula belongs to the axiom system GLD()
? For each
game , this requires determining membership in the sets S+
i(i) andS
i(i),
which are dened by existential quantication over simplices ( i), subject
to constraints based on the decision rules Di(). In the case of familiar deci-
sion paradigms like maximin or expected utility maximization, computing the
setsS+
i(i) andS
i(i) boils down to solving systems of linear inequalities. In
general, however, we must impose certain computability requirements on the de-
cision paradigms in order to be able to decide whether a formula is an instance
of an axiom. To take an extreme example: we could dene Di() depending on
whether the number of players in  lies in the halting set.
This kind of example suggests that we want to be more restrictive in the form
thatDican take. In particular, we may be able to get more traction on this
problem if we restrict attention to decision paradigms that can be expressed in
some limited language. All the standard decision rules|maximizing expected
utility, minimax regret, maximin, and so on|are of the form \choose strategy
20
ionly if
", where
is a collection of constraints expressible in some simple
language involving quantication over strategies, linear inequalities, etc. We
believe that by identifying appropriate languages and limiting the constraints
that can be used to dene decision paradigms to those expressible in these
languages, we may well be able to establish general decidability results that
apply to decision paradigms rather than merely decision rules.
Finally, through the introduction D-rationality in this paper, we hope to initiate
a broader research program investigating the representational power of endoge-
nous decision rules. Belief update in strategic scenarios is widely recognized as a
foundational issue in modern game theory; the additional structure of decision
rules associated to each state allows a player to learn not just about her op-
ponents' strategy choices and beliefs, but about the mechanism by which they
make decisions under uncertainty. As we have already suggested, this kind of
belief update is particularly relevant in a dynamic setting. Thus, a natural ex-
tension of the present work would be to formulate an extensive-form version of
D-rationalizability and investigate its relationship with standard extensive-form
solution concepts and methods of belief update.
A Proofs
A.1 Axiomatizing LB(d
)
Theorem A.1: GLd
is a sound axiomatization of the language LB(d
)with
respect to the class of all nite -structures.
Proof: Soundness of the axioms and rules of KD45 can be proved as usual. It
therefore suces to show that G1{G6are valid in all nite -structures.
Fix a nite -structure M= (
;(Pri)i2I;s). Soundness of G1andG2is an
immediate consequence of the fact that sis a (total) function. Soundness of
G3is a straightforward consequence of condition (C4), while soundness of G4
follows easily from the combination of conditions (C3) and (C4).
Now suppose that !j=playi(i)^RATi; theni2di(i;!). SetS:=
supp(i;!), and observe that S2 S+
i(i). For each i2S, it is easy to
see that!j=bBiplayi(i); therefore, we must have
!j=^
i2SbBiplayi(i):
Similarly, for each i=2S, we have!j=:bBiplayi(i), so
!j=^
i=2S:bBiplayi(i):
21
In other words, !j=i;S; this establishes the soundness of G5.
Finally, suppose that !j=playi(i)^:RATi, which implies that i=2di(i;!).
SetS:=supp(i;!); thenS2S
i(i) and, as above, we have !j=i;S, which
establishes soundness of G6.
We prove completeness by what is essentially the canonical model method , a
standard method for proving completeness of modal logics (see, e.g., [Fagin,
Halpern, Moses, and Vardi 1995] or any standard text on modal logic). Of
course, the full canonical model is not nite (for n > 1), so we modify the
construction by restricting attention to nite sub-languages. More precisely,
given a formula '2LB(d
), we identify a nite sub-language of LB(d
) such
that the corresponding canonical model refutes 'just in case GLd
6`'. This
technique is sometimes called ltration .
Fix a formula '2LB(d
). LetSub(') denote the collection of all subformulas
of'together with all subformulas of instances of the axiom schemes G1through
G6. Dene
Sub+
('):=Sub(')[f: : 2Sub(')g:
Note that there are only nitely many instances of G1through G6, and there-
foreSub+
(') is nite.
Let
'be the collection of all maximal, consistent (with respect to GLd
) subsets
ofSub+
('). Clearly
'is a nite set. Given XLB(d
), set
XBi:=f :Bi 2Xg;
and, forF2
', dene
Beli(F):=fG2
':GFBiandGBi=FBig:
For eachi2Iand eachF2
', we will dene a probability measure Pr'
i(F) on

'such that the support of this measure is precisely Bel i(F). Loosely speaking,
Beli(F) is the set of all G2
'that are compatible with the beliefs of player i
inF. More precisely, G2Beli(F) if and only if:
(a)Bi 2Fimplies 2G, and
(b)Bi 2FiBi 2G.
Condition (a) just says that everything player ibelieves in Fis true inG;
condition (b) says that player i's beliefs in Fare the same as her beliefs in G,
which is reasonable in light of the fact that we are working in a system with
positive and negative introspection. (In the full canonical model, (b) follows
from (a); here we must impose this condition explicitly because of the way the
language has been restricted.)
Since our aim is to dene probability measures with the sets Bel i(F) as their
supports, we must show that these sets are never empty.
22
Lemma A.2: For eachi2IandF2
',Beli(F)6=;.
Proof: GivenF2
', set
:=f :Bi 2Fg[fBi :Bi 2Fg[f:Bi ::Bi 2Fg:
It is easy to see that  Sub+
('). In addition, we show that  is consistent.
For suppose not; then
GLd
`:^
2=)GLd
`Bi:^
2
=)GLd
`:Bi^
2
=)GLd
`:^
2Bi;
which is a contradiction, since each formula Biwith2 is logically equivalent
to a formula in F, andFis consistent.
From this we can conclude that there exists a G2
'such thatG. It
follows immediately that GFBiand thatGBiFBi. Moreover, if  2GBi,
thenBi 2Gand so certainly:Bi =2G, from which it follows that :Bi =2
and thus:Bi =2F. Maximality of Fthen guarantees that Bi 2F, whence
 2FBi, and soGBiFBi. This establishes that G2Beli(F), as desired.
In the classical canonical model construction, it is sucient to dene Pr'
i(F)
to be the uniform distribution on Bel i(F). In the present context, however,
we need to be more careful, since Pr'
i(F) is used not only to interpret the
belief modalities Bi, but also the primitive propositions RATi. In essence, we
must dene Pr'
i(F) in a manner that agrees with whether or not player iis
best responding to her beliefs at F; not surprisingly, this is precisely where the
axiom schemes G5andG6come into play. At the same time, we have to dene
Pr'
ion
'in a systematic way so as to preserve the introspection condition
(C3). What follows is a formalization of this basic recipe, for which several more
lemmas and denitions are needed.
Lemma A.3: LetF2
'. IfbBi 2F, then there exists a G2Beli(F)with
 2G; if:bBi 2F, then for all G2Beli(F)we have =2G.
Proof: First suppose that bBi 2F, and set
:=f:Bi2Fg[fBi:Bi2Fg[f:Bi::Bi2Fg:
Assume for contradiction that  [f gis inconsistent. We then have
GLd
`^
2!: ;
23
from which it follows that
GLd
`^
2Bi!Bi: : (1)
As observed in Lemma A.2, each Biis equivalent to a formula in F, and
therefore (1) implies that Bi: 2F, contradicting our assumption that bBi 2
F. Thus [f gis consistent, and so can be extended to some G2
';
moreover, as we saw in Lemma A.2, G2Beli(F). This proves the rst statement
of the Lemma. The second statement follows immediately from the denition
of Beli(F): if:bBi 2F, then also Bi: 2F, and so for all G2Beli(F) we
have: 2G, whence =2G.
For eachi2i, dene
Beli(F;i):=fG2Beli(F):playi(i)2Gg:
GivenF2
', it is easy to see, using G1andG2, that there is a unique i2i
with playi(i)2F. If, in addition, RATi2F, then by G5we know that for
someS2S+
i(i),i;S2F(or, in the case where S+
i(i) =;, we know that no
suchFexists). Otherwise, if RATi=2F, then by G6we know that for some
S2S
i(i),i;S2F(or again, in the case where S
i(i) =;, that no such
Fexists). Thus, for each i2I, there is a unique set Si(F)isuch that
i;Si(F)2F, and moreover, Si(F)2S+
i(i) ifRATi2F, andSi(F)2S
i(i)
ifRATi=2F.
Lemma A.4: The collectionfBeli(F;i):i2igpartitions Beli(F);
moreover, Beli(F;i)6=;if and only if i2Si(F).
Proof: The rst statement is a straightforward consequence of G1andG2,
while the second is an immediate corollary of Lemma A.3 together with the fact
that
i;Si(F)^
i2Si(F)bBiplayi(i)^^
i=2Si(F):bBiplayi(i)2F:
For eachi2iandS2S+
i(i), let+
i;Sbe a xed probability measure
witnessing the fact that S2S+
i(i); that is, supp(+
i;S) =Sandi2di(+
i;S).
Likewise, for each i2iandS2S
i(i), let
i;Sbe a xed probability
measure witnessing the fact that S2S
i(i).
LetF2
', and suppose that playi(i)2F. In light of Lemma A.4, we can
dene Pr'
i(F) to be the unique probability measure on Bel i(F) such that, for
alli2i,
Pr'
i(F)(Beli(F;i)) =(
+
i;Si(F)(i) if RATi2F

i;Si(F)(i) if RATi=2F,
24
and which is uniform within each (nonempty) set Bel i(F;i).
Proposition A.5: Pr'
isatises the following:
(a) Pr'
i(F)(G)>0iG2Beli(F), and
(b)GBi=FBiimplies Pr'
i(G) =Pr'
i(F).
Proof:
(a) The forward implication is immediate from the denition. For the reverse
implication, suppose that G2Beli(F); then, by Lemma A.4, we know
thatG2Beli(F;i) for some i2Si(F), from which it follows that
Pr'
i(F)(G)>0 by denition.
(b) IfGBi=FBithen Beli(G) = Beli(F). Moreover, axioms G3andG4
guarantee that playi(i)2Fif and only if playi(i)2G, and likewise
RATi2Fif and only if RATi2G. Finally, it is not dicult to see that
Si(F) is completely determined by Bel i(F), soSi(F) =Si(G). Therefore,
by denition of Pr'
i, we can deduce that Pr'
i(G) =Pr'
i(F).
Finally, we dene a strategy function s':
'! by assigning to each F2
'
the unique strategy prole 2 such that play()2F.
Lemma A.6: The tupleM':= (
';(Pr'
i)i2I;s')is a-structure.
Proof: Conditions (C1) and (C2) have already been established. By Lemma
A.5(b), in order to see that (C3) holds it suces to observe that Pr'
i(F)(G)>
0 implies that GBi=FBi, which follows from Lemma A.5(a). This same
observation also establishes (C4), since by G3we know that GBi=FBiimplies
s'
i(G) =s'
i(F).
Lemma A.7: For all formulas  , for allF2
', if 2Sub+
(')thenF2
[ [ ] ]M'if and only if  2F.
Proof: The proof proceeds by induction on the structure of  . We prove here
the base cases corresponding to the primitive propositions; the inductive steps
can be proved in the standard way (see, e.g., [Fagin, Halpern, Moses, and Vardi
1995]).
First consider the primitive proposition playi(i). We have
F2[ [playi(i)] ]M'is'
i(F) =i
i playi(i)2F;
25
as a direct consequence of the denition of s'. Next consider the primitive
proposition RATi; we have
F2[ [RATi] ]M'is'
i(F)2di(i;F)
i RATi2F;
the last equivalence being a consequence of the denition of Pr'
i, which ensures
thats'
i(F) is a best response to (the pushforward of) Pr'
i(F) precisely when
RATi2F. This completes the proof.
Theorem A.8: GLd
is a complete axiomatization of the language LB(d
)with
respect to the class of all nite -structures.
Proof: Suppose that GLd
6`'. Thenf:'gis consistent and so can be extended
to a maximal consistent set F2
'. By Lemma A.7, this implies that F =2
[ ['] ]M'and so, in particular, that M'6j=', as desired.
A.2 Belief as lower probability
Theorem A.9: GLd
is a sound and complete axiomatization of the language
LB(d
)with respect to the class of all nite lower -structures.
Proof: The proof given in Appendix A.1 works here as well, modulo the obvious
minor alterations in keeping with the generalized denitions given in Section 3.1.
In particular, for each i2iandS2S+
i(i), we dene P+
i;Sto be a xed set
of probability measures such that supp(P+
i;S) =Sandi2di(P+
i;S); likewise,
for eachi2iandS2S
i(i), dene P
i;Sto be a xed set of probability
measures witnessing the fact that S2 S
i(i). Then, given F2
'with
playi(i)2F, dene PR'
i(F) as follows: for each 2P+
i;S[P
i;S, let ~be
the unique probability measure on Bel i(F) such that, for all i2i,
~(Beli(F;i)) =(i);
and which is uniform on each set Bel i(F;i); then set
PR'
i(F):=(
f~:2P+
i;SgifRATi2F
f~:2P
i;SgifRATi=2F.
A.3 Axiomatizing LB(D
)
To obtain a complete axiomatization of this language, it is not sucient to sim-
ply let G4{G6range over all decision rules di2Difor each player i; in general,
26
this system is sound but not complete. Roughly speaking, this is because it
is possible for both playi(i)^RATdi
iand playi(i)^RATd0
i
ito be consistent
withi;Sfor someSi, yet no measure with supp() =Sis such that
i2di()\d0
i(). In this case, the formula
playi(i)^RATdi
i^RATd0
i
i^i;S
is not satisable, but there is no way to prove its negation from the axioms.
However, provided that each set Diis nite, we can deal with this problem by
replacing G5andG6with the following collection of axioms for each player i,
each strategy i2i, and every subset DDi:

playi(i)^^
di2DRATdi
i^^
di=2D:RATdi
i
!_
S2SD
i(i)i;S;
whereSD
i(i) is the collection of all Sisuch that there exists a probability
measureonSsuch that supp() =Sand for every di2D,i2di(), and for
every di=2D,i=2di().G5andG6are special cases occurring when jDij= 1,
corresponding to D=DiandD=;, respectively.
Acknowledgements
Bjorndahl is supported in part by NSF grants IIS-0812045, CCF-1214844, DMS-
0852811, and DMS-1161175, and ARO grant W911NF-09-1-0281. Halpern is
supported in part by NSF grants IIS-0812045, IIS-0911036, and CCF-1214844,
by MURI grant FA9550-12-1-0040, and by ARO grant W911NF-09-1-0281. Pass
is supported in part by an Alfred P. Sloan Fellowship, a Microsoft Research Fac-
ulty Fellowship, NSF Awards CNS-1217821 and CCF-1214844, NSF CAREER
Award CCF-0746990, AFOSR YIP Award FA9550-10-1-0093, and DARPA and
AFRL under contract FA8750-11-2-0211. The views and conclusions contained
in this document are those of the authors and should not be interpreted as
representing the ocial policies, either expressed or implied, of the Defense Ad-
vanced Research Projects Agency or the US Government. This paper subsumes
and extends an earlier conference publication [ ?].
References
Aumann, R. J. (1999). Interactive epistemology II: probability. International
Journal of Game Theory 28 (3), 301{314.
Battigalli, P. and M. Siniscalchi (2002). Strong beliefs and forward-induction
reasoning. Journal of Economic Theory 106 , 356{391.
Bjorndahl, A., J. Y. Halpern, and R. Pass (2011). Reasoning about justied
belief. In Theoretical Aspects of Rationality and Knowledge: Proc. Thir-
teenth Conference (TARK 2011) , pp. 221{227.
27
Brandenburger, A. and E. Dekel (1987). Rationalizability and correlated equi-
libria. Econometrica 55 , 1391{1402.
Capra, M., J. K. Goeree, R. Gomez, and C. A. Holt (1999). Anomalous
behavior in a traveler's dilemma. American Economic Review 89 (3), 678{
690.
Fagin, R., J. Y. Halpern, and N. Megiddo (1990). A logic for reasoning about
probabilities. Information and Computation 87 (1/2), 78{128.
Fagin, R., J. Y. Halpern, Y. Moses, and M. Y. Vardi (1995). Reasoning About
Knowledge . Cambridge, Mass.: MIT Press. A slightly revised paperback
version was published in 2003.
Halpern, J. Y. (2003). Reasoning About Uncertainty . Cambridge, Mass.: MIT
Press.
Halpern, J. Y. and Y. Moses (2007). Characterizing solution concepts in
games using knowledge-based programs. In Proc. Twentieth International
Joint Conference on Articial Intelligence (IJCAI '07) , pp. 1300{1307.
Halpern, J. Y. and R. Pass (2012). Iterated regret minimization: a new solu-
tion concept. Games and Economic Behavior 74 (1), 194{207.
Halpern, J. Y. and R. Pass (2013). Conservative belief and rationality. Games
and Economic Behavior 80 , 186{192.
Hayashi, T. (2008). Regret aversion and opportunity dependence. Journal of
Economic Theory 139 (1), 242{268.
Osborne, M. J. and A. Rubinstein (1994). A Course in Game Theory . Cam-
bridge, Mass.: MIT Press.
Rosenthal, R. W. (1982). Games of perfect information, predatory pricing,
and the chain store paradox. Journal of Economic Theory 25 , 92{100.
Tan, T. and S. Werlang (1988). The Bayesian foundation of solution concepts
of games. Journal of Economic Theory 45 (45), 370{391.
28

Adam Bjorndahl
Joseph Y
Halpern
Rafael Pass