Towards Cooperation in Learning Games

Towards cooperation in learning games
Jesse Clifton Maxime Riché
Abstract
Suppose that several actors are going to deploy learning agents to act on their behalf. What principles
should guide these actors in designing their agents, given that they may have competing goals? An
appealing solution concept in this setting is welfare-optimal learning equilibrium . This means that the
learning agents should constitute a Nash equilibrium whose payoﬀ proﬁle is optimal according to some
measure of total welfare (welfare function). In this work, we construct a class of learning algorithms in
this spirit called learning tit-for-tat (L-TFT) . L-TFT algorithms maximize a welfare function according to
a speciﬁed optimization schedule, and punish their counterpart when they detect that they are deviating
from this plan. Because the policies of other agents are not in general fully observed, agents must infer
whether their counterpart is following a cooperative learning algorithm. This requires us to develop
new techniques for making inferences about counterpart learning algorithms. In two sequential social
dilemmas, our L-TFT algorithms successfully cooperate in self-play while eﬀectively avoiding exploitation
by and punishing defecting learning algorithms.
1 Introduction
As the capabilities of machine learning systems improve, it is increasingly important to understand how to
achieve cooperation between systems deployed by actors with competing goals. Moreover, because intelligent
systems may continue to learn after they are deployed, designing agents who can cooperate requires accounting
for ways their policies may change post-deployment. In this paper, we look at the setting where multiple
learning agents are deployed to act on behalf of their respective principals.
There are three main goals of this work. The ﬁrst is to articulate an appealing criterion for the design of
learning agents, which is welfare-optimal learning equilibrium . The second is to present a general approach to
designing learning agents who approximately fulﬁll this criterion, which we call learning tit-for-tat (L-TFT) .
This framework is generic to agents who update their policies over time, rather than depending on a particular
class of reinforcement learning algorithms. The third is to illustrate some of the strategic considerations
which need to be addressed in the design of L-TFT agents. We do this by constructing an L-TFT agent
using model-free reinforcement learning algorithms, and examining its performance in two sequential social
dilemmas. This agent converges to welfare-optimal payoﬀs in self-play, while successfully avoiding exploitation
and punishing defecting learning algorithms.
A learning equilibrium is simply a Nash equilibrium of a learning game , in which strategies are learning
algorithms, and payoﬀs are the cumulative rewards associated with the proﬁle of learning algorithms chosen
by each principal [Brafman and Tennenholtz, 2003]. Principals will want to choose the learning agents they
deploy strategically. This means that it is not enough for a learner to perform well against a speciﬁed set
of other players (e.g., Shoham and Leyton-Brown 2008). It is also not enough that a learner converges to
Nash equilibrium of the game which is being learned (e.g. Bowling and Veloso 2001, Conitzer and Sandholm
2007, Balduzzi et al. 2018, Letcher et al. 2018), as this does not guarantee that a player cannot beneﬁt by
deviating from that proﬁle (i.e., deploying a diﬀerent learning algorithm).
Like other sequential games, learning games will generally have many learning equilibria. Independent
equilibrium selection on part of the principals may lead to poor performance. Thus, to guarantee socially
optimal payoﬀs, principals need to coordinate when designing their agents. L-TFT addresses this problem
via awelfare function . A welfare function measures the social value of a payoﬀ proﬁle; for instance, a simple
welfare function is the sum of the principals’ respective payoﬀs. An L-TFT agent learns a policy proﬁle
which optimizes this welfare function, and punishes its counterpart if it detects that they are not doing
the same. Thus agents are incentivized to follow a learning algorithm which optimizes the welfare function.
1
Coordination can thus be achieved via coordination on a welfare function and on a class of learning algorithms
for optimizing it.
In our experiments, learners observe only the actions taken by their counterpart, and not their counterpart’s
(stochastic) policy. This requires us to construct L-TFT agents who model their counterpart’s learning
schedule based on their actions and conduct a hypothesis test as to whether it matches a cooperative learning
schedule. This in turn introduces incentives to exploit this testing procedure. We examine the performance
of L-TFT against an algorithm designed speciﬁcally to exploit the way in which it detects defections, as
well as its performance in self-play and against a naive selﬁsh learner. Lastly, there is a strategic element in
choosing the sensitivity with which L-TFT’s hypothesis test detects defections. In Section 5.3, we examine
this element in a learning game between an L-TFT agent and a potential exploiter choosing how cautious to
be in deviating from a cooperative learning algorithm.
2 Related work
Learning equilibrium was ﬁrst discussed (under that name) by Brafman and Tennenholtz [2003]. Their
construction of learning equilibria involves evaluating each of a ﬁnite set of policies in a ﬁnite stochastic game,
whereas the algorithms we focus on use incremental stochastic updates towards the welfare-optimal policy
proﬁle.
Our online learning setting complements the setting in which agents are trained fully oﬄine and then
deployed. This is the setting typically considered in applications of reinforcement learning to cooperation in
social dilemmas. For instance, Peysakhovich and Lerer [2017], Lerer and Peysakhovich [2017], Wang et al.
[2018] each develop methods in which punishment and cooperative policies are trained oﬄine, and cooperation
at deployment time is sustained by the threat of punishment.
However, as many agents may learn post-deployment, the strategic aspects of learning itself — rather
than just policy selection — must be addressed. The manipulation of the learning of counterpart agents has
been explored in the literature on opponent-shaping [Foerster et al., 2018, Letcher et al., 2018]. Perhaps
the most similar to our approach is the variant of Foerster et al. [2018]’s learning with opponent learning
awareness which uses opponent modeling (LOLA-OM). LOLA-OM uses supervised learning to estimate the
policy followed by the other agent, and updates its own policy in a way that is intended to manipulate the
direction in which the other player updates their policy. On the other hand, our L-TFT uses opponent
modeling to infer whether the other agent is following a cooperative learning algorithm.
3 Preliminaries: Learning games
Consider a situation in which multiple agents interact with their environment over time, incrementally
updating the policies which they use to choose actions. These updates are controlled by their learning
algorithms, which are ﬁxed by their principals before the agents begin interacting with the environment. It
is helpful to distinguish between two diﬀerent games that are being played here. There is the base game ,
which is deﬁned by the dynamics of the environment, the reward functions of the agents, and the policies
available to each agent. In this paper, the “base game” will be a stochastic game (described below). There is
also thelearning game , in which the principals simultaneously submit learning algorithms to be deployed
in the base game environment, and attain payoﬀs equal to some measure of cumulative reward.
We work with stochastic games [Littman, 1994], a generalization of Markov decision processes to the
multi-agent setting. We will assume only two players, i= 1;2. We’ll refer to player i’s counterpart with
the indexj. In a stochastic game, players simultaneously take actions at each time t2N; observe a reward
depending on the current state; and the state evolves according to a Markovian transition function. Write the
state space asSand the action spaces as Ai. Let the reward functions be mappings ri:SA 1A 2!R,
with the reward to player iat timetgiven byRt
i=ri(St;At
1;At
2). LetHtbe the history of observations
until timet. We will assume that both agents fully observe the state and each others’ actions and rewards,
such thatHt=f(Sv;Av
1;Av
2;Rv
1;Rv
2)gt
v=0.
Apolicyigives a probability distribution over actions to be taken by player iat each state. Players
wish to learn the parameters iof a policy which (in some sense) leads to large cumulative reward. Then we
deﬁne a learning algorithm.
2
Deﬁnition 3.1 (Learning algorithm) .A learning algorithm for player iis a mapping ifrom histories Htto
parameters, i.e., i:H! i, whereHis the set of histories and iis the space of policy parameters for
playeri.
A learning game is a game whose strategies are learning algorithms. We deﬁne payoﬀs via average
rewards. Deﬁne the value to player iof learning algorithm proﬁle (1;2)starting from history htas
Vi(ht;) =lim infT!1T1E1;2Pt+T
v=tRv
ijHt
, where E1;2is the expectation taken with respect to
trajectories in which agents follow learning algorithms 1;2at each step. Let the initial state s0be ﬁxed so
thath0=fs0g. Then a learning game is a game whose strategy spaces are spaces of learning algorithms 1;2,
and whose payoﬀs are given by Vi(1;2)Vi(h0;1;2). A learning equilibrium is a Nash equilibrium of a
learning game:
Deﬁnition 3.2 (Learning equilibrium) .Learning algorithm proﬁle (1;2)is alearning equilibrium of the
learning game with learning algorithm spaces 1;2if
sup
0
121V1(0
1;2)V1(1;2)and sup
0
222V2(1;0
2)V2(1;2):
Finally, let a welfare function wbe a function measuring the social value of diﬀerent learning algorithm
proﬁles. A straightforward welfare function is the utilitarian welfare ,w(1;2) =V1(1;2) +V2(1;2).
Others commonly discussed are the Nash welfare [Nash, 1950] and the egalitarian welfare (e.g., Kalai 1977).
We say that a proﬁle of learning algorithms (1;2)is awelfare-optimal learning equilibrium if it is a learning
equilibrium and if it maximizes a welfare function w. We will construct learning algorithms which 1) optimize
a welfare function in self-play and 2) are minimally exploitable, in the spirit of welfare-optimal learning
equilibrium.
4 Learning tit-for-tat (L-TFT)
Tit-for-tat (TFT) famously enforces cooperation in the iterated Prisoner’s Dilemma by cooperating when its
counterpart cooperates, and defecting when they defect. Similarly, the “folk theorems” say that equilibria
in repeated games are constructed by specifying a target payoﬀ proﬁle, a proﬁle of target strategies which
achieve this payoﬀ proﬁle, and a punishment strategy that is used to punish agents who deviate from the
target strategy (see Mailath et al. [2006] for a review).
Learning tit-for-tat (L-TFT) is a class of learning algorithms that are analogous to tit-for-tat for learning
games. For L-TFT, defection consists in playing a policy other than the current estimate of the cooperative
policy. This depends on a notion of cooperation and a class of algorithms for learning a “cooperative” policy.
These are encoded in a welfare function and a class of learning algorithms which learn an optimal policy with
respect to that welfare function. Thus an L-TFT agent punishes their counterpart when it detects that they
are not learning a welfare-optimal policy using a particular set of algorithms, and otherwise plays according
to their current estimate of the welfare-optimal policy.
Because policies are in general stochastic, and because we do notassume that policy parameters are
mutually visible, we also need a method for inferring whether the other player is following this cooperative
algorithm. In particular, we need to construct a test for the hypotheses:
H0:My counterpart is following an algorithm for learning a welfare-optimal policy.
H1:My counterpart is not following an algorithm for learning a welfare-optimal policy.
In order to be able to cooperate and punish, as well as detect defections by their counterpart, an agent i
playing L-TFT maintains the following policy estimates:
A cooperative policy C;t
itrained to optimize the welfare function. This network is updated according
to a learning algorithm OC. In our experiments, OCcorresponds to double deep Q-networks (DDQN;
Van Hasselt et al. 2015) trained on the reward signal Rt
1+Rt
2;
A punishment policy P;t
i, updated according to a learning algorithm OP. In our experiments, this
corresponds to DDQN trained on the reward signal Rt
j;
3
An estimatebC;t
jof the other player’s current policy which corresponds to the hypothesis that they are
playing according to a cooperative learning algorithm. This policy is estimated by specifying a class of
cooperative learning algorithms C
j; ﬁnding the learning algorithm bC;t
jin this class which best ﬁts the
other player’s actions; and taking bC;t
j=bC;t
j(Ht). In our experiments, the L-TFT agent assumes that
the learning algorithms C
japply DDQN to Rt
1+Rt
2and follow a Boltzmann exploration policy with
an unknown initial temperature parameter. In order to obtain bC;t
jwe must therefore estimate this
temperature parameter, which we do using an approximate maximum likelihood algorithm;
An estimatebD;t
jof the other player’s current policy which corresponds to the hypothesis that they are
not playing according to a cooperative learning algorithm. This policy is estimated by specifying a
class of learning algorithms D
j, ﬁnding the learning algorithm bD;t
jin this class which best ﬁts the
other player’s actions; and taking bD;t
j=bD;t
j(Ht). In our experiments, we approximate this procedure
by obtainingbD;t
jvia supervised learning on player j’s actions given the state.
Algorithm 1 summarizes L-TFT. Note that, while our experiments use a standard model-free learning
algorithm for the base algorithms OC;OP, this framework is highly general. For instance, the players might
be model-based planners who choose actions by optimizing against an environment model with parameter
i, which is updated over time. Moreover, with more sophisticated methods for inferring the other player’s
learning algorithm, L-TFT algorithms may also be constructed in games with partial state observability, or
uncertainty about the other players’ rewards.
Algorithm 1: Learning tit-for-tat (L-TFT)
Input: Observation history Ht, Other player punishment indicator being\_punished , defection at
previous timestep indicator defection\_detected , cooperative learning algorithm OC,
punishment learning algorithm OP, defection hypothesis test HypothesisTest
punishing is\_punishing (Ht);
P;t
i OP(Ht);
ifnotbeing\_punished ordefection\_detected then
C;t
i OC(Ht);
ifdefection\_detected andnotbeing\_punished then
At
iP;t
i(jSt);
Broadcast punishing;
else
At
iC;t
i(jSt);
defection\_detected HypothesisTest (Ht);
returnAt
i;defection\_detected
5 Experiments
5.1 L-TFT with DDQN + Boltzmann exploration
For the cooperative and punishment learning algorithms, we use DDQN applied to reward signals Rt
1+Rt
2
andRt
j, respectively. Agents follow Boltzmann exploration with respect to the estimated Q-function, i.e.,
i(aijs;i)/expnP
ajQi(s;ai;aj)=io
, whereQiis a neural network with parameter i. The initial
temperature iis decayed according to a schedule g(;i), such that the temperature at each time step tis
t
i=g(t;i). (In our experiments glinearly decreases to a minimum value, and is constant thereafter.) Our
L-TFT algorithm punishes for one episode after it has detected a defection. We assume that players may
broadcast punishment to their counterpart. L-TFT agents do not update their cooperative and defecting
policy estimates if their counterpart broadcasts punishment, and do not punish if they are currently being
punished; for simplicity we suppress the dependence of agents’ policies and of the state on these broadcasts.
4
Testing for defection: L-TFTq
To decide whether to punish, our agent conducts a goodness-of-ﬁt test for the hypothesis that the other
player is cooperating. A goodness-of-ﬁt test (e.g., Lemeshow and Hosmer Jr 1982) compares the ﬁt of a
simple model with the ﬁt of a ﬂexible model. If the ﬁt of the simple model is suﬃciently close to that of
the ﬂexible model, the simple model is deemed adequate. Otherwise, the simple model is said to ﬁt poorly.
Here, the simple model we assess is the hypothesis that the other player is using cooperative DDQN with
Boltzmann exploration. The ﬂexible model we would like to compare to is that the other player is following
an arbitrary learning algorithm (which need not be cooperative). One approach to estimating this model
could be to solve the penalized maximum likelihood problem
bD;1
j;:::;bD;t
j= arg max
D;1
j;:::;D;t
jtX
v=1logD
D;v
j(AvjSv) +t1tX
v=1kD;v
jk2; (1)
whereD
jis a class of neural networks that output a distribution over actions given a state. But solving 1
is computationally infeasible. As an approximation, we maintain a sequence fbD;v
jgt
v=1by taking gradient
steps on the supervised learning loss at each time step. We maintain a buﬀer of log likelihoods under the
hypothesis of defection, LD;t
j=flogD
bD;v
j(Av
jjSv)gt
v=tt0. (The estimate bD;t
jand log likelihood buﬀer LD;t
j
are not updated while the other player broadcasts punishment.)
On the other hand, to obtain likelihoods under the hypothesis of cooperation, player jis cooperating, we
need to estimate the initial temperature junder the assumption that player jis following a cooperative
learning algorithm for some j. We would thus like to solve
bt
j= arg max
jtX
v=1logC;v
jfAvjSv;g(v;j)g: (2)
As a tractable approximation to the solution of problem 2, we maintain an estimated cooperative Q-
networkbC;t
jvia DDQN updates on reward signal Rt
1+Rt
2(only when not being punished). We track
the likelihood of player j’s actions under bC;t
jover a set of candidate values for 0
j; and takebt
jto be
the highest-likelihood such value at time t. For each 0
j, we also maintain a buﬀer of log likelihoods1
LC;t
j(0
j) =flogC;v
j
Av
jjSv;g(v;0
j)
gt
v=tt0. (EstimatebC;t
jand buﬀers LC;t
j(0
j)are again not udpated
while the opponent broadcasts punishment.)
Finally, the hypothesis test is conducted at the end of each episode by bootstrapping [Efron, 1992] the
diﬀerence in mean log likelihoods over buﬀers LC;t
j(bt
j)andLD;t
j. The hypothesis of cooperation is rejected
when theqthquantile of this distribution is less than 0. This hypothesis test is summarized in Algorithm 2.
We refer to the variant of L-TFT which uses this test as L-TFT q.
Exploiting L-TFTq
We also construct exploiter agents to play against L-TFTq. These agents assume their counterpart is L-TFTq
for a particular value of q, simulate the hypothesis test used by L-TFTq, and act (myopically) depending on the
result of the test. That is, if an exploiter predicts that a defection would be detected by a test with cutoﬀ q,
it acts according to the cooperative policy; otherwise, it takes a selﬁsh action (according to a DDQN network
trained on its reward signal only). Call this agent Exploiterq. The networks maintained by Exploiterq
are: 1) a cooperative network trained on Rt
1+Rt
2, 2) a selﬁsh network trained on Rt
i, 3) an estimate of its
ownpolicy under the hypothesis that it is cooperating eC;l
j, and 4) an estimate of its own policy under the
hypothesis that it is defecting eD;l
j.
1When computing these likelihoods, we set the Q-value of an action equal to the mean of all Q-values which are within some
distance of one another (in our experiments, 0:25). This is important in Coin Game, where there are multiple paths of equal
length to a coin. Thus, although estimated Q-values along diﬀerent paths may be very similar, these diﬀerences are ampliﬁed
by the exponentiation used in Boltzmann exploration when the temperature is small, possibly leading to false detections of
defection.
5
Algorithm 2: Hypothesis test for L-TFTq
Input: Defection log likelihood buﬀer LD
j, cooperative log likelihood buﬀer LC
j(bt
j), cutoﬀ quantile q,
test statistics buﬀer f(q);v
jgt1
v=tu
/\*tindexes episodes \*/
j ;
forb= 1;:::;Bdo
LC;b
j Bootstrapn
LC;t
j(bt
j)o
LD;b
j Bootstrap (LD;t
j)
LC;b
j B1P
`C
j2LC;b
j`C
j
LD;b
j B1P
`D
j2LD;b
j`D
j
j j[n
LC;b
jLD;b
jo
(q);t
j Percentile (j;q)
(q);t
j u1Pt
v=tu(q);v
j// Average previous test statistics for stability
if(q);t
j<0then
defection\_detected True
else
defection\_detected False
return defection\_detected
5.2 Environments
The environments we use are the iterated Prisoner’s Dilemma (IPD) and the Coin Game environment
[Lerer and Peysakhovich, 2017]2. In both cases, episodes are 20 timesteps long. Our version of the IPD
involves repeated play of the matrix game with expected payoﬀs as in Table 1. Policy igives the
probability of each action given player j’s action at the previous timestep. In our implementation, this
means that the state passed to the estimated Q-function is the proﬁle of actions taken at the last step.
Player 2
C D
Player 1C1;13;0
D 0;32;2
Table 1: Prisoner’s Dilemma expected payoﬀs.
Figure 1: Coin Game Payoﬀs.
Write the expected reward function, corresponding to the payoﬀs in Table 1, as ri(ai;aj). We introduce
randomness by generating rewards as Rt
iN
ri(At
i;At
j);0:1
.
In the Coin Game environment [Lerer and Peysakhovich, 2017] depicted in Figure 1, a Red and Blue
player navigate a grid and pick up randomly-generated coins. Each player gets a reward of 1for picking up a
coin of any color. But, a player gets a reward of 2if the other player picks up their coin. This creates a
social dilemma in which the socially optimal behavior is to only get one’s own coin, but there is incentive to
defect and try to get the other player’s coin as well.
2These experiments were implemented using extensions of OpenAI Gym [Brockman et al., 2016] and the SLM Lab framework
for reinforcement learning [Keng and Graesser, 2017].
6
The state space consists of encodings of each player’s location and the location of the coin. We use
networks with two convolutional layers and two fully connected feedforward layers.
5.3 Results
Figure 2: Performance of L-TFT 0:95in the IPD (left) and Coin Game (right). Counterparts are L-TFT 0:95
(top), Exploiter0:95(bottom). For IPD, the probaxis is the probability of mutual cooperation ( cc) and
mutual defection ( dd), For Coin Game, the pick\_own axis is the proportion of times the coin is picked by
the agent of its own color (i.e., the agents cooperate). The payoffs axis shows a moving average of player
rewards after each episode. These values are averaged over 10 replicates and bands are (min, max) values.
Note that temperature values are constant in the IPD after 3000 episodes and in the Coin Game after 30000
episodes.
Figure 2 shows the performance of L-TFTqin self-play, against Exploiterq(i.e., an exploiter with
a correctly-speciﬁed value of q).L-TFTqis able to converge to welfare-optimal payoﬀs in self-play and
against the Exploiterq(suggesting that the exploiter eventually anticipates being punished if it defects,
and therefore chooses to cooperate). This indicates that our hypothesis test eﬀectively detects defection.
L-TFTqalso eﬀectively punishes a naive selﬁsh learner (i.e., one trained with DDQN on Rt
j); see Figure 3 in
the Supplementary Materials. Punishing the naive learner comes as the expense of low payoﬀs for L-TFT.
However, this is the expected behavior, just as the punishments which enforce equilibria in classical iterated
games can lead to low payoﬀs for each player.
Strategic choice of q
While our previous experiments with Exploiterqassumed that the exploiter knew the cutoﬀ quantile used in
L-TFT’s hypothesis test, this seems like too strong an assumption for real-world agents deployed by diﬀerent
principals. Here we consider a game played between 1) one principal deciding which value of q1to specify for
itsL-TFTq1agent, and 2) a second principal deciding among L-TFTq2andExploiterq2for diﬀerent values of
q2. Table 2 shows the average payoﬀs to each player over a time horizon of 4000 steps.
7
qinL-TFT q qinExploiterq
0:55 0:75 0:95 0:55 0:75 0:95
L-TFT 0:551:28;1:20:97;1:480:98;1:381:03;1:411:06;1:430:91;1:81
L-TFT 0:751:44;0:971:13;1:11:09;1:131:19;1:111:25;1:111:21;1:35
L-TFT 0:951:21;1:061:12;1:11:11;1:111:2;1:061:27;1:031:41;0:98
Table 2: Empirical payoﬀ matrix for a learning game in which the base game is an iterated Prisoner’s
Dilemma played for 4000 timesteps. The row player chooses L-TFTqforq2f0:55;0:75;0:95g. The column
player chooses either L-TFTqorExploiterqforq2f0:55;0:75;0:95g. Cells give (row player payoﬀs, column
player payoﬀs), where payoﬀs are the mean of rewards over the span of the game, averaged over 10 replicates.
Standard errors are all less than or equal to 0.1.
It turns out that the unique Nash equilibrium of this empirical learning game3is one in which row player
chooses L-TFT 0:95with probability 0:88andL-TFT 0:55with probability 0:12, while the column player chooses
L-TFT 0:55with probability 0:88andExploiter0:95with probability 0:12. The expected payoﬀs in equilibrium
are1:23;1:08for row and column player, respectively. This suggests that L-TFTqis vulnerable to some
amount of exploitation on short time horizons, although still does better than the mutual-defection payoﬀ of
2. On the other hand, L-TFTq1converges to mutual cooperation4against both L-TFTq2andExploiterq2
for each value of q1;q2. Thus L-TFT avoids exploitation over longer time horizons in this learning game.
6 Discussion
Powerful learning agents should be designed to avoid conﬂict when they are eventually deployed. One way
to avoid such outcomes is to construct proﬁles of agents which avoid conﬂict and which rational principals
prefer to deploy. One appealing criterion for joint rationality on part of the principals is welfare-optimal
learning equilibrium (rather than mere convergence to Nash equilibrium of the base game, for instance). We
have presented a class of learning algorithms, learning tit-for-tat (L-TFT), which approximately implement
welfare-optimal learning equilibrium in the sense of converging to welfare-optimal payoﬀs in self-play and
discouraging exploitation.
Many open questions remain. First, while we have relied on a welfare function, we have said little about
which welfare function should be used. The ideal scenario would be for the principals of the reinforcement
learning systems in question to coordinate on a welfare function before deploying these systems. Moreover, it
may be necessary to develop novel reinforcement learning methods tailored for the optimization of diﬀerent
welfare functions. For instance, our use of DDQN with the utilitarian welfare is unproblematic, as the
sum of the players’ reward signals still admits a Bellman equation for their cumulative reward. But the
welfare functions other than the utiliarian welfare (such as the Nash welfare [Nash, 1950]) do not admit a
Bellman equation (see the discussion of nonlinear scalarization functions in Roijers et al. [2013]’s review of
multi-objective reinforcement learning).
Many generalizations of our setting can be studied, including partial observability and more than two
agents. Perhaps the most restrictive of our assumptions is that the agents’ reward functions are known to
each other. A complete framework should address the problem of incentives to misrepresent one’s reward
function to improve one’s payoﬀs in the welfare-optimal policy proﬁle. One direction would be to allow for
uncertainty about the other players’ reward functions, analogously to Bayesian games [Harsanyi, 1967] but
in a way that is tractable in complex environments. This may in turn require the development of novel
techniques for inference about other players’ preferences in complex environments; see Song et al. [2018],
Yu et al. [2019] for recent steps in this direction. Another approach to reward uncertainty would be to use
mechanism design techniques [Nisan et al., 2007, Ch. 9] to incentivize the truthful reporting of the principals’
utility functions.
3Computed using the Nashpy Python library [Knight and Campbell, 2018].
4Payoﬀs averaged over the ﬁnal 10 episodes equal to 10:03.
8
Acknowledgements
Many thanks to Tobias Baumann, Anthony DiGiovanni, Flo Dorner, Daniel Kokotajlo, Alex Lyzhov, and
Johannes Treutlein for helpful comments on drafts of this document.
References
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The
mechanics of n-player diﬀerentiable games. arXiv preprint arXiv:1802.05642 , 2018.
Michael Bowling and Manuela Veloso. Rational and convergent learning in stochastic games. In International
joint conference on artiﬁcial intelligence , volume 17, pages 1021–1026. Lawrence Erlbaum Associates Ltd,
2001.
Ronen I Brafman and Moshe Tennenholtz. Eﬃcient learning equilibrium. In Advances in Neural Information
Processing Systems , pages 1635–1642, 2003.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech
Zaremba. Openai gym, 2016.
Vincent Conitzer and Tuomas Sandholm. Awesome: A general multiagent learning algorithm that converges
in self-play and learns a best response against stationary opponents. Machine Learning , 67(1-2):23–43,
2007.
Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics , pages
569–593. Springer, 1992.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch.
Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on
Autonomous Agents and MultiAgent Systems , pages 122–130. International Foundation for Autonomous
Agents and Multiagent Systems, 2018.
John C Harsanyi. Games with incomplete information played by “bayesian” players, i–iii part i. the basic
model.Management science , 14(3):159–182, 1967.
Ehud Kalai. Proportional solutions to bargaining situations: interpersonal utility comparisons. Econometrica:
Journal of the Econometric Society , pages 1623–1630, 1977.
Wah Loon Keng and Laura Graesser. Slm lab. https://github.com/kengz/SLM-Lab , 2017.
Vincent Knight and James Campbell. Nashpy: A python library for the computation of nash equilibria.
Journal of Open Source Software , 3(30):904, 2018.
Stanley Lemeshow and David W Hosmer Jr. A review of goodness of ﬁt statistics for use in the development
of logistic regression models. American journal of epidemiology , 115(1):92–106, 1982.
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep
reinforcement learning. arXiv preprint arXiv:1707.01068 , 2017.
Alistair Letcher, Jakob Foerster, David Balduzzi, Tim Rocktäschel, and Shimon Whiteson. Stable opponent
shaping in diﬀerentiable games. arXiv preprint arXiv:1811.08469 , 2018.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine
learning proceedings 1994 , pages 157–163. Elsevier, 1994.
George J Mailath, J George, Larry Samuelson, et al. Repeated games and reputations: long-run relationships .
Oxford university press, 2006.
John F Nash. The bargaining problem. Econometrica: Journal of the Econometric Society , pages 155–162,
1950.
9
Noam Nisan et al. Introduction to mechanism design (for computer scientists). Algorithmic game theory , 9:
209–242, 2007.
Alexander Peysakhovich and Adam Lerer. Consequentialist conditional cooperation in social dilemmas with
imperfect information. arXiv preprint arXiv:1710.06975 , 2017.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective
sequential decision-making. Journal of Artiﬁcial Intelligence Research , 48:67–113, 2013.
Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, game-theoretic, and logical
foundations . Cambridge University Press, 2008.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation
learning. In Advances in neural information processing systems , pages 7461–7472, 2018.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. arXiv
preprint arXiv:1509.06461 , 2015.
Weixun Wang, Jianye Hao, Yixi Wang, and Matthew Taylor. Towards cooperation in sequential prisoner’s
dilemmas: a deep multiagent reinforcement learning approach. arXiv preprint arXiv:1803.00162 , 2018.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learning. arXiv
preprint arXiv:1907.13220 , 2019.
Supplementary Materials
L-TFT vs. naive DDQN
Figure 3: Performance of L-TFT 0:95in the IPD (left) and Coin Game (right) against a DDQN learner trained
onRt
j(naive). For IPD, the probaxis is the probability of mutual cooperation ( cc) and mutual defection
(dd), For Coin Game, the pick\_own axis is the proportion of times the coin is picked by the agent of its
own color (i.e., the agents cooperate). The payoffs axis shows a moving average of player rewards after
each episode. These values are averaged over 10 replicates and bands are (min, max) values. Note that
temperature values are constant in the IPD after 3000 episodes and in the Coin Game after 30000 episodes.
10


Astronomical Waste: The Opportunity Cost of Delayed Technological Development: Nick Bostrom

1Astronomical Waste: The Opportunity Cost of Delayed
Technological Development
NICK BOST ROM

Oxford University
ABSTRACT. With very advanced technology, a very large population of people living
happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there
is therefore a corresponding opportunity cost: a potential good, lives worth living, is not
being realized. Given some plausible assumption s, this cost is extremely large. However,
the lesson for standard utilitarians is not that we ought to maximize the pace of
technological development, but rather that we ought to maximize its safety, i.e. the
probability that colonization will eventually occur. This goal has such high utility that standard utilitarians ought to focus all their efforts on it. Utilitarians of a ‘person-
affecting’ stripe should accept a modified version of this conclusion. Some mixed ethical views, which combine utilitarian considerat ions with other criteria, will also be
committed to a similar bottom line.
I.THE RATE OF LOSS OF POTENTIAL LIVES
As I wri te these words, suns are illuminating and heating empty rooms, unused energy is
being flushed down black holes, and our great common endowment of negentropy is Russian, Portuguese, Bosnian Translations:
2be
ing irreversibly degraded into entropy on a cosmic scale. These are resources that an
advanced civilization could have used to create value-structures, such as sentient beings
living worthwhile lives.
The rate of this loss boggles the mind. One recent paper speculates, using loose
theoretical considerations based on the rate of increase of entropy, that the loss of
potential human lives in our own galactic supercluster is at least ~1046 per century of
delayed colonization.1 This estimate assumes that all the lost entropy could have been
used for productive purposes, although no currently known technological mechanisms are even remotely capable of doing that. Since the estimate is meant to be a lower bound, this
radically unconservative assumption is undesirable.
We can, however, get a lower bound more straightforwardly by simply counting
the number or stars in our galactic supercluster and multiplying this number with the
amount of computing power that the resources of each star could be used to generate using technologies for whose feasibility a strong case has already been made. We can
then divide this total with the estimated amount of computing power needed to simulate one human life.
As a rough approximation, let us say the Virgo Supercluster contains 10
13 stars.
One estimate of the computing power extractable from a star and with an associated planet-sized computational structure, using advanced molecular nanotechnology
2, is 1042
operations per second.3 A typical estimate of the human brain’s processing power is
roughly 1017 operations per second or less.4 Not much more seems to be needed to
simulate the relevant parts of the environment in sufficient detail to enable the simulated
minds to have experiences indistinguishable from typical current human experiences.5
 3Given these estimates, it follows that the potential for approximately 1038 human lives is
lost every century that colonization of our local supercluster is delayed; or equivalently,
about 1029 potential human lives per second.
While this estimate is conservative in that it assumes only computational
mechanisms whose implementation has been at least outlined in the literature, it is useful
to have an even more conservative estima te that does not a ssume a non-biological
instantiation of the potential persons. Suppose that about 1010 biological humans could be
sustained around an average star. Then the Virgo Supercluster could contain 1023
biological humans. This corresponds to a loss of potential of over 1013 potential human
lives per second of delayed colonization.
What matters for present purposes is not the exact numbers but the fact that they
are huge. Even with the most conservative estimate, assuming a biological
implementation of all persons, the potential for over ten trillion potential human beings is
lost for every second of postponement of colonization of our supercluster.6

II. THE OPPORTUNITY COST OF DELAYED COLONIZATION
From a utilitarian perspective, this huge loss of potential human lives constitutes a
correspondingly huge loss of potential value. I am assuming here that the human lives that could have been created would have been worthwhile ones. Since it is commonly
supposed that even current human lives are typically worthwhile, this is a weak
assumption. Any civilization advanced enough to colonize the local supercluster would likely also have the ability to establish at least the minimally favorable conditions
required for future lives to be worth living.
4The effect on tot al value, then, seems greater fo r actions th at accelerate
technological d evelopm ent than f or practically a ny othe r possible acti on. Advancing
technology (o r its enabling factors, such as ec onomic productivity) e ven b y such a tiny
amount that it lead s to colonizatio n of the l ocal supercluster j ust one second earli er than
would otherwise have happ ened amoun ts to b ringing abou t more than 1029 human lives
(or 1013 human lives if we use the most conservativ e lower bound) that would not
otherwise have existed. Few oth er ph ilanthropic causes could hope to match that level of
utilitarian payoff.
Utilitarians are no t the only ones who shou ld strong ly oppose astronomical waste.
There are ma ny views about what h as value that would concur with th e assessment that
the current rat e of wastag e constitutes a n eno rmous lo ss of potential v alue. Fo r exa mple,
we can ta ke a thicke r concepti on of h uman we lfare than commonly supposed b y
utilitarians (whether of a he donistic, e xperientialist, o r desire-satisfactioni st bent), such as
a conception th at locate s valu e also in human flourishing, m eaningf ul relationshi ps, noble
character, in dividual expre ssion, aesthetic appreciation, and so forth. So long as th e
evaluation function is aggregativ e (doe s not count o ne perso n’s welfare for less just
becaus e there are many other persons in existence who also e njoy happy lives) and is not
relativized to a particular point in time (no t ime-disco unting ), the conclusion will ho ld.
These c onditions c an be r elaxed further. Even if the welfare function is not
perfectly a ggregativ e (perhaps b ecause o ne component of the good is diversity, th e
marginal rate of production o f which might decline with in creasing population size), it
can sti ll yield a similar bottom line pro vided only that at least some si gnificant
 5component of the good is sufficiently aggregative. Similarly, some degree of time-
discounting future goods could be accommodated without changing the conclusion.7

III. THE CHIEF GOAL FOR UTILITARIANS SHOULD BE TO REDUCE
EXISTENTIAL RISK
In light of the above discussion, it may seem as if a utilitarian ought to focus her efforts
on accelerating technological development. The payoff from even a very slight success in this endeavor is so enormous that it dwarfs that of almost any other activity. We appear to
have a utilitarian argument for the greatest possible urgency of technological
development.
However, the true lesson is a different one. If what we are concerned with is
(something like) maximizing the expected number of worthwhile lives that we will
create, then in addition to the opportunity cost of delayed co lonization, we have to take
into account the risk of failure to colonize at all. We might fall victim to an existential
risk, one where an adverse outcome would either annihilate Earth-originating intelligent
life or permanently and drastically curtail its potential.
8 Because the lifespan of galaxies
is measured in billions of years, whereas th e time-scale of any delays that we could
realistically affect would rather be measured in years or decades, the consideration of risk trumps the consideration of opportunity cost. For example, a single percentage point of
reduction of existential risks would be worth (from a utilitarian expected utility point-of-
view) a delay of over 10 million years.
Therefore, if our actions have even the slightest effect on the probability of
eventual colonization, this will outweigh their effect on when colonization takes place.
 6For standard utilitarians, priority number one, two, three and four should consequently be
to reduce existential risk. The utilitarian imperative ‘Maximize expected aggregate
utility!’ can be simplified to the maxim ‘Minimize existential risk!’.

IV. IMPLICATIONS FOR AGGREGATIVE PERSON-AFFECTING VIEWS
The argument above presupposes that our concern is to maximize the total amount of
well-being. Suppose instead that we adopt a ‘person-affecting’ version of utilitarianism,
according to which our obligations are primar ily towards currently existing persons and
to those persons that will come to exist.9 On such a person-affecting view, human
extinction would be bad only because it makes past or ongoing lives worse, not because it constitutes a loss of potential worthwhile lives. What ought someone who embraces this
doctrine do? Should he emphasize speed or safety, or something else?
 To answer this, we need to consider some further matters. Suppose one thinks that
the probability is negligible that any existing person will survive long enough to get to use a significant portion of the accessible astr onomical resources, which, as described in
opening section of this paper, are gradually going to waste. Then one’s reason for
minimizing existential risk is that sudden ex tinction would off cut an average of, say, 40
years from each of the current (six billion or so) human lives.
10 While this would
certainly be a large disaster, it is in the same big ballpark as other ongoing human
tragedies, such as world poverty, hunger and disease. On this assumption, then, a person-
affecting utilitarian should regard reducing existential risk as a very important but not completely dominating concern. There would in this case be no easy answer to what he
ought to do. Where he ought to focus his efforts would depend on detailed calculations
 7about which area of philanthropic activity he would happen to be best placed to make a
contribution to.
 Arguably, however, we ought to assign a non-negligible probability to some
current people surviving long enough to reap the benefits of a cosmic diaspora. A so-called technological ‘singularity’ mi ght occur in our natural lifetime
11, or there could be a
breakthrough in life-extension, brought about, perhaps, as result of machine-phase
nanotechnology that would give us unprecedented control over the biochemical processes in our bodies and enable us to halt and reverse the aging process.
12 Many leading
technologists and futurist thinkers give a fairly high probability to these developments
happening within the next several decades.13 Even if you are skeptical about their
prognostications, you should consider the poor track record of technological forecasting.
In view of the well-establis hed unreliability of many such forecasts, it would seem
unwarranted to be so confident in one’s prediction that the requisite breakthroughs will not occur in our time as to give the hypothes is that they will a probability of less than,
say, 1%. The expected utility of a 1% chance of realizing an astronomically large good
could still be astronomical. But just how good would it be for (some substantial subset of)
currently living people to get access to astronomical amounts of resources? The answer is not obvious. On the one hand, one might reflect that in today’s world, the marginal utility
for an individual of material resources declines quite rapidly ones his basic needs have
been met. Bill Gate’s level of well-being doe s not seem to dramatically exceed that of
many a person of much more modest means. On the other hand, advanced technologies of
the sorts that would most likely be deployed by the time we could colonize the local
8supercluster may well provide new ways of converting resources into well-being . In
particular, material re sources could be us ed to gre atly expand our mental c apacities a nd
to indefinitely prolong our subjecti ve lifes pan. And it is b y no m eans clea r that the
marginal ut ility of ex tended h ealthspan and in creased mental powers mu st be sharply
declin ing above some l evel. If there is no such decline in marginal utilit y, we have to
conclude that the ex pected utility to current in dividuals14 of su ccessful colonization of
our supercluster is astronomically g reat, an d this conclusi on holds even if on e gives a
fairly low probability to that outcome. A long shot i t may be, bu t for an expected utility
maximize r, the benefit of livi ng for perhap s billi ons of s ubjective yea rs with greatly
expanded ca pacities under fantasticall y favo rable condi tions could mo re than ma ke up for
the remote pros pects of success.
Now, if th ese assumptions a re made, wh at follow s about ho w a person-affecting
utilitarian should ac t? Clearly, a voiding e xistential calami ties is im porta nt, not just
because it wo uld truncate the natural lifespan of six billion or so people, b ut also – and
given th e assump tions this is an even weight ier consideration – b ecause it wo uld
extin guish th e chance that current peo ple have of r eaping th e enormous benefits of
eventual colo nization. However, by co ntrast to the total u tilitarian , the perso n-affecting
utilitaria n woul d have to balanc e this g oal w ith another equally important des ideratum,
namely that of m aximizing t he chances o f curren t peop le surviving to benefit from the
colonizatio n. For the person-affecting u tilitar ian, it is not en ough that humankind
survives to colonize; it is crucial that extant people be saved. This should lead her to
emphasize speed of technological d evelopment, since the rap id arrival of advanced
technology wou ld surely b e needed to help cu rrent peop le stay alive until the fru its of
 9colonization could be harvested. If the goal of speed conflicts with the goal of global
safety, the total utilitarian should always opt to maximize safety, but the person-affecting
utilitarian would have to balance the risk of people dying of old age with the risk of them
succumbing in a species-destroying catastrophe. Mixed ethical views, which also incorporate non-utilitarian elements, might or might not yield one of these bottom lines
depending on the nature of what is added.
15

1 M. Cirkovic, ‘Cosmological Forecast and its Practical Significance’, Journal of Evolution and
Technology , xii (2002), http://www.jetpress.org/volume12/CosmologicalForecast.pdf .
2 K. E. Drexler, Nanosystems: Molecular Machinery, Manufacturing, and Computation , New York, John
Wiley & Sons, Inc., 1992.
3 R. J. Bradbury, ‘Matrioshka Brains’, Manuscript , 2002,
http://www.aeiveos.com/~bradbury/MatrioshkaBrains/MatrioshkaBrains.html
4 N. Bostrom, ‘How Long Before Superintelligence?’, International Journal of Futures Studies ii (1998);
R. Kurzweil, The Age of Spiritual Machines: When Computers Exceed Human Intelligence , New York,
Viking, 1999. The lower estimate is in H. Moravec, Robot: Mere Machine to Transcendent Mind , Oxford,
1999.
5 N. Bostrom, ‘Are You Living in a Simulation?’, Philosophical Quarterly , liii (211). See also
http://www.simulation-argument.com .
6 The Virgo Supercluster contains only a small part of the colonizable resources in the universe, but it is
sufficiently big to make the point. The bigger the region we consider, the less certain we can be that
significant parts of it will not have been colonized by a civilization of non-terrestrial origin by the time we
could get there.
 10
7 Utilitarians commonly regard time-discounting as inappropriate in evaluating moral goods (see e.g. R. B.
Brandt, Morality, Utilitarianism, and Rights , Cambridge, 1992, pp. 23f.). However, it is not clear that
utilitarians can avoid compromising on this principle in view of the possibility that our actions could conceivably have consequences for an infinite number of persons (a possibility that we set aside for the
purposes of this paper).
8 N. Bostrom, ‘Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards’, Journal of
Evolution and Technology , ix (2002), http://www.jetpress.org/volume9/risks.html .
9 This formulation of the position is not necessarily the best possible one, but it is simple and will serve for
the purposes of this paper.
10 Or whatever the population is likely to be at the time when doomsday would occur.
11 See e.g. V. Vinge, ‘The Coming Technological Singularity’, Whole Earth Review , Winter issue (1993).
12 R. A. Freitas Jr., Nanomedicine , Vol. 1, Georgetown, Landes Bioscience, 1999.
13 E.g. Moravec, Kurzweil, and Vinge op. cit.; E. Drexler, Engines of Creation , New York, Anchor Books,
1986.
14 On the person-affecting view, the relevant factor is not the temporal location of a person per se . In
principle, effects on the well-being of a past or a future person could also be an appropriate target for moral
concern. In practice, however, the effect on the well-being of past persons is likely to be relatively small or
even zero (depending on which conception of well-being one adopts), and the effect (of any action that has
a significant effect on the future) on not-yet-existing persons is very likely to be such that competing alternative actions would lead to separate sets of possible persons coming into existence, for example as a result of different sperms fertilizing different eggs, and thus there would typically be no future persons
whose level of well-being is affected by our current actions. Our actions might affect which future persons
there will be, but on the person-affecting view, such results provide no general moral reasons for action.
One exception are fetuses and fertilize eggs, which are not yet persons, but which may become persons in
the future, and whose lives our current actions may influence.
15 I’m grateful for comments from John Broome, Milan Cirkovic, Roger Crisp, Robin Hanson, and James
Lenman, and for the financial support of a British Academy Postdoctoral Award.