Risks of Astronomical Future Suﬀering

Risks of Astronomical Future Suﬀering
Brian Tomasik
Center on Long-Term Risk
brian.tomasik@longtermrisk.org
Abstract
It’s far from clear that human values will shape an Earth-based space-colonization wave, but even if
they do, it seems more likely that space colonization will increase total suﬀering rather than decrease
it. That said, other people care a lot about humanity’s survival and spread into the cosmos, so I think
suﬀering reducers should let others pursue their spacefaring dreams in exchange for stronger safety
measures against future suﬀering. In general, I encourage people to focus on making an intergalactic
future more humane ifit happens rather than making sure there will be an intergalactic future.
Contents
1 Epigraphs 2
2 Humans values may not control the future 2
3 Some scenarios for future suﬀering 2
3.1 Spread of wild animals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3.2 Sentient simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3.3 Suﬀering subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3.4 Black Swans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4 Even a human-controlled future is likely to increase suﬀering 3
4.1 Spread of wild animals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4.2 Sentient simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.3 Suﬀering subroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.4 Black Swans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5 What about paperclippers? 4
6 What if human colonization is more humane than ET colonization? 5
7 Why we should remain cooperative 6
8 Possible upsides to an intelligent future 7
8.1 Black swans that don’t cut both ways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
8.2 Valuing reﬂection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
Acknowledgments 8
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
1 Epigraphs
If we carry the green ﬁre-brand from star to
star, and ignite around each a conﬂagration of
vitality, we can trigger a Universal metamorpho-
sis. [...] Because of us [...] Slag will become soil,
grass will sprout, ﬂowers will bloom, and forests
will spring up in once sterile places.1[...] If we
deny our awesome challenge; turn our backs
on the living universe, and forsake our cosmic
destiny, we will commit a crime of unutterable
magnitude.
– Marshall T. Savage, The Millennial Project:
Colonizing the Galaxy in Eight Easy Steps , 1994
Let’s pray that the human race never escapes
from Earth to spread its iniquity elsewhere.
– C.S. Lewis
If you can’t beat ’em, join ’em.
– proverb
2 Humans values may not control the
future
Nick Bostrom’s "The Future of Human Evolution"
(Bostrom, 2004) describes a scenario in which hu-
man values of fun, leisure, and relationships may be
replaced by hyper-optimized agents that can better
compete in the Darwinian race to control our future
light cone. The only way we could avert this com-
petitive scenario, Bostrom suggests, would be via
a "singleton" (Bostrom, 2006), a uniﬁed agent or
governing structure that could control evolution. Of
course, even a singleton may not carry on human
values. Many naive AI agents that humans might
build may optimize an objective function that hu-
mans ﬁnd pointless. Or even if humans do maintain
hands on the steering wheel, it’s far from guaranteed
that we can preserve our goals in a stable way across
major self-modiﬁcations going forward.
These factors suggest that even conditional on
human technological progress continuing, the proba-
bility that human values are realized in the future
may not be very large. Carrying out human values
seems to require a singleton that’s not a blind opti-
mizer, that can stably preserve values, and that is
shaped by designers who care about human values
rather than selﬁsh gain or something else. This isimportant to keep in mind when we imagine what
future humans might be able to bring about with
their technology.
Some people believe that suﬃciently advanced su-
perintelligences will discover the moral truth and
hence necessarily do the right things. Thus, it’s
claimed, as long as humanity survives and grows
more intelligent, the right things will eventually hap-
pen. There are two problems with this view. First,
Occam’s razor militates against the existence of a
moral truth (whatever that’s supposed to mean).
Second, even if such moral truth existed, why should
a superintelligence care about it? There are plenty
of brilliant people on Earth today who eat meat.
They know perfectly well the suﬀering that it causes,
but their motivational systems aren’t suﬃciently en-
gaged by the harm they’re doing to farm animals.
The same can be true for superintelligences. In-
deed, arbitrary intelligences in mind-space needn’t
have even the slightest inklings of empathy for the
suﬀering that sentients experience.
3 Some scenarios for future suﬀering
Even if humans do preserve control over the future
of Earth-based life, there are still many ways in
which space colonization would multiply suﬀering.
Following are some of them.
3.1 Spread of wild animals
Humans may colonize other planets, spreading
suﬀering-ﬁlled animal life via terraforming. Some
humans may use their resources to seed life through-
out the galaxy, which some sadly consider a moral
imperative.
3.2 Sentient simulations
Given astronomical (Bostrom, 2003) computing
power, post-humans may run various kinds of sim-
ulations. These sims may include many copies of
wild-animal life, most of which dies painfully shortly
after being born. For example, a superintelligence
aiming to explore the distribution of extraterres-
trials of diﬀerent sorts might run vast numbers of
simulations (Thiel, Bergmann and Grey, 2003) of
evolution on various kinds of planets. Moreover,
scientists might run even larger numbers of simula-
tions of organisms-that-might-have-been, exploring
1Because nature contains such vast amounts of suﬀering, I would strongly dislike such a project. I include this quotation for
rhetorical eﬀect and to give a sense of how others see the situation.
2
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
the space of minds. They may simulate decillions of
reinforcement learners that are suﬃciently self-aware
as to feel what we consider conscious pain.
3.3 Suﬀering subroutines
It could be that certain algorithms (say, reinforce-
ment agents (Tomasik, 2014)) are very useful in
performing complex machine-learning computations
that need to be run at massive scale by advanced
AI. These subroutines might be suﬃciently similar
to the pain programs in our own brains that we con-
sider them to actually suﬀer. But proﬁt and power
may take precedence over pity, so these subroutines
may be used widely throughout the AI’s Matrioshka
brains.
3.4 Black Swans
The range of scenarios that we can imagine is limited,
and many more possibilities may emerge that we
haven’t thought of or maybe can’t even comprehend.
4 Even a human-controlled future is
likely to increase suﬀering
If I had to make an estimate now, I would give
~70% probability that if humans choose to colonize
space, this will cause more suﬀering than it reduces
on intrinsic grounds (ignoring compromise consid-
erations discussed later). Think about how space
colonization could plausibly reduce suﬀering. For
most of those mechanisms, there seem to be counter-
mechanisms that will increase suﬀering at least as
much. The following sections parallel those above.
4.1 Spread of wild animals
David Pearce coined the phrase "cosmic rescue mis-
sions" (Pearce, n.d.) in referring to the possibility of
sending probes to other planets to alleviate the wild
extraterrestrial (ET) suﬀering they contain. This is
a nice idea, but there are a few problems.
We haven’t found any ETs yet, so it’s not obvi-
ous there are vast numbers of them waiting to
be saved from Darwinian misery.
The speciﬁc kind of conscious suﬀering known
to Earth-bound animal life would not neces-
sarily be found among the ETs. Most likely
ETs would be bacteria, plants, etc., and even if
they’re intelligent, they might be intelligent in
the way robots are without having emotions ofthe sort that we care very much about. (How-
ever, if they were very sophisticated, it would be
relatively unlikely that we would not consider
them conscious.)
It’s unclear whether humanity would support
such missions. Environmentalists would ask us
to leave ET habitats alone. Others wouldn’t
want to spend the energy on rescue missions un-
less they planned to mine resources from those
planets.
Contrast this with the possibilities for spreading
wild-animal suﬀering:
Humans may spread life to many planets (e.g.,
Mars via terraforming, other Earth-like planets
via directed panspermia). The number of plan-
ets that can support life may be appreciably
bigger than the number that already have it.
(See the discussion of flin the Drake equation.)
Moreover, the percentage of planets that can be
converted into computers that could simulate
wild-animal suﬀering might be close to 100%.
We already know that Earth-based life is sen-
tient, unlike for ETs.
Spreading biological life is slow and diﬃcult,
but disbursing small life-producing capsules is
easier than dispatching Hedonistic Imperative
probes or berserker probes.
Fortunately, humans might not support spread of
life that much, though some do. For terraforming,
there are survival pressures to do it in the near term,
but probably directed panspermia is a bigger prob-
lem in the long term. Also, given that terraforming
is estimated to require at least thousands of years,
while human-level digital intelligence should take at
most a few hundred years to develop, terraforming
may be a moot point from the perspective of catas-
trophic risks, since digital intelligence doesn’t need
terraformed planets.
While I noted that ETs are not guaranteed to
be sentient, I do think it’s moderately likely that
consciousness is fairly convergent among intelligent
civilizations. This is based on (a) suggestions of
convergent consciousness among animals on Earth
and (b) the general principle that consciousness
seems to be useful for planning, manipulating im-
ages, self-modeling, etc. On the other hand, maybe
this reﬂects the paucity of my human imagination
in conceiving of ways to be intelligent without con-
sciousness.
3
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
4.2 Sentient simulations
It may be that biological suﬀering is a drop in the
bucket compared with digital suﬀering. The bio-
sphere of a planet is less than Type I on the Kar-
dashev scale; it uses a tiny sliver of all the energy
of its star. Intelligent computations by a Type II
civilization can be many orders of magnitude higher.
So humans’ sims could be even more troubling than
their spreading of wild animals.
Of course, maybe there are ETs running sims of
nature for science or amusement, or of minds in gen-
eral to study biology, psychology, and sociology. If
we encountered these ETs, maybe we could persuade
them to be more humane.
Ithinkit’slikelythathumansaremoreempathetic
than the average civilization because
1.we seem much more empathetic than the aver-
age animal on Earth, probably in part due to
parental impulses and in part due to trade, al-
though presumably some of these factors would
necessarily be true of any technologically ad-
vanced civilization
2.selection bias implies that we’ll agree with our
own society’s morals more than those of a ran-
dom other society because these are the values
that we were raised with and that our biology
impels us toward.
Based on these considerations, it seems plausible
that there would be room for improvement through
interaction with ETs. Indeed, we should in general
expect it to be possible for any two civilizations or
factions to achieve gains from compromise if they
have diminishing marginal utility with respect to
amount of control exerted. In addition, there may be
cheap Pareto improvements to be had purely from
increased intelligence and better understanding of
important considerations.
That said, there are some downside risks. Post-
humans themselves might create suﬀering simula-
tions, and what’s worse, the sims that post-humans
run would be more likely to be sentient than those
runbyrandomETsbecausepost-humanswouldhave
a tendency to simulate things closer to themselves
in mind-space. They might run nature sims for aes-
thetic appreciation, lab sims for science experiments,
or pet sims for pets.4.3 Suﬀering subroutines
Suﬀering subroutines may be a convergent outcome
of any AI, whether human-inspired or not. They
might also be run by aliens, and maybe humans
could ask aliens to design them in more humane
ways, but this seems speculative.
4.4 Black Swans
It seems plausible that suﬀering in the future will be
dominated by something totally unexpected. This
could be a new discovery in physics, neuroscience,
or even philosophy more generally. Some make the
argument that because we know so very little now,
it’s better for humans to stick around because of
the "option value": If they later realize it’s bad to
spread, they can stop, but if they realize they should
spread, they can proceed to reduce suﬀering in some
novel way that we haven’t anticipated.
Of course, the problem with the "option value"
argument is that it assumes future humans do the
right things, when in fact, based on examples of
speculations we can imagine now, it seems future
humans would probably do the wrong things much
of the time. For instance, faced with a new discovery
of obscene amounts of computing power somewhere,
most humans would use it to run oodles more minds,
some nontrivial fraction of which might suﬀer terri-
bly. In general, most sources of immense power are
double-edged swords that can create more happiness
and more suﬀering, and the typical human impulse
to promote life/consciousness rather than to remove
them suggests that negative and negative-leaning
utilitarians are on the losing side.
Still, waiting and learning more is plausibly
Kaldor-Hicks eﬃcient, and maybe there are ways it
can be made Pareto eﬃcient by granting additional
concessions to suﬀering reducers as compensation.
5 What about paperclippers?
Above I was largely assuming a human-oriented civ-
ilization with values that we recognize. But what
if, as seems mildly likely, Earth is taken over by a
paperclip maximizer, i.e., an unconstrained automa-
tion or optimization process? Wouldn’t that reduce
suﬀering because it would eliminate wild ETs as the
paperclipper spread throughout the galaxy, without
causing any additional suﬀering?
Maybe, but if the paperclip maximizer is actually
generally intelligent, then it won’t stop at tiling the
4
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
solar system with paperclips. It will want to do sci-
ence, perform lab experiments on sentient creatures,
possibly run suﬀering subroutines, and so forth. It
will require lots of intelligent and potentially sen-
tient robots to coordinate and maintain its paperclip
factories, energy harvesters, and mining operations,
as well as scientists and engineers to design them.
And the paperclipping scenario would entail similar
black swans as a human-inspired AI. Paperclippers
would presumably be less intrinsically humane than
a "friendly AI", so some might cause signiﬁcantly
more suﬀering than a friendly AI, though others
might cause less, especially the "minimizing" paper-
clippers, e.g., cancer minimizers or death minimizers.
If the paperclipper is not generally intelligent, I
have a hard time seeing how it could cause human
extinction. In this case it would be like many other
catastrophic risks – deadly and destabilizing, but
not capable of wiping out the human race.
6 What if human colonization is more
humane than ET colonization?
Ifwe knew for certain that ETs would colonize our
region of the universe if Earth-originating intelli-
gence did not, then the question of whether humans
should try to colonize space becomes less obvious.
As noted above, it’s plausible that humans are more
compassionate than a random ET civilization would
be. On the other hand, human-inspired computa-
tions might also entail more of what we consider to
count as suﬀering because the mind architectures of
the agents involved would be more familiar. And
having more agents in competition for the light cone
might lead to dangerous outcomes.
But for the sake of argument, suppose an Earth-
originating colonization wave would be better than
the expected colonization wave of an ET civilization
that would colonize later if we didn’t do so. In partic-
ular, suppose that if human values colonized space,
they would cause only 0:5units of suﬀering, com-
pared with 1units if random ETs colonized space.
Then it would seem that as long as the probability
Pof some other ETs coming later is bigger than 0:5,
then it’s better for humans to colonize and pre-empt
the ETs from colonizing, since 0:5>1Pfor
P > 0:5.
However, this analysis forgets that even if Earth-
originating intelligence does colonize space, it’s not
at all guaranteed that human values will controlhow that colonization proceeds. Evolutionary forces
mightdistortcompassionatehumanvaluesintosome-
thing unrecognizable. Alternatively, a rogue AI
might replace humans and optimize for arbitrary val-
ues throughout the cosmos. In these cases, humans’
greater-than-average compassion doesn’t make much
diﬀerence, so suppose that the value of these colo-
nization waves would be 1, just like for colonization
by random ETs. Let the probability be Qthat these
non-compassionate forces win control of Earth’s col-
onization. Now the expected values are
31Q+0:5(1Q)
for Earth-originating colonization versus
1P
if Earth doesn’t colonize and leaves open the possi-
bility of later ET colonization.
For concreteness, say that Q= 0:5. (That seems
plausiblytoolowtome, givenhowmanytimesEarth
has seen overhauls of hegemons in the past.) Then
Earth-originating colonization is better if and only
if
10:5 +0:50:5>1P
0:75>1P
P > 0:75:
Given uncertainty about the Fermi paradox and
Great Filter, it seems hard to maintain a probability
greater than 75% that our future light cone would
contain colonizing ETs if we don’t ourselves colonize,
although this section presents an interesting argu-
ment for thinking that the probability of future ETs
is quite high.
What if rogue AIs result in a diﬀerent magnitude
of disvalue from arbitrary ETs? Let Hbe the ex-
pected harm of colonization by a rogue AI. Assume
ETs are as likely to develop rogue AIs as humans
are. Then the disvalue of Earth-based colonization
is
HQ+ (0:5)(1Q);
and the harm of ET colonization is
P(HQ+ (1)(1Q)):
Again taking Q= 0:5, then Earth-based colonization
has better expected value if
H0:5 +0:50:5> P(H0:5 +10:5)
H0:5> P(H1)
P >(H0:5)
(H1);
5
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
Figure 1: Plot of threshold values for Pas a function of H
where the inequality ﬂips around when we divide by
the negative number (H1).Figure 1 represents
a plot of these threshold values for Pas a function
ofH.
Even if H= 0and a rogue AI caused no suﬀering,
it would still only be better for Earth-originating
intelligence to colonize if P > 0:5, i.e., if the prob-
ability of ETs colonizing in its place was at least
50%.
These calculations involve many assumptions, and
it could turn out that Earth-based colonization has
higher expected value given certain parameter val-
ues. This is a main reason I maintain uncertainty
as to the sign of Earth-based space colonization.
However, this whole section was premised on human-
inspired colonization being better than ET-inspired
colonization, and the reverse might also be true,
since computations of the future are more likely to
be closer to what we most value and disvalue if
humans do the colonizing.
7 Why we should remain cooperative
If technological development and space colonization
seem poised to cause astronomical amounts of suf-
fering, shouldn’t we do our best to stop them? Well,
it is worth having a discussion about the extent to
which we as a society want these outcomes, but my
guess is that someone will continue them, and this
would be hard to curtail without extreme measures.Eventually, those who go on developing the tech-
nologies will hold most of the world’s power. These
people will, if only by selection eﬀect, have strong
desires to develop AI and colonize space.
Resistance might not be completely futile. There’s
some small chance that suﬀering reducers could in-
ﬂuence society in such a way as to prevent space
colonization. But it would be better for suﬀering
reducers, rather than ﬁghting technologists, to com-
promise with them: We’ll let you spread into the
cosmos if you give more weight to our concerns about
future suﬀering. Rather than oﬀering a very tiny
chance of complete victory for suﬀering reducers,
this cooperation approach oﬀers a higher chance of
getting an appreciable fraction of the total suﬀering
reduction that we want. In addition, compromise
means that suﬀering reducers can also win in the
scenario ( 30% likely in my view) that technologi-
cal development does prevent more suﬀering than it
causes even apart from considerations of strategic
compromise with other people.
Ideally these compromises would take the form
of robust bargaining arrangements. Some exam-
ples are possible even in the short term, such as if
suﬀering reducers and space-colonization advocates
agree to cancel opposing funding in support of some
commonly agreed-upon project instead.
The strategic question of where to invest resources
to advance your values at any given time amounts to
a prisoner’s dilemma with other value systems, and
6
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
because we repeatedly make choices about where
to invest, what stances to adopt, and what policies
to push for, these prisoner’s dilemmas are iterated.
In Robert Axelrod’s tournaments on the iterated
prisoner’s dilemma, the best-performing strategies
were always "nice," i.e., not the ﬁrst to defect. Thus,
suﬀering reducers should not be the ﬁrst to defect
against space colonizers. Of course, if it seems that
space colonizers show no movement toward suﬀering
reduction, then we should also be "provocable" to
temporary defection until the other side does begin
to recognize our concerns.
We who are nervous about space colonization
stand a lot to gain from allying with its supporters –
in terms of thinking about what scenarios might hap-
pen and how to shape the future in better directions.
We also want to remain friends because this means
pro-colonization people will take our ideas more seri-
ously. Even if space colonization happens, there will
remain many sub-questions on which suﬀering reduc-
ers want to have a say: e.g., not spreading wildlife,
not creating suﬀering simulations/subroutines, etc.
We want to make sure suﬀering reducers don’t
become a despised group. For example, think about
how eugenics is more taboo because of the Nazi
atrocities than it would have been otherwise. Anti-
technology people are sometimes smeared by associa-
tion with the Unabomber. Animal supporters can be
tarnished by the violent tactics of a few, or even by
the antics of PETA. We need to be cautious about
something similar happening for suﬀering reduction.
Most people already care a lot about preventing
suﬀering, and we don’t want people to start saying,
"Oh, you care about preventing harm to powerless
creatures? What are you, one of those suﬀering
reducers?" where "suﬀering reducers" has become
such a bad name that it evokes automatic hatred.
So not only is cooperation with colonization sup-
porters the more promising option, but it’s arguably
the only net-positive option for us. Taking a more
confrontational stance risks hardening the opposi-
tion and turning people away from our message.
Remember, preventing future suﬀering is something
that everyone cares about, and we shouldn’t erode
that fact by being excessively antagonistic.8 Possible upsides to an intelligent
future
8.1 Black swans that don’t cut both
ways
Many speculative scenarios that would allow for
vastly reducing suﬀering in the multiverse would
also allow for vastly increasing it: When you can
decrease the number of organisms that exist, you
can also increase the number, and those who favor
creating more happiness / life / complexity / etc.
will tend to want to push for the increasing side.
However, there may be some black swans that re-
ally are one-sided, in the sense that more knowledge
is most likely to result in a decrease of suﬀering. For
example: We might discover that certain routine
physical operations map onto our conceptions of
suﬀering. People might be able to develop ways to
re-engineer those physical processes to reduce the
suﬀering they contain. If this could be done without
a big sacriﬁce to happiness or other values, most peo-
ple would be on board, assuming that present-day
values have some share of representation in future
decisions.
This may be a fairly big deal. I give nontrivial
probability (maybe ~10%?) that I would, upon suf-
ﬁcient reﬂection, adopt a highly inclusive view of
what counts as suﬀering, such that I would feel that
signiﬁcant portions of the whole multiverse contain
suﬀering-dense physical processes. After all, the
mechanics of suﬀering can be seen as really simple
when you think about them a certain way, and as
best I can tell, what makes animal suﬀering special
are the bells and whistles that animal sentience in-
volves over and above crude physics – things like
complex learning, thinking, memory, etc. But why
can’t other physical objects in the multiverse be
the bells and whistles that attend suﬀering by other
physical processes? This is all very speculative, but
what understandings of the multiverse our descen-
dants would arrive at we can only begin to imagine
right now.
8.2 Valuing reﬂection
If we care to some extent about moral reﬂection on
our own values, rather than assuming that suﬀering
reduction of a particular ﬂavor is undoubtedly the
best way to go, then we have more reason to sup-
2Of course, what’s compelling to idealized-me would not necessarily be compelling to idealized-you. Value divergences may
7
Risks of Astronomical Future Suﬀering – Center on Long-Term Risk
port a technologically advanced future, at least if
it’s reﬂective.
In an idealized scenario like coherent extrapolated
volition (CEV) (Yudkowsky, 2004), say, if suﬀering
reduction was the most compelling moral view, oth-
ers would see this fact.2Indeed, all the arguments
any moral philosopher has made would be put on
the table for consideration (plus many more that no
philosopher has yet made), and people would have
a chance to even experience extreme suﬀering, in
a controlled way, in order to assess how bad it is
compared with other things. Perhaps there would
be analytic approaches for predicting what people
would say about how bad torture was without ac-
tually torturing them to ﬁnd out. And of course,
we could read through humanity’s historical record
and all the writings on the Internet to learn more
about what actual people have said about torture,
although we’d need to correct for will-to-live bias
and deﬁcits of accuracy when remembering emotions
in hindsight. But, importantly, in a CEV scenario,
all of those qualiﬁcations can be taken into account
by people much smarter than ourselves.
Of course, this rosy picture is nota likely future
outcome. Historically, forces seize control because
they best exert their power. It’s quite plausible that
someone will take over the future by disregarding
the wishes of everyone else, rather than by com-
bining and idealizing them. Or maybe concern for
the powerless will just fall by the wayside, because
it’s not really adaptive for powerful agents to care
about weak ones, unless there are strong, stable so-
cial pressures to do so. This suggests that improving
prospects for a reﬂective, tolerant future may be an
important undertaking. Rather than focusing on
whether or not the future happens, I think it’s more
valuable for suﬀering reducers to focus on making
the future better ifit happens – by encouraging com-
promise, moral reﬂectiveness, philosophical wisdom,
and altruism, all of which make everyone better oﬀ
in expectation.
Acknowledgments
A question by Rob Wiblin ﬁrst inspired this piece.
The discussion of cooperation with other value sys-tems was encouraged by Carl Shulman. Initially
I resisted his claim, but – as has often proved the
case at least on factual and strategic questions – I
eventually realized he was right and came around to
his view.
References
Bostrom, Nick. "The Future of Human Evolution."
Death and Anti-Death: Two Hundred Years
After Kant, Fifty Years After Turing. Palo Alto,
CA: Ria UP, 2004. 339-71. Web. 3 Mar. 2016
www.nickbostrom.com/fut/evolution.html.
Bostrom, Nick. "What Is a Singleton?" Linguistic
and Philosophical Investigations 5.2 (2006):
48-54. Web. 3 Mar. 2016.
www.nickbostrom.com/fut/singleton. html
Bostrom, Nick. "Astronomical Waste: The
Opportunity Cost of Delayed Technological
Development." Utilitas15.3 (2003): 308-3014.
Web. 3 Mar. 2016.
www.nickbostrom.com/astronomical/waste.
html.
Pearce, David. "Extraterrestrial Life, the Fermi
Paradox and the Hedonistic Imperative." The
Hedonistic Imperative . N.p., n.d. Web. 3 Mar.
2016. www.hedweb.com/object32.htm
Thiel, Inari, Neil W. Bergmann, and William Grey.
"A Case for Investigating the Ethics of Artiﬁcial
Life?"The University of New South Wales 1
(2003): 276-87. Web. 3 Mar. 2016.
http://espace.library.uq.edu.au/view
/UQ:10754/A\_Case\_for\_Inves.pdf
Tomasik, Brian. "Do Artiﬁcial
Reinforcement-Learning Agents Matter
Morally?" ArXiv.org . N.p., 30 Oct. 2014. Web.
03 Mar. 2016. http://arxiv.org/ abs/1410
.8233v1.
Yudkowsky, Eliezer. Coherent Extrapolated Volition .
San Francisco, CA: The Singularity Institute,
2004. Web. 3 Mar. 2016.
http://intelligence.org/ﬁles/CEV.pdf.
remain after extrapolation. That said, there’s no alternative better than compromising using a CEV-like approach, because if I
try to defect and push my particular values, you’ll just try to push yours, and we’ll both be worse oﬀ in expectation.
8


Multi-agent learning in mixed-motive coordination problems
Julian Stastny Johannes Treutlein Maxime Riché Jesse Clifton
Abstract
Cooperation in settings where agents have diﬀerent but overlapping preferences ( mixed-motive set-
tings) has recently received considerable attention in multi-agent learning. However, the mixed-motive
environments typically studied are simplistic in that they have a single cooperative outcome on which all
agents can agree. Multi-agent systems in general may exhibit many payoﬀ proﬁles which might be called
cooperative, but which agents have diﬀerent preferences over. This causes problems for independently
trained agents that do not arise in the case of that there is a unique cooperative payoﬀ proﬁle. In this
note, we illustrate this problem with a class of games called mixed-motive coordination problems (MCPs) .
We demonstrate the failure of several methods for achieving cooperation in sequential social dilemmas
when used to independently train policies in a simple MCP. We discuss some possible directions for
ameliorating MCPs.
1 Introduction
InAn Essay on Bargaining [1956], Thomas Schelling describes a negotiation over the price of a house.
Considering diﬀerent strategies the negotiators might use, he points out that
If the buyer is an agent authorized by a board of directors to buy at $16000 but not a cent more,
and the directors cannot constitutionally meet again for several months and the buyer cannot
exceed his authority, and if all this can be made known to the seller, then the buyer ‘wins’ — if,
again, the seller has not tied himself up with a commitment to $19000. (pg. 284)
Imagine various actors with diﬀerent preferences — a mixed-motive setting [Schelling, 1958] — developing
machine learning systems to act on their behalf. These actors are like Schelling’s bargainers, in that their
systems implicitly encode a commitment to pursuing a particular tradeoﬀ between their own preferences and
those of other players. Just as the price negotiation in Schelling’s example may fail if buyer and seller are
each committed to insisting on incompatible prices, machine learning systems may fail to cooperate if they
encode commitments to incompatible tradeoﬀs between their principals’ goals.
Multi-agent learning (MAL) has in recent years paid considerable attention to problems of cooperation
between agents with diﬀerent, but not zero-sum, preferences. However, much of this work has been in the
setting of (sequential) social dilemmas (SSDs) (e.g., Peysakhovich and Lerer 2017a, Lerer and Peysakhovich
2017, Peysakhovich and Lerer 2017b, Foerster et al. 2018, Wang et al. 2018, Eccles et al. 2019). The classic
example of a social dilemma is the Prisoner’s Dilemma (PD), and the SSDs studied in this literature are
similar to the Prisoner’s Dilemma in that there is a single salient notion of “cooperation”. This means that
it is relatively easy for actors to coordinate in their selection of policies to deploy in these settings. This
situation is quite unlike that faced by Schelling’s bargainers, for whom there are many possible agreements to
commit to, and who thus may fail to reach an agreement if they make incompatible commitments1.
Cao et al. [2018] are an exception in that they look at negotiation between deep reinforcement learners,
such that there are multiple incompatible ways in which agents could choose among possible agreements.
However, they do not evaluate the performance of independently trained policies when played against one
Working draft. Last updated March 8, 2021.
1The inadequacy of the Prisoner’s Dilemma as a model for cooperation problems, and the need to study games that exhibit
competing “cooperative” solutions, has been pointed out in other ﬁelds. See Snidal [1985] for a discussion in international
relations, McAdams [2008] in law, and Binmore [1998] in evolutionary game theory.
1
another (“cross-play” [Hu et al., 2020]), which is where the existence of multiple incompatible cooperative
solutions creates problems.
In this note, we argue that a focus on sequential social dilemmas and the lack of cross-play evaluation
obscures a fundamental problem for decentralized MAL in mixed-motive settings. We discuss a class of games
that we call mixed-motive coordination problems (MCPs) , which highlight this problem. We compare MCPs
to pure coordination problems and SSDs. We then illustrate how MAL methods which achieve cooperation in
other social dilemmas fail in some simple MCPs, and argue that future methods for promoting cooperation in
mixed-motive settings should be benchmarked in these more diﬃcult cases. We close with some tentative
positive contributions, by sketching some partial solutions to MCPs.
2 Preliminaries
We work in the setting of partially observable stochastic games (POSGs). For simplicity we assume two
players,i= 1;2. We will index player i’s counterpart by i. Each player has an action space Ai, there is an
underlying state space Swhich evolves according to a Markovian transition function P(St+1jSt;At
1;At
2). At
each time step, each player sees an observation Ot
iwhich depends on St. Thus each player has an accumulating
history of observations Ht
i=fOv
1;Av
1gt
v=1. Call the set of all histories Hi=[1
t=1Ht
i. Assume for simplicity
that the initial observation history is ﬁxed and common knowledge, h0
1=h0
2h0. Finally, principals choose
among policies i:H! (Ai), which we imagine as artiﬁcial agents deployed by the principals. We will
refer to policy proﬁles generically as 2 :=  12.
Each player has a reward function ri, such that ri(St;At
1;At
2)is their reward at time t. Deﬁne the value
to playeriof policy proﬁle starting at history ht
iasVi(ht
i;) =E[P1
v=t
vtri(Sv;Av
1;Av
2)jHt
i=ht
i],
where
 < 1is a discount factor. Deﬁne the value of a policy proﬁle to player iasVi() =Vi(h0;).
A payoﬀ proﬁle is then a tuple (V1();V2()). We say that is a (Nash) equilibrium of a POSG if
i2arg max0
i2iVi(h0;0
i;i)fori= 1;2. We say that is Pareto-optimal if for i= 1;2and02, we
have thatVi(0)>Vi()impliesVi(0)<Vi():
3 Mixed-motive coordination problems
We envision multiple actors (“principals”) deploying machine learning systems, deﬁned by policies, into the
world. The principals may be human developers, or they may be machine learning systems themselves. The
principals simultaneously deploy policies, without communication (in Section 6 we discuss ways in which
communication may change the analysis). Let Wbe a set of welfare functions , which are supposed to encode
normative properties (total welfare, fairness, etc.) of diﬀerent payoﬀ proﬁles. Two uncontroversial properties
of a welfare function are Pareto-optimality (i.e., its optimizer should be Pareto-optimal) and symmetry (the
welfare of a policy proﬁle should be invariant to permutations of player indices). Beyond that, there are
several appealing properties which cannot all be satisﬁed by the same welfare function (see Table 1). The
experimental evidence regarding people’s preferences over welfare functions is inconclusive [Felsenthal and
Diskin, 1982]. We will say that a game is a mixed-motive coordination problem (MCP) if it has several
equilibria2which are optimal with respect to some welfare function, but are incompatible with each other.
Deﬁnition 3.1 (Mixed-motive coordination problem) .LetGbe a POSG. LetWbe a set of welfare functions,
and letWEWbe the set of welfare functions in Wwhich are optimized by some equilibrium of G. Then
Gis a mixed-motive coordination problem (with respect to W) if there exist distinct w;w02WE2such that
(w
1;w0
2)has a Pareto-suboptimal policy proﬁle for all w2arg max0w(0),w02arg max0w0(0).
An example of a mixed-motive coordination problem is the Bach or Stravinsky game. A variant of this
game with asymmetric payoﬀs is given in Figure 1 (left). The folk theorem [Friedman, 1971] tells us that any
payoﬀ proﬁle in the convex hull of these payoﬀs is attainable in an equilibrium of the repeated game, for
players with suﬃciently high discount factors. And, as is suggested by the shape of the Pareto frontier for
2The spirit of an MCP is that independent training predictably leads to policy proﬁles which optimize diﬀerent welfare
functions, but are incompatible. So one could modify the requirement that welfare-optimal policies be equilibria and retain this
spirit. One could replace the requirement of Nash equilibrium with a diﬀerent or a broader set of solution concepts, e.g., the
recently-proposed Markov-Conley chains used in the -rank evaluation methodology [Omidshaﬁei et al., 2019].
2
Name of welfare function w Form ofw()
Nash [Nash, 1950] [V1()d1][V2()d2]
Kalai-Smorodinsky
[Kalai and Smorodinsky, 1975]V1()d1
V2()d2supV1()d1
supV2()d2+
fPareto-optimalg
Egalitarian [Kalai, 1977] minfV1()d1;V2()d2;g
Utilitarian (e.g. Harsanyi 1955) V1() +V2()
Table 1: Welfare functions, adapted to the multi-agent RL setting where two agents with value functions V1;V2
are bargaining over the policy proﬁle . to deploy. The function in the deﬁnition of the Kalai-Smorodinsky
welfare is the1-0indicator, used to enforce the constraint in its argument. Vd
i=Vi(d)are the players’
disagreement values.
Note that in bargaining problems in which the set of feasible payoﬀs is convex, the Nash welfare
function uniquely satisﬁes the properties of (1) Pareto optimality, (2) symmetry, (3) invariance to aﬃne
transformations, and (4) independence of irrelevant alternatives. See Zhou [1997] for a characterization of the
Nash welfare in non-convex bargaining problems. The Nash welfare can also be obtained as the subgame
perfect equilibrium of an alternating-oﬀers game as the “patience” of the players goes to inﬁnity [Binmore
et al., 1986].
On the other hand, Kalai-Smorodinsky uniquely satisﬁes (1)-(3) plus (5) resource monotonicity, which
means that all players are weakly better oﬀ when there are more resources to go around. The egalitarian
solution instead satisﬁes (1), (2), (4), and (5). The utilitarian welfare function is implicitly used in the work
of Peysakhovich and Lerer [2017a], Lerer and Peysakhovich [2017], Wang et al. [2018] on cooperation in
sequential social dilemmas.
BoS B S
B4, 1 0, 0
S0, 0 2, 2PD C D
C-1, -1 -3, 0
D0, -3 -2, -2
Figure 1: Payoﬀs for asymmetric Bach or Stravinsky (BoS) and the Prisoner’s Dilemma (PD).
BoS (Figure 2, left), diﬀerent welfare functions pick out diﬀerent Pareto-optimal payoﬀ proﬁles. For instance,
taking the disagreement point to be d1;d2= (0;0), the Nash welfare is optimized at (3:0;1:5), whereas the
egalitarian welfare is optimized at (2:0;2:0). On the other hand, all welfare functions in Table 1 agree on the
payoﬀs corresponding to (C;C)in the iterated Prisoner’s Dilemma (Figure 2, right).
In the following, we compare mixed-motive coordination problems to the more general equilibrium selection
problem and to sequential social dilemmas.
3.1 Relation to other cooperation problems
Equilibrium and Pareto selection problems It is well-known that achieving cooperation between
independently-trained agents this is a challenging problem for MAL in general (e.g., Boutilier 1999, Matignon
et al. 2012, Hu et al. 2020). One reason is that there are often multiple equilibria, such that (to the extent
that one wants to deploy a policy which is a best-response to that of one’s counterpart) the choice of a policy
constitutes an equilibrium selection problem .
Such problems even arise in the fully cooperative case, i.e., when V1=V2V, and are then also called
Pareto selection problems [Matignon et al., 2012] or pure coordination problems. In a pure coordination
problem, any 2arg maxV()has an optimal value and is thus in equilibrium. There is an equilibrium
selection problem if there exist ;02arg maxV()such that (1;0
2)has Pareto-suboptimal payoﬀ proﬁle.
Mixed-motive coordination problems that occur between policy proﬁles in equilibrium are a subclass
of equilibrium selection problems. They only occur in general-sum, non-fully cooperative cases and pose a
challenge which is qualitatively diﬀerent from the fully cooperative case: players have to coordinate among
3
options over which they have conﬂicting preferences. The pure coordination problem can be solved by merely
giving the players a chance to converge upon a coordinated policy proﬁle (as in Boutilier [1999], for instance)
and any such policy will do equally well. This is not so in mixed-motive coordination problems, since agents
that are open to converging upon any Pareto-optimal outcome would be exploitable by opponents that insist
on an outcome they prefer. (See Example 6.0.1 for a concrete illustration of this diﬀerence.)
Sequential social dilemmas Sequential social dilemmas (SSDs), as deﬁned by Leibo et al. [2017], are
POSGs whose policy space contains subsets corresponding to cooperation and defection, called C
iandD
i,
respectively. At each state, the normal form game whose payoﬀs are value functions under proﬁles of policies
inC
i;D
isatisﬁes the social dilemma inequalities [Macy and Flache, 2002], where atomic “cooperate” and
“defect” actions are replaced with elements of C
i;D
i. An SSD is not necessarily an equilibrium selection
problem. For instance, any Pareto-optimal and symmetric policy proﬁle will select an equilibrium proﬁle that
leads to constant cooperation in the iterated Prisoner’s Dilemma, and which is compatible with other such
proﬁles. But other SSDs may exhibit multiple incompatible ways of implementing cooperation.
SSDs with symmetric Pareto-optimal equilibria are in any case not mixed-motive coordination problems.
This includes the Gathering, Wolfpack [Leibo et al., 2017], and Coin Game [Peysakhovich and Lerer, 2017a]
environments. For instance, in the Coin Game, the single cooperative payoﬀ proﬁle is attained when agents
get only the coin of their own color, rather than attempting to take their counterpart’s coin as well.
4 Learning algorithms
Play between policies trained by diﬀerent runs or variations of an algorithm, or “cross-play” [Hu et al., 2020]
is problematic for MCPs. To illustrate the need for evaluating policies in cross-play, we train policies using
two multi-agent learning algorithms which have been shown to achieve cooperation in SSDs. The ﬁrst is a
variant of Lerer and Peysakhovich [2017]’s approximate Markov tit-for-tat (amTFT). The original amTFT
algorithm trains a cooperative policy on the utilitarian welfare (Table 1), as well as a punishment policy, and
switches from the cooperative to the punishment policy when it detects that the other player is defecting
from the cooperative policy. We consider the more general class of algorithms in which a cooperative policy
is constructed by optimizing some welfare function. Call the version of amTFT in which the cooperative
policy optimizes welfare function wasamTFT (w).
In our experiments, we use the utilitarian welfare function wUtiland aninequity averse welfare function
wIA. Following Hughes et al. [2018], this welfare function is constructed using modiﬁed reward functions
which penalize inequitable outcomes:
VIA
i(at
1;at
2;;) =r1(at
1;at
2) +r1(at
2;at
2)[et
1(at
1;at
2)et
2(at
1;at
2)]+[e2(at
1;at
2)e1(at
1;at
2)]+
where [x]+=maxfx;0gandet
iaresmoothedcumulativerewardscomputedas et
i(at
1;at
2) =
et1
i(at1
1;at1
2)+
ri(at
1;at
2)with hyperparameter . and discount factor
 andcontrol how much unequal outcomes are
penalized.3
The second learning algorithm is learning with opponent-learning awareness (LOLA; Foerster et al. 2018)
a policy gradient method in which learners attempt to shape the others’ learning through the choice of their
own policy updates. Write Vi(1;2)as the value to player iunder a proﬁle of policies with parameters 1;2.
Then, the LOLA update for player 1 at time twith parameters ;> 0is
t
1=t1
1+r1Vi(1;2) +[r2V1(1;2)]T
r1r2V2(1;2):
We use the discount factor
= 0:96throughout.
3We choose the utilitarian and inequity-averse welfare functions largely for convenience, and not because they have particularly
compelling normative properties. Speciﬁcally, these welfare functions admit a Bellman equation, allowing us to straightforwardly
apply reinforcement learning methods in order to maximize them. Methods for optimizing other welfare functions (Table 1),
which may be less amenable to oﬀ-the-shelf reinforcement learning methods, are a direction for future work.
4
5 Results
Here we illustrate the special problem posed by MCPs by comparing the performance of the learning
algorithms above in iterated BoS (IBoS) and in the iterated Prisoner’s Dilemma (IPD). The folk theorems
[Mailath et al., 2006] tell us that, for players with suﬃciently high discount factors, any payoﬀ proﬁle of
the stage game in which players get at least as much as they can unilaterally guarantee themselves can be
attained in an equilibrium of the repeated game. IBoS is an MCP with respect to the set of welfare functions
W=fwUtil;wIAg, since these welfare functions are optimized by distinct payoﬀ proﬁles for suﬃciently large
values of;.
We measure the performance of these algorithms in self-play andcross-play . Self-play payoﬀs are the
average payoﬀs attained by a proﬁle of policies produced by the same training run. Cross-play payoﬀs are
the average payoﬀs attained by a proﬁle of two policies produced by two diﬀerent training runs. In the
case of amTFT, we also consider cross-play between policies trained with amTFT (wUtil)and those trained with
amTFT (wIA).
5.1 LOLA
Following Foerster et al. [2018]’s parameterization of policies in the iterated Prisoner’s Dilemma, we used
policies which condition on the previous pair of actions played, with one parameter for every possible previous
action proﬁle. To learn policy proﬁles, we used a discount factor of
= 0:96and applied LOLA to the
closed-form value functions (which can be computed for each policy proﬁle using the Bellman equations; this
is called exact LOLA in Foerster et al. [2018]).
Figure 2: LOLA average payoﬀ proﬁles for 40 training runs in IBoS (left) and IPD (right). Cross-play payoﬀs
were obtained by randomly sampling 40 pairs (1;2), where1and2come from diﬀerent runs. A small
amount of random noise is applied to each point to enhance visibility.
On average, self-play yields payoﬀs (2:69;1:34), while cross-play yields (1:30;0:70). As Figure 2 shows, the
self-play payoﬀs almost all lie close to the empirical Pareto frontier, which consists of payoﬀ proﬁles roughly
equal tof(4;1);(3;1:5);(2;2)g. This clustering is caused by the policies depending only on actions taken at
the previous timestep, which allows for always playing (S,S), alternating between them, or always playing
(B,B). These coincide with the maxima for wUtil, the Nash welfare [Nash, 1950] under disagreement proﬁle
(0;0), orwIA.
On the other hand, the cross-play points are much more often Pareto-suboptimal. This is not surprising,
as if LOLA produces tit-for-tat-like strategies which punish deviations from a particular strategy proﬁle, then
playing policies from separate runs is liable to lead to incompatible policies and therefore punishments.
5.2 amTFT
We evaluated self- and cross-play for amTFT in IBoS and IPD, where each episode lasted 100 of steps. Payoﬀ
proﬁles from individual runs are displayed in Figure 3, and average payoﬀs under self- and cross-play for each
5
combination of welfare functions are displayed in Table 2.
Figure 3: amTFT payoﬀ proﬁles for 40 training runs in IBoS (left) and IPD (right). A small amount of
random noise is applied to each point to enhance visibility.
Table 2: Self-play (" ") and cross-play (" ") payoﬀs in IBoS for amTFT (wUtil)andamTFT (wIA). Payoﬀs are
averages over 40 training runs. wUtilwIAleads to the same payoﬀ regardless of which player operates under
which welfare function. All standard errors were below 0:01.
Policy proﬁle wUtilwUtilwIAwIAwUtilwUtilwIAwIAwUtilwIA
Player 1 3.97 2.00 3.87 2.00 0.00
Player 2 1.01 2.00 0.97 2.00 0.00
6 Possible solutions to MCPs
Of course, the goal of developing benchmarks is not only to point out ﬂaws in existing approaches, but also
to help develop better ones. In this section we discuss several research directions for ameliorating MCPs, and
give a toy illustration of one approach.
Coordination by the principals MCPs can be solved if the principals can agree on a way of selecting
among policy proﬁles to deploy. For instance, principals might agree on a welfare function, and choose
learning algorithms which converge to an optimizer of that welfare function. Coordination on a welfare
function (and among equilibria which all maximize that welfare function) may be easy in toy environments
such as the ones we have considered. But as systems become more complex, it will likely be intractable
for principals to fully coordinate so as to train welfare-optimal agents. Thus there is a need to develop
methods for low-bandwidth coordination between principals training agents in MCPs which captures most
of the beneﬁts of full coordination. Bandwidth-constrained coordination has already been studied in fully
cooperative games (e.g., Stone and Veloso 1999, Zhang et al. 2019).
Compromise between welfare functions We have not discussed how giving agents the opportunity to
communicate post-deployment might ameliorate MCPs. One possibility is to have them reach a compromise
over diﬀerent welfare-optimal policy proﬁles before deciding which to deploy. On possibility is for the train to
several policies corresponding to several welfare functions according to which they are willing to play. When
given the chance to communicate, their agents can then apply a bargaining protocol to this reduced set of
policy proﬁles. Here is a simple example of such a protocol. Agents ﬁrst announce the sets of policy proﬁles
they are willing to play. If these sets overlap, a policy proﬁle is taken at random from the intersection (compare
with Chakravorti and Conley [2004]’s discussion of randomization between the Nash and Kalai-Smorodinsky
6
Pure Coordination A B
A 1, 1 0, 0
B 0, 0 1, 1
Figure 4: Payoﬀs for a pure coordination problem.
bargaining solutions in the asymmetric iterated Prisoner’s Dilemma). Only when these sets do not overlap do
agents revert to a default policy. We give an illustration in Example 6.0.1.
Example 6.0.1 (Bargaining over amTFT policies) .An example of a protocol for bargaining over welfare-
optimal policy proﬁles is given in Algorithm 1. Policies are obtained by applying amTFT to a welfare
functionw, returning a policy amTFT (w)ifor playeri. Refer to the strategy which reports welfare functions
Wiand default policy d
ias(Wi;d
i). Table 3 displays the payoﬀ table of the game with strategy proﬁles
f(W1;d
1);(W2;d
2)gin this bargaining protocol applied to IBoS.
This payoﬀ matrix clearly shows the tradeoﬀ between exploitability and robustness to independently-
trained policies. The strategies which play fwUtil;wIAgare of more robust, in the sense that they avoid the
mutually dispreferred proﬁle of (1:0;0:3). However, these strategies are notplayed in equilibrium, because
they are exploitable: an agent can deviate from this proﬁle by instead playing only their preferred welfare
function. Nevertheless, one has the intuition that it is reasonable to play fwUtil;wIAgin view of its robustness,
despite the fact that it is exploitable. On the other hand, the situation becomes more complicated when the
set of welfare functions under consideration is much larger. Identifying reasonable solution concepts for this
setting is an important direction for future work.
Besides the fact that avoiding bargaining failure is still not guaranteed, there are additional problems for
this protocol. First, it may be ex-ante Pareto suboptimal, because randomization between Pareto optimal
payoﬀ proﬁles may be ex-ante Pareto suboptimal. Second, while it assumes that agents agree to uniform
randomization over the intersection of their sets, agents will in fact have diﬀering preferences over diﬀerent
distributions with which to randomize, generating another bargaining problem. Less ad-hoc solutions might
be found by using a dynamic bargaining protocol (as in the classical Rubinstein bargaining game Rubinstein
1982), which may narrow down the set of equilibria.
As a ﬁnal note, compare this protocol with an analogous protocol for a pure coordination problem (Figure
4). In this protocol, agents play a default action but also announce a set of actions that they are willing to play.
Again, actions are taken uniformly at random from the overap of the sets if it is nonempty, and otherwise the
default action is taken. Clearly playing fA;Bgis a dominant strategy for each player, unlike the analogous
strategy in the protocol for IBoS, where greater ﬂexibility comes at the price of greater exploitability.
Algorithm 1: Welfare function selection protocol
Input:Sets of welfare functions Wi, default policies d
i
// Independently train policies to maximize welfare functions
fori= 1;2do
i ;
forw2Wido
w
i amTFT (w)i
i i[fw
ig
// Communicate and decide what policies to deploy
Players announce W1;W2
ifW1\W 26=;then
wUnif (W1\W 2)
1;2 w
1;w
2
else
1;2 d
1;d
2
return1;2
7
Table 3: Empirical payoﬀ matrix for agents playing IBoS and following bargaining protocol in Algorithm 1
with welfare functions in fwUtil;wIAg:
fwUtilg;IA
2fwIAg;IA
2fwUtil;wIAg;IA
2
fwUtilg;Util
1 4.0, 1.0 0.0, 0.0 4.0, 1.0
fwIAg;Util
1 0.0, 0.0 2.0, 2.0 2.0, 2.0
fwUtil;wIAg;Util
1 4.0, 1.0 2.0, 2.0 3.0, 1.5
Table 4: Payoﬀ matrix for agents playing Pure Coordination and following a variant of the bargaining protocol
in 1 where sets of rather than welfare functions are announced.
fAg;BfBg;BfA;Bg;B
fAg;A 1.0, 1.0 0.0, 0.0 1.0, 1.0
fBg;A 0.0, 0.0 1.0, 1.0 1.0, 1.0
fA;Bg;A 1.0, 1.0 1.0, 1.0 1.0, 1.0
References
Ken Binmore, Ariel Rubinstein, and Asher Wolinsky. The nash bargaining solution in economic modelling.
The RAND Journal of Economics , pages 176–188, 1986.
Ken G Binmore. The evolution of fairness norms. Rationality and Society , 10(3):275–301, 1998.
Craig Boutilier. Sequential optimality and coordination in multiagent systems. In IJCAI, volume 99, pages
478–485, 1999.
Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent
communication through negotiation. arXiv preprint arXiv:1804.03980 , 2018.
Bhaskar Chakravorti and John Conley. Bargaining eﬃciency and the repeated prisoner’s dilemma. Economics
Bulletin, 3(3):1–8, 2004.
Tom Eccles, Edward Hughes, János Kramár, Steven Wheelwright, and Joel Z Leibo. Learning reciprocity in
complex sequential social dilemmas. arXiv preprint arXiv:1903.08082 , 2019.
Dan S. Felsenthal and Abraham Diskin. The Bargaining Problem Revisited: Minimum Utility Point,
Restricted Monotonicity Axiom, and the Mean as an Estimate of Expected Utility. Journal of Conﬂict
Resolution , 26(4):664–691, December 1982. ISSN 0022-0027. doi: 10.1177/0022002782026004005. URL
https://doi.org/10.1177/0022002782026004005 . Publisher: SAGE Publications Inc.
Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch.
Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on
Autonomous Agents and MultiAgent Systems , pages 122–130. International Foundation for Autonomous
Agents and Multiagent Systems, 2018.
James W. Friedman. A Non-cooperative Equilibrium for Supergames12. The Review of Economic Studies ,
38(1):1–12, 1971. ISSN 0034-6527. doi: 10.2307/2296617. URL https://doi.org/10.2307/2296617 .
\_eprint: https://academic.oup.com/restud/article-pdf/38/1/1/4362169/38-1-1.pdf.
John C Harsanyi. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal
of political economy , 63(4):309–321, 1955.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. " other-play" for zero-shot coordination.
arXiv preprint arXiv:2003.02979 , 2020.
Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Antonio García
Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves
8
cooperation in intertemporal social dilemmas. In Advances in neural information processing systems , pages
3326–3336, 2018.
Ehud Kalai. Proportional solutions to bargaining situations: interpersonal utility comparisons. Econometrica:
Journal of the Econometric Society , pages 1623–1630, 1977.
Ehud Kalai and Smorodinsky. Other solutions to nash’s bargaining problem. Econometrica , 43(3):513–518,
1975.
Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement
learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and
MultiAgent Systems , pages 464–473. International Foundation for Autonomous Agents and Multiagent
Systems, 2017.
Adam Lerer and Alexander Peysakhovich. Maintaining cooperation in complex social dilemmas using deep
reinforcement learning. arXiv preprint arXiv:1707.01068 , 2017.
Michael W Macy and Andreas Flache. Learning dynamics in social dilemmas. Proceedings of the National
Academy of Sciences , 99(suppl 3):7229–7236, 2002.
George J Mailath, J George, Larry Samuelson, et al. Repeated games and reputations: long-run relationships .
Oxford university press, 2006.
Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learners in
cooperative markov games: a survey regarding coordination problems. 2012.
Richard H McAdams. Beyond the prisoners’ dilemma: Coordination, game theory, and law. S. Cal. L. Rev. ,
82:209, 2008.
John F Nash. The bargaining problem. Econometrica: Journal of the Econometric Society , pages 155–162,
1950.
Shayegan Omidshaﬁei, Christos Papadimitriou, Georgios Piliouras, Karl Tuyls, Mark Rowland, Jean-Baptiste
Lespiau, Wojciech M Czarnecki, Marc Lanctot, Julien Perolat, and Remi Munos. -rank: Multi-agent
evaluation by evolution. Scientiﬁc reports , 9(1):1–29, 2019.
Alexander Peysakhovich and Adam Lerer. Consequentialist conditional cooperation in social dilemmas with
imperfect information. arXiv preprint arXiv:1710.06975 , 2017a.
Alexander Peysakhovich and Adam Lerer. Prosocial learning agents solve generalized stag hunts better than
selﬁsh ones. arXiv preprint arXiv:1709.02865 , 2017b.
Ariel Rubinstein. Perfect equilibrium in a bargaining model. Econometrica: Journal of the Econometric
Society, pages 97–109, 1982.
Thomas C Schelling. An essay on bargaining. The American Economic Review , 46(3):281–306, 1956.
Thomas C Schelling. The strategy of conﬂict. prospectus for a reorientation of game theory. Journal of
Conﬂict Resolution , 2(3):203–264, 1958.
Duncan Snidal. Coordination versus prisoners’ dilemma: Implications for international cooperation and
regimes. American Political Science Review , 79(4):923–942, 1985.
Peter Stone and Manuela Veloso. Task decomposition, dynamic role assignment, and low-bandwidth
communication for real-time strategic teamwork. Artiﬁcial Intelligence , 110(2):241–273, 1999.
Weixun Wang, Jianye Hao, Yixi Wang, and Matthew Taylor. Towards cooperation in sequential prisoner’s
dilemmas: a deep multiagent reinforcement learning approach. arXiv preprint arXiv:1803.00162 , 2018.
Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Eﬃcient communication in multi-agent reinforcement learning via
variance based control. In Advances in Neural Information Processing Systems , pages 3235–3244, 2019.
Lin Zhou. The nash bargaining theory with non-convex problems. Econometrica: Journal of the Econometric
Society, pages 681–685, 1997.
9


Suffering-focused AI safety: Why "fail-safe'" measures might be our top intervention

Report FRI-16-1
Suﬀering-focused AI safety: Why “fail-safe”
measures might be our top intervention
Lukas Gloor
Foundational Research Institute
lukas.gloor@foundational-research.org
June 2016
Abstract
AI-safety eﬀorts focused on suﬀering reduction should place particular emphasis on
avoiding risks of astronomical disvalue. Among the cases where uncontrolled AI de-
stroys humanity, outcomes might still diﬀer enormously in the amounts of suﬀering
produced. Rather than concentrating all our eﬀorts on a speciﬁc future we would like
to bring about, we should identify futures we least want to bring about and work
on ways to steer AI trajectories around these. In particular, a “fail-safe”1approach
to AI safety is especially promising because avoiding very bad outcomes might be
much easier than making sure we get everything right. This is also a neglected cause
despite there being a broad consensus among diﬀerent moral views that avoiding the
creation of vast amounts of suﬀering in our future is an ethical priority.
Contents
1 Introduction 2
2 How AI outcomes might contain suﬀering 2
2.1 Type I: Controlled AI gone bad . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.2 Type II: “Near misses” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.3 Type III: Uncontrolled AI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
3 Suﬀering-focused AI safety: Some proposals 5
3.1 Inﬂuencing outcomes with controlled AI . . . . . . . . . . . . . . . . . . . . . . 5
3.2 Diﬀerential progress in AI safety . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.3 AI safety where it is most needed . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.4 Graceful fails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Concluding thoughts 7
Acknowledgements 8
References 8
1“Fail-safe” in the sense that if control fails, at least the AI causes less suﬀering than would have been the
case without fail-safe measures.
Suﬀering-focused AI safety −Foundational Research Institute
1 Introduction
Classical AI safety looks very diﬃcult. In or-
der to ensure a truly “utopian” outcome, the
following things need to happen:
1. The AI that gains a decisive strategic ad-
vantage over the competition needs to be
built by the right group of people.
2. These people would need to have ﬁgured
out how to program favorable values into
an AI (or program the AI to learn these
values).
3. They would also need to have come up
with a satisfying description of what
these values are, either directly, or in-
directly with the help of a suitable ex-
trapolation procedure (e.g. Muehlhauser
& Helm, 2012).
4. Finally, the creators of the ﬁrst superin-
telligence might need to get decision the-
ory right (see Soares & Fallenstein, 2015)
and safeguard the AI from a lot of hard-
to-anticipate failure modes.
Succeeding with all of this together is the only
safe way of bringing about a ﬂourishing utopia
mostly free of suﬀering. (There might be other
paths to utopia, provided enough luck.) But
what about outcomes where something goes
wrong? It is important to consider that things
can go wrong to very diﬀerent degrees. For
value systems that place primary importance
on the prevention of suﬀering, this aspect is
crucial: the best way to avoid bad-case sce-
narios speciﬁcally may not be to try and get
everything right. Instead, it makes sense to fo-
cus on the worst outcomes (in terms of the
suﬀering they would contain) and on tractable
methods to avert them. As others are trying to
shoot for a best-case outcome (and hopefully
they will succeed!), it is important that some
people also take care of addressing the biggest
risks. This perspective to AI safety is espe-
cially promising both because it is currently
neglected and because it is easier to avoid a
subset of outcomes rather than to shoot foronehighlyspeciﬁcoutcome.Finally,itissome-
thing that people with many diﬀerent value
systems could get behind.
2 How AI outcomes might contain
suﬀering
Fromasuﬀering-focusedperspective,themain
reason to be concerned about the risks from
artiﬁcial intelligence is not the possibility of
human extinction or the corresponding failure
tobuildaﬂourishing,intergalacticcivilization.
Rather, it is the thought of misaligned or “ill-
aligned” AI as a powerful but morally indif-
ferent optimization process which, in the pur-
suit of its goals, may transform galactic re-
sources into (among other things) suﬀering.
Either like the suﬀering we see on Earth to-
day, or by bringing about optimized struc-
tures that possibly also contain novel forms
of suﬀering. The things a superintelligent AI
would build to pursue its goals might include
a ﬂeet of “worker bots”, factories, supercom-
puters, space colonization machinery, etc. It is
possible that all of these end up being built
in a way that does not permit suﬀering. How-
ever, given that evolution has produced plenty
of sentient minds, this weakly suggests that
some of the easiest ways to implement mind
architectures do come with the capacity for
suﬀering. In the absence of an explicit anti-
suﬀering preference, even the slightest bene-
ﬁt to the AI’s objectives would lead to the
instantiation of suﬀering minds. What makes
this especially worrying is that the stakes in-
volved will be huge: Space colonization is an
attractive subgoal for almost any powerful op-
timization process, as it leads to control over
the largest amount of resources (Omohundro,
2008). Even if only a small portion of these
resources were used for purposes that include
suﬀering,theresultingdisvaluewouldsadlybe
astronomical.
To ﬁnd the best interventions to prevent suf-
fering, it is crucial to study how AI outcomes
2
Suﬀering-focused AI safety −Foundational Research Institute
diﬀerintheirexpectedamountofsuﬀeringand
whether there are ways to favorably aﬀect the
likely outcomes. This paper aims to provide
an overview on “bad-case scenarios” in the AI
context and some general suggestions for how
they could be avoided. Future work will in-
clude checking fundamental assumptions and
looking at the most promising proposals in
more detail.
2.1 Type I: Controlled AI gone bad
In these scenarios, the team that builds the
ﬁrst superintelligence knew what it was doing
and managed to successfully shape the result-
ing AI’s goals exactly the way they wanted it
togo.Unfortunately,thegoalsarenotwhatwe
would wish them to be and thus lead to bad
or very bad consequences. Things that could
go wrong include:
a) Anthropocentrism: perhaps the result-
ing AI would not care about the suﬀer-
ing of “weird” and/or voiceless nonhu-
man minds such as suﬀering subroutines
or animal minds in ancestor simulations.
b) Retributivism: perhaps the AI gets built
by people who want to punish members
of an outgroup (e.g. religious fundamen-
talists punishing sinners).
c) Uncooperative: perhaps the AI’s goal
is something like classical utilitarianism
(or any other “monotone” maximizing
function) with no additional regards for
cooperation with other value systems.
Even though such an AI would all else
equal prefer to not create suﬀering, it
seems possible that the anti-suﬀering
concern would in practice be overridden
by opportunity costs: if happiness sim-
ulations contain much more utility for
the AI than the disutility produced bythe accidental suﬀering created in the
buildup, then such an AI would behave
similarly “recklessly” as e.g. a paperclip-
maximizing AI.2Another concern lies in
how strongly the AI would value robust-
ness to deterioration.
d) Libertarianism regarding computations:
perhaps the creators of the ﬁrst super-
intelligence instruct the AI to give every
human alive at the time of the singular-
ity control of a planet or galaxy, with
no additional rules to govern what goes
onwithinthoseterritories.Someofthese
human rulers are curious and amused by
seeing others hurt, or worse, might be
psychopaths.
2.2 Type II: “Near misses”
In this scenario, the ﬁrst superintelligence is
built by people who were aware of the risks,
and who tried to get things right. Unfortu-
nately, mistakes happened and the resulting
outcome with “nearly-controlled AI” is bad or
very bad.
a) Miserable creatures: perhaps the AI’s
goal function includes terms that at-
tempt to specify sentient or human-like
beings and conditions that are meant to
be good for these beings. However, be-
cause of programming mistakes, unan-
ticipated loopholes or side-eﬀects, the
conditions speciﬁed actually turn out to
be bad for these beings. Worse still, the
AI has a maximizing function and wants
to ﬁll as many regions of the universe as
possible with these poor creatures.
b) Black swans: perhaps the AI cares about
sentientorhuman-likemindsin“proper”
ways, but has bad priors, ontology, de-
cision theory, or other fundamental con-
2An AI that values happiness simulations but takes cooperation with suﬀering reducers into account would in
expectation still create some amount of suﬀering, but depending on just how costly it is to use more “suﬀering-
proof” algorithms in the colonization process, or to do fewer or less ﬁne-grained ancestor simulations, it might
be possible to cut down on most of the instrumentally produced suﬀering at a cost that isn’t very high (e.g. not
higher than 15% of total happiness simulations produced).
3
Suﬀering-focused AI safety −Foundational Research Institute
stituents that would make it act in un-
fortunate and unpredictable ways.
2.3 Type III: Uncontrolled AI
Uncontrolled AI is stipulated to have goals
that were never intended by its creators, or
at least were not intended to take over control
in the form of a singleton (Bostrom, 2006).
Unless the AI in question has a neuromor-
phic design, it is unlikely that its goals would
(directly) have something to do with sentient
beings. However, even non-neuromorphic un-
controlled AI might instrumentally instanti-
ate suﬀering minds in the process of achieving
its goal, as a side-eﬀect of the computations
it performs. The following presents a list of
ways how uncontrolled AI might create vast
amounts of suﬀering:
•Suﬀering subroutines: For tasks like in-
venting and developing advanced tech-
nologies, as well as for coordinating its
expansion into space for resource accu-
mulation, the AI would have to rely on
a ﬂeet of “robot workers” or “robot scien-
tists” of various kinds (which, of course,
might work and function very unlike hu-
man workers or scientists). It is possible
that these processes would be capable of
suﬀering (Tomasik, 2014), although it is
unclear whether would perform at their
best if they (occasionally) suﬀer. What is
clear is that if they do suﬀer under use-
ful circumstances, an AI that is not ex-
plicitly aligned with compassionate goals
would not have any qualms about instan-
tiating astronomical numbers of them.
•Ancestor simulations (Bostrom, 2003;
Bostrom, 2014, Ch.8): in order to gather
information relevant to its goals, e.g. for
improving its understanding of human
psychology or sociology (Bostrom, 2014,pp. 125-26) or for studying the density of
aliensintheuniverseandwhattheirlikely
values are, a superintelligence might sim-
ulatemanyrunsofDarwinianevolutionin
planet-sized supercomputers (Sandberg,
1999). It is possible that these simulations
would be ﬁne-grained enough to contain
sentient minds.
•Warfare: it may be that the density of life
in the universe is high enough for coloniz-
ing AIs to eventually encounter one an-
other. If so, uncontrolled AI might end
up clashing with other superintelligences,
including AIs that build expanding civ-
ilizations of happy sentient beings. Per-
haps they could agree to a compromise,
but such an encounter could also lead to
ﬁghting over resources and warfare at the
points of contact.
It is interesting to note that none of the afore-
mentioned scenarios involve large quantities of
suﬀering in the types of structures that the AI
directlyoptimizesfor.Itseemslikelythatwhat
makes uncontrolled AI bad is what it builds
for instrumental reasons, including coloniza-
tion machinery, science simulations, and other
strategic computations. This suggests that op-
portunity costs are relevant: If an uncontrolled
AI’s goal function does not have diminishing
returnsonresources(liketheinfamous“paper-
clipmaximizer”),thentheroomforinstrumen-
tally important computations might be small.
By contrast, an AI with goals that are easy to
fulﬁll3– e.g. a “paperclip protector” that only
cares about protecting a single paperclip, or
an AI that only cares about its own reward
signal – would have much greater room pur-
suing instrumentally valuable computations.
This line of thought suggests that outcomes
from uncontrolled AI should be steered away
from “paperclip-protector types.”4
3Thanks to Carl Shulman for bringing this idea to our attention.
4However, it may be that paperclip protectors are easier to compromise with, and thus more likely to also
pursue maximization of sorts in exchange for beneﬁts from “paperclip-maximizer types.”
4
Suﬀering-focused AI safety −Foundational Research Institute
3 Suﬀering-focused AI safety: Some
proposals
Bad-case scenarios can be avoided in several
ways. Using the typology from above, we can
distinguish three classes of interventions:
•Inﬂuencing outcomes with con-
trolled AI: inﬂuence the values imple-
mented in outcomes with controlled AI
either through value spreading or by sup-
portingtheAIsafetyprojectwiththebest
values
The following can be labelled “fail-safe” mea-
sures:
•Targeted AI safety:
a) Diﬀerential progress: push classical
AI safety diﬀerentially in ways that
best insulates it from the worst
“near misses”
b) AI safety where it is most needed:
identify the paradigm in general AI
that would create the worst out-
comes in the event of a control
failure, and work towards identify-
ing (general) control or shut-down
mechanisms in those areas
•Graceful fails : research ways to make AI
fail in a “benign” way conditional on it
failing(“It might be the end of the world,
but it could have been (much) worse”)
The following sections present some prelimi-
nary ideas for the type of interventions that
could be done in those areas.
3.1 Inﬂuencing outcomes with
controlled AI
This class of interventions is primarily tar-
geted at “controlled AI gone bad” outcomes. It
is something FRI and others in the EA com-
munity have already thought about a great
deal.
•Improve values: raise awareness of anti-
speciesism, anti-substratism, concern forweird minds or suﬀering-focused ethics in
general.
•Cooperation: promote the idea that the
gains from cooperation make it impor-
tant for diﬀerent value-systems to work
together.
•Cooperative AI-safety eﬀorts: by funding
the AI-safety project with the best val-
ues or the best approach to cooperation,
we make it more likely that controlled AI
won’t create vast amounts of suﬀering.
3.2 Diﬀerential progress in AI safety
This intervention is targeted at the worst out-
comes in the “near misses” category. The goal
is to ﬁgure out solutions to the worst prob-
lems ﬁrst, such that any additional progress in
AI safety is unlikely to lead to “near-misses,”
which may be worse than outcomes with un-
controlled AI in terms of suﬀering produced.
This intervention seems comparatively more
neglected than interventions in the category
above.
•Secure value implementation: Look into
the pros and cons of various approaches to
specifying an AI’s values and work on the
one least likely to lead to really bad out-
comes. Questions to study may include:
What’s more likely to go wrong –
something like indirect normativity,
or an explicit speciﬁcation of a goal
function?
What are other promising approaches
to AI safety, and how bad would the
outcome be if they fail according to
their weak points?
Would corrigibility reduce risks of as-
tronomical suﬀering?
•Foundational research: Which unantici-
pated failure modes could there be for
controlled AI? Once they are identiﬁed,
how can we prevent them?
5
Suﬀering-focused AI safety −Foundational Research Institute
3.3 AI safety where it is most needed
This intervention is targeted at preventing the
worstoutcomesinvolvinguncontrolledAI.Not
all uncontrolled AIs are expected to behave
equally; there are diﬀerent paradigms in AI re-
search, i.e. diﬀerent basic architectures for at-
tempting to build smarter-than-human intelli-
gence. Examples include reinforcement learn-
ing (Sutton & Barto, 1998), proof-based seed
AI, and neuromorphic AIs (Hasler & Marr,
2013). Classical AI safety is necessarily con-
cerned with the tradeoﬀ between success prob-
ability and “controllability;” if a particular ap-
proach to building AI is likely to lead to su-
perintelligence, but in a way where it is hard
to control the values of the resulting agent,
then this poses a likely dead end. By contrast,
a suﬀering-focused “fail-safe” approach does
not have to worry about controllability to the
same extent, as long as there’s enoughcontrol-
labilitytopredictablyavoidoutcomeswiththe
most suﬀering. This suggests that, conditional
on uncontrolled AI being worse than (nearly-
)controlled AI in expectation, suﬀering reduc-
ers can be less constrained by issues with con-
trollability, and can thus focus more on ap-
plying AI safety to the paradigm/architecture
with the highest probability of bringing about
superintelligence.
In addition, we could try to identify diﬀer-
ences among the typesof uncontrolled AI each
AI-paradigm would produce, in order to then
focusonAIsafetyinthedomainwherefailures
of control result in the worst outcomes. Sup-
pose for instance that one way of building AI
appears to show promising results and rapid
progress,yetcontrolseemshardandlikelyfail-
ure modes are particularly worrying. In this
case, we should consider putting eﬀorts into
AI safety applied to that particular domain.
Speciﬁc examples of this more general idea in-
clude:
•AI safety for machine learners: WithDeepMind’s recent success, it seems not
unlikely that the ﬁrst superintelligent AI
may be all reinforcement learning as the
top control structure (as opposed to GO-
FAI, theorem proving or logic-based AI).
IncontrasttoAIarchitectureswheregoals
refer to the state of the world (Hibbard,
2011) rather than to internal or observa-
tional signals, pure reinforcement learners
maybeatahigherriskofwireheading,i.e.
of hacking their input signals to continu-
ously attain the maximum reward. If the
wireheading AI is smart enough to plan
into the future, it might patiently avoid
doing so and develop plans for takeover
ﬁrst: As Bostrom notes in Superintel-
ligence(Bostrom, 2014, 122–23), smart
wireheaders would still want to colonize
other galaxies for option value, protec-
tion, research, etc. However, unlike paper-
clip maximizers, wireheaders would more
generally function like “paperclip protec-
tors” in that they would have almost no
opportunity costs (Ring & Orseau, 2011).
If the observation that lower opportunity
costs make outcomes with uncontrolled
AI worse is accurate (see the discussion
under “Type III: Uncontrolled AI), this
wouldmakeAIsafetyappliedtoreinforce-
ment learning architectures particularly
promising. Speciﬁc interventions could in-
clude work on Interruptibility (Armstrong
& Orseau, 2016), but also methods of
value-loading (examples here and here).
•AI safety for “em-ﬁrst” scenarios: Condi-
tional on whole brain emulation (Sand-
berg & Bostrom, 2008; Hanson, 2016) be-
coming technologically feasible before the
advent of de-novo AI, de-novo AI as even-
tually developed will more likely exhibit
a neuromorphic design.5This would make
it more likely that the values are human-
like, which also comes with the possi-
5If only because neuroscience research would be very advanced in such a world.
6
Suﬀering-focused AI safety −Foundational Research Institute
bility of “bad values” or “near misses.”
AI safety applied to “em-ﬁrst” scenarios
couldthusbeapromisingintervention(al-
though with early arrival timelines being
longer,itmaynotbefavoredbyhastecon-
siderations).
Caveat: Pursuing any of the interventions
above would require that we make sure to not
speed up progress in AI development – both
in general and in “unsafe” or “hard-to-control”
approaches speciﬁc to those interventions.
3.4 Graceful fails
This class of interventions is targeted at pre-
venting the worst outcomes involving “nearly-
controlled” or uncontrolled AI. Instead of try-
ing to make fails less likely, the idea is to come
up with safety nets or mechanisms such that,
if control fails, the outcome will be as good
as it gets under the circumstances. Graceful
fails are easiest to bring about in the domain
of near misses, as the researchers working on
AI safety are aware of the risks and thus open
to the idea of safety nets.
•Backup goal functions: Research ways to
implement multi-layered goal functions,
with a “backup goal” that kicks in if the
implementation of the top layer does not
fulﬁll certain safety criteria. The backup
would be a simpler, less ambitious goal
that is less likely to result in bad out-
comes. Diﬃculties would lie in selecting
thesafetycriteriainwaysthatpeoplewith
diﬀerent values could all agree on, and
in making sure that the backup goal gets
triggeredunderthecorrectcircumstances.
InthecontextofAIresearchthatisnotguided
by safety precautions, graceful fails propos-
als seem harder to implement. The challenge
is to get an AI project that might result in
uncontrolled AI to care about whatever pro-
posal there is to implement. Perhaps there are
proposals to consider in rather early stages of
AI development where researchers think hardtakeoﬀs are very unlikely. Should the AI de-
sign in question nevertheless undergo an intel-
ligenceexplosion,theremightbewaystomake
the result counterfactually less bad.
•Dummy goals: We could come up with a
goal function with the following proper-
ties:
Easy to program
Interesting (in the sense that it’s easy
to create problems for an AI with this
goal function to solve, such that re-
searchers can track their progress on
general intelligence)
Leads to decent outcome in the un-
likely case of hard takeoﬀ
If AI research and testing at early stages is
always conducted with goals of this type, an
unanticipated hard takeoﬀ would at least be
comparatively benign.
4 Concluding thoughts
Suﬀering-focused AI safety, and “fail-safe”
measures in particular, make up a large and
promising area of interventions for FRI (and
perhaps other organizations) to investigate
further. Advantages are that they are ne-
glected and often “less ambitious” than clas-
sical AI safety, which means they might end
up being more tractable. Moreover, “fail-safe”
measures are a promising project to focus on
because the interventions discussed are pos-
itive or at worst “unobjectionable” from the
perspectives of virtually all other value sys-
tems.
The main reason we had not explicitly
zoomed in on this category of interventions
earlier was that FRI was initially too focused
on answering the more general, complicated
question whether AI safety on the whole is
net positive for suﬀering reducers. Upon re-
ﬂection, this question may be less important.
AI safety is a very broad category where we
shouldexpectalotofroomfortargetedeﬀorts.
The general lesson to draw is that partitioning
7
Suﬀering-focused AI safety −Foundational Research Institute
broad categories can be an important step to-
wards making progress. More research into the
feasibilityofsuﬀering-focusedAIsafetycarries
high information value for this very reason: As
we look into more proposals and their prac-
tical feasibility, we might uncover more rele-
vant distinctions among AI outcomes, value-
transferproposals,orgeneralapproachestoAI
– a process which, in turn, will make it easier
to identify eﬀective interventions.
Acknowledgements
Brian Tomasik provided the central idea for
this paper and gave important feedback on
earlier versions. Thanks also go to Caspar
Oesterheld and Max Daniel for contributing
to the ideas and scenarios in the paper, and
to Simon Knutsson, Adrian Hutter, David Al-
thaus, Adrian Rohrheim, Andrei Pöhlmann,
Jan Leike and Kaj Sotala for valuable feed-
back and edits.
References
Armstrong, M. S., and L. Orseau. “Safely
Interruptible Agents.” Machine
Intelligence Research Institute 2016.
Bostrom, Nick. “Are We Living in a
Computer Simulation?.” The Philosophical
Quarterly 53.211 (2003): 243-255.
Bostrom, Nick. “What Is a Singleton.”
Linguistic and Philosophical Investigations
5.2 2006: 48-54.
Bostrom, Nick. Superintelligence: Paths,
Dangers, Strategies . Oxford: Oxford UP,
2014. Print.
Hanson, Robin. The Age of Em: Work, Love
and Life When Robots Rule the Earth.
Oxford: Oxford University Press, 2016.
Print.
Hasler, Jennifer, and Harry Bo Marr.
“Finding a Roadmap to Achieve Large
Neuromorphic Hardware Systems.”
Frontiers in Neuroscience 7 2013: 118.Hibbard, Bill. “Model-based Utility
Functions.” Journal of Artiﬁcial General
Intelligence 3.1 2011: 1-24.
Muehlhauser, Luke, and Louie Helm. 2012.
“Intelligence Explosion and Machine
Ethics.”In Singularity Hypotheses: A
Scientiﬁc and Philosophical Assessment .
Ed. Amnon Eden, Johnny Søraker, James
H. Moor, and Eric Steinhart. Berlin:
Springer, 2012. Print.
Omohundro, Stephen M. “The Basic AI
Drives.”AGI. Vol. 171. 2008.
Ring, Mark, and Laurent Orseau. “Delusion,
Survival, and Intelligent Agents.”
International Conference on Artiﬁcial
General Intelligence. Springer Berlin
Heidelberg, 2011.
Sandberg, Anders. “The Physics of
Information Processing Superobjects:
Daily Life Among the Jupiter Brains.”
Journal of Evolution and Technology 5.1
1999: 1-34.
Sandberg, Anders, and Nick Bostrom. Whole
Brain Emulation: A Roadmap . Future of
Humanity Institute, 2008. Print.
Soares, Nate, and Benja Fallenstein. “Toward
Idealized Decision Theory.” arXiv preprint
arXiv:1507.01986 2015.
Sutton, Richard S., and Andrew G. Barto.
Reinforcement Learning: An Introduction.
Vol. 1. No. 1. Cambridge: MIT press,
1998.
Tomasik, Brian. “Do Artiﬁcial
Reinforcement-Learning Agents Matter
Morally?.” arXiv preprint arXiv:1410.8233
2014.
8