Von Neumann–Morgenstern utility theorem

Toggle the table of contents

Toggle the table of contents

Von Neumann–Morgenstern utility theorem
=======================================

3 languages

\* [Français](https://fr.wikipedia.org/wiki/Th%C3%A9or%C3%A8me\_d%27utilit%C3%A9\_de\_von\_Neumann-Morgenstern "Théorème d'utilité de von Neumann-Morgenstern – French")
\* [עברית](https://he.wikipedia.org/wiki/%D7%9E%D7%A9%D7%A4%D7%98\_%D7%A4%D7%95%D7%9F\_%D7%A0%D7%95%D7%99%D7%9E%D7%9F-%D7%9E%D7%95%D7%A8%D7%92%D7%A0%D7%A9%D7%98%D7%A8%D7%9F "משפט פון נוימן-מורגנשטרן – Hebrew")
\* [Nederlands](https://nl.wikipedia.org/wiki/Von\_Neumann-Morgenstern-nutsfunctie "Von Neumann-Morgenstern-nutsfunctie – Dutch")

[Edit links](https://www.wikidata.org/wiki/Special:EntityPage/Q4358367#sitelinks-wikipedia "Edit interlanguage links")

\* [Article](/wiki/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem "View the content page [c]")
\* [Talk](/wiki/Talk:Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem "Discuss improvements to the content page [t]")

English

\* [Read](/wiki/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem)
\* [Edit](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&action=edit&oldid=1044421624 "Edit this page [e]")
\* [View history](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&action=history "Past revisions of this page [h]")

Tools

Tools
move to sidebar
hide

 Actions

\* [Read](/wiki/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem)
\* [Edit](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&action=edit&oldid=1044421624)
\* [View history](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&action=history)

 General

\* [What links here](/wiki/Special:WhatLinksHere/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem "List of all English Wikipedia pages containing links to this page [j]")
\* [Related changes](/wiki/Special:RecentChangesLinked/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem "Recent changes in pages linked from this page [k]")
\* [Upload file](/wiki/Wikipedia:File\_Upload\_Wizard "Upload files [u]")
\* [Special pages](/wiki/Special:SpecialPages "A list of all special pages [q]")
\* [Permanent link](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&oldid=1044421624 "Permanent link to this revision of this page")
\* [Page information](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&action=info "More information about this page")
\* [Cite this page](/w/index.php?title=Special:CiteThisPage&page=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&id=1044421624&wpFormIdentifier=titleform "Information on how to cite this page")
\* [Wikidata item](https://www.wikidata.org/wiki/Special:EntityPage/Q4358367 "Structured data on this page hosted by Wikidata [g]")

 Print/export

\* [Download as PDF](/w/index.php?title=Special:Book&bookcmd=render\_article&arttitle=Von+Neumann%E2%80%93Morgenstern+utility+theorem&returnto=Von+Neumann%E2%80%93Morgenstern+utility+theorem&oldid=1044421624&writer=rl "Download this page as a PDF file")
\* [Printable version](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&printable=yes "Printable version of this page [p]")

 Print/export

From Wikipedia, the free encyclopedia

\*\*This is an [old revision](/wiki/Help:Page\_history "Help:Page history") of this page, as edited by [2601:445:4380:7dd0::6b8c](/wiki/Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C "Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C") ([talk](/w/index.php?title=User\_talk:2601:445:4380:7DD0:0:0:0:6B8C&action=edit&redlink=1 "User talk:2601:445:4380:7DD0:0:0:0:6B8C (page does not exist)")) at 04:35, 15 September 2021. The present address (URL) is a [permanent link](/wiki/Help:Permanent\_link "Help:Permanent link") to this revision, which may differ significantly from the [current revision](https://en.wikipedia.org/wiki/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem).\*\*

Revision as of 04:35, 15 September 2021 by [2601:445:4380:7dd0::6b8c](/wiki/Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C "Special:Contributions/2601:445:4380:7DD0:0:0:0:6B8C") ([talk](/w/index.php?title=User\_talk:2601:445:4380:7DD0:0:0:0:6B8C&action=edit&redlink=1 "User talk:2601:445:4380:7DD0:0:0:0:6B8C (page does not exist)"))([diff](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&diff=prev&oldid=1044421624 "Von Neumann–Morgenstern utility theorem")) [← Previous revision](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&direction=prev&oldid=1044421624 "Von Neumann–Morgenstern utility theorem") | [Latest revision](/wiki/Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem "Von Neumann–Morgenstern utility theorem") ([diff](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&diff=cur&oldid=1044421624 "Von Neumann–Morgenstern utility theorem")) | [Newer revision →](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&direction=next&oldid=1044421624 "Von Neumann–Morgenstern utility theorem") ([diff](/w/index.php?title=Von\_Neumann%E2%80%93Morgenstern\_utility\_theorem&diff=next&oldid=1044421624 "Von Neumann–Morgenstern utility theorem"))
Any individual whose preferences satisfy four axioms has a utility function
In [decision theory](/wiki/Decision\_theory "Decision theory"), the \*\*von Neumann–Morgenstern\*\* (\*\*VNM\*\*) \*\*utility theorem\*\* shows that, under certain [axioms](/wiki/Axiom "Axiom") of [rational behavior](/wiki/Rationality "Rationality"), a decision-maker faced with [risky](/wiki/Risk "Risk") (probabilistic) outcomes of different choices will behave as if he or she is maximizing the [expected value](/wiki/Expected\_value "Expected value") of some function defined over the potential outcomes at some specified point in the future. This function is known as the von Neumann–Morgenstern utility function. The theorem is the basis for [expected utility theory](/wiki/Expected\_utility\_theory "Expected utility theory").

In 1947, [John von Neumann](/wiki/John\_von\_Neumann "John von Neumann") and [Oskar Morgenstern](/wiki/Oskar\_Morgenstern "Oskar Morgenstern") proved that any individual whose [preferences](/wiki/Preference\_(economics) "Preference (economics)") satisfied four axioms has a [utility function](/wiki/Utility\_function "Utility function");[[1]](#cite\_note-VNM-1) such an individual's preferences can be represented on an [interval scale](/wiki/Interval\_scale "Interval scale") and the individual will always prefer actions that maximize expected utility. That is, they proved that an agent is (VNM-)rational \*if and only if\* there exists a real-valued function \*u\* defined by possible outcomes such that every preference of the agent is characterized by maximizing the expected value of \*u\*, which can then be defined as the agent's \*VNM-utility\* (it is unique up to adding a constant and multiplying by a positive scalar). No claim is made that the agent has a "conscious desire" to maximize \*u\*, only that \*u\* exists.

The [expected utility hypothesis](/wiki/Expected\_utility\_hypothesis "Expected utility hypothesis") is that rationality can be modeled as maximizing an [expected value](/wiki/Expected\_value "Expected value"), which given the theorem, can be summarized as "\*rationality is VNM-rationality\*". However, the axioms themselves have been critiqued on various grounds, resulting in the axioms being given further justification.[[2]](#cite\_note-2)

VNM-utility is a \*decision utility\* in that it is used to describe \*decision preferences\*. It is related but not equivalent to so-called \*E-utilities\*[[3]](#cite\_note-KWS-3) (experience utilities), notions of utility intended to measure happiness such as that of [Bentham](/wiki/Jeremy\_Bentham "Jeremy Bentham")'s [Greatest Happiness Principle](/wiki/Greatest\_happiness\_principle "Greatest happiness principle").

Set-up
------

In the theorem, an individual agent is faced with options called [\*lotteries\*](/wiki/Lottery\_(probability) "Lottery (probability)"). Given some [mutually exclusive](/wiki/Mutually\_exclusive "Mutually exclusive") outcomes, a lottery is a scenario where each outcome will happen with a given [probability](/wiki/Probability "Probability"), all probabilities summing to one. For example, for two outcomes \*A\* and \*B\*,

L
=
0.25
A
+
0.75
B

{\displaystyle L=0.25A+0.75B}

![{\displaystyle L=0.25A+0.75B}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/65609f0204eae5e293e04be54a8765d4e6076871)
denotes a scenario where \*P\*(\*A\*) = 25% is the probability of \*A\* occurring and \*P\*(\*B\*) = 75% (and exactly one of them will occur). More generally, for a lottery with many possible outcomes \*Ai\*, we write:

L
=
∑

p

i

A

i

,

{\displaystyle L=\sum p\\_{i}A\\_{i},}

![{\displaystyle L=\sum p\_{i}A\_{i},}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/b8d0fce14b6b440fc698a747b407009c68822f1e)
with the sum of the

p

i

p\\_{i}

![p\_{i}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/cc755a12730f445caa917980ff8400c8f7bea900)s equalling 1.

The outcomes in a lottery can themselves be lotteries between other outcomes, and the expanded expression is considered an equivalent lottery: 0.5(0.5\*A\* + 0.5\*B\*) + 0.5\*C\* = 0.25\*A\* + 0.25\*B\* + 0.50\*C\*.

If lottery \*M\* is preferred over lottery \*L\*, we write

L
≺
M

{\displaystyle L\prec M}

![{\displaystyle L\prec M}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/0182f1e20f44b51055fcd59eb81757897e9294af), or equivalently,

M
≻
L

{\displaystyle M\succ L}

![{\displaystyle M\succ L}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/80803d03780883af17aff238ddf94a958413dbe9). If the agent is indifferent between \*L\* and \*M\*, we write the \*indifference relation\*[[4]](#cite\_note-Kreps-4)

L
∼
M
.

L\sim M.

![L\sim M.](https://wikimedia.org/api/rest\_v1/media/math/render/svg/000e3009e7e8125a4bc901e6db9b875b16351d98) If \*M\* is either preferred over or viewed with indifference relative to \*L\*, we write

L
⪯
M
.

L\preceq M.

![L \preceq M.](https://wikimedia.org/api/rest\_v1/media/math/render/svg/835b0639c4609862a7277aeb9514d2a2c69aebfe)

The axioms
----------

The four axioms of VNM-rationality are then \*completeness\*, \*transitivity\*, \*continuity\*, and \*independence\*.

Completeness assumes that an individual has well defined preferences:

\*\*Axiom 1 (Completeness)\*\* For any lotteries \*L,M\*, exactly one of the following holds:

L
≺
M

{\displaystyle \,L\prec M}

![{\displaystyle \,L\prec M}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/4d342c7334631e2f25929ce5b4c0fa5a237dc2b8),

M
≺
L

{\displaystyle \,M\prec L}

![{\displaystyle \,M\prec L}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/f90b9bdfd3b1f7de1c8e264c260d9b32d9e317f1), or

L
∼
M

\,L\sim M

![\, L \sim M ](https://wikimedia.org/api/rest\_v1/media/math/render/svg/a2894248bb820486601bd5a05a5276ff46d4eca7)
(either \*M\* is preferred, \*L\* is preferred, or the individual is indifferent[[5]](#cite\_note-nop-5)).

[Transitivity](/wiki/Transitive\_relation "Transitive relation") assumes that preferences are consistent across any three options:

\*\*Axiom 2 (Transitivity)\*\* If

L
≺
M

{\displaystyle \,L\prec M}

![{\displaystyle \,L\prec M}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/4d342c7334631e2f25929ce5b4c0fa5a237dc2b8) and

M
≺
N

{\displaystyle \,M\prec N}

![{\displaystyle \,M\prec N}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/8d01b1e70f36e74bb88cb96d0017fad1fe0ca714), then

L
≺
N

{\displaystyle \,L\prec N}

![{\displaystyle \,L\prec N}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/7bf6cb42fbbd0b49d21b2e8e309d00efe3fa5662), and similarly for

∼
\sim

![\sim ](https://wikimedia.org/api/rest\_v1/media/math/render/svg/db54f75bdf7b940dd3626a5448357a2453c9e27c).
Continuity assumes that there is a "tipping point" between being \*better than\* and \*worse than\* a given middle option:

\*\*Axiom 3 (Continuity):\*\* If

L
⪯
M
⪯
N

{\displaystyle \,L\preceq M\preceq N}

![{\displaystyle \,L\preceq M\preceq N}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/db759acfaed5c7544ea659645bf6aa4d03cb9876), then there exists a probability

p
∈
[
0
,
1
]

{\displaystyle \,p\in [0,1]}

![{\displaystyle \,p\in [0,1]}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/4449572ac985c35af802cf3a3f8e28594be10554) such that

p
L
+
(
1
−
p
)
N

∼

M

{\displaystyle \,pL+(1-p)N\,\sim \,M}

![{\displaystyle \,pL+(1-p)N\,\sim \,M}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/27b8381ac45f6fd337b78643a7cdf72ed7646231)
where the notation on the left side refers to a situation in which \*L\* is received with probability \*p\* and \*N\* is received with probability (1–\*p\*).

Instead of continuity, an alternative axiom can be assumed that does not involve a precise equality, called the [Archimedean property](/wiki/Archimedean\_property "Archimedean property").[[4]](#cite\_note-Kreps-4) It says that any separation in preference can be maintained under a sufficiently small deviation in probabilities:

\*\*Axiom 3′ (Archimedean property):\*\* If

L
≺
M
≺
N

{\displaystyle \,L\prec M\prec N}

![{\displaystyle \,L\prec M\prec N}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/3a0072df9c734c8ad431711fd38d0a4d5519cd43), then there exists a probability

ε
∈
(
0
,
1
)

\,\varepsilon \in (0,1)

![\,\varepsilon\in(0,1)](https://wikimedia.org/api/rest\_v1/media/math/render/svg/0833c6a52751dd5b490ac24bc457bf77f246723c) such that

(
1
−
ε
)
L
+
ε
N

≺

M

≺

ε
L
+
(
1
−
ε
)
N
.

{\displaystyle \,(1-\varepsilon )L+\varepsilon N\,\prec \,M\,\prec \,\varepsilon L+(1-\varepsilon )N.}

![{\displaystyle \,(1-\varepsilon )L+\varepsilon N\,\prec \,M\,\prec \,\varepsilon L+(1-\varepsilon )N.}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/64ff3f2253aa81b3f3e3f9841800f05b774e57f5)
Only one of (3) or (3′) need to be assumed, and the other will be implied by the theorem.

[Independence of irrelevant alternatives](/wiki/Independence\_of\_irrelevant\_alternatives "Independence of irrelevant alternatives") assumes that a preference holds independently of the possibility of another outcome:

\*\*Axiom 4 (Independence):\*\* For any

N

\,N

![\,N](https://wikimedia.org/api/rest\_v1/media/math/render/svg/e8ee3605551b781498ac5a9bd710c483921baa09) and

p
∈
(
0
,
1
]

{\displaystyle \,p\in (0,1]}

![{\displaystyle \,p\in (0,1]}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/590a2ef3670309007fe88b3d8cb8b752eaba85be),

L
⪯
M

iff

p
L
+
(
1
−
p
)
N
⪯
p
M
+
(
1
−
p
)
N
.

{\displaystyle \,L\preceq M\qquad {\text{iff}}\qquad pL+(1-p)N\preceq pM+(1-p)N.}

![{\displaystyle \,L\preceq M\qquad {\text{iff}}\qquad pL+(1-p)N\preceq pM+(1-p)N.}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/e8a6623aac916e0cd10fbc893d1064d3ee4deced)
The independence axiom implies the axiom on reduction of compound lotteries:[[6]](#cite\_note-6)

\*\*Axiom 4′ (Reduction of compound lotteries):\*\* For any lotteries

L
,

L
′

,
N
,

N
′

{\displaystyle L,L',N,N'}

![{\displaystyle L,L',N,N'}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/5b655f53c05fe6b42158783a5d51f756fe957320) and any

p
,
q
∈
[
0
,
1
]

{\displaystyle p,q\in [0,1]}

![{\displaystyle p,q\in [0,1]}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/4ce8c8abe5c6cff2c531084637e986f132a54703),

if

L
∼
q

L
′

+
(
1
−
q
)

N
′

,

{\displaystyle {\text{if}}\qquad L\sim qL'+(1-q)N',}

![{\displaystyle {\text{if}}\qquad L\sim qL'+(1-q)N',}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/c80efa13cc1d99b6588914665b7a5e7524cf8a82)

then

p
L
+
(
1
−
p
)
N
∼
p
q

L
′

+
p
(
1
−
q
)

N
′

+
(
1
−
p
)
N
.

{\displaystyle {\text{then}}\quad pL+(1-p)N\sim pqL'+p(1-q)N'+(1-p)N.}

![{\displaystyle {\text{then}}\quad pL+(1-p)N\sim pqL'+p(1-q)N'+(1-p)N.}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/707a8a608dcee7289d3ff1b05c397190a4774002)
To see how Axiom 4 implies Axiom 4', set

M
=
q

L
′

+
(
1
−
q
)

N
′

{\displaystyle M=qL'+(1-q)N'}

![{\displaystyle M=qL'+(1-q)N'}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/d68a56fdfd003375cf2f70b4e684316f42631e1a) in the expression in Axiom 4, and expand.

The theorem
-----------

For any VNM-rational agent (i.e. satisfying axioms 1–4), there exists a function \*u\* which assigns to each outcome \*A\* a real number \*u(A)\* such that for any two lotteries,

L
≺
M

i
f

a
n
d

o
n
l
y

i
f

E
(
u
(
L
)
)
<
E
(
u
(
M
)
)
,

{\displaystyle L\prec M\qquad \mathrm {if\,and\,only\,if} \qquad E(u(L))
u
(
L
)

u(M)>u(L)

![u(M)>u(L)](https://wikimedia.org/api/rest\_v1/media/math/render/svg/4e5ea77670926be26b5de061d8d44ffa3b11a44f), a rational decision maker would prefer the lottery

M
M

![M](https://wikimedia.org/api/rest\_v1/media/math/render/svg/942a5b14fdd65b6e39c13e87904c3634947c5521) over the lottery

L
L

![L](https://wikimedia.org/api/rest\_v1/media/math/render/svg/2a719782d8d9ccfb2790d743cd337b9aeebe042f), because it gives him a larger chance to win the best outcome.

Hence:

L
≺
M

{\displaystyle L\prec M\;}

![{\displaystyle L\prec M\;}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/54144dccfa8f037f645c635f0c41d0f90b2fa40c) if and only if

E
(
u
(
L
)
)
<
E
(
u
(
M
)
)
.

{\displaystyle E(u(L))
> "Many economists will feel that we are assuming far too much ... Have we not shown too much? ... As far as we can see, our postulates [are] plausible ... We have practically defined numerical utility as being that thing for which the calculus of mathematical expectations is legitimate." – \*VNM 1953, § 3.1.1 p.16 and § 3.7.1 p. 28\*[[1]](#cite\_note-VNM-1)
>
>
>
>

Thus, the content of the theorem is that the construction of \*u\* is possible, and they claim little about its nature.

Consequences
------------

### Automatic consideration of risk aversion

.mw-parser-output .hatnote{font-style:italic}.mw-parser-output div.hatnote{padding-left:1.6em;margin-bottom:0.5em}.mw-parser-output .hatnote i{font-style:normal}.mw-parser-output .hatnote+link+.hatnote{margin-top:-0.5em}Main article: [Risk aversion](/wiki/Risk\_aversion "Risk aversion")
It is often the case that a person, faced with real-world [gambles](/wiki/Gamble "Gamble") with money, does not act to maximize the expected value of their \*dollar assets.\* For example, a person who only possesses $1000 in savings may be reluctant to risk it all for a 20% chance odds to win $10,000, even though

20
%
(
$
10

000
)
+
80
%
(
$
0
)
=
$
2000
>
100
%
(
$
1000
)

{\displaystyle 20\%(\$10\,000)+80\%(\$0)=\$2000>100\%(\$1000)}

![{\displaystyle 20\%(\$10\,000)+80\%(\$0)=\$2000>100\%(\$1000)}](https://wikimedia.org/api/rest\_v1/media/math/render/svg/167ca3f9cd2649a8a8811e2fe1f36573b31d6f00)
However, \*if\* the person is VNM-rational, such facts are automatically accounted for in their utility function \*u\*. In this example, we could conclude that

20
%
u
(
$
10

000
)
+
80
%
u
(
$
0
)
<
u
(
$
1000
)

{\displaystyle 20\%u(\$10\,000)+80\%u(\$0)
> "The axioms should not be too numerous, their system is to be as simple and transparent as possible, and each axiom should have an immediate intuitive meaning by which its appropriateness may be judged directly. In a situation like ours this last requirement is particularly vital, in spite of its vagueness: we want to make an intuitive concept amenable to mathematical treatment and to see as clearly as
> possible what hypotheses this requires." – \*VNM 1953 § 3.5.2, p. 25\*[[1]](#cite\_note-VNM-1)
>
>
>
>

As such, claims that the expected utility hypothesis does not characterize rationality must reject one of the VNM axioms. A variety of [generalized expected utility](/wiki/Generalized\_expected\_utility "Generalized expected utility") theories have arisen, most of which drop or relax the independence axiom.

### Implications for ethics and moral philosophy

Main article: [Consequentialism](/wiki/Consequentialism "Consequentialism")
Because the theorem assumes nothing about the nature of the possible outcomes of the gambles, they could be morally significant events, for instance involving the life, death, sickness, or health of others. A von Neumann–Morgenstern rational agent is capable of acting with great concern for such events, sacrificing much personal wealth or well-being, and all of these actions will factor into the construction/definition of the agent's VNM-utility function. In other words, both what is naturally perceived as "personal gain", and what is naturally perceived as "altruism", are implicitly balanced in the VNM-utility function of a VNM-rational individual. Therefore, the full range of [agent-focussed to agent-neutral](/wiki/Consequentialism#Agent-focused\_or\_agent-neutral "Consequentialism") behaviors are possible with various VNM-utility functions[\*[clarification needed](/wiki/Wikipedia:Please\_clarify "Wikipedia:Please clarify")\*].

If the utility of

N
N

![N](https://wikimedia.org/api/rest\_v1/media/math/render/svg/f8e2d8185ce44f49f1cf6dcce4cfdebe27c9a8d3) is

p
M

pM

![pM](https://wikimedia.org/api/rest\_v1/media/math/render/svg/60d1dc4cdc74e97275af9a5c9b7a09f6f1845544), a von Neumann–Morgenstern rational agent must be indifferent between

1
N

1N

![1N](https://wikimedia.org/api/rest\_v1/media/math/render/svg/fb0b54b7f5b41c8dfc4f73479db0a79c7088cfee) and

p
M
+
(
1
−
p
)
0

pM+(1-p)0

![pM+(1-p)0](https://wikimedia.org/api/rest\_v1/media/math/render/svg/540d62628f1b20f0df75bb5bd169ddf104d51512). An agent-focused von Neumann–Morgenstern rational agent therefore cannot favor more equal, or "fair", distributions of utility between its own possible future selves.

### Distinctness from other notions of utility

Some [utilitarian moral theories](/wiki/Utilitarianism#Average\_v\_total "Utilitarianism") are concerned with quantities called the "total utility" and "average utility" of collectives, and characterize morality in terms of favoring the utility or happiness of others with disregard for one's own. These notions can be related to, but are distinct from, VNM-utility:

\* 1) VNM-utility is a \*decision utility\*:[[3]](#cite\_note-KWS-3) it is that according to which one decides, and thus by definition cannot be something which one disregards.
\* 2) VNM-utility is not canonically additive across multiple individuals (see Limitations), so "total VNM-utility" and "average VNM-utility" are not immediately meaningful (some sort of normalization assumption is required).

The term \*E-utility\* for "experience utility" has been coined[[3]](#cite\_note-KWS-3) to refer to the types of "hedonistic" utility like that of [Bentham](/wiki/Jeremy\_Bentham "Jeremy Bentham")'s [greatest happiness principle](/wiki/Greatest\_happiness\_principle "Greatest happiness principle"). Since morality affects decisions, a VNM-rational agent's morals will affect the definition of its own utility function (see above). Thus, the morality of a VNM-rational agent can be characterized by \*correlation\* of the agent's VNM-utility with the VNM-utility, E-utility, or "happiness" of others, among other means, but not by \*disregard\* for the agent's own VNM-utility, a contradiction in terms.

Limitations
-----------

### Nested gambling

Since if \*L\* and \*M\* are lotteries, then \*pL\* + (1 − \*p\*)\*M\* is simply "expanded out" and considered a lottery itself, the VNM formalism ignores what may be experienced as "nested gambling". This is related to the [Ellsberg problem](/wiki/Ellsberg\_paradox "Ellsberg paradox") where people choose to avoid the perception of \*risks about risks\*. Von Neumann and Morgenstern recognized this limitation:

> "...concepts like a \*specific utility of gambling\* cannot be formulated free of contradiction on this level. This may seem to be a paradoxical assertion. But anybody who has seriously tried to axiomatize that elusive concept, will probably concur with it." – \*VNM 1953 § 3.7.1, p. 28\*.[[1]](#cite\_note-VNM-1)
>
>

### Incomparability between agents

Since for any two VNM-agents \*X\* and \*Y\*, their VNM-utility functions \*uX\* and \*uY\* are only determined up to additive constants and multiplicative positive scalars, the theorem does not provide any canonical way to compare the two. Hence expressions like \*uX\*(\*L\*) + \*uY\*(\*L\*) and \*uX\*(\*L\*) − \*uY\*(\*L\*) are not canonically defined, nor are comparisons like \*uX\*(\*L\*) < \*uY\*(\*L\*) canonically true or false. In particular, the aforementioned "total VNM-utility" and "average VNM-utility" of a population are not canonically meaningful without normalization assumptions.

### Applicability to economics

The [expected utility hypothesis](/wiki/Expected\_utility\_hypothesis "Expected utility hypothesis") is shown to have limited predictive accuracy in a set of lab based empirical experiments, such as the [Allais paradox](/wiki/Allais\_paradox "Allais paradox").
Which leads some people to interpret as evidence that

\* humans are not always rational, or
\* VNM-rationality is not an appropriate characterization of rationality, or
\* some combination of both, or
\* humans \*do\* behave VNM-rationally but the objective evaluation of \*u\* and the construction of \*u\* are [intractable](/wiki/Intractability\_(complexity) "Intractability (complexity)") problems.

References and further reading
------------------------------

.mw-parser-output .reflist{font-size:90%;margin-bottom:0.5em;list-style-type:decimal}.mw-parser-output .reflist .references{font-size:100%;margin-bottom:0;list-style-type:inherit}.mw-parser-output .reflist-columns-2{column-width:30em}.mw-parser-output .reflist-columns-3{column-width:25em}.mw-parser-output .reflist-columns{margin-top:0.3em}.mw-parser-output .reflist-columns ol{margin-top:0}.mw-parser-output .reflist-columns li{page-break-inside:avoid;break-inside:avoid-column}.mw-parser-output .reflist-upper-alpha{list-style-type:upper-alpha}.mw-parser-output .reflist-upper-roman{list-style-type:upper-roman}.mw-parser-output .reflist-lower-alpha{list-style-type:lower-alpha}.mw-parser-output .reflist-lower-greek{list-style-type:lower-greek}.mw-parser-output .reflist-lower-roman{list-style-type:lower-roman}
1. ^ [\*\*\*a\*\*\*](#cite\_ref-VNM\_1-0) [\*\*\*b\*\*\*](#cite\_ref-VNM\_1-1) [\*\*\*c\*\*\*](#cite\_ref-VNM\_1-2) [\*\*\*d\*\*\*](#cite\_ref-VNM\_1-3) [Neumann, John von](/wiki/John\_von\_Neumann "John von Neumann") and [Morgenstern, Oskar](/wiki/Oskar\_Morgenstern "Oskar Morgenstern"), \*[Theory of Games and Economic Behavior](/wiki/Theory\_of\_Games\_and\_Economic\_Behavior "Theory of Games and Economic Behavior")\*. Princeton, NJ. Princeton University Press, 1953.
2. \*\*[^](#cite\_ref-2)\*\* Peterson, Chapter 8.
3. ^ [\*\*\*a\*\*\*](#cite\_ref-KWS\_3-0) [\*\*\*b\*\*\*](#cite\_ref-KWS\_3-1) [\*\*\*c\*\*\*](#cite\_ref-KWS\_3-2) .mw-parser-output cite.citation{font-style:inherit;word-wrap:break-word}.mw-parser-output .citation q{quotes:"\"""\"""'""'"}.mw-parser-output .citation:target{background-color:rgba(0,127,255,0.133)}.mw-parser-output .id-lock-free a,.mw-parser-output .citation .cs1-lock-free a{background:url("//upload.wikimedia.org/wikipedia/commons/6/65/Lock-green.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-limited a,.mw-parser-output .id-lock-registration a,.mw-parser-output .citation .cs1-lock-limited a,.mw-parser-output .citation .cs1-lock-registration a{background:url("//upload.wikimedia.org/wikipedia/commons/d/d6/Lock-gray-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .id-lock-subscription a,.mw-parser-output .citation .cs1-lock-subscription a{background:url("//upload.wikimedia.org/wikipedia/commons/a/aa/Lock-red-alt-2.svg")right 0.1em center/9px no-repeat}.mw-parser-output .cs1-ws-icon a{background:url("//upload.wikimedia.org/wikipedia/commons/4/4c/Wikisource-logo.svg")right 0.1em center/12px no-repeat}.mw-parser-output .cs1-code{color:inherit;background:inherit;border:none;padding:inherit}.mw-parser-output .cs1-hidden-error{display:none;color:#d33}.mw-parser-output .cs1-visible-error{color:#d33}.mw-parser-output .cs1-maint{display:none;color:#3a3;margin-left:0.3em}.mw-parser-output .cs1-format{font-size:95%}.mw-parser-output .cs1-kern-left{padding-left:0.2em}.mw-parser-output .cs1-kern-right{padding-right:0.2em}.mw-parser-output .citation .mw-selflink{font-weight:inherit}Kahneman; Wakker; Sarin (1997). ["Back to Bentham? Explorations of Experienced Utility"](http://repub.eur.nl/pub/23011). \*[Quarterly Journal of Economics](/wiki/Quarterly\_Journal\_of\_Economics "Quarterly Journal of Economics")\*. \*\*112\*\* (2): 375–406. [doi](/wiki/Doi\_(identifier) "Doi (identifier)"):[10.1162/003355397555235](https://doi.org/10.1162%2F003355397555235). [hdl](/wiki/Hdl\_(identifier) "Hdl (identifier)"):[1765/23011](https://hdl.handle.net/1765%2F23011).
4. ^ [\*\*\*a\*\*\*](#cite\_ref-Kreps\_4-0) [\*\*\*b\*\*\*](#cite\_ref-Kreps\_4-1) [Kreps, David M.](/wiki/David\_M.\_Kreps "David M. Kreps") \*Notes on the Theory of Choice\*. Westview Press (May 12, 1988), chapters 2 and 5.
5. \*\*[^](#cite\_ref-nop\_5-0)\*\* Implicit in denoting indifference by equality are assertions like if

L
≺
M
=
N

{\displaystyle L\prec M=N}

![L\prec M = N](https://wikimedia.org/api/rest\_v1/media/math/render/svg/41c97165bae2eb24ce9448c456fb22c4884b6a32) then

L
≺
N

{\displaystyle L\prec N}

![L\prec N](https://wikimedia.org/api/rest\_v1/media/math/render/svg/8ff0ed9533e882a709e3bf19099ad043398cad26). To make such relations explicit in the axioms, Kreps (1988) chapter 2 denotes indifference by

∼

{\displaystyle \,\sim }

![{\displaystyle \,\sim }](https://wikimedia.org/api/rest\_v1/media/math/render/svg/a2d29304f09810413194433497d0d8d2007a3639), so it may be surveyed in brief for intuitive meaning.
6. \*\*[^](#cite\_ref-6)\*\* EconPort, "Von Neumann–Morgenstern Expected Utility Theory"
7. \*\*[^](#cite\_ref-KeeneyRaiffa1993\_7-0)\*\* Keeney, Ralph L.; Raiffa, Howard (1993). \*Decisions with Multiple Objectives\*. [ISBN](/wiki/ISBN\_(identifier) "ISBN (identifier)") [0-521-44185-4](/wiki/Special:BookSources/0-521-44185-4 "Special:BookSources/0-521-44185-4").
8. \*\*[^](#cite\_ref-8)\*\* \*Specimen theoriae novae de mensura sortis\* or \*Exposition of a New Theory on the Measurement of Risk\*

.mw-parser-output .refbegin{font-size:90%;margin-bottom:0.5em}.mw-parser-output .refbegin-hanging-indents>ul{margin-left:0}.mw-parser-output .refbegin-hanging-indents>ul>li{margin-left:0;padding-left:3.2em;text-indent:-3.2em}.mw-parser-output .refbegin-hanging-indents ul,.mw-parser-output .refbegin-hanging-indents ul li{list-style:none}@media(max-width:720px){.mw-parser-output .refbegin-hanging-indents>ul>li{padding-left:1.6em;text-indent:-1.6em}}.mw-parser-output .refbegin-columns{margin-top:0.3em}.mw-parser-output .refbegin-columns ul{margin-top:0}.mw-parser-output .refbegin-columns li{page-break-inside:avoid;break-inside:avoid-column}
\* [Nash, John F., Jr.](/wiki/John\_Forbes\_Nash\_Jr. "John Forbes Nash Jr.") (1950). "The Bargaining Problem". \*[Econometrica](/wiki/Econometrica "Econometrica")\*. \*\*18\*\* (2): 155–162. [doi](/wiki/Doi\_(identifier) "Doi (identifier)"):[10.2307/1907266](https://doi.org/10.2307%2F1907266). [JSTOR](/wiki/JSTOR\_(identifier) "JSTOR (identifier)") [1907266](https://www.jstor.org/stable/1907266).
\* Anand, Paul. \*Foundations of Rational Choice Under Risk\* Oxford, Oxford University Press. 1993 reprinted 1995, 2002
\* [Fishburn, Peter C.](/wiki/Peter\_C.\_Fishburn "Peter C. Fishburn") \*Utility Theory for Decision Making\*. Huntington, NY. Robert E. Krieger Publishing Co. 1970. [ISBN](/wiki/ISBN\_(identifier) "ISBN (identifier)") [978-0-471-26060-8](/wiki/Special:BookSources/978-0-471-26060-8 "Special:BookSources/978-0-471-26060-8")
\* [Sixto Rios](/wiki/Sixto\_Rios "Sixto Rios") (1998) [Some problems and developments in decision science](http://www.mat.ucm.es/serv/revmat/vol11-1/vol11-1g.html), \*Revista Matematica Complutense\* 11(1):113–41.
\* Peterson, Martin (2009). \*An Introduction to Decision Theory (Cambridge Introductions to Philosophy)\*. Cambridge: Cambridge University Press.

![](//en.wikipedia.org/wiki/Special:CentralAutoLogin/start?type=1x1)
Retrieved from ""
[Categories](/wiki/Help:Category "Help:Category"): \* [Theorems](/wiki/Category:Theorems "Category:Theorems")
\* [Game theory](/wiki/Category:Game\_theory "Category:Game theory")
\* [Utility](/wiki/Category:Utility "Category:Utility")
\* [John von Neumann](/wiki/Category:John\_von\_Neumann "Category:John von Neumann")
Hidden categories: \* [Articles with short description](/wiki/Category:Articles\_with\_short\_description "Category:Articles with short description")
\* [Short description is different from Wikidata](/wiki/Category:Short\_description\_is\_different\_from\_Wikidata "Category:Short description is different from Wikidata")
\* [Wikipedia articles needing clarification from March 2016](/wiki/Category:Wikipedia\_articles\_needing\_clarification\_from\_March\_2016 "Category:Wikipedia articles needing clarification from March 2016")


ICLR Safe ML Workshop Report

ICLR Safe ML Workshop Report
============================

![](https://futureoflife.org/wp-content/uploads/2019/06/cropped-pagodas-1024x267.jpg)PublishedJune 18, 2019AuthorViktoriya Krakovna
#### Contents

This year the ICLR conference hosted topic-based workshops for the first time (as opposed to a single track for workshop papers), and I co-organized the [Safe ML workshop](https://sites.google.com/corp/view/safeml-iclr2019/home?authuser=0). One of the main goals was to bring together near and long term safety research communities.

![](https://futureoflife.org/wp-content/uploads/2019/06/near-long-term.png)The workshop was structured according to a taxonomy that incorporates both near and long term safety research into three areas — specification, robustness, and assurance.

\*\*Specification:\*\* define the purpose of the system

\* Reward hacking
\* Side effects
\* Preference learning
\* Fairness

\*\*Robustness:\*\* design system to withstand perturbations

\* Adaptation
\* Verification
\* Worst-case robustness
\* Safe exploration

\*\*Assurance:\*\* monitor and control system activity

\* Interpretability
\* Monitoring
\* Privacy
\* Interruptibility

We had an invited talk and a contributed talk in each of the three areas.

Talks
-----

In the \*\*specification\*\* area, Dylan Hadfield-Menell spoke about [formalizing the value alignment problem](https://slideslive.com/38915783/formalizing-the-value-alignment-problem-in-ai) in the Inverse RL framework.

![](https://futureoflife.org/wp-content/uploads/2019/06/formalizing-value-alignment.png)

David Krueger [presented](https://slideslive.com/38915784/misleading-metaobjectives-and-hidden-incentives-for-distributional-shift) a [paper](https://drive.google.com/uc?export=download&id=1k93292JCoIHU0h6xVO3qmeRwLyOSlS4o) on hidden incentives for the agent to shift its task distribution in the meta-learning setting.

![](https://futureoflife.org/wp-content/uploads/2019/06/distributional-shift.png)

In the \*\*robustness\*\* area, Ian Goodfellow argued for [dynamic defenses against adversarial examples](https://slideslive.com/38915790/the-case-for-dynamic-defenses-against-adversarial-examples) and encouraged the research community to consider threat models beyond small perturbations within a norm ball of the original data point.

![](https://futureoflife.org/wp-content/uploads/2019/06/threat-models.png)

Avraham Ruderman [presented](https://slideslive.com/38915789/uncovering-surprising-behaviors-in-reinforcement-learning-via-worstcase-analysis) a [paper](https://drive.google.com/uc?export=download&id=1z\_d1EjvKWlh2L39xxGXYDWP1wQwnlNLQ) on worst-case analysis for discovering surprising behaviors (e.g. failing to find the goal in simple mazes).

![](https://futureoflife.org/wp-content/uploads/2019/06/mazes.png)

In the \*\*assurance\*\* area, Cynthia Rudin argued that interpretability doesn’t have to trade off with accuracy (especially in applications), and that it is helpful for solving research problems in all areas of safety.

![](https://futureoflife.org/wp-content/uploads/2019/06/interpretability-taxonomy.png)

Beomsu Kim [presented](https://slideslive.com/38915785/bridging-adversarial-robustness-and-gradient-interpretability) a [paper](https://drive.google.com/uc?export=download&id=1FUlKR07jf1VQb6M-4VVTxxu3oDkK0x-8) explaining why adversarial training improves the interpretability of gradients for deep neural networks.

![](https://futureoflife.org/wp-content/uploads/2019/06/gradient-interpretability.png)

Panels
------

The workshop panels discussed possible overlaps between different research areas in safety and research priorities going forward.

In terms of \*\*overlaps\*\*, the main takeaway was that advancing interpretability is useful for all safety problems. Also, adversarial robustness can contribute to value alignment – e.g. reward gaming behaviors can be viewed as a system finding adversarial examples for its reward function. However, there was a cautionary point that while near- and long-term problems are often similar, solutions might not transfer well between these areas (e.g. some solutions to near-term problems might not be sufficiently general to help with value alignment).

![](https://futureoflife.org/wp-content/uploads/2019/06/panel.jpg)

The \*\*research priorities\*\* panel recommended more work on adversarial examples with realistic threat models (as mentioned above), complex environments for testing value alignment (e.g. creating new structures in Minecraft without touching existing ones), fairness formalizations with more input from social scientists, and improving cybersecurity.

Papers
------

Out of the 35 accepted papers, 5 were on long-term safety / value alignment, and the rest were on near-term safety. Half of the near-term paper submissions were on adversarial examples, so the resulting pool of accepted papers was skewed as well: 14 on adversarial examples, 5 on interpretability, 3 on safe RL, 3 on other robustness, 2 on fairness, 2 on verification, and 1 on privacy. Here is a summary of the value alignment papers:

[Misleading meta-objectives and hidden incentives for distributional shift](https://drive.google.com/uc?export=download&id=1k93292JCoIHU0h6xVO3qmeRwLyOSlS4o) by Krueger et al shows that RL agents in a meta-learning context have an incentive to shift their task distribution instead of solving the intended task. For example, a household robot whose task is to predict whether its owner will want coffee could wake up its owner early in the morning to make this prediction task easier. This is called a ‘self-induced distributional shift’ (SIDS), and the incentive to do so is a ‘hidden incentive for distributional shift’ (HIDS). The paper demonstrates this behavior experimentally and shows how to avoid it.

![](https://futureoflife.org/wp-content/uploads/2019/06/coffee-robot.png)

[How useful is quantilization for mitigating specification-gaming?](https://drive.google.com/uc?export=download&id=13qAfOm8McRvXS33MCNH0ia4ApMIClZP9) by Ryan Carey introduces variants of several classic environments (Mountain Car, Hopper and Video Pinball) where the observed reward differs from the true reward, creating an opportunity for the agent to game the specification of the observed reward. The paper shows that a quantilizing agent avoids specification gaming and performs better in terms of true reward than both imitation learning and a regular RL agent on all the environments.

![](https://futureoflife.org/wp-content/uploads/2019/06/spec-gaming.png)

[Delegative Reinforcement Learning: learning to avoid traps with a little help](https://drive.google.com/uc?export=download&id=1xa7UpGGODl6mszNWkA4XQGPyeopsNuWu) by Vanessa Kosoy introduces an RL algorithm that avoids traps in the environment (states where regret is linear) by delegating some actions to an external advisor, and achieves sublinear regret in a continual learning setting. (Summarized in [Alignment Newsletter #57](http://eepurl.com/gtZ8DD))

![](https://futureoflife.org/wp-content/uploads/2019/06/trap.gif)

[Generalizing from a few environments in safety-critical reinforcement learning](https://drive.google.com/uc?export=download&id=1Q\_lGKZzJwc7h2f8oYgZDBAmRttMBkgCk) by Kenton et al investigates how well RL agents avoid catastrophes in new gridworld environments depending on the number of training environments. They find that both model ensembling and learning a catastrophe classifier (used to block actions) are helpful for avoiding catastrophes, with different safety-performance tradeoffs on new environments.

![](https://futureoflife.org/wp-content/uploads/2019/06/dist-shift-catastrophes.png)

[Regulatory markets for AI safety](https://drive.google.com/uc?export=download&id=1bFPiwLrZc7SQTMg2\_bW4gt0PaS5NyqOH) by Clark and Hadfield proposes a new model for regulating AI development where regulation targets are required to choose regulatory services from a private market that is overseen by the government. This allows regulation to efficiently operate on a global scale and keep up with the pace of technological development and better ensure safe deployment of AI systems. (Summarized in [Alignment Newsletter #55](http://eepurl.com/gp5MFP))

![](https://futureoflife.org/wp-content/uploads/2019/06/regulatory-markets.png)

The workshop got a pretty good turnout (around 100 people). Thanks everyone for participating, and thanks to our reviewers, sponsors, and my fellow organizers for making it happen!

![](https://futureoflife.org/wp-content/uploads/2019/06/audience-1250x630.jpg)

\*(Cross-posted from the [Deep Safety blog](https://vkrakovna.wordpress.com/2019/06/18/iclr-safe-ml-workshop-report/).)\*

Our contentRelated posts
-------------

If you enjoyed this, you also might like:[Our content](https://futureoflife.org/our-content/)![](https://futureoflife.org/wp-content/uploads/2023/06/Podcast-thumbnails-45-1-1024x576.jpg)#### [Dan Hendrycks on Why Evolution Favors AIs over Humans](https://futureoflife.org/podcast/dan-hendrycks-on-why-evolution-favors-ais-over-humans/)

June 8, 2023![](https://futureoflife.org/wp-content/uploads/2023/05/Podcast-thumbnails-1-1-1024x576.jpg)#### [Nathan Labenz on How AI Will Transform the Economy](https://futureoflife.org/podcast/nathan-labenz-on-how-ai-will-transform-the-economy/)

May 12, 2023![](https://futureoflife.org/wp-content/uploads/2023/03/FLi\_Banner-05-1024x576.jpg)#### [FAQs about FLI’s Open Letter Calling for a Pause on Giant AI Experiments](https://futureoflife.org/ai/faqs-about-flis-open-letter-calling-for-a-pause-on-giant-ai-experiments/)

Disclaimer: Please note that these FAQ’s have been prepared by FLI and do not necessarily reflect the views of the […]March 31, 2023


Research priorities for robust and beneficial artificial intelligence: an open letter

[All Open Letters](https://futureoflife.org/?page\_id=39741)Research Priorities for Robust and Beneficial Artificial Intelligence: An Open Letter
=====================================================================================

There is now a broad consensus that AI research is progressing steadily, and that its impact on society is likely to increase. The potential benefits are huge, since everything that civilization has to offer is a product of human intelligence. Because of the great potential of AI, it is important to research how to reap its benefits while avoiding potential pitfalls.Signatures11251PublishedOctober 28, 2015Click here to see this page in other languages: \*\*[Chinese](https://futureoflife.org/ai-open-letter-chinese/)  [![](https://futureoflife.org/wp-content/uploads/2016/06/china\_flag-e1464798319604.png)](https://futureoflife.org/ai-open-letter-chinese/) [German](https://futureoflife.org/ai-open-letter-german/)[![](https://futureoflife.org/wp-content/uploads/2016/06/Germany\_flag.jpg?x57718)](https://futureoflife.org/ai-open-letter-german/) [Japanese](https://futureoflife.org/ai-open-letter-japanese/) [![](https://futureoflife.org/wp-content/uploads/2016/02/red\_circle-1.jpg)](https://futureoflife.org/ai-open-letter-japanese/) [Russian](https://futureoflife.org/ai-open-letter-russian/) [![](https://futureoflife.org/wp-content/uploads/2016/02/Russian\_Flag.jpg?x56934)](https://futureoflife.org/ai-open-letter-russian/)\*\*

Artificial intelligence (AI) research has explored a variety of problems and approaches since its inception, but for the last 20 years or so has been focused on the problems surrounding the construction of intelligent agents - systems that perceive and act in some environment. In this context, "intelligence" is related to statistical and economic notions of rationality - colloquially, the ability to make good decisions, plans, or inferences. The adoption of probabilistic and decision-theoretic representations and statistical learning methods has led to a large degree of integration and cross-fertilization among AI, machine learning, statistics, control theory, neuroscience, and other fields. The establishment of shared theoretical frameworks, combined with the availability of data and processing power, has yielded remarkable successes in various component tasks such as speech recognition, image classification, autonomous vehicles, machine translation, legged locomotion, and question-answering systems.

As capabilities in these areas and others cross the threshold from laboratory research to economically valuable technologies, a virtuous cycle takes hold whereby even small improvements in performance are worth large sums of money, prompting greater investments in research. There is now a broad consensus that AI research is progressing steadily, and that its impact on society is likely to increase. The potential benefits are huge, since everything that civilization has to offer is a product of human intelligence; we cannot predict what we might achieve when this intelligence is magnified by the tools AI may provide, but the eradication of disease and poverty are not unfathomable. Because of the great potential of AI, it is important to research how to reap its benefits while avoiding potential pitfalls.

The progress in AI research makes it timely to focus research not only on making AI more capable, but also on maximizing the societal benefit of AI. Such considerations motivated the AAAI 2008-09 Presidential Panel on Long-Term AI Futures and other projects on AI impacts, and constitute a significant expansion of the field of AI itself, which up to now has focused largely on techniques that are neutral with respect to purpose. We recommend expanded research aimed at ensuring that increasingly capable AI systems are robust and beneficial: our AI systems must do what we want them to do. The attached [research priorities document](https://futureoflife.org/static/data/documents/research\_priorities.pdf) gives many examples of such research directions that can help maximize the societal benefit of AI. This research is by necessity interdisciplinary, because it involves both society and AI. It ranges from economics, law and philosophy to computer security, formal methods and, of course, various branches of AI itself.

In summary, we believe that research on how to make AI systems robust and beneficial is both important and timely, and that there are concrete research directions that can be pursued today.

\*If you have questions about this letter, please contact [Max Tegmark](mailto:max@futureoflife.org).\*

Signatories
-----------

[Click here](https://futureoflife.org/open-letter/ai-open-letter-signatories/) to view the full list of signatories.

To date, the open letter has been signed by over 8,000 people. The list of signatories includes:

### Prominent Signatories

\*\*Stuart Russell\*\*, Berkeley, Professor of Computer Science, director of the Center for Intelligent Systems, and co-author of the standard textbook Artificial Intelligence: a Modern Approach.

\*\*Tom Dietterich\*\*, Oregon State, President of AAAI, Professor and Director of Intelligent Systems

\*\*Eric Horvitz\*\*, Microsoft research director, ex AAAI president, co-chair of the AAAI presidential panel on long-term AI futures

\*\*Bart Selman\*\*, Cornell, Professor of Computer Science, co-chair of the AAAI presidential panel on long-term AI futures

\*\*Francesca Rossi\*\*, Padova & Harvard, Professor of Computer Science, IJCAI President and Co-chair of AAAI committee on impact of AI and Ethical Issues

\*\*Demis Hassabis\*\*, co-founder of DeepMind

\*\*Shane Legg\*\*, co-founder of DeepMind

\*\*Mustafa Suleyman\*\*, co-founder of DeepMind

\*\*Dileep George\*\*, co-founder of Vicarious

\*\*Scott Phoenix\*\*, co-founder of Vicarious

\*\*Yann LeCun\*\*, head of Facebook’s Artificial Intelligence Laboratory

\*\*Geoffrey Hinton\*\*, University of Toronto and Google Inc.

\*\*Yoshua Bengio\*\*, Université de Montréal

\*\*Peter Norvig\*\*, Director of research at Google and co-author of the standard textbook Artificial Intelligence: a Modern Approach

\*\*Oren Etzioni\*\*, CEO of Allen Inst. for AI

\*\*Guruduth Banavar\*\*, VP, Cognitive Computing, IBM Research

\*\*Michael Wooldridge\*\*, Oxford, Head of Dept. of Computer Science, Chair of European Coordinating Committee for Artificial Intelligence

\*\*Leslie Pack Kaelbling\*\*, MIT, Professor of Computer Science and Engineering, founder of the Journal of Machine Learning Research

\*\*Tom Mitchell\*\*, CMU, former President of AAAI, chair of Machine Learning Department

\*\*Toby Walsh\*\*, Univ. of New South Wales & NICTA, Professor of AI and President of the AI Access Foundation

\*\*Murray Shanahan\*\*, Imperial College, Professor of Cognitive Robotics

\*\*Michael Osborne\*\*, Oxford, Associate Professor of Machine Learning

\*\*David Parkes\*\*, Harvard, Professor of Computer Science

\*\*Laurent Orseau\*\*, Google DeepMind

\*\*Ilya Sutskever\*\*, Google, AI researcher

\*\*Blaise Aguera y Arcas\*\*, Google, AI researcher

\*\*Joscha Bach\*\*, MIT, AI researcher

\*\*Bill Hibbard\*\*, Madison, AI researcher

\*\*Steve Omohundro\*\*, AI researcher

\*\*Ben Goertzel\*\*, OpenCog Foundation

\*\*Richard Mallah\*\*, Cambridge Semantics, Director of Advanced Analytics, AI researcher

\*\*Alexander Wissner-Gross\*\*, Harvard, Fellow at the Institute for Applied Computational Science

\*\*Adrian Weller\*\*, Cambridge, AI researcher

\*\*Jacob Steinhardt\*\*, Stanford, AI Ph.D. student

\*\*Nick Hay\*\*, Berkeley, AI Ph.D. student

\*\*Jaan Tallinn\*\*, co-founder of Skype, CSER and FLI

\*\*Elon Musk\*\*, SpaceX, Tesla Motors

\*\*Steve Wozniak\*\*, co-founder of Apple

\*\*Luke Nosek\*\*, Founders Fund

\*\*Aaron VanDevender\*\*, Founders Fund

\*\*Erik Brynjolfsson\*\*, MIT, Professor at and director of MIT Initiative on the Digital Economy

\*\*Margaret Boden\*\*, U. Sussex, Professor of Cognitive Science

\*\*Martin Rees\*\*, Cambridge, Professor Emeritus of Cosmology and Astrophysics, Gruber & Crafoord laureate

\*\*Huw Price\*\*, Cambridge, Bertrand Russell Professor of Philosophy

\*\*Nick Bostrom\*\*, Oxford, Professor of Philosophy, Director of Future of Humanity Institute (Oxford Martin School)

\*\*Stephen Hawking\*\*, Director of research at the Department of Applied Mathematics and Theoretical Physics at Cambridge, 2012 Fundamental Physics Prize laureate for his work on quantum gravity

\*\*Luke Muehlhauser\*\*, Executive Director of Machine Intelligence Research Institute (MIRI)

\*\*Eliezer Yudkowsky\*\*, MIRI researcher, co-founder of MIRI (then known as SIAI)

\*\*Katja Grace\*\*, MIRI researcher

\*\*Benja Fallenstein\*\*, MIRI researcher

\*\*Nate Soares\*\*, MIRI researcher

\*\*Paul Christiano\*\*, Berkeley, Computer Science graduate student

\*\*Anders Sandberg\*\*, Oxford, Future of Humanity Institute researcher (Oxford Martin School)

\*\*Daniel Dewey\*\*, Oxford, Future of Humanity Institute researcher (Oxford Martin School)

\*\*Stuart Armstrong\*\*, Oxford, Future of Humanity Institute researcher (Oxford Martin School)

\*\*Toby Ord\*\*, Oxford, Future of Humanity Institute researcher (Oxford Martin School), Founder of Giving What We Can

\*\*Neil Jacobstein\*\*, Singularity University

\*\*Dominik Grewe\*\*, Google DeepMind

\*\*Roman V. Yampolskiy\*\*, University of Louisville

\*\*Vincent C. Müller\*\*, ACT/Anatolia College

\*\*Amnon H Eden\*\*, University Essex

\*\*Henry Kautz\*\*, University of Rochester

\*\*Boris Debic\*\*, Google, Chief History Officer

\*\*Kevin Leyton-Brown\*\*, University of British Columbia, Professor of Computer Science

\*\*Trevor Back\*\*, Google DeepMind

\*\*Moshe Vardi\*\*, Rice University, editor-in-chief of Communications of the ACM

\*\*Peter Sincak\*\*, prof. TU Kosice, Slovakia

\*\*Tom Schaul\*\*, Google DeepMind

\*\*Grady Booch\*\*, IBM Fellow

\*\*Alan Mackworth\*\*, Professor of Computer Science, University of British Columbia. Ex AAAI President

\*\*Andrew Davison\*\*, Professor of Robot Vision, Director of the Dyson Robotics Lab at Imperial College London

\*\*Daniel Weld\*\*, WRF / TJ Cable Professor of Computer Science & Engineering, University of Washington

\*\*Michael Witbrock\*\*, Cycorp Inc & AI4Good.org

\*\*Stephen L. Reed\*\*, ai-coin.com

\*\*Thomas Stone\*\*, Co-founder of PredictionIO

\*\*Dan Roth\*\*, University of Illinois, Editor in Chief of The Journal of AI Research (JAIR)

\*\*Babak Hodjat\*\*, Sentient Technologies

\*\*Vincent Vanhoucke\*\*, Google, AI researcher

\*\*Itamar Arel\*\*, Stanford University, Prof. of Computer Science

\*\*Ramon Lopez de Mantaras\*\*, Director of the Artificial Intelligence Research Institute, Spanish National Research Council

\*\*Antoine Blondeau\*\*, Sentient Technologies

\*\*George Dvorsky\*\*, Contributing Editor, io9; Chair of the Board, Institute for Ethics and Emerging Technologies

\*\*George Church\*\*, Harvard & MIT

\*\*Klaus-Dieter Althoff\*\*, University of Hildesheim, Professor of Artificial Intelligence; Head of Competence Center Case-Based Reasoning, German Research Center for Artificial Intelligence, Kaiserslautern; Editor-in-Chief German Journal on Artificial Intelligence

\*\*Christopher Bishop\*\*, Distinguished Scientist, Microsoft Research

\*\*Jen-Hsun Huang\*\*, NVIDIA CEO

[Close](http://)### How does verification work?

Verified signatures are those which we have taken one or more extra steps to confirm as legitimate:
• \*\*Direct contact\*\* - We have been in direct contact with this person to verify that they have signed the letter.
• \*\*Declaration URL\*\* - This person has made a public declaration of signing the open letter which can be viewed online.
All published signatures, ‘verified’ or otherwise, are subject to several forms of verification: email verification, spam and duplicate filters, and a review by a member of our data vetting team.

OPEN LETTERSRelated posts
-------------

If you enjoyed this, you also might like:[Our Open Letters](https://futureoflife.org/fli-open-letters/)Signatories31810#### [Pause Giant AI Experiments: An Open Letter](https://futureoflife.org/open-letter/pause-giant-ai-experiments/)

We call on all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.March 22, 2023Signatories998#### [Open Letter Against Reckless Nuclear Escalation and Use](https://futureoflife.org/open-letter/open-letter-against-reckless-nuclear-escalation-and-use/)

The abhorrent Ukraine war has the potential to escalate into an all-out NATO-Russia nuclear conflict that would be the greatest catastrophe in human history. More must be done to prevent such escalation.October 18, 2022SignatoriesClosed#### [Foresight in AI Regulation Open Letter](https://futureoflife.org/open-letter/foresight-in-ai-regulation-open-letter/)

The emergence of artificial intelligence (AI) promises dramatic changes in our economic and social structures as well as everyday life […]June 14, 2020Signatories276#### [Autonomous Weapons Open Letter: Global Health Community](https://futureoflife.org/open-letter/medical-lethal-autonomous-weapons-open-letter/)

Given our commitment to do no harm, the global health community has a long history of successful advocacy against inhumane weapons, and the World and American Medical Associations have called for bans on nuclear, chemical and biological weapons. Now, recent advances in artificial intelligence have brought us to the brink of a new arms race in lethal autonomous weapons.March 13, 2019


2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy | Global Catastrophic Risk Institute

![](http://gcrinstitute.org/wp-content/uploads/2021/01/AGI-2020-map-1024x508.png)
[View the paper “2020 Survey of Artificial General Intelligence Projects for Ethics, Risk, and Policy”](https://gcrinstitute.org/papers/055\_agi-2020.pdf)

In 2017, GCRI published [the first-ever survey](https://gcrinstitute.org/a-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/) of artificial general intelligence (AGI) research and development (R&D) projects for ethics, risk, and policy. This paper updates the 2017 survey. The 2020 survey features improved methodology, enabling it to find more projects than the 2017 survey and characterize them more precisely. The 2020 survey also evaluates how the landscape of AGI R&D projects has changed from 2017 to 2020.

AGI is AI that can reason across a wide range of domains. Most current AI R&D is narrow, but as the 2017 and 2020 surveys both document, there is a significant amount of dedicated AGI R&D. AGI is important because, if built, it could have major consequences. Depending on how it is designed and built, it may be able to help solve many of the world’s problems, including problems involving global catastrophic risk, or it could itself cause global catastrophe. Therefore, it is important to monitor AGI R&D and identify opportunities to orient it in better directions.

The 2017 and 2020 surveys characterize AGI R&D projects in terms of seven attributes:

 • The type of institution the project is based in

 • Whether the project publishes open-source code

 • Whether the project has military connections

 • The nation(s) that the project is based in

 • The project’s goals for its AGI

 • The extent of the project’s engagement with AGI safety issues

 • The overall size of the project

To accomplish this, the surveys use openly published information as found in scholarly publications, project websites, popular media articles, and other websites. The 2020 survey uses information from the 2017 survey as well as the past three years of the \*Journal of Artificial General Intelligence\*, the past three years of AGI conference proceedings (the 2017 survey covered prior content from the \*Journal of Artificial General Intelligence\* and the AGI conference proceedings), keyword searches in Google web search, Google Scholar, Crunchbase, GitHub, the authors’ prior knowledge, suggestions from readers of the 2017 survey, and additional literature and webpages identified via all of the above. The use of Crunchbase, GitHub, and reader suggestions is new to the 2020 survey.

Whereas the 2017 survey identified 45 AGI R&D projects spread across 30 countries, the 2020 survey finds that, in 2020, there are 72 AGI R&D projects spread across 37 countries. The 2020 further finds that in 2017, there were 70 AGI R&D projects spread across 36 countries. 57 of the projects active in 2017 remain active in 2020, with an additional 15 projects new to 2020. The projects vary widely in size, with the largest being over 100 times larger than the smallest as measured in terms of the number of project personnel.

Relative to the 2017 survey, the AGI R&D projects presented in the 2020 survey tend to be smaller, more geographically diverse, less open-source, less focused on intellectual goals, more focused on humanitarian goals, and more concentrated in private corporations.

The 2020 survey also finds that, from 2017 to 2020, there has been a decrease in academic projects, an increase in private corporation projects, an increase in projects stating humanitarian goals, a decrease in projects with military connections, and a decrease in projects based in the United States (though the US remains the dominant country in AGI R&D); all of these changes are relatively small compared to the differences between the 2017 and 2020 surveys.

The projects active in 2020 are diverse, with three major clusters: (1) corporate projects that are active on AGI safety and state that their goals are to benefit humanity, (2) academic projects that are not active on AGI safety and state that their goals are to advance the forefront of knowledge, and (3) small private corporations that are not active on AGI safety and state a range of different goals. Governments and nonprofits play relatively minor roles in AGI R&D. The 2020 survey continues to observe an absence of large government AGI R&D projects, including military projects. The small handful of projects with military connections mostly involve basic research. The data show no indication of militaries or other government divisions pursuing AGI R&D for major strategic purposes.

The data suggest the following conclusions:

\*\*Regarding ethics,\*\* the two most common goals are to benefit humanity and to advance knowledge. This is the same as in the 2017 survey, but in the 2020 survey, the order is reversed, with there now being more projects seeking to benefit humanity. The 2020 survey also finds a large increase in the number of corporate projects. These projects seldom state a goal of pursuing profit, but they may nonetheless have profit as a motivation.

\*\*Regarding risk,\*\* the proliferation of corporate projects relative to the 2017 survey heightens the concern that these projects could put profit ahead of safety and the public interest. Additionally, academic projects remain relatively inattentive to safety. On the other hand, many projects are active on safety. Additionally, the partial consensus on ethics, the concentration of projects in the US and its allies, and the various interconnections between different projects all suggest potential for cooperation on safety issues; these matters are unchanged from the 2017 survey.

\*\*Regarding policy,\*\* the proliferation of corporate projects suggests an important role for corporate governance and attention to the political economy of AGI R&D. The modest decline of academic projects suggests a smaller but still significant role for academic research policy. Additionally, as in the 2017 survey, international policy is facilitated by the concentration of projects in the US and its allies, though the preponderance of projects with open-source code complicates the political geography of AGI R&D. Finally, the absence of large government AGI R&D projects suggests that governments may be involved in AGI R&D primarily as regulators of private-sector R&D instead of as drivers of the R&D.

As with the 2017 survey, the 2020 survey has some limitations, meaning that the actual state of AGI R&D may differ from what is presented in the surveys. This is due to the fact that the surveys are based exclusively on openly published information. It is possible that some AGI R&D projects were missed by the surveys. Indeed, the 2020 survey documents many projects that were missed by the 2017 survey. Therefore, the number of projects identified in the 2020 survey should be taken as a lower bound. Furthermore, it is possible that projects’ actual attributes differ from those found in openly published information. For example, most corporate projects did not state the goal of profit, even though many presumably seek profit. Therefore, this study’s results should not be assumed to necessarily reflect the actual current state of AGI R&D. That said, the study nonetheless provides the most thorough description yet of AGI R&D in terms of ethics, risk, and policy.

This paper was recently covered in the media article [What Is AGI?](https://www.fierceelectronics.com/sensors/what-agi).

Academic citation:
Fitzgerald, McKenna, Aaron Boddy, and Seth D. Baum, 2020. [2020 survey of artificial general intelligence projects for ethics, risk, and policy](https://gcrinstitute.org/papers/055\_agi-2020.pdf). Global Catastrophic Risk Institute Technical Report 20-1.

 Tagged with [artificial general intelligence](https://gcrinstitute.org/tag/artificial-general-intelligence/)


Artificial Intelligence Needs Environmental Ethics | Global Catastrophic Risk Institute

![](http://gcrinstitute.org/wp-content/uploads/2021/06/ComputerLeaf\_Buckyball\_2021-06-04.jpg)
[View the paper “Artificial Intelligence Needs Environmental Ethics”](https://gcrinstitute.org/papers/059\_ai-environmental-ethics.pdf)

Artificial intelligence is an interdisciplinary topic. As such, it benefits from contributions from a wide range of disciplines. This short paper calls for greater contributions from the discipline of environmental ethics and presents several types of contributions that environmental ethicists can make.

First, environmental ethicists can raise the profile of the environmental dimensions of AI. For example, discussions of the ethics of autonomous vehicles have thus far focused mainly on “trolley problem” scenarios in which the vehicle must decide whom to harm in hypothetical crash scenarios. Less attention has been paid to the environmental impacts of autonomous vehicles, even though the environmental impacts are arguably much more important. Environmental ethicists could make the moral case for attention to this and to other environmental issues involving AI.

Second, environmental ethicists can help analyze novel ethical situations involving AI. These situations specifically involve artificial versions of phenomena that have long been studied in environmental ethics research. For example, AI and related technology could result in the creation of artificial life and artificial ecosystems. Environmental ethicists have often argued that “natural” life and ecosystems have important moral value. Perhaps the same reasoning would apply to artificial life and ecosystems, or perhaps it would not due to their artificiality. Environmental ethicists can help evaluate these sorts of novel ethical issues. Such work is especially important because existing work on AI ethics has focused more narrowly on issues centered on humans; environmental ethicists can help make the case for a broader scope.

Third, environmental ethicists can provide valuable perspectives on the future-orientation of certain AI issues. Within the communities of people working on AI issues, there is a divide between people focused on near-term AI issues and people focused on long-term AI issues. Global catastrophic risks associated with AI are often linked to the long-term AI issues. Similar debates exist on environmental issues, due to the long-term nature of major environmental issues such as global warming, natural resource depletion, and biodiversity loss. Environmental ethicists have made considerable progress on the ethics of the future, which can be applied to debates about AI.

The paper builds on prior GCRI research and experience as environmental ethicists working on AI. [Moral consideration of nonhumans in the ethics of artificial intelligence](https://gcrinstitute.org/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence) documents the tendency for work on AI ethics to focus on humans and calls for more robust attention to nonhumans. [Reconciliation between factions focused on near-term and long-term artificial intelligence](https://gcrinstitute.org/reconciliation-between-factions-focused-on-near-term-and-long-term-artificial-intelligence) describes the debate between those favoring attention to near-term AI issues and those favoring attention to long-term AI issues. [Artificial intelligence, systemic risks, and sustainability](https://gcrinstitute.org/artificial-intelligence-systemic-risks-and-sustainability) analyzes risks associated with near-term applications of AI in sectors related to environmental sustainability such as agriculture and forestry.

Academic citation:
Baum, Seth D. and Andrea Owe, 2023. [Artificial intelligence needs environmental ethics](https://gcrinstitute.org/papers/059\_ai-environmental-ethics.pdf). \*Ethics, Policy, & Environment\*, vol. 26, no. 1, pages 139-143, [DOI 10.1080/21550085.2022.2076538](https://doi.org/10.1080/21550085.2022.2076538).

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/), [ethics](https://gcrinstitute.org/tag/ethics/)


Collective Action on Artificial Intelligence: A Primer and Review | Global Catastrophic Risk Institute

![](http://gcrinstitute.org/wp-content/uploads/2021/07/Motion-earth3d-1\_Scr-4-1024x576.jpg)
[View the paper “Collective Action on Artificial Intelligence: A Primer and Review”](https://gcrinstitute.org/papers/058\_collective-action.pdf)

The development of safe and socially beneficial artificial intelligence (AI) will require collective action: outcomes will depend on the actions that many different people take. In recent years, a sizable but disparate literature has looked at the challenges posed by collective action on AI, but this literature is generally not well grounded in the broader social science literature on collective action. This paper advances the study of collective action on AI by providing a primer on the topic and a review of existing literature. It is intended to get an interdisciplinary readership up to speed on the topic, including social scientists, computer scientists, policy analysts, government officials, and other interested people.

The primer describes the theory of
collective action and relates it to different types of AI collective action
situations. A primary distinction is between situations in which individual and
collective interests diverge, as in the [prisoner’s
dilemma](https://en.wikipedia.org/wiki/Prisoner's\_dilemma) or adversarial AI competition, and in
which they converge, as in [coordination
problems](https://en.wikipedia.org/wiki/Coordination\_game) such as establishing common
platforms for AI. In general, collective action is easier to achieve when
interests converge, because individual actors pursuit of their own
self-interest can lead to outcomes that are worse for the group as a whole. The
primer also explains how AI collective action situations depend both on whether
the goods involved are [excludable](https://en.wikipedia.org/wiki/Excludability) or [rivalrous](https://en.wikipedia.org/wiki/Rivalry\_(economics)) and
whether they hinge on the action of a single actor or on some combination of
actors.

One major focus of the AI collective action literature identified in this paper are potentially dangerous AI race scenarios. AI races are not necessarily dangerous and might even hasten the arrival of socially beneficial forms of AI, but could be dangerous if individual actors’ interest in developing AI quickly diverges from the collective interest in ensuring that AI is safe and socially beneficial. The paper looks at both near-term and long-term AI races. The literature identified in this paper looks in particular at near-term races to develop military applications and at long-term AI races to develop advanced forms of AI such as artificial general intelligence and superintelligence. The two types of races are potentially related, since near-term races could affect the long-term development of AI.

Finally, the paper evaluates different
types of potential solutions to collective action problems. The collective
action literature identifies three major types of solution: government
regulation, private markets, and community self-organizing. All three types of
solution can advance AI collective action, but no single type is likely to
address the entire range of AI collective action problems. Instead of looking
for narrow, silver-bullet solutions it may be to pursue a mix of solutions that
AI collective action issues in different ways and at different scales.
Governance regimes should also account for other factors that could affect
collective action, such as the extent to which AI developers are transparent
about their technology.

AI collective action issues are
increasingly pressing. Collective action will be necessary to ensure that AI
serves the public interest rather than just the narrow private interests of
those who develop it. Collective action will also be necessary to ensure that
AI is developed with adequate safety measures and risk management protocols. Further
work could provide more detailed analysis and support practical progress on AI
collective action issues.

This paper has also been [summarized](https://montrealethics.ai/collective-action-on-artificial-intelligence-a-primer-and-review/) in the [AI Ethics Brief #71](https://brief.montrealethics.ai/p/beginners-guide-collective-action-accountability?token=eyJ1c2VyX2lkIjo3OTA2OTI2LCJwb3N0X2lkIjo0MDc1ODcxNCwiXyI6IkVDTlJPIiwiaWF0IjoxNjMwNTI4MjIyLCJleHAiOjE2MzA1MzE4MjIsImlzcyI6InB1Yi0yOTk5OSIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ.6tyOCFcObM3E0BJQMQKRmDnFcdqCTZcVPMphh\_pPlj8) of the Montreal AI Ethics Institute.

This paper extends GCRI’s interdisciplinary [research on AI](https://gcrinstitute.org/ai/). It builds on GCRI’s prior work on the governance of AI, particularly the papers [On the promotion of safe and socially beneficial artificial intelligence](https://gcrinstitute.org/on-the-promotion-of-safe-and-socially-beneficial-artificial-intelligence/) and [Lessons for artificial intelligence from other global risks](https://gcrinstitute.org/lessons-for-artificial-intelligence-from-other-global-risks/).

Academic citation:
de Neufville, Robert and Seth D. Baum, 2021. [Collective action on artificial intelligence: A primer and review](https://gcrinstitute.org/papers/058\_collective-action.pdf). \*Technology in Society\*, vol. 66, (August), article 101649, [DOI 10.1016/j.techsoc.2021.101649.](https://doi.org/10.1016/j.techsoc.2021.101649)

\*Image credit: [Volodymyr Goinyk](https://creativemarket.com/Goinyk)\*

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/)


![](http://gcrinstitute.org/wp-content/uploads/2020/07/UN-Patrick-Gruban.jpg)
[View the paper “Minimizing Global Catastrophic and Existential Risks from Emerging Technologies Through International Law”](https://gcrinstitute.org/papers/006\_international-law.pdf)

Mankind is rapidly developing “emerging technologies” in the fields of bioengineering, nanotechnology, and artificial intelligence that have the potential to solve humanity’s biggest problems, such as curing all disease, extending human life, or mitigating massive environmental problems like climate change. However, if these emerging technologies are misused or have an unintended negative effect, the consequences could be enormous, potentially resulting in serious, global damage to humans (known as “global catastrophic harm”) or severe, permanent damage to the Earth—including, possibly, human extinction (known as “existential harm”). The chances of a global catastrophic risk or existential risk actually materializing are relatively low, but mankind should be careful when a losing gamble means massive human death and irreversible harm to the planet. While international law has become an important source of global regulation for other global risks like climate change and biodiversity loss, emerging technologies do not fall neatly within existing international regimes, and thus any country is more or less free to develop these potentially dangerous technologies without practical safeguards that would curtail the risk of a catastrophic event. In light of these problems, this paper serves to discuss the risks associated with bioengineering, nanotechnology, and artificial intelligence; review the potential of existing international law to regulate these emerging technologies; and propose an international regulatory regime that would put the international world in charge of ensuring that low-probability, high-risk disasters never materialize.

Academic citation:
Wilson, Grant S., 2013. [Minimizing global catastrophic and existential risks from emerging technologies through international law.](https://gcrinstitute.org/papers/006\_international-law.pdf) \*Virginia Environmental Law Journal\*, vol. 31, no. 2, pages 307-364.

\*Image credit: [Patrick Gruban](https://www.flickr.com/photos/19473388@N00/336920038)\*

---

\*This blog post was published on 28 July 2020 as part of a website overhaul and backdated to reflect the time of the publication of the work referenced here.\*

 Tagged with [emerging technologies](https://gcrinstitute.org/tag/emerging-technologies/), [international law](https://gcrinstitute.org/tag/international-law/)


Moral Consideration of Nonhumans in the Ethics of Artificial Intelligence | Global Catastrophic Risk Institute

![](http://gcrinstitute.org/wp-content/uploads/2021/06/ComputerLeaf\_Buckyball\_2021-06-04.jpg)
[View the paper “Moral Consideration of Nonhumans in the Ethics of Artificial Intelligence”](https://link.springer.com/article/10.1007/s43681-021-00065-0)

In the ethics of artificial intelligence, a major theme is the challenge of aligning AI to human values. This raises the question of the role of nonhumans. Indeed, AI can profoundly affect the nonhuman world, including nonhuman animals, the natural environment, and the AI itself. Given that large parts of the nonhuman world are already under immense threats from human affairs, there is reason to fear potentially catastrophic consequences should AI R&D fail to account for nonhumans, for example with AI systems for industrial and commercial infrastructure, or future artificial general intelligence (AGI). This paper documents the state of attention to nonhumans within the field of AI ethics and presents an argument for giving nonhumans adequate attention.

The paper specifically examines
the extent to which nonhumans are given moral consideration in AI ethics, and
the extent to which they should be. Moral consideration of nonhumans means
actively valuing nonhumans for their own sake—in philosophy terms,
intrinsically valuing them. Unfortunately, the paper finds that most work in AI
ethics ignore nonhumans, or value nonhumans only for the effects they have on
humans. This leaves apt opportunity for the development and use of AI that
adversely impacts nonhumans.

The paper documents moral consideration of nonhumans in academic AI ethics research, statements of AI ethics principles, [AGI R&D projects](https://gcrinstitute.org/2020-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/), and select initiatives to design, build, apply, and govern AI. Aside from a line of research on the moral status of AI, the field of AI ethics generally fails to give moral consideration to nonhumans: The paper finds no attention to nonhumans in 76 of 84 sets of AI ethics principles surveyed by [Jobin et al.](https://www.nature.com/articles/s42256-019-0088-2), 40 of 45 AGI R&D projects surveyed by [Baum](https://gcrinstitute.org/a-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/), 38 of 44 chapters in the [\*Oxford Handbook of Ethics of AI\*](https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/9780190067397.001.0001/oxfordhb-9780190067397), and 13 of 17 chapters in the anthology [\*Ethics of Artificial Intelligence\*](https://global.oup.com/academic/product/ethics-of-artificial-intelligence-9780190905033). In the latter two, any dedicated attention is on the moral status of AI. No other types of nonhumans are given dedicated attention.

More could be done. The [Microsoft
AI for Earth](https://www.microsoft.com/en-us/ai/ai-for-earth) program is a good example of AI used in ways that benefit
nonhumans. It supports several programs for environmental protection and
biodiversity conservation that give explicit moral consideration to nonhumans,
including [Wild
Me](https://news.microsoft.com/2018/06/14/wild-me-joins-ai-for-earth/),
[eMammal](https://www.microsoft.com/en-us/ai/ai-for-earth-emammal), [NatureServe](https://www.microsoft.com/en-us/ai/ai-for-earth-natureserve), and [Zamba
Cloud](https://www.microsoft.com/en-us/ai/ai-for-earth-zamba-cloud).
Other AI groups could run similar programs. Within AI ethics research, the
paper outlines ideas for nonhuman algorithmic bias, such as by applying [ecolinguistics](https://en.wikipedia.org/wiki/Ecolinguistics) to biases in
natural language processing. While algorithmic bias is a major topic in AI
ethics, other literature has focused on social biases, but research in
ecolinguistics show that English—the primary language for AI system design—contains
biases in favor of humans over nonhumans.

Given the limited moral
consideration of nonhumans in the current field of AI ethics, the paper argues
for more consistent and extensive moral consideration of nonhumans. The
argument draws on concepts of ontological and ethical anthropocentrism as
developed in environmental ethics. Humans are members of the animal kingdom and
part of nature, and there is no sound basis for restricting moral consideration
exclusively to humans. There are important questions of how much moral
consideration to give to nonhumans relative to humans. The paper sets these
questions aside to call for a more basic improvement in moral consideration to
nonhumans across AI ethics.

This paper extends GCRI’s interdisciplinary [research on AI](https://gcrinstitute.org/ai/). It builds on prior GCRI research on AI ethics, especially the paper [Social Choice Ethics in Artificial Intelligence](https://gcrinstitute.org/social-choice-ethics-in-artificial-intelligence/). It uses ethics data compiled in our [2017](https://gcrinstitute.org/a-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/) and [2020](https://gcrinstitute.org/2020-survey-of-artificial-general-intelligence-projects-for-ethics-risk-and-policy/) surveys of AGI R&D projects, especially project goals. It also continues our tradition of applying the rich body of environmental research to new AI issues, as previously done in our papers [On the Promotion of Safe and Socially Beneficial Artificial Intelligence](https://gcrinstitute.org/on-the-promotion-of-safe-and-socially-beneficial-artificial-intelligence/) and [Lessons for artificial intelligence from other global risks](https://gcrinstitute.org/lessons-for-artificial-intelligence-from-other-global-risks/).

This paper has also been [summarized](https://montrealethics.ai/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence/) in the [AI Ethics Brief #73](https://brief.montrealethics.ai/p/deepfake-voices-embedding-values-bravery?token=eyJ1c2VyX2lkIjo3OTA2OTI2LCJwb3N0X2lkIjo0MTI2MTU2MSwiXyI6ImdSclZwIiwiaWF0IjoxNjMxNjUwMjU5LCJleHAiOjE2MzE2NTM4NTksImlzcyI6InB1Yi0yOTk5OSIsInN1YiI6InBvc3QtcmVhY3Rpb24ifQ.jJUM6lOMgU4U9eRB8CDDAqW27zZG85YoDUmYh6rSwNs) of the Montreal AI Ethics Institute, and in the [blog](https://mahb.stanford.edu/blog/moral-consideration-of-nonhumans-in-the-ethics-of-artificial-intelligence/) of [Stanford MAHB](https://mahb.stanford.edu/). It is also included in the 2022 [The AI Ethics Report](https://montrealethics.ai/volume6/) and is discussed in the MEDIUM article [“Is 2022 the Year that AI Ethics Takes Sustainability Seriously?”](https://josh-gellers.medium.com/is-2022-the-year-that-ai-ethics-takes-sustainability-seriously-8a10953105e9).

Academic citation:
Owe, Andrea and Seth D. Baum, 2021. [Moral consideration of nonhumans in the ethics of artificial intelligence](https://link.springer.com/article/10.1007%2Fs43681-021-00065-0). \*AI & Ethics,\* vol. 1, no. 4 (November), pages 517-528, [DOI 10.1007/s43681-021-00065-0](https://dx.doi.org/10.1007/s43681-021-00065-0). [ReadCube](https://rdcu.be/cl2Zv).

\*Image credit: Buckyball
Design\*

 Tagged with [artificial intelligence](https://gcrinstitute.org/tag/artificial-intelligence/), [ethics](https://gcrinstitute.org/tag/ethics/)


AI Safety Open Problems

Created: 2018-11-08 | Updated: 2019-11-02 | Suggestions: please make suggestions directly in this Doc | List maintainer: Mati Roy ([contact@matiroy.com](mailto:contact@matiroy.com))

AI Safety Open Problems

Technical AGI safety research outside AI: [https://forum.effectivealtruism.org/posts/2e9NDGiXt8PjjbTMC/technical-agi-safety-research-outside-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/2e9NDGiXt8PjjbTMC/technical-agi-safety-research-outside-ai&sa=D&source=editors&ust=1689706826553344&usg=AOvVaw2wC58mX6Qz2pXHL6mOOH9e)

Concrete problems in AI safety: [https://arxiv.org/abs/1606.06565](https://www.google.com/url?q=https://arxiv.org/abs/1606.06565&sa=D&source=editors&ust=1689706826554047&usg=AOvVaw3wIh5im83NVoSfttPv1SoZ)

MIRI: Agent Foundations for Aligning Superintelligence with Human Interests: [https://intelligence.org/files/TechnicalAgenda.pdf](https://www.google.com/url?q=https://intelligence.org/files/TechnicalAgenda.pdf&sa=D&source=editors&ust=1689706826554551&usg=AOvVaw3uDiOS2ilv-Q1XToZGh39t)

MIRI: Alignment for Advanced Machine Learning Systems: [https://intelligence.org/files/AlignmentMachineLearning.pdf](https://www.google.com/url?q=https://intelligence.org/files/AlignmentMachineLearning.pdf&sa=D&source=editors&ust=1689706826555038&usg=AOvVaw1GOtuwSSGzZ5pb1jHHyPck)

Research Priorities for Robust and Beneficial Artificial Intelligence: [ttps://arxiv.org/pdf/1602.03506.pdf](https://www.google.com/url?q=https://arxiv.org/pdf/1602.03506.pdf&sa=D&source=editors&ust=1689706826555496&usg=AOvVaw1yzJZUj2X2QrZdi4ayVXAa)

Luke Muehlhauser: How to study superintelligence strategy: [http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/](https://www.google.com/url?q=http://lukemuehlhauser.com/some-studies-which-could-improve-our-strategic-picture-of-superintelligence/&sa=D&source=editors&ust=1689706826556085&usg=AOvVaw0tHLsd8tdZkyIVS7sQE7J8)

Andrew Critch: Abstract open problems in AI alignment: [http://acritch.com/abstract-open-problems/](https://www.google.com/url?q=http://acritch.com/abstract-open-problems/&sa=D&source=editors&ust=1689706826556763&usg=AOvVaw0MoHIkSOPMCsR0ABt7se2n)

Foundational Research Institute: Open Research Questions: [https://foundational-research.org/open-research-questions/](https://www.google.com/url?q=https://foundational-research.org/open-research-questions/&sa=D&source=editors&ust=1689706826557395&usg=AOvVaw2Rk7WKKNtdUOft7bFrSjfC)

AI Impacts: List of multipolar research projects: [https://aiimpacts.org/multipolar-research-projects/](https://www.google.com/url?q=https://aiimpacts.org/multipolar-research-projects/&sa=D&source=editors&ust=1689706826557969&usg=AOvVaw26a9nbWT8glsD3G3PlyftT)

AI Impacts: Promising research projects: [https://aiimpacts.org/promising-research-projects/](https://www.google.com/url?q=https://aiimpacts.org/promising-research-projects/&sa=D&source=editors&ust=1689706826558479&usg=AOvVaw2BBlLN76VPufAggTX00\_jQ)

AI Impacts: Research Problems: [https://aiimpacts.org/category/research-problems/](https://www.google.com/url?q=https://aiimpacts.org/category/research-problems/&sa=D&source=editors&ust=1689706826558972&usg=AOvVaw2ZKOFhyH31v\_Hfd4ikJk\_B)

Effective thesis in computer science: [http://effectivethesis.com/theses/?discipline=computer+science](https://www.google.com/url?q=http://effectivethesis.com/theses/?discipline%3Dcomputer%2Bscience&sa=D&source=editors&ust=1689706826559505&usg=AOvVaw2DlrL4wSD13Gu640Wjyjss)

Other ideas in the comment section here: [http://effective-altruism.com/ea/18p/concrete\\_project\\_lists/](https://www.google.com/url?q=http://effective-altruism.com/ea/18p/concrete\_project\_lists/&sa=D&source=editors&ust=1689706826560042&usg=AOvVaw1zVo9dyc5JdtAOXDEekM5d)

Other ideas about Ryan Carey again: Improving long-run civilisational robustness: [http://effective-altruism.com/ea/xg/improving\\_longrun\\_civilisational\\_robustness/](https://www.google.com/url?q=http://effective-altruism.com/ea/xg/improving\_longrun\_civilisational\_robustness/&sa=D&source=editors&ust=1689706826560583&usg=AOvVaw2Qg82H3fHiuHlfkVcwAp2T)

Problems in AI Alignment that philosophers could potentially contribute to: [https://www.lesswrong.com/posts/rASeoR7iZ9Fokzh7L/problems-in-ai-alignment-that-philosophers-could-potentially](https://www.google.com/url?q=https://www.lesswrong.com/posts/rASeoR7iZ9Fokzh7L/problems-in-ai-alignment-that-philosophers-could-potentially&sa=D&source=editors&ust=1689706826561145&usg=AOvVaw0pxyD9bsdU9kC-nFsP7Fw8)

Cognitive Science/Psychology As a Neglected Approach to AI Safety: [https://forum.effectivealtruism.org/posts/WdMnmmqqiP5zCtSfv/cognitive-science-psychology-as-a-neglected-approach-to-ai](https://www.google.com/url?q=https://forum.effectivealtruism.org/posts/WdMnmmqqiP5zCtSfv/cognitive-science-psychology-as-a-neglected-approach-to-ai&sa=D&source=editors&ust=1689706826561712&usg=AOvVaw3HW39gpkfCeLItmEJIowJ5) (related: [https://ought.org/](https://www.google.com/url?q=https://ought.org/&sa=D&source=editors&ust=1689706826562018&usg=AOvVaw1YuA02ZALnDhwuv5dFTIhn))

Machine Learning Projects on IDA: [https://www.alignmentforum.org/posts/Y9xD78kufNsF7wL6f/machine-learning-projects-on-ida](https://www.google.com/url?q=https://www.alignmentforum.org/posts/Y9xD78kufNsF7wL6f/machine-learning-projects-on-ida&sa=D&source=editors&ust=1689706826562579&usg=AOvVaw0CcThmXftK9MzlACLInU0g)

Addendum

Landscape of current work on potential risks from advanced AI: [https://docs.google.com/document/d/16Te6HnZN2OEviYFA-42Tf9Pal\\_Idovtgr5Y1RGEPW\\_g/](https://www.google.com/url?q=https://docs.google.com/document/d/16Te6HnZN2OEviYFA-42Tf9Pal\_Idovtgr5Y1RGEPW\_g/&sa=D&source=editors&ust=1689706826563327&usg=AOvVaw1pk957VS-ffiqOTGhECS9r)