On the promotion of safe and socially beneficial artificial intelligence

OPEN FORUM
On the promotion of safe and socially beneﬁcial artiﬁcial
intelligence
Seth D. Baum1
Received: 29 July 2016 / Accepted: 20 September 2016
/C211Springer-Verlag London 2016
Abstract This paper discusses means for promoting artiﬁ-
cial intelligence (AI) that is designed to be safe and beneﬁcial
for society (or simply ‘‘beneﬁcial AI’’). The promotion of
beneﬁcial AI is a social challenge because it seeks to moti-
vate AI developers to choose beneﬁcial AI designs. Cur-
rently, the AI ﬁeld is focused mainly on building AIs that are
more capable, with little regard to social impacts. Two types
of measures are available for encouraging the AI ﬁeld to shift
more toward building beneﬁcial AI. Extrinsic measures
impose constraints or incentives on AI researchers to induce
them to pursue beneﬁcial AI even if they do not want to.
Intrinsic measures encourage AI researchers to want to
pursue beneﬁcial AI. Prior research focuses on extrinsic
measures, but intrinsic measures are at least as important.
Indeed, intrinsic factors can determine the success of
extrinsic measures. Efforts to promote beneﬁcial AI must
consider intrinsic factors by studying the social psychology
of AI research communities.
Keywords Artiﬁcial intelligence /C1Beneﬁcial
artiﬁcial intelligence /C1Artiﬁcial intelligence safety /C1
Social psychology
1 Introduction
The challenge of building technologies that are safe and
beneﬁcial for society is really two challenges in one. There
is the technical challenge of developing safe and beneﬁcialtechnology designs, and there is the social challenge of
ensuring that such designs are used. The two challenges are
interrelated. Motivating technologists to pursue safe and
beneﬁcial designs is itself a social challenge. Furthermore,
motivating people to use safe and beneﬁcial designs is
made easier when the designs also have other attractive
features such as low cost and ease of use; creating these
features is a technical challenge.
This paper is concerned with the social challenge.
Speciﬁcally, the paper examines a range of approaches to
motivating technologists to pursue safe and beneﬁcial
technology designs. The paper focuses on artiﬁcial intel-
ligence (AI) technologies, including both near-term AI and
the proposed future ‘‘strong’’ or ‘‘superintelligent’’ AI that
some posit could bring extreme social beneﬁts or harms
depending on its design. Much of the paper’s discussion
also applies to other technologies.
That AI has signiﬁcant social impacts is now beyond
question. AI is now being used in ﬁnance, medicine, mil-
itary, transportation, and a range of other critical sectors.
The impact is likely to grow over time as new technologies
are adopted, such as autonomous vehicles and lethal
autonomous weapons (unless the latter are banned or
heavily restricted). The prospects for strong AI are con-
troversial; this paper takes the position that the stakes are
sufﬁciently high that it warrants careful attention even if
the probability of achieving it appears to be low. Regard-
less, while the paper is motivated in part by the risk of
strong AI, the insights are more general.1
For brevity, the paper uses the term ‘‘beneﬁcial AI’’ to
refer to AI that is safe and beneﬁcial for society. It also
uses the term ‘‘promoting beneﬁcial AI’’ to refer to efforts
 &Seth D. Baum
seth@gcrinstitute.org
1Global Catastrophic Risk Institute, PO Box 40364,
Washington, DC 20016, USA1For perspectives on near-term AI impacts, see, e.g., Lin et al.
(2011 ). For strong AI, see e.g., Eden et al. ( 2013 ).
123AI & Soc
DOI 10.1007/s00146-016-0677-0
to encourage technologists to design and build beneﬁcial
AI, or to have them avoid designing and building AI that is
not beneﬁcial. The technologists include AI researchers/
designers/developers (the paper uses these terms more orless interchangeably) as well as adjacent personnel in
management, business development, etc.
The paper’s implicit value judgment is that AI should be
built so as to have net beneﬁts for the whole of society—or,
in the face of uncertainty, net expected beneﬁts. This is to
say that AI should not be built just for the sake of making it
more capable or more intellectually interesting. Also, AI
should not be built for the beneﬁt of its builders if thiscomes at the expense of society as a whole. These positions
may seem to cut against ideals of academic freedom,
intellectual progress, and capitalist entrepreneurship. Thepaper takes the position that these ideals are only worth
pursuing to the extent that doing so beneﬁts society.
Were it the case that the ﬁeld of AI was already
focused on beneﬁcial design, efforts to promote it would
be unnecessary. Unfortunately, this is not the case. The
ﬁeld is largely focused on building systems that are morecapable, regardless of whether this capability is used for
social good. This tendency and the need to shift it are
articulated, for example, by distinguished AI researcherStuart Russell:
I think the right approach is to build the issue [ben-
eﬁcial AI] directly into how practitioners deﬁne whatthey do. No one in civil engineering talks about
‘‘building bridges that don’t fall down.’’ They just
call it ‘‘building bridges.’’ Essentially all fusionresearchers work on containment as a matter of
course; uncontained fusion reactions just aren’t use-
ful. Right now we have to say ‘‘AI that is probably
beneﬁcial,’’ but eventually that will just be called
‘‘AI.’’ [We must] redirect the ﬁeld away from itscurrent goal of building pure intelligence for its own
sake, regardless of the associated objectives and their
consequences (Bohannon 2015 :252).
This paper takes on the challenge of how to shift the AI
ﬁeld toward greater emphasis on social impacts. The paper
reviews and critiques existing proposals for promotingbeneﬁcial AI and lays out a wider portfolio of techniques.
A core criticism is that existing proposals neglect human
psychology: They seek to inﬂuence AI researchers withoutthinking carefully about how AI researchers are inﬂuenced.
Neglect of human psychology limits the portfolio of tech-
niques that get considered for promoting beneﬁcial AI andreduces the effectiveness of those techniques that are
considered. In some cases, measures taken in ignorance of
human psychology can even backﬁre, resulting in lessbeneﬁcial AI than would have existed without any mea-
sures taken.Broadly speaking, there are two types of measures for
promoting beneﬁcial AI. Extrinsic measures are imposed
on AI designers from the outside so that they adopt bene-
ﬁcial designs even if they do not want to. These measuresinclude constraints that require or forbid certain designs,
incentives to encourage or discourage certain designs, and
compliance measures to make sure that constraints orincentives are being followed. Intrinsic measures are cul-
tivated within AI designers so that they want to adopt
beneﬁcial designs. These measures include the cultivation
of social norms and the framing of communications. There
can also be intrinsic effects of extrinsic measures, such aswhen a technology ban sparks backlash, making designers
less interested in adopting beneﬁcial designs. Extrinsic and
intrinsic measures are discussed in Sects. 2and 3,
respectively.
2
Prior discussions of the promotion of beneﬁcial AI focus
almost exclusively on extrinsic measures.3However, both
types of measures can help. Indeed, strategies based purely
on extrinsic measures run a signiﬁcant risk of having no net
effect or even being counterproductive. As this paper dis-cusses, the success of extrinsic measures often depends
heavily on intrinsic factors. Meanwhile, pure intrinsic
strategies can be quite effective, as can hybrid extrinsic–intrinsic strategies. The bottom line is that the promotion of
beneﬁcial AI demands attention to human psychology.
2 Extrinsic measures
2.1 Constraints
Constraints are perhaps the simplest means of promoting
beneﬁcial AI, and the most simplistic. The logic is direct: If
a design feature is beneﬁcial, require it; if it is harmful, ban
it. A ban on dangerous AI technologies is implicit in Joy’s(2000 ) call for relinquishment of dangerous AI, and it is
explicit in other work (Posner 2004 ; Wilson 2013 ; Yam-
polskiy and Fox 2013 ). Requirements for beneﬁcial AI
designs are less common in discussions of AI. Require-
ments could be used to insist that AI developers adopt
certain beneﬁcial designs such as veriﬁcation, validity,security, and control (Russell et al. 2015 ) and avoiding
negative side effects, avoiding reward hacking, scalable
oversight, safe exploration, and robustness to distributionalshift (Amodei et al. 2016 ).
2On the extrinsic/intrinsic distinction, see e.g., Markowitz and Sharif
(2012 :246) and references therein.
3See Sotala and Yampolskiy ( 2014 , Sect. 3) for a review in the
context of strong AI. Russell et al. ( 2015 ) also discuss a range of
predominantly extrinsic measures. A notable exception to the focuson extrinsic measures is Russell’s emphasis on shifting ‘‘howpractitioners deﬁne what they do’’ (Bohannon 2015 :252).AI & Soc
123
When constraints work, they guarantee that AI designs
are beneﬁcial. However, they also limit the freedom and
ﬂexibility of AI designers. This can provoke backlash by
AI designers, which is one example of an intrinsic effect ofextrinsic measures (Sect. 2.4). Even without backlash,
enacting constraints can require extensive institutional and
political changes, which makes them difﬁcult toimplement.
Constraints pose other challenges as well. One is that
unless they are carefully designed, they can unwittingly
constrain the wrong features, resulting in AI that is less
beneﬁcial. Designing successful AI constraints can thusrequire close interaction between AI experts and policy
makers. A related issue is that constraints may need to be
constantly updated as AI technology evolves. An AI designattribute that was harmful in early AI may be beneﬁcial in
later AI, and vice versa. New design attributes will also
emerge; these could merit new constraints. One potentialsolution is to phrase constraints in more general terms
(Moses 2007 ); for AI, this could mean requiring AI
designers to select the most beneﬁcial available design.Such an approach makes constraints more durable as AI
technology evolves, but it comes at the expense of making
it more difﬁcult to verify compliance.
2.2 Incentives
Incentives are the primary extrinsic alternative to con-
straints. Unlike constraints, incentives let AI developers
keep the freedom to pursue whatever designs they desire.Incentives act by changing the rewards or penalties for
speciﬁc designs, so as to push developers in different
design directions. The AI literature has focused mainly onmonetary incentives, such as by offering funding for ben-
eﬁcial AI research (McGinnis 2010 ) or by making AI
companies pay compensation when found liable for theconsequences of harmful AI (Gurney 2013 ). However,
incentives can also take on other forms, such as social
praise/scorn or professional advancement/sanction fordeveloping beneﬁcial/harmful AI.
Incentives hold several advantages over constraints. By
giving AI designers more freedom, they are less likely toprovoke backlash, which can make them easier to imple-
ment.
4Policy makers can also avoid the need to identify
beneﬁcial design attributes by applying incentives tocompleted technologies, as in liability schemes (which
impose penalties for AIs that turn out to be harmful) and inprize competitions (which can offer rewards for AIs that
turn out to be beneﬁcial).
5
The core disadvantage of incentives is that they do not
guarantee that beneﬁcial AI designs would be chosen. AnAI developer could simply choose to forgo the reward or
pay the penalty and continue to develop harmful AI. The
logical response to this is to strengthen the incentive,though this can provoke more backlash and can even erode
the distinction between incentives and constraints. Indeed,
a constraint could be deﬁned as an incentive with an inﬁ-
nite or maximal reward/penalty.
2.3 Compliance
Constraints and incentives are generally built on the pre-
mise that AI designers do not want to choose beneﬁcial
designs. Otherwise, they would not need to be constrained
or incentivized. AI designers thus have reason to avoidcomplying with the constraint or the incentive. When this
is the case, mechanisms for achieving compliance are
needed, including mechanisms for monitoring for non-compliance and mechanisms for enforcing penalties for
noncompliance.
A simple approach to monitoring is to require AI groups
to submit research proposals to review boards prior to
conducting the research. AI review boards could be anal-
ogous to the review boards that already exist at manyuniversities and other institutions for reviewing medical
and social science research (Yampolskiy and Fox 2013).
Existing review boards are focused mainly on harms thatcould be caused by the conduct of the research, in partic-
ular through abuse of human research subjects. AI review
boards would need an expanded scope that includes thesocietal impacts of the products of research. Such an
expansion would be in line with a more general expansion
of research ethics to include ethical assumptions embeddedwithin the research (such as ethical positions implicit in AI
objective functions) and ethical aspects of the societal
impacts of research (Schienke et al. 2009 ,2011 ).
One challenge for the review boards proposal is that
some AI groups may not be at institutions that have review
boards and thus could go undetected. Hard-to-monitorgroups can include private companies, especially startup
companies, and groups in unregulated countries. Indeed,
there is some concern that national AI regulations couldsimply push AI research to unregulated countries. This
problem can be addressed via international AI treaties
(Posner 2004 ; Wilson 2013 ), though this is easier said than
4Incentives can nonetheless provoke signiﬁcant backlash. For
example, in the United States, environmentalists have long pursued
incentive-based policies such as taxes on pollution in order to appealto industry interests that do not want constraints, yet industry has beenlargely successful at avoiding these incentive-based policies.5Incentives for completed technologies are less relevant for AIs that
could be catastrophic because there may be no penalty that couldadequately compensate for damages and, in the extreme case, no onealive to process the penalty.AI & Soc
123
done. Another approach in some AI monitoring proposals
is to implement a draconian mass surveillance regime in
order to ﬁnd any harmful AI group wherever they are (e.g.,
Shulman 2009 ). Sufﬁce to say, such surveillance poses
extreme problems for privacy, intellectual property, and
trustful geopolitical relations. It is a downside of extrinsic
measures that such problematic surveillance mechanismswould even be considered.
If monitoring succeeds and harmful AI groups are
identiﬁed, the next step is to enforce whatever penalty is to
be applied.
6Enforcement should in general be less of a
challenge than monitoring because AI groups have limitedmeans of resisting penalties. Government penalties can be
imposed through the threat or application of force. Insti-
tutional penalties can be imposed via the threat or appli-cation of measures such as ﬁring noncompliant personnel.
These sorts of actions could succeed at achieving compli-
ance to extrinsic measures, but, in addition to beingintrinsically regrettable (i.e., one should not want AI
developers to lose their jobs or suffer physical harm), they
can also alienate AI developers, provoke backlash, andmotivate them to relocate to unregulated places. The net
effect of the typical extrinsic measure is to unwittingly
create an antagonistic relationship between AI developersand those who seek beneﬁcial AI, which makes beneﬁcial
AI more difﬁcult to achieve. The essential solution to this
predicament is to consider intrinsic factors, i.e., the psy-chology of AI developers.
2.4 Intrinsic aspects of extrinsic measuresConsider two different extrinsic measures: a ban on ﬂag
burning and a requirement that dog owners cleanup aftertheir dogs. Flag burning is legal in several countries,
including the USA. The USA has repeatedly considered a
constitutional amendment to ban ﬂag burning. Such a banhas never passed, but here is one analysis of what would
happen if it did:
Few people have burned the American ﬂag in recent
years, and it is reasonable to suppose that a consti-
tutional amendment making it possible to criminalizeﬂag burning would have among its principal conse-
quences a dramatic increase in annual acts of ﬂag
burning. In fact, adopting a constitutional amendmentmay be the best possible way to promote the inci-
dence of ﬂag burning (Sunstein 1996 :2023).
Why would a ban on ﬂag burning increase the rate of ﬂag
burning? One potential mechanism is that the ban woulddraw attention to ﬂag burning, which is otherwise something
that not many people think about. Some portion of people
who think about it may then go on to do it. Another potential
mechanism is that the ban changes the social meaning of ﬂagburning. Without the ban, ﬂag burning is seen as distasteful
and anti-patriotic, whereas with the ban, ﬂag burning
becomes a patriotic rebellion against a bad law.
The story of dog cleanup is exactly the opposite. The
following describes the effect of dog cleanup laws in
Berkeley. Similar effects have been observed in other
locations, such as New York City (Krantz et al. 2008 ).
After the Berkeley town council enacted an ordinance
requiring owners to cleanup after their dogs, the
sidewalks became much cleaner, even though ofﬁ-
cials never issued citations for breaking the law. Thelaw apparently tipped the balance in favor of informal
enforcement. Citizens became more aggressive about
complaining to inconsiderate dog owners, and,anticipating this fact, dog owners became more
considerate (Cooter 2000 :11).
The dog cleanup story is notable because it achieved
positive outcomes without enforcing compliance . There
was no draconian surveillance, and no need to worry aboutdog owners relocating to places that lacked cleanup laws.
Instead, the law prompted dog owners and their neighbors
to police themselves.
The point of the comparison between ﬂag burning and dog
cleanup is that people can react in different ways to different
extrinsic measures and that this can signiﬁcantly affect out-comes. Indeed, how people react can be the difference
between negative outcomes (i.e., the extrinsic measure is
counterproductive) and positive outcomes. These two casesare directly applicable to extrinsic measures for promoting
beneﬁcial AI. In short, extrinsic measures for promoting
beneﬁcial AI should strive to be like dog cleanup, not likeﬂag burning. This means that extrinsic measures should aim
to be considered desirable to AI developers—measures
should be something that AI developers would want tocomply with, not something they would want to push back
against. In order to ﬁgure out what the effect of an extrinsic
measure would be, it is necessary to consider not just theextrinsic measure itself, but also the intrinsic factors, i.e., the
social psychology of AI communities.
A fuller accounting of intrinsic factors is presented in
Sect. 3, but here some intrinsic factors that are speciﬁc to
extrinsic measures. One recurrent ﬁnding is that monetary
incentives can reduce intrinsic motivation (Deci 1971 ;
Vohs et al. 2006 ). This means that once the money is gone,
people become less motivated to perform some task than
they would have been if there never was any money. Incontrast, social praise and encouragement can increase
intrinsic motivation (Deci 1971 ). Finally, carefully
6Conversely, when beneﬁcial AI groups are identiﬁed, rewards are to
be applied, though this is less of a challenge because AI groups arelikely to seek rewards, not dodge them.AI & Soc
123
designed extrinsic measures can use a psychological phe-
nomenon called cognitive dissonance to increase people’s
intrinsic motivation for a given type of activity (Dickerson
et al. 1992 ; Sect. 3.6).
3 Intrinsic factors and intrinsic measures
This section presents a variety of intrinsic phenomena of
relevance to promoting beneﬁcial AI. Some of the phe-nomena are dedicated intrinsic measures for promoting
beneﬁcial AI, and some of the phenomena are factors that
can play a role in a range of intrinsic and extrinsic mea-
sures. All of the phenomena are oriented toward motivating
AI developers to want to pursue beneﬁcial designs.
3.1 Social context and social meaning
People often behave differently depending on the social
setting or context that they are in. For example, some
people go to the library to study because the presence ofother studious people compels them to focus more, whereas
if they stayed at home, they would let themselves get
distracted.
Efforts to promote beneﬁcial AI can be more effective in
certain social contexts. For example, some beneﬁcial AI
measures beneﬁt from cooperation across AI groups inorder to avoid some groups choosing harmful designs that
give them competitive advantage. Research in other con-
texts has found that it is often easier to achieve cooperationwhen people are together in a group than when they are in
isolation (Krantz et al. 2008 ). Therefore, workshops, con-
ferences, and other meetings, or even group phone callscould all be more effective at achieving cooperation among
AI groups than private conversations or written statements
that would be read in private.
Second, promoting beneﬁcial AI can be more effective
in social contexts where beneﬁcial AI is considered desir-
able. For any given AI researcher, he or she is more likelyto support beneﬁcial AI if he or she is in a group of people
who openly support beneﬁcial AI than if he or she is alone
or in a group of people who do not openly support bene-ﬁcial AI. Support for beneﬁcial AI can thus be expanded by
creating and expanding groups of open beneﬁcial AI sup-
porters. Openness is important: other people will not beinﬂuenced by the group’s support for beneﬁcial AI if they
are not in any way aware of this support.
Related to social context is the concept of social
meaning. An act or idea can have a different meaning
depending on the social context. For example, the act of
ﬂag burning can have an anti-patriotic meaning if it is notbanned or a patriotic meaning if it is banned (Sect. 2.4).
Efforts to promote beneﬁcial AI should aim for it to have apositive social meaning. This can be accomplished, for
example, by testing prospective measures using focus
groups of AI researchers in order to gauge their reactions.
3.2 Social norms
Norms are that which is considered normal. Social norms
are norms held by groups of people about the normal
behaviors of people in that group, including which
behaviors normally are practiced (‘‘descriptive norms’’)
and which behaviors normally should be practiced (‘‘in-
junctive norms’’) (Lapinski and Rimal 2005 ). Norms can
vary from group to group and place to place. For example,
in some places, it is normal for pedestrians to cross the
street whenever it is safe, whereas in other places, it isnormal to wait for the streetlight to turn green. Norms can
also change over time. For example, slavery was once
considered normal throughout much of the world, but thisnorm has reversed in many places. Speciﬁc social contexts
can also activate certain social norms. For example, the
same group of people may show different norms in alibrary than in a nightclub.
Beneﬁcial AI can be a norm, i.e., concern for social
impacts can be considered normal among AI developers.Beneﬁcial AI as a social norm is implicitly at the heart of
Stuart Russell’s call for the AI community to abandon ‘‘its
current goal of building pure intelligence for its own sake,regardless of the associated objectives and their conse-
quences’’ and instead ‘‘build the issue [beneﬁcial AI]
directly into how practitioners deﬁne what they do’’ (Bo-hannon 2015 :252). It is difﬁcult to overstate the importance
of social norms for beneﬁcial AI. If beneﬁcial AI is con-
sidered normal, then it will be easier to achieve complianceon extrinsic measures and easier to succeed with intrinsic
measures, and there will be less need for either sort of
measure in the ﬁrst place because many AI researchers willalready be pursuing beneﬁcial designs.
How can one go about shifting social norms in AI?
Answering this question would beneﬁt from dedicatedresearch on AI social norms, but some insights can be
gained from other issues. For example, Posner
(2000 :1784–1785) lists several ways in which social norms
for paying taxes can be strengthened, including showing
that other people also pay taxes, creating social sanctions
for not paying taxes,
7and reminding people of their civic
obligation to pay taxes. Schultz et al. ( 2007 ) ﬁnd that
descriptive norms messages can reduce deviance but can
also have a counterproductive ‘‘boomerang’’ effect forpeople with better-than-normal behavior and that this effect
7Social sanctions are an extrinsic measure, speciﬁcally an incentive
using a social penalty, though they can also cultivate certain socialnorms.AI & Soc
123
can be attenuated with injunctive norms messages of social
approval for good behavior. These approaches could be
adapted for promoting beneﬁcial AI by showing that other
AI researchers also support it, creating social sanctions forthose who do not support it and social approval for those
who do, and cultivating a sense of duty for AI researchers
to attend to the social impacts of AI.
3.3 Messengers and alliesIn promoting beneﬁcial AI, it is not just what is said that
matters, but also who says it . This is because people
interpret meanings differently depending on who is con-
veying a message. This holds for AI researchers just as
much as it does for anyone else. The fact that this happenscuts against the scientiﬁc ideal of objectivity, but scientists
are humans too, and try as we might to avoid it, the identity
of the messenger still matters for how we react tomessages.
One important class of messenger is the fellow AI
researcher. Prior research has found that when conveyingmessages about ethics and social impacts to young scien-
tists (e.g., graduate students or postdocs), it is important
that the messages be delivered by established researchers inthat ﬁeld instead of by outside ethics professionals
(Schienke et al. 2009 ,2011 ). Using in-ﬁeld researchers
shows that ethics and social impacts is something that‘‘we’’ (i.e., people in the ﬁeld) care about and is not just
something that ‘‘they’’ (i.e., people outside the ﬁeld) want
‘‘us’’ to care about. This is reﬂective of a more generaltendency for people to respond better to messages from
other people in their ‘‘in group.’’ Thus, to the extent pos-
sible, messages should be selected from respected AIresearchers. The quotes in this paper from Stuart Russell
offer one example of this.
Sometimes, using ‘‘out-group’’ messengers can also
succeed. This can occur when the out-group is seen as
having some sort of high status. For example, funders,
institutional leadership, and policy makers can ﬁt this rolebecause they have some control over AI researchers’ pro-
fessional success and some credentials for setting social
norms and research directions. Celebrities—includingacademic and business celebrities like Stephen Hawking
and Bill Gates—can also ﬁt this role because they can be
perceived as successful, important, and inﬂuential. It isimportant that these people deliver thoughtful messages so
as to not come off as ‘‘ignorant blowhards’’, but when they
do deliver thoughtful messages, their inﬂuence can besubstantial.
As AI becomes more widely used across society, new
out-group allies will also emerge. For example, the auto-motive industry is currently applying AI to autonomous
vehicles. Automobiles must be safe, otherwise they will notsell and manufacturers can face steep liability claims. The
automotive industry likewise has a safety culture that is
currently pushing back against the AI culture of rapid
product development and postlaunch debugging. As FordCEO Mark Fields put it in a recent interview, ‘‘You can’t
hit control-alt-delete when you’re going 70 miles an hour’’
(Grifﬁth 2016 ). Insofar, as AI researchers would like the
business opportunities of autonomous vehicles, they may
be motivated to listen to the safety messages from mes-
sengers like Fields.
Another important potential ally could be found in
militaries. This may seem surprising, since militaries areassociated with violence and potential misuse of AI.
However, militaries can be inﬂuential to AI researchers
because they provide extensive AI research funding. Fur-thermore, militaries can be more safety conscious than is
commonly believed. One study of AI researcher opinion
found that a slight majority viewed the US military as themost likely institution to produce harmful AI, but that
‘‘experts who estimated that the US military scenario is
relatively safe noted that the US military faces strong moralconstraints, has experience handling security issues, and is
very reluctant to develop technologies that may backﬁre
(such as biological weapons)’’ (Baum et al. 2011 :193). For
these reasons, military ofﬁcials will often be motivated to
promote beneﬁcial AI. For their messages to resonate with
AI researchers, they must achieve trust, which could bedifﬁcult given AI researchers’ negative perceptions of the
military. If trust can be achieved, then the military could
offer another powerful ally in efforts to promote beneﬁcialAI.
3.4 FramingTo frame is to present a message in a certain way. Framing
is a matter of not what is said buthow it is said . Skillful
communication frames messages to achieve certain effects
for certain audiences. For example, climate change is
commonly framed as an environmental issue, which res-onates with liberals more than with conservatives. For
conservative audiences, climate change is sometimes
framed as a threat to national security or to the economy, orframed as an injustice that compels religious duty (Shome
and Marx 2009 ). Using the wrong frame for the wrong
audience can lead people to reject a cause that they mightotherwise support.
AI technologies can be framed in a variety of ways as
well. Unfortunately, existing messages about beneﬁcial AIare not always framed well. One potentially counterpro-
ductive frame is the framing of strong AI as a powerful
winner-takes-all technology. This frame is implicit (andsometimes explicit) in discussions of how different AI
groups might race to be the ﬁrst to build strong AI (e.g.,AI & Soc
123
Shulman 2009 ; Armstrong et al. 2016 ). The problem with
this frame is that it makes a supposedly dangerous tech-
nology seem desirable. If strong AI is a winner-takes-all
technologies race, then AI groups will want to join the raceand rush to be the ﬁrst to win. This is exactly the opposite
of what the discussions of strong AI races generally
advocate—they postulate (quite reasonably) that the rush towin the race could compel AI groups to skimp on safety
measures, thereby increasing the probability of dangerous
outcomes. Instead of framing strong AI as a winner-takes-
all race, those who are concerned about this technology
should frame it as a dangerous and reckless pursuit thatwould quite likely kill the people who make it. AI groups
may have some desire for the power that might accrue to
whoever builds strong AI, but they presumably also desireto not be killed in the process.
Another potentially counterproductive frame is the
framing of AI researchers as people who do not want topursue beneﬁcial designs. This framing is implicit in the
existing literature’s emphasis on extrinsic measures:
Extrinsic measures are used because AI researchers wouldnot want to pursue beneﬁcial designs. In the worst case,
heavy-handed extrinsic measures could counterproduc-
tively instill a social norm of AI researchers not pursuingbeneﬁcial measures. This would be a reaction like ‘‘I didn’t
think I was someone who ignores social impacts, but since
you mention it, I guess I am.’’ Light penalties, cooperativerelationships, and positive framing of AI researchers could
make them more inclined to pursue beneﬁcial designs.
Finally, extreme proposals like draconian global
surveillance can inadvertently frame efforts to promote
beneﬁcial AI as being the problem, not the solution. In
other words, it could give the impression that the efforts aremisguided and causing more harm than good. The potential
for aggressive beneﬁcial AI efforts to be perceived as
conspiratorial should not be discounted. Conspiracy theo-ries are already prominent in the perceptions of global
warming held by policy makers and the public (Le-
wandowsky et al. 2015 ). If AI succumbs to similar con-
spiracy theories, this could make it more difﬁcult to
promote beneﬁcial AI. And even without conspiracy the-
ories, ﬂoating extreme proposals can make the beneﬁcialAI cause seem at best out of touch, and at worst outright
harmful.
3.5 Stigmatization
Stigmatization is a type of framing oriented toward making
an object or an activity feel socially undesirable or even
taboo. Stigmatization can be an effective technique for
preventing the use of dangerous technologies. For example,stigmatization has been used repeatedly for international
arms control, most notably to achieve the 1997 OttawaTreaty banning landmines and the 2008 Convention on
Cluster Munitions banning cluster bombs, and currently to
promote nuclear disarmament (Borrie 2014 ). The experi-
ence with landmines and cluster munitions is notable inpart because the treaties have wider compliance than they
have ratiﬁcation. That is, some countries (e.g., the USA)
have not ratiﬁed the treaties, yet they still act in compliance
with them, even though they have no legal obligation to do
so. The effort to stigmatize landmines and cluster muni-
tions was so successful that the legal requirements are not
necessary to achieve the social goal.
The international community’s experience with stigma-
tization could be applied to dangerous AI. Stigmatizing
dangerous AI can help build support for extrinsic measures
such as national regulations and international treaties.However, even in the absence of any extrinsic measures,
stigmatization can still lead people to avoid building dan-
gerous AI. A successful stigmatization effort causes peopleto not want to do the stigmatized activity. Stigmatization thus
complements extrinsic measures. Indeed, it is difﬁcult to
imagine bans on dangerous AI technologies succeedingwithout an effective stigmatization effort.
In order for stigmatization to work, it must be based on a
convincing argument. The landmine and cluster munitionscampaigns were so successful because there was a sound
moral and legal argument against these weapons, speciﬁ-
cally that they cause indiscriminate harm to civilian pop-ulations. This argument was crucial for convincing
countries to reject the weapons even though they had
previously been accustomed to using them. Likewise, anyattempts to stigmatize speciﬁc AI technologies must be
based on some compelling reason. The potential for an AI
technology to cause a massive global catastrophe is anexample of such a reason.
Another challenge of stigmatization is that it can be
alienating to those who disagree with it and/or those whoare involved in a stigmatized activity. People do not like to
think of themselves as being involved in a stigmatized
activity and can resent being accused. This is seen, forexample, in current debates about nuclear weapons, in
which the nuclear-armed states distance themselves from
efforts to stigmatize nuclear weapons even while they sharean underlying concern about the weapons’ catastrophic
impacts. Likewise, efforts to stigmatize harmful AI should
distinguish between the harms of the AI and the characterof the AI designer so that the AI designers know that they
are not seen as bad people and that they would be embraced
if they switch to beneﬁcial designs.
3.6 Cognitive dissonance
Cognitive dissonance occurs when a person holds con-
ﬂicting beliefs in her or his mind. People typically seek toAI & Soc
123
resolve the dissonance of conﬂicting beliefs by rejecting
one of them. For example, people might reject reports that
a seemingly good person committed a terrible harm on
grounds that ‘‘He or she couldn’t possibly have done that.’’
An example of relevance to beneﬁcial AI is in the
relation between economic activity and beliefs about cli-
mate change. Around 2008, public belief in the scientiﬁcevidence of climate change declined. One explanation for
the decline is that the economic recession induced cogni-
tive dissonance. In response to the recession, people want
the economy to grow. However, economic activity typi-
cally increases greenhouse gas emissions, thereby wors-ening climate change. This creates dissonance between the
belief that there should be more economic growth and the
belief that climate change is a problem. Some data suggestthat some people handled this dissonance by rejecting the
scientiﬁc evidence of climate change, even though the
evidence itself is about the natural environment, not theeconomy (Scruggs and Benegal 2012 ).
Similarly, cognitive dissonance could lead AI
researchers to reject claims that AI could be harmful. Thepotential for harmful AI could imply that AI research
should be restricted, bringing AI researchers diminished
intellectual freedom and business opportunities and insome cases can even threatening their livelihoods. Just as
people may reject the science of climate change when the
economy is bad, AI researchers may reject evidence orargument about harmful AI when their welfare is at stake.
Like all people, AI researchers can have ‘‘motivated rea-
soning’’ in which they are motivated not by a goal ofaccuracy but instead by other goals, such as the goal of
believing that they are a good person (Kunda 1990 ).
Therefore, in order to improve salience, messaging aboutbeneﬁcial AI should strive to be sympathetic to AI
researchers’ intellectual and professional interests, and
extrinsic and intrinsic measures alike should strive tominimize the intellectual and professional downsides that
AI researchers could face.
Cognitive dissonance can also be used to promote cer-
tain beneﬁcial activities. Dickerson et al. ( 1992 ) studied the
combined effect of people making a public commitment to
conserving water and people being told that they had takenlong showers. People took shorter showers only when they
made a public commitment and were told they had taken
long showers. If only one of the two conditions were pre-sent, they took longer showers. The explanation is that
people had a cognitive dissonance between their public
commitment and their self-perception of taking longshowers, which they resolved by taking shorter showers.
Similar effects have been observed in other contexts, with
the strongest effect coming when people make publiccommitments or advocacy of an action and then are pri-
vately reminded of their own failures to perform that action(Stone and Fernandez 2008 ). This model could be adapted
for beneﬁcial AI by having AI researchers make public
commitments to beneﬁcial AI (such as via professional
societies or in classrooms) and then being privatelyinformed that some of their designs are not beneﬁcial. This
technique is likely to be more effective if an injunctive
social norm for beneﬁcial AI is already in place, becausethen AI researchers would be motivated to resolve the
cognitive dissonance in favor of more beneﬁcial AI.
4 Conclusion
As the societal impacts of AI continue to increase, it
becomes more and more important to promote the devel-opment of AI that is safe and beneﬁcial to society—ab-
breviated throughout this paper as ‘‘beneﬁcial AI’’.
Thus far, discussions of how to promote beneﬁcial AI
have focused mainly on extrinsic measures that are
imposed on AI designers even if they do not want to pursue
beneﬁcial AI. These measures come in the form of con-straints and incentives, and they are often accompanied by
measures for monitoring and enforcing compliance.
Extrinsic measures can be successful at promoting beneﬁ-cial AI, but they can be difﬁcult to implement and can be
resisted by AI developers. The success of extrinsic mea-
sures can also depend heavily on intrinsic factors, i.e., onhow AI developers react to the measures. If the reaction is
favorable, AI developers could comply on their own
without external monitoring and enforcement. Alterna-tively, if the reaction is unfavorable, AI developers could
even pursue less beneﬁcial designs than they would if there
were no extrinsic measures in place.
Meanwhile, a range of dedicated intrinsic measures are
available; these encourage AI developers to want to pursue
beneﬁcial designs. Social norms be shifted toward caringabout beneﬁcial design. Messengers can be selected, and
messages can be framed to resonate with AI developers and
entice them to want to pursue beneﬁcial designs. HarmfulAI designs can be stigmatized such that AI developers want
to avoid them. AI developers can make commitments to
choosing beneﬁcial designs that can, via cognitive disso-nance, lead them to do so.
This paper draws heavily from research on other issues
due to a lack of prior research on intrinsic aspects ofbeneﬁcial AI. The paper makes especially heavy use of
research on environmental issues because these issues have
seen robust social and psychological research. Manyinsights from these other issues apply to beneﬁcial AI, but
beneﬁcial AI will inevitably have its own unique charac-
teristics. Therefore, dedicated research on the social psy-chology of AI research communities is needed to
understand the effectiveness of both extrinsic and intrinsicAI & Soc
123
measures. Such research should be included in broader
research agendas for beneﬁcial AI.
One potential objection to intrinsic measures is that they
are unreliable because they depend on each AI researcherto cooperate. There is some truth to this. However,
extrinsic measures can also be unreliable—hence the
effectiveness of extrinsic measures can depend on intrinsicfactors. Regardless, 100 % success is an inappropriate
goal. The aim of any measure should be to reduce the
harms and increase the beneﬁts of AI to society. A measure
that does this should be pursued, even if it still leaves some
potential for harm or for loss of beneﬁt. Given the stakesinvolved in AI, all effective measures for promoting ben-
eﬁcial AI should be pursued.
Acknowledgments This paper has beneﬁted from conversations with
many people including Dario Amodei, Miles Brundage, Sean
Legassick, Richard Mallah, and Jaan Tallinn, and from time spent atthe Columbia University Center for Research on EnvironmentalDecisions. Tony Barrett, Steven Umbrello, and Peter Howe provided
helpful comments on an earlier draft. Any errors or shortcomings in
the paper are the author’s alone. Work on this paper was funded inpart by a grant from the Future of Life Institute. The views in thispaper are the author’s and are not necessarily the views of the Future
of Life Institute or the Global Catastrophic Risk Institute.
References
Amodei D, Olah C, Steinhardt J, Christiano P, Schulman J, Mane ´D
(2016) Concrete problems in AI safety. arXiv:1606.06565
Armstrong S, Bostrom N, Shulman C (2016) Racing to the precipice:
a model of artiﬁcial intelligence development. AI & Soc31(2):201–206
Baum SD, Goertzel B, Goertzel TG (2011) How long until human-
level AI? Results from an expert assessment. Technol ForecastSoc Change 78(1):185–195
Bohannon J (2015) Fears of an AI pioneer. Science 349(6245):252
Borrie J (2014) Humanitarian reframing of nuclear weapons and the
logic of a ban. Int Aff 90(3):625–646
Cooter RD (2000) Three effects of social norms on law: expression,
deterrence, and internalization. Oregon Law Rev 79:1–22
Deci EL (1971) Effects of externally mediated rewards on intrinsic
motivation. J Pers Soc Psychol 18:105–115
Dickerson CA, Thibodeau R, Aronson E, Miller D (1992) Using
cognitive dissonance to encourage water conservation. J Appl
Soc Psychol 22:841–854
Eden AH, Moor JH, Soraker JH, Steinhart E (2013) Singularity
hypotheses: a scientiﬁc and philosophical assessment. Springer,
Berlin
Grifﬁth E (2016) Who will build the next great car company? Fortune
Magazine, 23 June. http://fortune.com/self-driving-cars-silicon-
valley-detroit . Accessed 26 Sept 2016
Gurney JK (2013) Sue my car not me: products liability and accidents
involving autonomous vehicles. Univ Ill J Law Technol Policy2013(2):247–277
Joy B (2000) Why the future doesn’t need us. Wired 8(4):238–263
Krantz DH, Peterson N, Arora P, Milch K, Orlove B (2008)
Individual values and social goals in environmental decisionmaking. In: Smith JC, Connolly T, Son YJ (eds) Kugler T.
Decision modeling and behavior in complex and uncertain
environments New York, Springer, pp 165–198
Kunda Z (1990) The case for motivated reasoning. Psychol Bull
108(3):480–498
Lapinski MK, Rimal RN (2005) An explication of social norms.
Commun Theory 15(2):127–147
Lewandowsky S, Cook J, Oberauer K, Brophy S, Lloyd EA, Marriott
M (2015) Recurrent fury: conspiratorial discourse in the
blogosphere triggered by research on the role of conspiracistideation in climate denial. J Soc Political Psychol 3(1):142–178
Lin P, Abney K, Bekey GA (eds) (2011) Robot ethics: the ethical and
social implications of robotics. MIT Press, Cambridge
Markowitz EM, Shariff AF (2012) Climate change and moral
judgement. Nat Clim Change 2(4):243–247
McGinnis JO (2010) Accelerating AI. Northwest Univ Law Rev
104:366–381
Moses LB (2007) Recurring dilemmas: the law’s race to keep up with
technological change. Univ Ill J Law Technol Policy
2007(2):239–285
Posner EA (2000) Law and social norms: the case of tax compliance.
Va Law Rev 86:1781–1819
Posner RA (2004) Catastrophe: risk and response. Oxford University
Press, Oxford
Russell S, Dewey D, Tegmark M (2015) Research priorities for robust
and beneﬁcial artiﬁcial intelligence. AI Mag 36(4):105–114
Schienke EW, Tuana N, Brown DA, Davis KJ, Keller K, Shortle JS,
Stickler M, Baum SD (2009) The role of the NSF Broader
Impacts Criterion in enhancing research ethics pedagogy. SocEpistemol 23(3–4):317–336
Schienke EW, Baum SD, Tuana N, Davis KJ, Keller K (2011)
Intrinsic ethics regarding integrated assessment models forclimate management. Sci Eng Ethics 17(3):503–523
Schultz PW, Nolan JM, Cialdini RB, Goldstein NJ, Griskevicius V
(2007) The constructive, destructive, and reconstructive power
of social norms. Psychol Sci 18(5):429–434
Scruggs L, Benegal S (2012) Declining public concern about climate
change: can we blame the great recession? Glob Environ Change
22(2):505–515
Shome D, Marx S (2009) The psychology of climate change
communication: a guide for scientists, journalists, educators,political aides, and the interested public. Columbia University
Center for Research on Environmental Decisions, New York
Shulman C (2009) Arms control and intelligence explosions. In: 7th
European Conference on Computing and Philosophy (ECAP),
Bellaterra, Spain, July 2–4
Sotala K, Yampolskiy RV (2014) Responses to catastrophic AGI risk:
a survey. Physica Scripta 90(1):018001. doi: 10.1088/0031-8949/
90/1/018001
Stone J, Fernandez NC (2008) To practice what we preach: the use of
hypocrisy and cognitive dissonance to motivate behavior change.Soc Pers Psychol Compass 2(2):1024–1051
Sunstein CR (1996) On the expressive function of law. Univ Pa Law
Rev 144(5):2021–2053
Vohs KD, Mead NL, Goode MR (2006) The psychological conse-
quences of money. Science 314:1154–1156
Wilson G (2013) Minimizing global catastrophic and existential risks
from emerging technologies through international law. VaEnviron Law J 31:307–364
Yampolskiy R, Fox J (2013) Safety engineering for artiﬁcial general
intelligence. Topoi 32(2):217–226AI & Soc
123