Social choice ethics in artificial intelligence

ORIGINAL ARTICLE
Social choice ethics in artiﬁcial intelligence
Seth D. Baum1
Received: 17 July 2016 / Accepted: 16 September 2017
/C211Springer-Verlag London Ltd. 2017
Abstract A major approach to the ethics of artiﬁcial
intelligence (AI) is to use social choice, in which the AI is
designed to act according to the aggregate views of society.
This is found in the AI ethics of ‘‘coherent extrapolated
volition’’ and ‘‘bottom–up ethics’’. This paper shows that
the normative basis of AI social choice ethics is weak due
to the fact that there is no one single aggregate ethical view
of society. Instead, the design of social choice AI faces
three sets of decisions: standing, concerning whose ethics
views are included; measurement, concerning how their
views are identiﬁed; and aggregation, concerning how
individual views are combined to a single view that will
guide AI behavior. These decisions must be made up front
in the initial AI design—designers cannot ‘‘let the AI ﬁg-
ure it out’’. Each set of decisions poses difﬁcult ethical
dilemmas with major consequences for AI behavior, with
some decision options yielding pathological or even
catastrophic results. Furthermore, non-social choice ethics
face similar issues, such as whether to count future gen-
erations or the AI itself. These issues can be more impor-
tant than the question of whether or not to use social choice
ethics. Attention should focus on these issues, not on social
choice.
Keywords Artiﬁcial intelligence /C1Ethics /C1Social
choice /C1Standing /C1Measurement /C1Aggregation1 Introduction
As artiﬁcial intelligences (AIs) become more powerful and
more embedded in important social systems, it becomes
more important to program them to have ethical frame-
works built in. In the words of Picard ( 1997 : 134), ‘‘The
greater the freedom of a machine, the more it will need
moral standards.’’ To take one of many examples, the
development of autonomous vehicles puts AIs in life-and-
death situations, including ethically difﬁcult situations such
as when a collision is inevitable and the AI driver must
decide which dangerous collision to have (Lin 2016 ).
Outcomes of such situations can depend on AI design
decisions, making it incumbent upon AI developers to
carefully choose the ethics that are built in.
This paper discusses an approach to AI ethics in which
the AI is designed to act according to the aggregate ethical
views of society. This approach underlies at least two
signiﬁcant lines of thinking in AI ethics. One is ‘‘coherent
extrapolated volition’’ or CEV (Yudkowsky 2004 ;
Muehlhauser and Helm 2012 ; Bostrom 2014 ). CEV was
formulated speciﬁcally for the ethics of superpowerful,
superintelligent AI (ASI) that would or could take over the
world. CEV abstains from selecting an ethical view for the
initial programming and instead seeks to have the AI derive
its values from the values of other ethical agents. CEV
speciﬁcally seeks to extrapolate beyond agents’ existing
ethical views, essentially to ﬁgure out the views that the
agents would ideally have if they were as smart as the ASI.
&Seth D. Baum
seth@gcrinstitute.org
1Global Catastrophic Risk Institute, PO Box 40364,
Washington, DC 20016, USA
123AI & Soc
DOI 10.1007/s00146-017-0760-1
The other line of thinking is the concept of ‘‘bottom–up’’
ethics (Allen et al. 2000 ;2005 ; Wallach et al. 2008 ;
Wallach and Allen 2008 ). An AI with bottom–up ethics is
designed to learn ethics as it interacts with its environmentand with other ethical agents, similar to how human chil-
dren learn ethics as they grow up.
1While bottom–up ethics
can be consistent with a range of ethical frameworks, someinstantiations of it attempt ‘‘to train or evolve agents whose
behavior emulates morally praiseworthy human behavior’’
(Allen et al. 2005 : 149). Bottom–up ethics is contrasted
with ‘‘top–down’’ ethics in which the AI is programmed to
have a speciﬁc ethical view from the start and thus does notseek to identify the views of society or any of its members.
Where bottom–up ethics is based on learning from other
ethical agents, the ethics views it ends up with will be someaggregation of the agents it learns from.
Though it is not explicitly identiﬁed in their respective
literatures, CEV and bottom–up ethics have the essentialstructure of what is known in economics, ethics, and
political science as social choice: the procedure for deriv-
ing group decisions from individual ethical views.
2The
ethics of social choice is rooted in certain notions of pro-
cedural justice, and it underlies both democracy, in which
individual preferences are expressed through voting, andcapitalism, in which individual preferences are expressed
through market behaviors (‘‘voting with dollars’’). The
study of social choice dates to the early work by politicaltheorist de Condorcet ( 1785 ); modern scholarship largely
traces to Arrow ( 1951 ), and today is a robust ﬁeld as fea-
tured in dedicated journals such as Social Choice and
Welfare .
Work on social choice often falls under the term ‘‘social
choice theory’’. This paper uses the term ‘‘social choiceethics’’ to refer to the view that the right ethical framework
to use is that which corresponds to the aggregate ethical
views of society. Social choice theory concerns theoreticalissues that arise in the context forming aggregate views.
Social choice ethics also includes empirical issues regard-
ing what ethical views individuals hold. In addition,whereas social choice theory often focuses on the aggre-
gation of individual preferences, social choice ethics can
include ethical views that are not readily described in termsof preference.This paper also uses the term ‘‘predetermined ethical
views’’ to refer to top–down, non-social choice ethics. The
views are predetermined in that they are set prior to any
observations of society’s views and are used regardless ofwhat society’s aggregate views turn out to be. For example,
one could attempt to program an AI to maximize the
welfare of all sentient beings, as in welfarist utilitarianism(e.g., Ng 1990 ), or to maximize the ﬁtness of ecosystems,
as in ecocentric consequentialism (e.g., Holbrook 1997 ).
While this paper is not concerned with the merits of various
predetermined ethical views, the reader should note their
availability as alternatives to social choice ethics.
CEV and bottom–up ethics have the structure of social
choice in that they identify ethical views held by various
ethical agents and combining these into some aggregateview to use for decision-making. In CEV, the agents are
those whose volition is to be extrapolated. In bottom–up
ethics—or at least in certain instantiations of it—the agentsare those whose behavior is to be emulated. It is thus
remarkable that the literatures on CEV and bottom–up
ethics have not discussed issues of social choice in anylength. This paper seeks to address this gap. As we will see,
some nuances of social choice make CEV and bottom–up
ethics more difﬁcult to implement in AI and less ethicallydesirable than they might initially appear, which, in turn,
makes predetermined ethical views more desirable. Fur-
thermore, some issues faced in social choice ethics parallelthose faced in predetermined ethical views; how these
issues are resolved can be more important than whether AI
is designed with social choice or predetermined viewethics.
2 Some preliminaries
Why might one favor AIs with social choice ethics? Sev-eral justiﬁcations can be found:
1. The procedural justiﬁcation . This is the procedural
justice intuition that individuals should have a say in
decisions that affect them, as epitomized by the classic
American dictum ‘‘no taxation without representa-tion’’. Per this, it would be unfair for AI designers to
impose their own ethics views on everyone else by
programming AIs with their choice of predetermined,top–down views. Yudkowsky ( 2004 : 17) makes this
rationale explicit in stating that AI programmers ‘‘do
not deserve, a priori, to cast a vote larger than anyoneelse’’ in how an AI is designed.
2. The abstention justiﬁcation . Some individuals may
rather not wrestle with ethics for themselves, so theyabstain from choosing ethics and ‘‘pass the buck’’ onto
society as a whole. Per this, AI designers choose a1Note that while consciousness may play a role in ethics learning
among human children, it is not essential for AI. The essential featureis that ethics is learned via interaction with the environment,
regardless of whether that interaction involves consciousness.
2One exception, in which social choice is (brieﬂy) discussed in the
context of CEV, is Tarleton ( 2010 ). Keyword searches in Google
Scholar identiﬁed no other discussions of social choice in CEV or
bottom-up ethics. There is a more extensive study of ‘‘computationalsocial choice’’ relating aspects of social choice theory and computerscience (Brandt et al. 2015 ).AI & Soc
123
bottom–up ethics design in hopes of producing an
ethical AI without themselves thinking about ethics.This justiﬁcation is consistent with AI veteran Stuart
Russell’s observation that the AI ﬁeld has shown a
little interest in social impacts (Bohannon 2015 ), and
the more general observation that research ethics has
concentrated on process issues like plagiarism instead
of ethics embedded in the research itself and ethics ofthe social impacts of research (Schienke et al.
2009 ,2011 ). Similarly, Muehlhauser and Helm
(2012 ) observe that all moral theories thus far devel-
oped by humans raise signiﬁcant objections and
propose that an ASI would be more capable at
determining a better moral theory.
3. The wise crowd justiﬁcation . It is sometimes posited
that better results are achieved when using the views of
many individuals, as in the maxim ‘‘wisdom of thecrowd’’. Per this, an AI’s ethics is likely to be ‘‘better’’
according to some neutral standard if its ethics come
from many individuals instead of from one. Thus, amarket democracy could outperform a communist
dictatorship, because it empowers many people to
contribute their unique insights; ditto for an open,interactive scientiﬁc community vs. individual scien-
tists working in isolation. Likewise, having an AI
aggregate across the ethical views of many individualscould ‘‘smooth out the rough edges’’ of humanity—
that is, unless only humanity’s edges are smooth, i.e.,
unless ‘‘large segments of humanity have base or evilpreferences’’ (Bostrom 2014 : 217), in which case
social choice approaches could yield worse results.
There is an inherent tension between the procedural and
wise crowd justiﬁcations. The wise crowd justiﬁcation
depends on having some notion of ‘‘better results’’. Thisnotion cannot come from the aggregate views of many
individuals—that would be circular logic (‘‘The aggregate
views of many individuals achieve better results accordingto the aggregate views of many individuals’’). Instead, it
must come from some predetermined ethical view, which is
precisely what the procedural justiﬁcation seeks to avoid.The wise crowd justiﬁcation is essentially an empirical
claim about the instrumental value of social choice: given a
predetermined ethical view, querying large crowds willtend to deliver better results per that view. The truth of this
empirical claim is beyond the scope of this paper. If it is
true, it still leaves open the question of which predeter-mined ethical view to design AIs with.
The rest of this paper is mainly concerned with the
procedural and abstention justiﬁcations. Speciﬁcally, thepaper argues that these justiﬁcations are severely limited,
because it is impossible for AI designers to avoid embed-
ding certain ethics views into an AI. This is because thereis no one single aggregate ethical view of society. Instead,
there are many aggregate views depending on how the
views are aggregated. These different aggregations can
have very different consequences, some of which could beconsidered pathological or even catastrophic. Therefore,
choosing a social choice ethics for AI does not absolve the
AI designer from thinking about ethics, or from makingdecisions about ethics similar to those required for
designing AI with predetermined, top–down ethical views.
This weakens the case for social choice ethics and likewise
strengthens the case for using predetermined ethical views.
Implementations of social choice ethics must make three
types of choices, each of which create their own set of
ethical dilemmas (Baum 2009 ):
1.Standing Who or what is included in the group to have
its values factored into the AI?
2.Measurement What procedure is used to obtain values
from each member of the selected group?
3.Aggregation How are the values of individual group
members combined to form the aggregated groupvalues?
The AI ethics literature has focused mainly on mea-
surement, and the social choice theory literature tradition-ally focuses on aggregation. However, all three pose
serious challenges.
The issues of standing, measurement, and aggregation
arise in all implementations of social choices, not just those
involving AI. Likewise, many of the ethical issues of
standing, measurement, and aggregation that are faced byAI designers are also faced by people designing social
choice systems in other contexts. The paper thus draws on
scholarship and experience from other contexts (including,but not limited to, social choice theory) and relates them to
AI. Meanwhile, AI poses some novel issues for social
choice. First, with AI, the social choice process is con-ducted by machines, not by humans. With AI, a machine
can be sent out into society to ﬁgure out what society wants
it to do and then attempt to do it. Human designers of AImust make decisions about standing, measurement, and
aggregation and translate this into the technology. Second,
there is the question of whether the AI itself should have
standing. Arguably, a sufﬁciently advanced AI should. This
raises distinct ethical questions. Thus, another aim of thispaper is show how AI expands the scope of social choice
ethics.
Before diving into the details, it should be explained that
questions of standing, measurement, and aggregation must
be answered by AI designers at the beginning of the social
choice process—the questions cannot be delegated to theprocess. Social choice processes can consider issues of
standing, measurement, and aggregation. However, how
these issues are considered is determined by how theAI & Soc
123
process was initially set up. In democracies, this is the issue
of who gets to vote on who gets to vote, and how that vote
is held. For example, in the United States and other
countries, women were ﬁrst enfranchised when men votedto enfranchise them. For this reason, AI designers cannot
simply ‘‘let the AI ﬁgure it out’’. AI designers who use
social choice ethics must make choices about standing,measurement, and aggregation.
3 Standing
The term ‘‘standing’’ comes from the legal context in
which to have standing is to have the right to bring a case
to court or otherwise participate in the case in some legallysigniﬁcant way. To have legal standing, one must be a legal
person—generally a human citizen of the jurisdiction in
question—and one must have sufﬁcient connection to thecase to justify participation. While this is the origin of the
term ‘‘standing’’ as it is used in this paper, the paper uses it
in a slightly different way.
In this paper, to have standing is to have one’s ethics
included in a social choice process used to determine the
ethics of an AI.
3In social choice processes, those who have
standing are those whose values are factored in. Alterna-
tively, if a predetermined ethical view is used instead of a
social choice ethics, then the individuals with standing arewhoever decides which view to build into the AI. For
example, if a person builds an AI on her or his own, and
that person makes a unilateral, top–down decision onwhich ethics to build in, then that person is the only one
who has standing for that AI. Biases can be introduced, all
the more so, because the demographics of AI tilt heavilytowards males of certain backgrounds (Clark 2016 ). Taken
in this context, the ﬁrst major decision related to social
choice ethics is the decision to use social choice ethics inthe ﬁrst place. Assuming this decision has, indeed, been
made, a series of standing decisions must then be resolved.
To help clarify the issue of standing, let us take the
concrete example of autonomous vehicles. Some driving
decisions pose trade-offs between vehicle occupants and
other individuals. For example, should the vehicle travelfaster, so as to minimize travel times (good for the occu-
pants), or slower, so as to minimize energy consumption
and environmental harms (good for almost everyone else)?One study found a 13% variation in per-mile energy efﬁ-
ciency of autonomous vehicles depending on how the
vehicle is programmed (Mersky and Samaras 2016 ).
Aggregated across a global ﬂeet of vehicles, this is anenormous difference for energy and the environment—a
difference that can depend on how the AI handles standing.
Autonomous vehicles may be designed to give their
occupants the choice of how the vehicle should drive,especially if the occupants own the vehicle. This gives
standing to the occupants but not to everyone else. Fur-
thermore, sometimes, the vehicle itself will need to makethe choice, because sometimes, occupants will neglect to
choose: the vehicle needs default drive settings. An AI with
social choice ethics would learn from the tendencies of
whoever is setting its drive mode and make its own driving
choices accordingly. Designing an AI to learn its drivingvalues from its occupants denies standing to everyone else.
This could lead the vehicle to travel faster and pollute the
environment more, causing the rest of the world consid-erable harm.
With this example in mind, we turn now to the general
questions of who or what gets standing.
Discussions of social choice AI ethics typically propose
giving standing to some (often unspeciﬁed) portion of
humanity. An example of this appears above in the quote‘‘[Bottom–up ethics] attempts to train or evolve agents
whose behavior emulates morally praiseworthy human
behavior’’ (Allen et al. 2005 : 149; emphasis added). Sim-
ilarly, Yudkowsky ( 2004 : 5) argues that ‘‘the initial
dynamic should implement the coherent extrapolated
volition of humankind ’’ (emphasis added). However, which
humans are to be included? Martin ( 2017 ) considers whe-
ther AI ethics should be set by its designers, its users, or by
human society as a whole.
4
Should the antisocial psychopaths be included—the
Hitlers and bin Ladens and serial killers and rapists and
other ‘‘very bad people’’ of the world? These are peoplewho might, for example, train autonomous vehicles to
drive in especially dangerous ways. One could make a case
for excluding them as unﬁt, just as convicted felons aredisenfranchised in some democracies. Indeed, if ‘‘large
segments of humanity have base or evil preferences’’
(Bostrom 2014 : 217), then including them could foul up the
entire social choice process, causing AIs to conduct horriﬁc
and unspeakable acts. However, excluding the antisocial
requires deﬁning who they are. As Foucault ( 1961 ) docu-
ments, conceptions of madness and mental illness have
changed dramatically over time. Current conceptions are
thus historically contingent and not necessarily correct. TheAI designer must choose which antisocial people, if any, to
exclude from the social choice process.
Another challenging case is children. Children are rou-
tinely excluded from elections on grounds that they lack
the intellectual and moral capacity to make reasoned
3This is similar to the ‘‘boundary problem’’ in democracy (Arrhenius
2005 ).4Martin ( 2017 ) also considers having AIs set their own ethics or the
ethics of other AIs; more on this below.AI & Soc
123
judgments about whom to vote for. In the United States, for
example, the minimum age for voting is 18. However,
about a quarter of the current human population is under
age 15, which is a lot to exclude. Excluding them isarguably inherently unfair, and it could bias an AI’s values
in certain directions, for example, because young people
generally more accepting of lesbian, gay, bisexual, andtransgender (LGBT) people (e.g., Pew Research Center
2017 ). Views about LGBT people may be insigniﬁcant for
the AI inside autonomous vehicles, but they could be
important for, say, AIs that chat with people on social
media, such as Microsoft’s notorious chatbot Tay (Gibbs2016 ), or AIs in robots that enforce proper behavior in
public, such as the Knightscope K5 security robot (Metz
2014 ).
5
Excluding children may not be hugely consequential as
long as their parents have standing. Parents invest heavily
in the success of their children, both privately and publicly(i.e., through governments and community groups). How-
ever, future generations are at greater risk. The moral
psychology of time preference and temporal discountingtypically ﬁnds that people value future events and future
generations less than the present (Frederick et al. 2002 ).
Likewise, philosophers sometimes consider that futuregenerations may have no value at all (Arrhenius and
Rabinowicz 2015 ). For example, Marglin ( 1963 : 97) writes
‘‘I consider it axiomatic that a democratic governmentreﬂects only the preferences of the individuals who are
presently members of the body politic’’. Thus, unless future
generations are given standing at the outset of a socialchoice process, there is a strong likelihood that their
interests would be given little consideration by the result-
ing AI. (There is also the question of how an AI can learnthe values of future generations, but this is a matter of
measurement, not standing).
It is difﬁcult to overstate how much is at stake with
standing for future generations. The population of future
generations could be extremely large, vastly dwarﬁng the
present population. Earth will remain habitable for on theorder of a few billion more years (O’Malley-James et al.
2014 ), and the rest of the universe will remain habitable for
much longer (Adams 2008 ). If AI designers only give
standing to the present generation, they risk biasing out-
comes against this astronomically large future. For exam-
ple, this could mean gluttonous consumption ofnonrenewable natural resources and pollution of global
ecosystems by the present generation (such as in autono-
mous vehicles designed to travel quickly instead of energyefﬁciently), with future generations left to struggle through
the ensuing mess. An AI that is programmed to only give
standing to the present generation could deliver an extreme
intergenerational injustice.
These are the ethical issues faced if standing is given to
some portion of humanity, as is commonly prescribed in
the AI ethics literature. However, this presupposes thatstanding should be given only to humanity. However,
humans are not the only individuals that could merit
standing. Standing has been considered for nonhuman
animals (Sunstein 2000 ), plants and ecosystems (Stone
1972 ; Hannon 1998 ), abiotic environments (Rolston 1986 ),
extraterrestrials (Cockell 2007 ), technologically enhanced
‘‘posthumans’’ (Buchanan 2009 ), and AIs (Hubbard 2011 ).
Each of these entities poses its own set of ethical chal-lenges for the issue of standing in an AI social choice
process.
Let us consider nonhuman animals. The more advanced
nonhuman animals (e.g., other primates) have cognitive
abilities approaching those of humans, a fact that has been
the basis of attempts to grant them some legal standing,such as the Great Ape Project. Sentience appears to decline
gradually across the animal kingdom—for example,
research on the sentience of ﬁsh is inconclusive (Rose et al.2014 ), while some research suggests the sentience of
invertebrate crayﬁsh (Fossat et al. 2014 ). Failing to give
standing to sentient animals would risk mass atrocitiescommitted against them—atrocities on par with or worse
than the widespread and horriﬁc abuse of livestock animals
in factory farms today (e.g., Anomaly 2015 ).
Robots are already managing livestock in some loca-
tions (Klein 2016 ), and it may only be a matter of time
before livestock is managed primarily by robots. Workingin factory farms is dangerous and unpleasant for humans
and also somewhat repetitive, making it an ideal candidate
for automation. However, if the robots are programmed tolearn their values from humans only, then they could per-
petuate the violence against livestock animals. Indeed,
robots could even make the violence worse if the empathyof human farmhands is replaced by the callousness of farm
management. Alternatively, if the robots learn values from
humans and livestock alike, then they may ﬁnd ways totreat the livestock animals better. In the best-case scenario,
this could drastically reduce or even eliminate the mass
abuse of livestock animals in factory farms while main-taining or even improving the supply of livestock products
to humans. However, the robots may ﬁnd some trade-offs
unavoidable, in which case giving standing to livestockcould worsen the supply of livestock products—in the
extreme case, the robots could even conclude that the
livestock should be set free.
Similar logic applies to plants, ecosystems, and abiotic
environments: AIs designed to learn values from humans
5Tay was programmed to learn from (and thus give standing to)
Twitter users who interact with it, which quickly devolved into
deviance and obscenity as Twitter users taught it to misbehave.Microsoft has since been wrestling with the question of how to givestanding to a more appropriate mix of people.AI & Soc
123
only could end up acting for human beneﬁt at the expense
of anyone or anything else. The AI designer must choose
sides on these issues when choosing who or what gets
standing in the social choice process.
A different logic applies to posthumans, because these
entities do not currently exist. Consider the extreme case of
a superpowerful ASI that has taken over the world and ismanaging it according to the CEV of humanity. It is pos-
sible—indeed it may be probable—that humanity ulti-
mately prefers remaining human and not being ‘‘enhanced’’
into posthumans. Indeed, a recent public opinion poll found
a majority of Americans indicating opposition to a range ofhuman ‘‘enhancement’’ technologies (Funk et al. 2016 ). In
this case, whether posthumans ever come into existence
could depend on whether the ASI (or its precursor seed AI)is designed to give standing to posthumans. If posthumans
have standing, then the desire of posthumans to come into
existence could be balanced against the desire of humans tostay human.
6
Finally, there is the AI itself. Hubbard ( 2011 ) argues that
if AIs possess the attributes that are considered essential forlegal standing, then they should have it. Hubbard proposes
three criteria for an AI meriting legal standing: complex
interaction skills, self-consciousness, and the ability topursue group beneﬁts. If anything, standards for moral
standing in a social choice process should be lower than
this, because legal standing implies a certain ability tofunction in courts and other legal contexts, whereas moral
standing does not. Martin ( 2017 ) considers that sufﬁciently
advanced AIs could gain capacity for ethical reasoning thatmatches or even exceeds that of humans, making them
independent ethical agents. In contrast, Yampolskiy ( 2013 :
393) argues that AIs that merit standing should not be builtin the ﬁrst place, and instead that ‘‘machines should be
inferior by design; they should have no rights and should
be expendable as needed, making their use as tools muchmore beneﬁcial for their creators’’.
Once again, much is at stake. To abstain from building
certain types of AIs could signiﬁcantly limit the extent towhich the beneﬁts of AI could be realized. AIs that would
merit standing may be some of the most sophisticated and
capable AIs; abstaining from building them could be anespecially large loss. Alternatively, if humans deny stand-
ing for sophisticated AIs, this puts core human values at
risk. As Hubbard ( 2011 ) argues, to deny standing to entities
that are at least as deserving of it as we are cuts against the
liberal, secular, rationalist, and egalitarian foundations of
democratic human society.However, AIs pose a novel and sizable complication. As
Yampolskiy ( 2013 ) explains, AIs can be readily copied in
large numbers. If AIs have standing, they could easily
drown out the human population or any other biologicalpopulation. Some existing nonhuman entities raise the
same complication, for example, if standing was given to
bacteria, which massively outnumber humans and otherlarge organisms. However, only AI can combine massive
population sizes with cognitive sophistication equal or
greater to humans. It would arguably be unjust to disen-
franchise AIs simply on the grounds of getting outvoted—
such a situation would be analogous to the apartheid ofSouth Africa, in which the white minority disenfranchised
the black majority. However, to grant AIs standing could
mean letting them control the social choice process. Ulti-mately, whether they would control, it depends on the
details of measurement and aggregation.
4 Measurement
Measurement in this context refers to the process throughwhich an individual’s ethical views are identiﬁed for
inclusion in the social choice process. In a typicaldemocracy, measurement is done via voting. In capitalism,
measurement is done via buying and selling. Measurement
can also be done via observing behavior, as in the eco-nomics concept of ‘‘revealed preference’’, via surveys and
interviews, such as in public opinion polling, or even via
brain imaging, such as in neuropsychology research.
If human beings—or whatever else was being mea-
sured—had one single, consistent set of ethical views, then
measurement would be a relatively simple issue. Eachmeasurement procedure would yield more or less the same
results. The only challenge would be obtaining results in
sufﬁcient detail to inform the issue in question. This is anontrivial challenge, but it pales in comparison to the
actual challenge of measurement. Humans do not have a
single, consistent set of ethical views, and different mea-surement procedures can yield different answers to the
same ethical question.
To take one example, consider the ethical issue of time
preference. This concerns how people value present gains
and losses relative to future gains and losses. Frederick
et al. ( 2002 , Table 1) review 41 different studies that
measure human time preference using either behavior
observation or survey methods. The studies show that
humans discount future gains and losses at rates rangingfrom -6t o ??%. Some of this variation is
attributable to differences in the nature of the gains and
losses being evaluated (e.g., money vs. health), but much ofit appears to be due to differences in the procedures used to
measure time preference.
6There is a certain irony that some proponents of CEV speak in
terms of giving standing only to humanity but also favor a transitionto posthumanity (e.g., Bostrom 2008 ).AI & Soc
123
Discount rates from -6t o ??% are an extremely
wide range. Negative discount rates imply that future gains
and losses are more valuable than present ones. A discount
rate of ??% implies that future gains and losses hold
zero value. Thus, depending on the choice of measurement
procedure, an AI could end up valuing future gains and
losses a lot or not at all. For example, this could be thedifference between an autonomous vehicle AI traveling
faster (if future gains and losses are not valued at all) or
instead more energy efﬁciently (if future gains and losses
are valued a lot). Or, for a superpowerful ASI, this could be
the difference between lavishing resources on the presentgeneration, future be damned, or preparing for the distant
future, even if that requires present sacriﬁce.
The wide range of time preferences that have been
measured in humans is one instance of a broader tendency
for humans to show inconsistent and often incoherent
ethics views. Even moral philosophers struggle withinconsistency, as evidenced by their various ‘‘impossibility
theorems’’ in which they ﬁnd it impossible to craft a moral
theory that meets a series of seemingly plausible moralintuitions (e.g., Arrhenius 2011 ). For this reason, an effort
to measure an individual’s ethics views is likely to yield
only one particular variation on his or her views.
This diversity of an individual’s values poses a thorny
challenge for social choice AI ethics. As an example,
consider an environmentalist who also happens to be a fastdriver. This may seem hypocritical, since driving fast is
worse for the environment—perhaps, this person just
thinks that other people should drive slower. However, it is
often observed that people disagree with their own
behavior on these sorts of matters (Stone and Fernandez
2008 ). In such a situation, what is a social choice AI to do?
The AI in an autonomous vehicle could measure this per-
son’s ethics by observing her driving behavior, in which
case the AI would end up traveling quickly, or by askingher how she thinks she should drive, in which case the AI
would end up traveling energy efﬁciently. Both measure-
ment approaches are technologically simple to implement:one involves recording driving data and the other involves
a simple questionnaire. Either approach could lay some
claim to measuring the person’s ‘‘true’’ view.
One option available to AI designers is to select the
measurement approach that they believe delivers the better
result. This is analogous to the concept of libertarianpaternalism or ‘‘nudges’’ (Thaler and Sunstein 2008 ; in the
context of robotics, see Borenstein and Arkin 2016 ). In this
context, to nudge someone is to structure their decisions tomake it more likely that they make the ‘‘better’’ decision.
For example, states can make it the default option that
people are organ donors, which increases the rate of organdonors. This is libertarian, because anyone is still free to
opt out of organ donation, and it is paternalistic, because itstructures the decision, so that people tend to make the
‘‘better’’ decision of being an organ donor.
Analogously, AI designers could select the measurement
option that they think delivers the better result. Indeed,designers could even give people the choice of measure-
ment option, in which case designers could inﬂuence
results by choosing which measurement option is thedefault. But which to choose? Is it better for people to
travel faster or more energy efﬁciently? Traveling faster
may be better for that person, while traveling more energy
efﬁciently may be better for society as a whole. This is a
difﬁcult ethics question in its own right, and it is exactlythe sort of question that AI designers seek to avoid by
choosing social choice ethics. Because humans display
multiple ethics views, the choice of measurement approachfor social choice ethics can require AI designers to make
unilateral ethical choices.
In response to the seeming incoherence of human ethics,
some AI ethicists call for measuring an idealized version of
human ethics instead of what is observed in actual humans.
For example, Muehlhauser and Helm ( 2012 : 114) posit that
an advantage of CEV for ASI is that it would deliver ‘‘what
a person would want after reaching reﬂective equilibrium
with respect to his or her values, rather than merely whateach person happens to want right now’’, and that this
‘‘may dissolve the contradictions within each person’s
current preferences’’. This follows calls in moral philoso-phy for the use of ‘‘idealized preferences’’ (e.g., Harsanyi
1996 ;N g 1999 ; and references in Muehlhauser and Helm
2012 ). The hope is that an ASI would be able to ﬁgure out
which ethics any given human would really want if he or
she was able to think the matter through as carefully as an
ASI could, and that following this idealized ethics wouldyield better results.
It should, indeed, be expected that a human’s idealized
ethics would differ from his or her original ethics. This isseen, for example, in differences between the ethics views
held by moral philosophers and those held by the lay
public. However, it may be a matter of opinion whether theprocess of idealization tends to deliver ethics that are in
some neutral sense ‘‘better’’. Moral philosophers are
known to hold a range of idiosyncratic and potentially
catastrophic views, as follows from their inclination to
following speciﬁc moral intuitions to extreme logical
conclusions, wherever that may lead. In some cases, per-haps many cases, it may appear that their pre-moral phi-
losophy, pre-idealization, common sense moral views
would be better.
As an illustrative example, consider the view of Benatar
(2006 ) that it is wrong for humans to procreate. Benatar’s
view is the logical conclusion of his carefully consideredbelief that it is bad to harm people by bringing them into
existence, but it is not good to beneﬁt people by bringingAI & Soc
123
them into existence. Put differently, procreation can be
bad, but it cannot be good. The extreme logical conclusion
of this view is that humanity should die out after the pre-
sent generation. Benatar’s views may well constitute aminority position among moral philosophers,
7but this is
beside the point. The point is that at least in this one case,
the process of idealization would appear to deliver worseresults. This presumes that, prior to putting more thought
into his moral philosophy, Benatar did not wish for
humanity’s extinction, which seems likely: few people with
‘‘common sense’’ moral views would wish such a thing.
Therefore, for cases like Benatar, measuring idealizedvalues instead of ‘‘common sense’’ initial values can deli-
ver worse results. Whether these cases are outliers or the
norm is an empirical question for which an answer may notyet exist. Regardless, it remains the case that the decision
to measure ethics using idealized human values is not an
ethical ‘‘no brainer’’ but instead is a complex and con-tentious decision on which much depends.
If measurement is to use an idealization, there remains
the question of which idealization process to use. Onevariable is whether idealization proceeds via solitary
reﬂection or group deliberation. Yudkowsky ( 2004 : 7) calls
for group deliberation for an idealization of the ethics ofhumans if they ‘‘had grown up farther together’’ instead of
the ethics of ‘‘the person you’d become if you made your
decisions alone in a padded cell’’. Yudkowsky posits thatthe social interactions of group deliberation would provide
‘‘social forces contributing to niceness’’.
However, there is reason to doubt that group delibera-
tion would tend to yield better results. Groups can suffer
from such pathologies as groupthink and in-group favor-
itism (Baron 2005 ; Balliet et al. 2014 ). Meanwhile, solitary
reﬂection can do well. Indeed, much of the most distin-
guished moral philosophy produced across human history
was produced by individuals working largely in solitude.AI designers who choose idealized measurement must
further make the nontrivial choice between idealization via
solitary reﬂection or group deliberation. Other idealizationchoices include how long to let the idealization process
play out and how to handle instances in which idealization
leaves some contradictory incoherences intact.
The various measurement issues discussed thus far
pertain mainly to present-generation humans. While the
issues are substantial, they pale in comparison to the issuespresented by other beings.
For starters, how would a social choice AI measure the
values of sentient nonhuman animals? Some standardhuman measurement techniques could not be used—for
example, cows and frogs cannot meaningfully respond to
survey questionnaires. Other techniques could be used—nonhuman animals’ behavior can be observed, and their
brains can be scanned with imaging technologies—but
interpreting the results is a major challenge. Human AI
designers do not know what it is like to be a cow or a frog,so they must make sweeping assumptions in programming
an AI to infer cow or frog ethics from any given mea-
surement technique.
What about nonsentient animals, plants, and other living
organisms, and nonliving, abiotic matter? The case for
giving them standing is weaker, because they (presumably)
cannot suffer or experience any sort of pleasure. However,
if they are given standing, this presents the challenge ofmeasuring their ethics views. Yet, there is no clear proce-
dure for extrapolating the values of nonsentient beings.
They have no brains to scan. Some of them (e.g., rocks)have no (or barely any) behavior to observe. The social
choice AI is left without the standard empirical techniques
of measurement. The same holds for future generations ofhumans or posthumans. These beings do not yet exist, and
so they cannot be surveyed, observed, scanned, or mea-
sured via any standard empirical technique.
When empirical techniques are inadequate or unavail-
able, an alternative (perhaps the only alternative) is to use
measurement by proxy. The essence of proxy measurementis to have an available being attempt to describe the views
that an unavailable being would have if he, she, or it was
available to be measured. For example, in some humanvoting systems, a person who cannot attend a vote can give
his or her proxy vote to a trusted associate. The associate
then gets to vote on behalf of the absentee person. Simi-larly, proxy measurement is at the heart of proposals to
build representation of future generations into present
democracies (e.g., Tonn 1996 ). Likewise, attempts to
measure the interests of nonsentient beings ultimately
come down to humans seeking to infer what those beings
really ‘‘want’’.
Therefore, if standing is given to nonsentient or future
beings, a social choice AI may need some procedure for
measuring their ethics via proxy. This raises its own set ofquestions: Who to give proxy to? How to measure their
proxy? These are difﬁcult questions, but they at least
resemble the more general questions of standing andmeasurement and thus are both familiar and tractable.
However, the stakes are quite large. Nonsentient and/or
future beings could vastly outnumber present sentientbeings. Therefore, different handlings of proxy could yield
massively different results, not just for each individual’s
ethics measurement but for the entire social choice process.How the entire social choice process would play out would
also depend on the technique used for aggregation.
If AIs have standing, they pose different measurement
issues—or rather, measurement opportunities. It may be
possible to infer an AI’s ethical views from its source code.
7For an argument against Benatar’s views, see Baum ( 2008 ).AI & Soc
123
This is not necessarily a straightforward task: the relevant
code could be large and complex. However, the inference
may nonetheless be feasible, perhaps with the assistance of
an AI designed for the task. In this case, it may be possibleto achieve a high degree of precision in measurement, at
least for this measurement technique. Other techniques
may still yield other descriptions of AIs’ ethical views, justas different techniques can yield different descriptions for
humans and other types of entities.
5 Aggregation
In a social choice process, no matter who has standing and
how their ethics views are measured, once the measure-ments are in, the ﬁnal step is to aggregate these individual
ethics views into a single societal ethics view. For AI, a
single view is needed to determine how the AI is to makedecisions. In other words, the aggregate ethics view is what
the AI is supposed to use to decide how it should behave.
Classic social choice theory recognizes that how to
aggregate is not always straightforward. The simplest case
occurs when three individuals (labeled X, Y, and Z) each
have a different ranking of three choice options (labeled A,B, and C), as shown in Table 1. When the vote is between
A and B, A wins; between A and C, C wins; between B and
C, B wins. Thus, it would appear that the group prefers Ato B, B to C, and C to A. In other words, the preferences are
intransitive. This case is known as the voting paradox or
Condorcet’s paradox, having been documented by Con-dorcet ( 1785 ).
One solution to Condorcet’s paradox is to also consider
the strength of individual preferences for each choiceoption. For example, perhaps X and Y have only a slight
preference for B over C, yet Z has a strong preference for
C, in which case C might be the best option. This solutionis hinted at in Yudkowsky’s ( 2004 : 8) argument that ‘‘A
minor, muddled preference of 60% of humanity might be
countered by a strong, unmuddled preference of 10% ofhumanity.’’ However, this imposes a greater burden on
measurement by requiring the strength of ethics views and
not just ranked orderings. It also makes aggregation moredemanding, because it requires being able to compare
‘‘ethics view strength’’ between individuals.
The inter-individual comparison of ethics view strength
is complicated by the fact that people often hold ‘‘pro-tected’’ or ‘‘sacred’’ values that they believe should not be
traded off against other values, much like deontological
rules (Ginges et al. 2007 ; Ritov and Baron 1999 ). To form
a single ethics view, an aggregation process must resolve
conﬂicts between different people’s protected values. For
example, suppose that one person believes that people
should be categorically prohibited from putting excessive
pollution into the environment, while another believes thatit should be categorically prohibited to deny anyone the
right to drive however they would like. An AI’s attempt to
measure the strength of these views could ﬁnd that theyhave inﬁnite strength pointing in opposite directions. The
AI’s aggregation process requires some means of resolving
such disagreements. One option is to insist that no one’sviews can have inﬁnite strength, but this imposes a dis-
tortion on the views of people who hold protected values.
Another aggregation challenge comes from aggregation
procedures that cluster individuals into distinct groups. In
modern democracies, voters are commonly clustered into
geographic districts. How elections turn out can dependheavily on how district lines are drawn. This is seen in the
concept of ‘‘gerrymandering’’, in which unintuitive district
lines are drawn so as to maximize advantage for a politicalparty, or more generally in the modiﬁable areal unit
problem, in which different demarcations of spatial units
yield different results (Openshaw 1983 ). Similar quirks of
clustering underlie the fact that a US Presidential candidate
can win the popular vote but lose the electoral college and
hence the overall election,
8as well the fact that each US
state has two Senators regardless of its population, giving
proportionately more Senate representation to residents of
small states. The AI designer may be tempted to rejectclustering and stick to a strict ‘‘one individual, one vote’’
principle. However, this can facilitate ‘‘tyranny of the
majority’’ at the expense of minority rights and interests.Indeed, it was to protect the interests of small states that US
Senate representation was designed with two Senators per
state, as per the 1787 Connecticut Compromise (Yazawa2016 ).
Many AIs with bottom–up ethics are designed in a rather
different fashion, updating their ethics each time someoneinteracts with it. This is essentially a ‘‘vote early and often’’
principle. Hence, for example, Microsoft’s chatbot Tay was
trained to speak vulgarities, even though many people did
8This happened in 2000 and 2016, when Al Gore and Hillary
Clinton, respectively, received more votes from individual voters, butGeorge W. Bush and Donald Trump, respectively, received morevotes in the electoral college.Table 1 A hypothetical set of choice rankings from three different
individuals
First choice Second choice Third choice
Individual X A B C
Individual Y B C AIndividual Z C A BAI & Soc
123
not want this—Tay’s training was driven mainly by the
‘‘vocal minority’’ who repeatedly engaged with it. In this
case, the vocal minority aggregation scheme yielded
pathological results. A ‘‘one individual, one vote’’ aggre-gation procedure would have given everyone equal say in
Tay’s training and may have yielded better results.
9
Further complications come when nonhumans have
standing. Do their views count the same as humans? A
‘‘one individual, one vote’’ principle may make a little
sense, for example, if standing is given to the massive
population of bacteria. An argument can be made for
giving nonhuman animals and other living beings less of avote than humans, because humans are more cognitively
sophisticated. However, the same argument would suggest
that humans should have less of a vote if standing isgranted to more sophisticated beings, such as posthumans
or ASIs.
Aggregation for AIs poses an additional complication
that derives from AI reproduction. An AI can be copied
extensively; if each copy has standing, and each has equal
say in the aggregation process, then this can drown outeveryone else’s vote. An aggregation process could face a
dilemma between disenfranchising AIs or disenfranchising
everyone else. A similar issue is posed by future genera-tions, given their astronomically large potential population.
It should be noted that these various aggregation chal-
lenges only exist to the extent that different individualshave different ethics views. Where measurement yields
consensus, aggregation is irrelevant. Some AI theorists
posit that superintelligent beings all tend towards the sameuniversal set of values (Goertzel 2016 ). Whether or not this
is the case remains to be seen and cannot a priori be
counted on. Without consensus, issues of aggregation mustbe addressed.
6 Discussion and conclusion
Having considered issues of standing, measurement, andaggregation in detail, let us now revisit the procedural and
abstention justiﬁcations for social choice ethics in AI
(Recall that the wise crowd justiﬁcation requires a prede-termined ethical view).
The procedural justiﬁcation maintains that individuals
should have a say in decisions that affect them instead ofAI designers imposing their own views on them. However,
this ideal leaves open questions about standing, measure-
ment, and aggregation; AI designers cannot avoid imposingtheir own views on these questions. At most, social choiceethics reduces the extent to which AI designers impose
their own views. Furthermore, it is often difﬁcult or even
impossible to give everyone a say. When AI has global
effects, such as in autonomous vehicles affecting globalwarming or ASI that takes over the world, a massive
number of individuals are affected, most of whom may not
have direct interaction with the AI. When AI affects futuregenerations of humans, future entities such as posthumans
or new forms of AI, or less intelligent nonhuman entities,
giving them a say becomes inherently difﬁcult. Thus, while
the procedural justiﬁcation may capture an admirable eth-
ical sentiment, it provides weak support and limited guid-ance for AI ethics design.
The abstention justiﬁcation maintains that by using
social choice ethics, AI designers can abstain from makingethics decisions and instead let the AI ﬁgure out what
society’s aggregate views are. AI designers may take
comfort in this insofar as they are unaccustomed or disin-clined to think about ethics. However, use of social choice
ethics requires decisions about standing, measurement, and
aggregation, which can in turn substantially affect out-comes of AI decisions. There is simply no way for AI
designers to successfully abstain from ethical decision-
making. Furthermore, the ethical issues in standing, mea-surement, and aggregation are numerous and profound.
Using social choice ethics requires much more than a
minimal amount of ethical thought. The abstention justiﬁ-cation fails.
AI designers are left with two options. They can use
social choice ethics and make the profound ethical deci-sions of standing, measurement, and aggregation. They
may justify their use of social choice ethics with a weak-
ened form of the procedural justiﬁcation; their decisionsregarding standing, measurement, and aggregation would
require separate justiﬁcation. Or, they can abandon social
choice ethics in favor of a predetermined ethical view. Apredetermined view would require its own justiﬁcation, but
this may not be much different than justifying decisions of
standing, measurement, and aggregation.
Indeed, depending on the details, the distinction between
social choice ethics and predetermined view ethics may be
insigniﬁcant. For example, consider the question ofstanding for future generations. This is broadly similar to
the question of whether to count the welfare of future
generations in welfarist utilitarianism. An AI may make thesame decisions under either social choice ethics or wel-
farist utilitarianism as long as it does (or does not) count
future generations. For example, an autonomous vehiclethat counts future generations may drive in an energy
efﬁcient fashion to reduce its greenhouse gas emissions,
while an ASI may embark on long-term projects at theexpense of short-term splurges. Or, consider the question of
whether to give standing to AIs. A social choice ethics may
9There is no indication that Tay was designed with bottom–up ethics
in mind, but the net result is the same in that Tay acquired itsprinciples for behavior via input from the people it interacted with.AI & Soc
123
give AIs standing if they have the capacity to form ethical
views. Similarly, welfarist utilitarianism may count AIs if
they have the capacity to experience pleasure and pain. In
both cases, the moral calculus could be dominated by theoverwhelmingly large number of AIs that can be created by
copying AI software ad inﬁnitum.
In these and other cases, the distinction between social
choice ethics and predetermined view ethics is unimpor-
tant. Thus, proposals such as CEV and bottom–up ethics do
not actually do much to resolve the important decisions to
be made in the design of AI ethics. These are inherently
decisions that must be made by AI designers—one cannot‘‘let the AI ﬁgure it out’’, because the decisions concern
how the AI would ﬁgure it out. Focus should likewise be on
the important decisions, not on whether AI uses some sortof social choice ethics.
Acknowledgements Anders Sandberg provided helpful discussion
for the development of this paper. Tony Barrett and two anonymous
reviewers provided helpful feedback on earlier drafts. Any errors or
shortcomings in the paper are the author’s alone. Work on this paperwas funded in part by Future of Life Institute Grant Number2015-143911. The views in this paper are the author’s and are not
necessarily the views of the Future of Life Institute or the Global
Catastrophic Risk Institute.
References
Adams FC (2008) Long-term astrophysical processes. In: Bostrom N,
C´irkovic ´MM (eds) Global catastrophic risks. Oxford University
Press, Oxford, pp 33–47
Allen C, Varner G, Zinser J (2000) Prolegomena to any future
artiﬁcial moral agent. J Exp Theor Artif Intell 12:251–261
Allen C, Smit I, Wallach W (2005) Artiﬁcial morality: top-down,
bottom-up, and hybrid approaches. Ethics Inf Technol7(3):149–155
Anomaly J (2015) What’s wrong with factory farming? Public Health
Ethics 8(3):246–254
Arrhenius G (2005) The boundary problem in democratic theory. In:
Tersman F (ed) Democracy unbound: basic explorations I.
Filosoﬁska Institutionen, Stockholm, pp 14–29
Arrhenius G (2011) The impossibility of a satisfactory population
ethics. In: Dzhafarov E, Lacey P (eds) Descriptive and normative
approaches to human behavior. World Scientiﬁc, Singapore,
pp 1–26
Arrhenius G, Rabinowicz W (2015) The value of existence. In: Hirose
I, Olson J (eds) The Oxford handbook of value theory. Oxford
University Press, Oxford, pp 424–443
Arrow KJ (1951) Social choice and individual values. Wiley, New
York
Balliet D, Wu J, De Dreu CKW (2014) Ingroup favoritism in
cooperation: a meta-analysis. Psychol Bull 140(6):1556–1581
Baron RS (2005) So right it’s wrong: groupthink and the ubiquitous
nature of polarized group decision making. Adv Exp Soc
Psychol 37:219–253
Baum SD (2008) Better to exist: a reply to Benatar. J Med Ethics
34(12):875–876
Baum SD (2009) Description, prescription and the choice of discount
rates. Ecol Econ 69(1):197–205Benatar D (2006) Better never to have been: the harm of coming into
existence. Oxford University Press, Oxford
Bohannon J (2015) Fears of an AI pioneer. Science 349(6245):252Borenstein J, Arkin R (2016) Robotic nudges: the ethics of
engineering a more socially just human being. Sci Eng Ethics
22(1):31–46
Bostrom N (2008) Why I want to be a posthuman when I grow up. In:
Gordijn B, Chadwick R (eds) Medical enhancement and
posthumanity. Springer, Berlin, pp 107–136
Bostrom N (2014) Superintelligence: paths, dangers, strategies.
Oxford University Press, Oxford
Brandt F, Conitzer V, Endriss U, Lang J, Procaccia AD (2015)
Handbook of computational social choice. Cambridge University
Press, Cambridge
Buchanan A (2009) Moral status and human enhancement. Philos
Public Aff 37(4):346–381
Clark J (2016) Artiﬁcial intelligence has a ‘sea of dudes’ problem.
Bloomberg, New York
Cockell CS (2007) Originism: ethics and extraterrestrial life. J Br
Interplanet Soc 60:147–153
de Condorcet M (1785) Essai sur l’Application de l’Analyse a `la
Probabilite ´des De ´cisions Rendues a `la Pluralite ´des Voix.
L’imprimerie Royale, Paris
Fossat P, Bacque ´-Cazenave J, De Deurwaerde `re P, Delbecque JP,
Cattaert D (2014) Anxiety-like behavior in crayﬁsh is controlledby serotonin. Science 344(6189):1293–1297
Foucault M (1961) Folie et De ´raison: Histoire de la Folie a `l’aˆge
Classique. Plon, Paris
Frederick S, Loewenstein G, O’donoghue T (2002) Time discounting
and time preference: a critical review. J Econ Lit 40(2):351–401
Funk C, Kennedy B, Podrebarac Sciupac E (2016) U.S. public wary
of biomedical technologies to ‘enhance’ human abilities. PewResearch Center
Gibbs S (2016) Microsoft’s racist chatbot returns with drug-smoking
Twitter meltdown. The Guardian
Ginges J, Atran S, Medin D, Shikaki K (2007) Sacred bounds on
rational resolution of violent political conﬂict. Proc Natl AcadSci 104(18):7357–7360
Goertzel B (2016) Infusing advanced AGIs with human-like value
systems: two theses. J Evol Technol 26(1):50–72
Hannon B (1998) How might nature value man? Ecol Econ
25:265–279
Harsanyi JC (1996) Utilities, preferences, and substantive goods. Soc
Choice Welf 14(1):129–145
Holbrook D (1997) The consequentialistic side of environmental
ethics. Environ Values 6:87–96
Hubbard FP (2011) ‘Do androids dream?’: Personhood and intelligent
artifacts. Temple Law Rev 83:405–441
Klein A (2016) Robot ranchers monitor animals on giant Australian
farms. New Scientist
Lin P (2016) Why ethics matters for autonomous cars. In: Maurer M,
Gerdes JC, Lenz B, Winner H (eds) Autonomous driving:technical, legal and social aspects. Springer, Berlin, pp 69–85
Marglin SA (1963) The social rate of discount and the optimal rate of
investment. Q J Econ 77(1):95–111
Martin D (2017) Who should decide how machines make morally
laden decisions? Sci Eng Ethics 23(4):951–967
Mersky AC, Samaras C (2016) Fuel economy testing of autonomous
vehicles. Transp Res Part C Emerg Technol 65:31–48
Metz R (2014) Startup Knightscope is preparing to roll out human-
size robot patrols. MIT Technol Rev
Muehlhauser L, Helm L (2012) Intelligence explosion and machine
ethics. In: Eden A, Søraker J, Moor JH, Steinhart E (eds)Singularity hypotheses: a scientiﬁc and philosophical assess-
ment. Springer, Berlin, pp 101–126AI & Soc
123
Ng YK (1990) Welfarism and utilitarianism: a rehabilitation. Utilitas
2(2):171–193
Ng YK (1999) Utility, informed preference, or happiness: following
Harsanyi’s argument to its logical conclusion. Soc Choice Welf
16(2):197–216
O’Malley-James JT, Cockell CS, Greaves JS, Raven JA (2014)
Swansong biospheres II: the ﬁnal signs of life on terrestrialplanets near the end of their habitable lifetimes. Int J Astrobiol
13:229–243
Openshaw S (1983) The modiﬁable areal unit problem. Geo Books,
Norwich
Pew Research Center (2017) Changing attitudes on gay marriage
Picard R (1997) Affective computing. MIT Press, Cambridge
Ritov I, Baron J (1999) Protected values and omission bias. Organ
Behav Hum Decis Process 79(2):79–94
Rolston H III (1986) The preservation of natural value in the solar
system. In: Hargrove EC (ed) Beyond spaceship Earth: environ-
mental ethics and the solar system. Sierra Club Books, SanFrancisco, pp 140–182
Rose JD, Arlinghaus R, Cooke SJ, Diggles BK, Sawynok W, Stevens
ED, Wynne CDL (2014) Can ﬁsh really feel pain? Fish Fish15(1):97–133
Schienke EW, Tuana N, Brown DA, Davis KJ, Keller K, Shortle JS,
Stickler M, Baum SD (2009) The role of the NSF Broader
Impacts Criterion in enhancing research ethics pedagogy. SocEpistemol 23(3–4):317–336
Schienke EW, Baum SD, Tuana N, Davis KJ, Keller K (2011)
Intrinsic ethics regarding integrated assessment models for
climate management. Sci Eng Ethics 17(3):503–523Stone C (1972) Should trees have standing? Toward legal rights for
natural objects. South Calif Law Rev 45:450–501
Stone J, Fernandez NC (2008) To practice what we preach: the use of
hypocrisy and cognitive dissonance to motivate behavior change.
Soc Personal Psychol Compass 2(2):1024–1051
Sunstein CR (2000) Standing for animals. UCLA Law Rev
47(5):1333–1368
Tarleton N (2010) Coherent extrapolated volition: a meta-level
approach to machine ethics. The Singularity Institute, Berkeley,
CA
Thaler R, Sunstein C (2008) Nudge: improving decisions about
health, wealth, and happiness. Yale University Press, New
Haven
Tonn B (1996) A design for future-oriented government. Futures
28(5):413–431
Wallach W, Allen C (2008) Moral machines: teaching robots right
from wrong. Oxford University Press, Oxford
Wallach W, Allen C, Smit I (2008) Machine morality: bottom-up and
top-down approaches for modelling human moral faculties. AI &
Soc 22(4):565–582
Yampolskiy RV (2013) Artiﬁcial intelligence safety engineering: why
machine ethics is a wrong approach. In: Mu ¨ller VC (ed)
Philosophy and theory of artiﬁcial intelligence. Springer, Berlin,
pp 389–396
Yazawa M (2016) Contested conventions: the struggle to establish the
constitution and save the union, 1787–1789. Johns HopkinsUniversity Press, Baltimore
Yudkowsky E (2004) Coherent extrapolated volition. The Singularity
Institute, San FranciscoAI & Soc
123