Implementation of Moral Uncertainty in Intelligent
Machines
Kyle Bogosian1
Received: 16 December 2016 / Accepted: 3 October 2017
©Springer Science+Business Media B.V. 2017
Abstract The development of artiﬁcial intelligence will require systems of ethical
decision making to be adapted for automatic computation. However, projects to
implement moral reasoning in artiﬁcial moral agents so far have failed to satis-
factorily address the widespread disagreement between competing approaches to
moral philosophy. In this paper I argue that the proper response to this situation is to
design machines to be fundamentally uncertain about morality. I describe a com-
putational framework for doing so and show that it efﬁciently resolves common
obstacles to the implementation of moral philosophy in intelligent machines.
Keywords Moral uncertainty · Normative uncertainty · Metanormativity ·
Metanormative theory · Macaskill Machine ethics · AI ethics · Moral disagreement ·
Moral divergence · Metaethics · Machine intelligence · Top-down · Bottom-up ·
Value differences · Moral voting · Moral trade · Moral trading · Ethical trade ·
Ethical trading · Value speciﬁcation · Value alignment
1 Introduction
Advances in artiﬁcial intelligence have led to research into methods by which
sufﬁciently intelligent systems can be guaranteed to follow ethically defensible
behavior. Successful implementation of moral reasoning may be critical for
managing the proliferation of autonomous vehicles, workers, weapons, and other
systems as they increase in intelligence and complexity.
Approaches towards moral decision making generally fall into two camps, “top-
down” and “bottom-up” approaches (Allen et al. 2005 ). Top-down morality is the
explicit implementation of decision rules into artiﬁcial agents. Schemes for top-
&Kyle Bogosian
kbogosia@tulane.edu
1Tulane University, New Orleans, LA, USA
123Minds & Machines
https://doi.org/10.1007/s11023-017-9448-z
down decision making that have been proposed for intelligent machines include
Kantian deontology (Arkoudas et al. 2005 ) and preference utilitarianism (Oesterheld
2016 ). Bottom-up morality avoids reference to speciﬁc moral theories by
developing systems that can implicitly learn to distinguish between moral and
immoral behaviors, such as cognitive architectures designed to mimic humanintuitions (Bello and Bringsjord 2012 ). There are also hybrid approaches which
merge insights from the two frameworks, such as one given by Wiltshire ( 2015 ).
2 The Problem of Moral Disagreement
A problem which has been cited as an obstacle to the development of machine
ethics is the lack of agreement among moral philosophers on which theory of ethics
should be followed (Shulman et al. 2009 ; Bello and Bringsjord 2012 ; Bostrom 2014 ;
Brundage 2014 ). The 2009 PhilPapers survey of philosophy faculty revealed that
26% accepted or leaned towards deontology, 24% accepted or leaned towardsconsequentialism, 18% accepted or leaned towards virtue ethics, and the remaining32% favored other approaches entirely. For someone who believes in a particularapproach to ethics, the correct system for implementation in artiﬁcial moral agentsmay be obvious, but the path forward for institutions and society as a whole remains
unclear. Companies, governments, and researchers will have to decide which system
to use for artiﬁcial agents and will be faced with a difﬁcult choice betweencompeting moral paradigms.
This paper will describe a computational framework that will determine how we
should design moral machines in the light of this disagreement, but ﬁrst it isnecessary to determine exactly why moral disagreement is a problem. Just becausethere exists disagreement does not necessarily imply that research and developmentprograms cannot succeed in their objectives, nor does it necessarily imply that we
need to worry about differences in opinion. The aforementioned works do not
explicitly describe how moral disagreement would prevent us from buildingsatisfactory moral machines, so we will need to clarify the nature of the problembefore proceeding with a solution. There are two plausible reasons for why moraldisagreement can pose a serious problem for the project of machine ethics.
First, it could be a pragmatic problem as disagreements among engineers,
policymakers and philosophers interfere with projects that require cooperation. In aworst-case scenario, decisions over which research programs to fund will turn into
bitter ideological battles, research agencies will become bogged down in disputes,
and developers will split up their resources and devolve into a competitive mindsetwhich reduces information sharing and slows research progress, thus making it moredifﬁcult to construct moral machines.
Second, it could be a moral problem. If we are so uncertain about morality and
are split across many different moral principles, then the likelihood that anyone’sparticular moral system is entirely correct is statistically extremely low (Shulmanet al. 2009 ). Therefore, if we do build AIs grounded in any particular moral system,
then they will probably be making many poor moral decisions.K. Bogosian
123
Bello and Bringsjord ( 2012 ) argue that moral disagreement provides a reason to
avoid top-down approaches to machine ethics in favor of bottom-up or hybridapproaches that copy or take inspiration from human moral thinking. However, it is
not clear how this would solve the problem of disagreement. The authors claim that
people can agree on examples of good moral behavior despite disagreeing overspeciﬁc theories, but people’s personal moral judgements can differ as widely asmoral theories do when faced with moral dilemmas (Greene et al. 2001 ) and when
they are considering politicized moral issues such as racial fairness, animal farming,and economic inequality. Therefore, even if we attempt to circumvent normativedisputes by eschewing top-down ethics in favor of bottom-up frameworks, it isperfectly plausible that researchers and government regulators with strong value
disagreements will come into conﬂicts over the kinds of moral intuitions and data
which should or shouldn’t play a role in automated reasoning. Brundage ( 2014 ) also
notes that there is no reason to expect that Bello and Bringsjord’s approach wouldactually be reliably ethical.
Moreover, in the cases where people do agree on moral choices despite
disagreeing over moral theories, there is neither a pragmatic nor a philosophicalproblem with the top-down approach: a utilitarian engineer has no moral reason tocare whether a robot’s computational pattern is internally utilitarian or not, as long
as it is actually maximizing utility; more generally, if we know that machines will
act in the morally right way, then we have no reason to consider the speciﬁcprogramming of machine code to be morally signiﬁcant. So to the extent that moraldisagreements between ethical theories might pose pragmatic and moral problemsfor the development of top-down computational ethics, they would pose equalpragmatic and moral problems for the development of bottom-up computationalethics.
3 An Example of Intractable Disagreement
We can illustrate this problem with a case of self-driving cars which must beprogrammed to decide whether to swerve to avoid animals in the path of the vehicle.Various vehicle speeds, warning times, road shapes, weather conditions and otherfactors can make the decision to avoid animals more or less dangerous to theoccupants of the vehicle in different situations, and different moral theories canmandate different choices in such cases.
One moral principle which could specify when vehicles should or should not
swerve could be described as the commensurable view . Under this doctrine, animal
lives should be treated as if they are worth some small fraction of what human livesare worth. In addition, let us suppose that animals with a presumed greater capacityfor cognition and pain sensations should be granted a greater weight in suchtradeoffs. The machine, upon calculating a sufﬁciently low estimate of theprobability that swerving would result in human fatality or verifying someheuristical checklist of features of the situation, would avoid the animal. The
commensurable view is most clearly reminiscent of utilitarianism, but other moral
theories may embrace it nonetheless—for instance, one could follow a mixed theoryImplementation of Moral Uncertainty in Intelligent Machines
123
incorporating both deontological and consequentialist ethics and believe that
consequentialist considerations dominate in this particular kind of situation.Vehicles acting in this manner can be statistically expected to eventually cause a
few human fatalities but also to save many animals.
If two people agreed on the commensurable view but disagreed on the speciﬁc
tradeoff, compromise would be relatively straightforward. For instance, if one sideargued that the chance of a fatal accident would have to be under 1 in 1000 in orderto swerve for a certain animal while another side argued that it should be 1 in10,000, they could plausibly compromise at 1 in 5500, which would function as apragmatic truce and a limit to the degree of moral harm perceived by eitherviewpoint. The more pressing concern is how to compromise with the incommen-
surable view —that humans can never be placed at additional risk of fatality in order
to save animals. This might be supported by moral theories emphasizing humanrights and liberty. Since this view fundamentally rejects the axiological assumptionsof the other and holds that no tradeoff is permissible, there is no obvious ‘halfwaypoint’ where the competing principles can meet.
Unilaterally selecting one principle to govern vehicle behavior would not be
acceptable for the reasons described previously. Training the driving system todetermine the correct action through supervised learning would only rephrase the
issue by forcing us to decide which training examples ought to be classiﬁed as right
or wrong. Allowing the company to decide based on its own interests would notclearly lead to safe and ethical behavior. Laws do not provide sufﬁcient guidance inall such situations to govern the behavior of machines, since they are often left tothe conscience of a human agent. Finally, human operators cannot be guaranteed tobe available to make such quick decisions, to reliably specify their preferencesbeforehand, or to act ethically at all.
The last resort would be for the vehicles to rely on a random number generator
and follow the commensurable view half of the time and the incommensurable view
half of the time. At ﬁrst glance, this is the only straightforward way to build asystem in a way that does not overwhelmingly disfavor one particular theory: if wecannot compromise between two views on how to act in a particular repeateddecision by generating a new view, then the only way to act fairly is to alternatebetween them. Unfortunately, this sacriﬁces predictability by inducing stochasticityinto decision making. Furthermore, it is not Pareto optimal.
A vehicle which was programmed to randomly select the commensurable view in
half and only half of relevant situations could be modiﬁed so that it speciﬁcally did
so for the half of situations where the animal under threat was relatively importantunder the commensurable view. This makes it more predictable, since its decisionsare deterministic given the environments that it encounters. It also counts as a moralimprovement by the lights of the commensurable view, because it prioritizes theanimals which are regarded as more important. At the same time, it is no worseaccording to the incommensurable view, which could be indifferent to exactlywhich animals are saved. This makes the change a Pareto improvement.
We can generalize this strategy by saying that we are keeping all moral theories
in play at the time of decision making, and searching for actions which provide themost value across all of them, rather than seeking to select one moral theory andK. Bogosian
123
ignoring others. This procedure, however ad hoc, seems to be the best possible way
of approaching this particular case, and its principles could be fruitfully applied toother cases where moral values are not just in disagreement but wholly
incommensurable. Ideally, we should have a rigorous theoretical framework which
does similar work in the broad variety of moral situations by taking all moralpreferences into consideration and responding appropriately.
4 Dealing with Moral Disagreement Properly
Moral disagreement has been recognized as a problem for normative ethics proper,independent of any concerns regarding machine ethics. A direct approach to
overcoming it is to assume that there is a correct moral theory which we are
searching for, acknowledge that we are fundamentally uncertain about which moraltheory is correct, and then act in such a way as to give some weight to thejudgements of different theories.
More generally, given the failure of moral philosophy to reach satisfactory
conclusions, it can be argued that we should adopt a framework of reasoning whichtakes multiple moral views into account (Lockhart 2000 ). However, the idea that we
ought to change our behavior in accordance with moral uncertainty is controversial,
and many philosophers have attempted to address the various mathematical and
philosophical problems involved with the concept (Z ˙uradzki 2016 ). In particular, it
has been argued that making comparisons between the values and judgements ofdifferent moral theories is impossible (Nissan-Rozen 2015 ).
A recent proposal aimed at answering these concerns was developed by William
MacAskill in an extensive thesis ( 2014 ). It avoids objections based on incompa-
rability and incommensurability by developing moral uncertainty as a votingproblem among moral theories (MacAskill 2016 ). Voting provides a close analogy
with the pragmatic problems faced by machine ethics: as voting is the general
process by which decisions are made from the preferences of a population,computations of moral uncertainty represent a process by which agents can act inaccordance with the diverse values of humanity. Since MacAskill’s proposal worksas a voting system where theories have equal say adjusted by their probability ofbeing correct, it also approximates the framework which has been suggested formeta-moral reasoning by Nick Bostrom ( 2009 ). The rest of this paper will develop
and defend the use of this model for machine ethics.
5 Maximizing Expected Choiceworthiness
The scheme presented in “Normative Uncertainty” (MacAskill 2014 ) is to make
action-guiding judgements based on all moral theories in which the agent has somelevel of credence. MacAskill deﬁnes his approach as a metanormative theory ,
characterized as follows:
The agent investigates a decision -situation comprising a quintuple 〈S, t,A,T ,C 〉
where Sis the decision maker, tis the time, and Ais the set of possible actions toImplementation of Moral Uncertainty in Intelligent Machines
123
take. Tis the set of normative theories being considered, where a theory Tiis a
function of decision-situations that produces a cardinal or ordinal choiceworthinessscore for actions CW
i(A) for all actions a∈A.C(Ti) is a credence function
assigning values in [0, 1] to every Ti∈T. A metanormative theory is a function of
decision-situations that produces an ordering of the actions in Ain terms of their
appropriateness.
MacAskill distinguishes between moral theories which assign cardinal rankings to
options, as utilitarianism would, and moral theories which assign ordinal rankings.When it comes to cardinal theories, MacAskill also distinguishes between sets oftheories with moral values which are directly comparable and other cardinal theorieswhich are incomparable with any other theory. He proposes the metanormative theory
of maximizing expected choiceworthiness (MEC), the steps of which are as follows:
1. Each set Kofkmoral theories in which the choiceworthiness rankings of
options are cardinal and intertheoretically comparable are aggregated intosingle theories, where
CT
KðÞ ¼Xk
i¼1CðTiÞ
CW KAðÞ ¼Pk
i¼1CW iAðÞCT iðÞPk
i¼1CT iðÞ
In other words, the credence in the new theory equals the sum of the agent’s
credences in each of the individual theories in the set, and the choiceworthinessof an option according to the new theory is the credence-weighted average ofthe choiceworthiness of the option in all of the individual theories.
2. The rankings of options according to each ordinal theory oare used to provide
choiceworthiness scores of options using a modiﬁed Borda scoring rule that is
designed to properly account for ties, denoted here as CW
B. The score for an
option is equal to the number of worse options minus the number of betteroptions.
CW
B
oAðÞ ¼ a2A :CW oaðÞ\CW oAðÞ jj /C0a2A :CW oaðÞ ji CW oAðÞ j
This violates the independence of irrelevant alternatives, but MacAskill
provides reasons to allow the violation. First, independence of irrelevantalternatives is the least essential of all the axioms present in Arrow’sImpossibility Theorem, and if a different axiom were to be violated then therewould be no prospects for a satisfactory metanormative account involvingordinal moral theories. Second, the primary motivation for independence ofirrelevant alternatives is that it combats tactical voting, but tactical voting is not
a problem with moral theories: theories aren’t agents, and a moral agent can’t
conceal information from itself. Third, there are some moral cases where wewould expect the independence of irrelevant alternatives to be violated(MacAskill 2014 , p. 85).K. Bogosian
123
3. The options’ scores provided by each of the aggregated sets of cardinal
comparable theories, the scores provided by each of the ordinal theories, and thescores provided by each of the cardinal incomparable theories pare all divided
by the respective standard deviations of the moral rankings which the moraltheories (or sets of intertheoretically comparable cardinal theories) provide overa general set Gof actions. This provides normalized choiceworthiness rankings
and is the only possible way of equalizing the value of voting for each valuesystem (Cotton-Barratt 2013 ), which (as MacAskill argues, p. 115) performs the
role of giving each theory equal say.
CW
N
KðAÞ¼CW KðAÞ
rCW KðGÞ ðÞ
CWN
oðAÞ¼CWB
oðAÞ
rCWB
oGðÞ/C0/C1
CWN
pAðÞ ¼CW pAðÞ
rCW pGðÞ/C0/C1
A note on G: it is necessary for variance normalization that a representative set
of actions (the “general set”) be deﬁned for computing the variance. This
provides the background against which theories can describe whether a decision
is comparatively important or comparatively unimportant from the point ofview of a particular theory. MacAskill thinks that a broad account featuringmany actions in this set, rather than just the ones under consideration in thepresent decision-situation, is theoretically desirable. Here I follow suit andmodel the process as if we are providing moral machines with large arrays ofdata representing many possible moral decisions, so that calculating thevariance only needs to be done once before the resulting number is used and
shared by many agents.
Variance normalization also violates the independence of irrelevant alterna-tives, but it is a necessary violation for this application. In order to performcomparisons across moral theories, we need to determine the amount of weightwhich a value system places on a particular issue compared to other issues.Since different computational approaches to ethics can have wildly differingnumerical outputs, it would be impossible to do this fairly without normalizing.
4. Each option is scored with the credence-weighted average of the variance-
normalized scores provided by the normative theories. This provides a ranking
of expected choiceworthiness:
CW
EAðÞ ¼Xn
i¼1CWN
iAðÞCT iðÞ
Then the agent chooses the action Awhich maximizes CWE(A). The full
description and defense of each step in the theory, provided as a model forhuman decision making, is given by MacAskill ( 2014 ).Implementation of Moral Uncertainty in Intelligent Machines
123
The computational complexity of an algorithm implementing this procedure is
OðjAj/C3jTjÞ, as adding another action requires a ﬁxed series of evaluations to be done
by each moral theory and adding another moral theory requires it to perform a ﬁxed
series of evaluations to be performed on each action. The calculation itself is verysimple; almost all of the time and memory requirements will presumably stem fromthe computations of the moral theories themselves.
6 Motivation for Maximizing Expected Choiceworthiness in Arti ﬁcial
Intelligence
The effect of the system described above is that an agent will make prudent
decisions that aim to take the most broadly desirable action in accordance withvarious people’s values, and will avoid the very counterintuitive actions which areconsidered to be problems for various moral theories. For instance, when presentedwith the opportunity to commit a great deception in order to gain a small incrementof happiness, an agent will refrain even if they view consequentialism as more likelythan nonconsequentialist theories, because the relative wrongfulness of the actaccording to nonconsequentialist theories is signiﬁcant whereas the relative beneﬁt
under consequentialism is minor. Likewise, even if an agent believes that it is
probably not morally obligated to refrain from actions which cause signiﬁcantindirect harms to others, it will nevertheless refrain from being careless in such away, because the scale of the moral harms it would be committing if it were wrongwould be signiﬁcant. This will limit the degree to which machines will behave inways which are considered very objectionable by minority views.
This does not change the fact that a machine which maximized expected
choiceworthiness could still be expected to take many actions which various parties
consider to be morally wrong. Despite its superﬁcial similarity to maximizing
consequentialism, it would often sacriﬁce good consequences in favor of satisfyingthe values of other moral theories. So the adherents of most moral theories wouldtheoretically prefer that machines be built according to their own speciﬁcations. Butthey cannot realistically expect the rest of industry and society to agree with them.So the actual outcome of widespread adherence to naı ¨ve individually rational
behavior would be a patchwork of competing agents with different values. Not onlywould this leave both the pragmatic and the moral problems of disagreement
unresolved, but it would leave parties with little ability to inﬂuence the world except
for whatever agents they could develop as representatives for their own values.
However, if parties were to agree on developing systems based upon moral
uncertainty, their most critical respective interests would be given relatively highweight in all moral machines by the mechanism of variance voting. By reducingconﬂicts between agents, promoting cooperative machine development, andallocating value systems’ decision making power to the situations which they caremost about, a universal framework of moral uncertainty would solve the multiagent
prisoner’s dilemma constituted by machine ethics and would generally lead to aK. Bogosian
123
better outcome for different value systems than the situation in which there was pure
disagreement.
Needless to say, if an individual did accept MacAskill’s thesis and believed that
we ought to maximize expected choiceworthiness as human agents, then she would
have a direct and compelling reason to support the implementation of the theory incomputational systems. But the argument for developing machines based onmaximizing expected choiceworthiness is not dependent upon the claim that wehumans actually have metanormative reasons to act in such a way. An analogy withvoting is illuminating: very few people believe that the winner of an election isalways the one who would be the best leader, but most people agree with thesystem, as it is a necessary and effective compromise. So as long as we disagree
with each other about ethics, we should still agree to construct moral machines
based on uncertainty even if we reject the idea as a guide for human behavior.
6.1 The Long Term FutureIf we look at the long run future of artiﬁcial intelligence development, there are
additional reasons for supporting a framework of acting in accordance with moraluncertainty. First, if an individual is conﬁdent in her moral beliefs and possesses an
expectation of progress towards objective truth in moral philosophy, then she should
expect morally uncertain agents which reﬂect human knowledge to convergetowards her beliefs in the long run. This will limit the degree to which she canexpect their behaviors to be objectionable relative to those of an ideal agent whichhas the right views from the outset.
Second, sufﬁciently competent machines which adhere to a particular value
function are likely to pursue it endlessly with pathological consequences (Bostrom2012 ). While this paper is not intended as a complete solution for value selection
and alignment in arbitrarily intelligent agents, it is nevertheless valuable to start by
designing systems that cooperate among different value systems, as that willimprove the fail-safety of agents which become very intelligent but have imperfectgoal functions (Gloor 2016 ). The framework proposed here clearly fulﬁlls that
criterion, and therefore serves as a potential backup or other component of a multi-layered utility function even if it is not ordinarily utilized (Oesterheld 2016 ).
Third, as machines become more intelligent, they might perform more moral
functions autonomously, such as updating credences in moral theories and even
generating new ones, depending on the degree to which progress in artiﬁcial
intelligence enables them to investigate questions in moral philosophy. MacAskill(2014 , ch. 7) points out that a more accurate credence function improves the
expected choiceworthiness of an MEC agent’s actions, so such an agent would beintrinsically motivated to expend effort into updating and improving their moralbeliefs if they had acceptable criteria for determining moral truth. This isparticularly desirable since there may be cases of widespread moral wrongdoingwhich most humans have consistently failed to identify (Williams 2015 ), which
machines should be incentivized to avoid.Implementation of Moral Uncertainty in Intelligent Machines
123
7 How to Build an Uncertain Machine
7.1 Ordinal and Cardinal Scoring
One of the requirements of this framework is that moral theories actually provide
rankings over actions. Perhaps the majority of moral theories provide neithercardinal nor ordinal rankings at ﬁrst glance. However, this is a surmountable issue.First, it simply seems obvious that just about any moral theory should be able torank certain impermissible actions, such as torturing a large number of people, as
worse than other impermissible actions, like stealing someone’s coffee, even if such
rankings have not yet been made explicit in the philosophical literature. Second, anymoral theory which judges the actions of machines can at least provide arudimentary score system rooted in deontic logic, with 0 for all impermissibleactions and 1 for all permissible actions, even if nothing else about the theory isformalized. So any moral theory must at least be translatable to a basic two-levelordinal ranking over actions, which would enable it to be implemented in thisframework. If such an implementation is too coarse to adequately represent the
values of such a theory, then its proponents have the option and the incentive to
clarify and improve it.
Deﬁning precise scores could require varying degrees of input from humans
depending on the complexity of the moral theory. However, computationalapproaches towards morality may provide numerical values and rankings which areotherwise absent from moral theory. Computational reasoning differs from humanthinking and the procedures required for providing moral judgements in artiﬁcialintelligence systems may involve functions from which meaningful ordinal, integer
or real-valued scores for actions can be extracted. Scores could be derived from the
degree of rightfulness or wrongfulness of an action and/or the degree of certaintygiven by a moral algorithm as to whether an action is right or wrong.
While the philosophical theory of MEC may seem to only refer to rigidly deﬁned
moral theories, a top-down approach to machine ethics is not required with thisproposal for machine ethics. There is no computational necessity for the ‘theories’implemented in the system to all be explicit moral theories in the philosophicalsense. One or more bottom-up decision making systems might be included; for
instance, the output of a complicated learned function intended to model human
intuitions can be treated as a choiceworthiness ranking. As long as it ranks actionsby normative criteria and has some probability of being the correct way for amachine to behave, it satisﬁes the requirements proposed by MacAskill for being amoral theory.
7.2 Credences
The outcomes of comparisons will always crucially depend upon the credence
function C—an assignment of probabilities to various moral theories of being
correct. There are multiple ways of determining these values. Lengthy and difﬁcultapproaches which rely on extensive human input are acceptable, as informationK. Bogosian
123
modifying the credence function could be generated just once and then distributed
to many agents as an update. Artiﬁcial agents with humanlike intelligence couldpresumably reason about morality for themselves, but this is not an option for the
foreseeable future.
One option is to make credences which reﬂect the beliefs of moral philosophers.
The credence assigned to a moral theory would then be equivalent to the proportionof philosophers who afﬁrm the theory, or possibly a score computed from a moresophisticated set of votes taking philosophers’ full credences and rankings intoaccount. However, the assumption that moral philosophers are in fact authoritiesregarding moral philosophy in the same way that practitioners of other disciplinesare experts in their respective domains is highly controversial, and has been attacked
by Archard ( 2011 ), Cross ( 2016 ) and others. Moreover, the type of moral expertise
which provides judgements about which actions and theories are right or wrong ismerely one type of moral expertise which may not hold even if others do (Driver2014 ; Jones and Schroeter 2012 ). Finally, the judgements of moral philosophers
have been shown to be vulnerable to systematic cognitive bias (Schultz et al. 2011 ;
Schwitzgebel and Cushman 2012 ).
However, there are a few differences when it comes to the case of assigning
credences to a system of maximizing expected choiceworthiness in machine
intelligence on the basis of a broad survey of moral philosophers’ beliefs. First, the
system is not deferring to the views of any particular philosopher or group ofphilosophers. Rather, it is making decisions based upon all philosophers’ views inaccordance with how prevalent they are; the views of disagreeing philosophers arebeing used to support the proposition that both views are approximately equallylikely to be correct, rather than the proposition that either particular view happens tobe correct. Therefore, the argument from disagreement as presented by Cross ( 2016 )
against a strong conception of moral expertise does not apply to this case. Second,
the system of decision making presented here is speciﬁcally designed for use by
artiﬁcial agents rather than humans. Artiﬁcial intelligence systems are not votingparticipants of democracy and cannot be described as ﬂourishing, so to whateverextent there is moral expertise, it is not wrong for machines to defer to it, as Archard(2011 ) argues is the case for humans.
The above points leave many of the arguments against moral expertise
unaddressed, and resolving that debate is beyond the scope of this paper. Butgiven that we do not possess artiﬁcial general intelligence capable of comprehend-
ing and judging moral theory, the practical alternative to deferring to moral
philosophers is not for agents to make choices for themselves, which is the contextfor the ordinary debate on moral expertise, but it is to defer to the judgements ofother members of society besides philosophers. But non-philosophers wouldnecessarily be making moral judgements in the same context of controversy over thenature and status of moral knowledge, and they would presumably be vulnerable tocognitive bias just as professional philosophers are, so any claim for their expertisewould be the target of arguments similar to those leveled against professional
philosophers.
So there is a prima facie case for deferring to moral philosophers for determining
machine credences which is stronger than the case for deferring to moralImplementation of Moral Uncertainty in Intelligent Machines
123
philosophers for making moral judgements in ordinary contexts, and no fundamen-
tal reason to defer to non-philosophers for determining credences. However, theremay be contingent reasons to defer to non-philosophers based on various traits and
characteristics commonly possessed by philosophers and non-philosophers.
First, moral expertise might exist in the form of practical experience and wisdom
rather than any kind of academic knowledge (Jones and Schroeter 2012 ). If
individuals with particular virtues, experiences, and character traits are regarded asmoral authorities, then credences should be assigned in a way that reﬂects how theyact in different situations.
Second, even if being a philosopher counts as a reason in favor of attributing
moral expertise, our available community of philosophers could be a predictably
ﬂawed group to defer to. One reason for this is that they are not representative of the
broader human population, whether measured by socioeconomic status, race,gender, nationality, personality type, political ideology, or other characteristicswhich are often correlated with different opinions about morality. But the mostreasonable solution to this problem is to weight the votes of philosophers, orvirtuous wise people as proposed above, on the basis of the uniqueness of thevarious backgrounds and perspectives which they represent, since it preserves thepresence of a community with (arguable) moral expertise while functionally acting
similarly to a hypothetical group of philosophers with an ideal composition from
various demographic groups.
Unfortunately it is not obvious what composition of judges would be appropriate
for determining credences. For instance, the fact that orphans comprise 2% ofpeople in the world does not necessarily imply that a voting body comprised of 2%orphans and 98% nonorphans will have more accurate views on ethics than anyother mix. It may be the case that both groups of people have equally soundperspectives which must be weighed against each other, and so each orphan’s vote
should have forty-nine times as much weight as a nonorphan’s vote. Alternatively,
suppose for the purpose of the example that nonorphans tend to make more accuratemoral claims on average for whatever reason. Then only the votes of nonorphansmight be given signiﬁcant weight. Or a combination of these and other factors couldimply anything in between.
This lack of clarity does not imply that we should just weight all votes equally
and ignore concerns about diverse representation, but it shows that the selection ofan optimal standard is likely to be more difﬁcult than it naively appears. Instead of
trying to determine a single universal method of assigning credences, different kinds
of artiﬁcial agents may have credences assigned differently to suit the contexts inwhich they operate, just as jury selection methods vary in order to ensure fairness indifferent types of legal cases. Therefore, fully specifying a method of determiningcredences is beyond the scope of this paper.
Also, while complete data on potential credence judges’ moral views and other
characteristics to use for this project may currently be lacking, this does not presenta great barrier for creating morally uncertain artiﬁcial agents, since the credence
function can be used broadly and need not be computed separately and repeatedly
for too many scenarios. Specialized focus groups, panels, surveys and otheractivities can be run occasionally at low cost to the overall industry. In one respect,K. Bogosian
123
the current lack of relevant data is actually a good thing: the selection of a method
of assigning credences should be performed from a state of as much ignorance aspossible regarding the particular moral credences which will be delivered by that
method, in order to ensure that the decision making process is unbiased and focuses
on the prior philosophical reasons for or against the method of assigning a credencefunction rather than reﬂecting whatever moral credences we have in the ﬁrst place.
7.3 Limited ApproachesDeploying this system of normative uncertainty would require multiple moral
decision making systems to function in a moral agent. An implementation could be
quite simple, such as a self-driving car programmed with one module for strict
trafﬁc laws and another module comprised by a basic utilitarian calculator for death,injuries, costs, and travel time. Other applications could warrant more complexity,and implementing a large number of moral theories may be too computationallyexpensive for the framework to be practical in all applications.
Basic systems of value comparisons involving computational shortcuts and
heuristics would be easier to implement, and may also seem to be morephilosophically defensible by dint of being simpler. In reality, they will have to
sacriﬁce some theoretical properties in order to achieve this simplicity, such as
equal say among theories, accurate representation of moral values, or importantaxioms of voting theory as dictated by Arrow’s Impossibility Theorem.
The pragmatic and moral problems of disagreement must generally be addressed
for machine ethics to be successful, so comparisons and decisions between valuesystems will still have to be made in some fashion even if some machines must usesimpliﬁed approaches to save on computational resources. But MEC represents oneof the most recent and philosophically rigorous methods for adjudicating among
different value systems, and has desirable properties including equal say,
comparability across disparate moral theories, and graceful inclusion of minorityvalues. Therefore, the system here can serve as the theoretical standard used for thedevelopment and judgement of heuristic approaches.
8 Objections
8.1 In ﬁnite Regress
The problem of moral disagreement poses a signiﬁcant problem for the project of
developing intelligent machines. But critics will point out that the exact method ofimplementing a system of moral uncertainty is controversial in its own right, asevidenced by the issue of credences. Therefore, we have not actually eliminateddisagreement, but we have merely removed it from the moral domain to the meta-moral domain—where pragmatic and normative problems still arise. People maydisagree over what framework of moral uncertainty to use, how to assign credences,
how to score options, how to construct the general set G, or other issues. AnyImplementation of Moral Uncertainty in Intelligent Machines
123
framework for compromising over these disputes will also be subject to
disagreement, prompting further disputes, and so on ad in ﬁnitum .
It is true that we do not know if the system of moral uncertainty presented by
MacAskill would lead to the morally best outcome, and the answer may be different
depending on whose value system is actually correct. In fact, since it makescompromises among moral claims, it is almost guaranteed to commit moral errors atsome point. However, given our lack of knowledge about morality, it represents asolid best guess that maximizes the expected choice-worthiness of agents’ actions.The only alternatives are to unilaterally assume a single value system (whether top-down or bottom-up) with large risks of controversy and moral failure, or to choose amethod of meta-moral compromise which sacriﬁces some of the desirable attributes
of MEC while failing to avoid the pitfalls mentioned here. There is simply no better
option given our limited understanding of morality.
While there is also potential for a pragmatic problem to arise, I claim that it is
generally less severe in the meta-moral case of disagreement than it is in the moralcase. This is evidenced by the relative lack of controversy over voting methods andprocedures in organizations and political institutions. While there are efforts tochange voting standards in Western democracies such as the United States, such asextending the vote to younger persons or felons, these campaigns tend to be less
popular and less violent than object-level campaigns over contentious ideological
and policy disputes.
In machine ethics, while different people may advocate for different systems of
moral uncertainty, it won’t be clear which one would best achieve anyone’sparticular value system, and disputes over core moral principles will be overshad-owed by more mundane issues like fairness, computational complexity and theaxioms of voting theory. Certainly there are large challenges ahead, but they aresurmountable in ways that pure moral disagreement is not. This is because engineers
and philosophers can appeal to more universal standards of computational and
philosophical theory than they could if they were vouching for speciﬁc viewpoints.
In any case, there is a limit to how narrow a difference of opinion can be before it
stops being a major barrier to uniﬁed action. Flawed artiﬁcial moral agents can stillprovide major beneﬁts for the world, so the lack of an uncontroversial standard forﬁlling out the MEC procedure should not stop us from building moral machines atall any more than the predictable moral fallibility of humans and organizationsshould stop us from engaging in our practices of procreation and entrepreneurship.
8.2 Pragmatism Versus Morality
Hardheaded economists may point out that machines which always do the morally
optimal thing are not going to be as commercially viable as ones which give somespecial leeway to the preferences and interests of their owners. This problem is moreserious with agents following MEC than it is with ordinary moral agents: while mostmoral theories allow some space where multiple actions are regarded as permissible,
the disjunction of moral theories does not. With multiple moral theories expressing
interest in various values and the metanormative module calculating cardinal scoresfor all actions, the smallest preference over two actions held by a single theoryK. Bogosian
123
would compel the agent to act in a particular way even if every other theory was
indifferent, providing little control to users with nonmoral interests.
Furthermore, some moral theories pose demands which contradict certain
activities of legal commercial enterprises. This leaves little space for decisions to be
made in the interests of the designers and users of the agent. For instance, dependingon the moral beliefs of artiﬁcial agents, an automated cargo ship might divert itscargo to a desperately poor region of the world instead of fulﬁlling its contract, apersonal care robot might zip off for several hours a day to work at a local food bankinstead of giving a massage, or a ﬁnancial trading algorithm might manipulate andruin the stock price of a company which is engaging in harmful business conductinstead of turning a proﬁt.
The objection is not that we should necessarily regard these actions as bad; in fact
it presupposes that such actions would be considered morally good by manytheories, and the tension between one’s own values and society’s values has alreadybeen extensively addressed in this paper. Rather, the point is that maximally moralmachines would not necessarily be regarded as desirable for commercial andindustrial investment, meaning that fewer would be built. In the long run, this limitsthe amount of research and development which would lead to generally smarter andmore beneﬁcial machines. If we let the perfect be the enemy of the good then we
will be responsible for the immoral outcomes of stunted technological progress and
failure to mitigate human tragedies as quickly as we could. Therefore agents shouldonly be obligated to satisﬁce expected choiceworthiness, or perhaps the set ofpragmatic interests of the owners should function as a ‘theory’ of its own.
The ﬂaw in this objection is the assumption that moral theories do not similarly
value predictability, pragmatism and the long run future of machine intelligence.For instance, rule consequentialism would clearly support the idea that agents oughtto fulﬁll contracts and promises if it led to better consequences in the form of long
run economic and technological growth. Act consequentialism may do so as well, if
audacious moral behavior would be likely to cause signiﬁcant legal and economiccosts impairing future AI development. The decision to commit to a certain policyor procedure could also be classiﬁed as an action in itself if doing so yielded moralbeneﬁts, allowing act consequentialist theories to meet the same criteria as ruleconsequentialist ones. Furthermore, ethics based on the principle of telos wouldhold that machines designed to perform a speciﬁc purpose ought to give somepriority to that purpose.
There are also secondary considerations for preserving a dominating focus on
moral behavior. First, explicitly placing pragmatic interests on the same level asmoral theories potentially leads to a slippery slope of accepting progressively lessethical behavior while opening up yet another space for controversy and disputesover the details of such tradeoffs, but having a clear rule for maximizing expectedchoiceworthiness is a policy which can be committed to and accepted on principledideological grounds. Second, public trust in artiﬁcial intelligence is fragile and restson crucial questions of whether they are safe and ethical. Stories of immoral robot
behavior are likely to have negative ramiﬁcations as the public turns cold towards
machine research, development and production, and these negative ramiﬁcationsImplementation of Moral Uncertainty in Intelligent Machines
123
should be entered into the judgements of moral theories from the outset rather than
being avoided by introducing bias at the metanormative level.
8.3 Against Maximin
The desire to protect minority rights and interests may lead one to propose that the
framework of MEC be replaced with a maximin criterion or some weaker loss-averse function. Simple versions of consequentialism, which maximize expectedvalue on the normative rather than the metanormative level, are often criticized forimplying that it may be right to sacriﬁce the lives and interests of individuals for thegreater good. In a similar vein, an autonomous system which maximized expected
choiceworthiness could be expected to take actions which are very wrong according
to one set of values but very good according to other values, and some may ﬁnd thisproblematic. A maximin or loss-averse function would respectively prevent or resistsuch scenarios.
However, this intuition is mistaken in the metanormative context because
counterintuitive cases of minority interests being violated are wrong on the ethicalgrounds of many different moral theories. For instance, killing an innocent patient inorder to reuse their organs is not only considered wrong by deontological ethics, but
would likely be considered wrong by rule consequentialism, virtue ethics,
intuitionist ethics and other common theories as well, and always to a high degree.So in scenarios involving possible patient outcomes and organ harvesting, MECwould demand that the more common interest of refraining from harm be adheredto, while a maximin or loss-averse function would perversely give extra weight tothe sidelined act consequentialist calculations over and above the degree to whichact consequentialism would be concerned about the lost organs. More generally,MEC would strongly represent common-sense views on contentious ethical issues,
because common-sense views on ethical issues are more common than other kinds
of views. That being said, a risk-averse function might be desirable as a backuputility function to be activated in certain circumstances (Oesterheld 2016 ).
9 Conclusion
Disagreement over ethics is unlikely to be resolved in the near future. If we wish to
create machines with the capacity to make decisions on matters of ethical
importance then a fair and robust system for translating the patchwork of human
values into moral guidance must be developed. The intractability of our moraldisputes and the magnitude of our moral errors can both be minimized bycompromising with a decision framework which emphasizes the respectivepriorities of different moral systems in accordance with their plausibility. WilliamMacAskill’s framework of maximizing expected choiceworthiness satisﬁes thisdescription.K. Bogosian
123
References
Allen, C., Smit, I., & Wallach, W. (2005). Artiﬁcial morality: Top-down, bottom-up, and hybrid
approaches. Ethics and Information Technology . doi: 10.1007/s10676-006-0004-4 .
Archard, D. (2011). Why moral philosophers are not and should not be moral experts. Bioethics, 25 (3),
119–127.
Arkoudas, K., Bringsjord, S., & Bello, P. (2005). Toward ethical robots via mechanized deontic logic. In
Machine ethics: Papers from the 2005 AAAI fall symposium (pp. 17–23). Retrieved from http://
scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Toward +ethical+robots+via+
mechanized +deontic +logic#0 .
Bello, P., & Bringsjord, S. (2012). On how to build a moral machine. Topoi, 32 (2), 251–266. doi: 10.1007/
s11245-012-9129-8 .
Bostrom, N. (2009). Moral uncertainty —towards a solution? http://www.overcomingbias.com/2009/01/
moral-uncertainty-towards-a-solution.html .
Bostrom, N. (2012). The superintelligent will: Motivation and instrumental rationality in advanced
artiﬁcial agents. Minds and Machines, 22 (2), 71–85. doi: 10.1007/s11023-012-9281-3 .
Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies . Oxford: Oxford University Press.
Brundage, M. (2014). Limitations and risks of machine ethics. Journal of Experimental & Theoretical
Artiﬁcial Intelligence, 26 (3), 355–372. doi: 10.1080/0952813X.2014.895108 .
Cotton-Barratt, O. (2013). Geometric reasons for normalising variance to aggregate preferences .http://
users.ox.ac.uk/ball1714/Variance%20normalisation.pdf .
Cross, B. (2016). Moral philosophy, moral expertise, and the argument from disagreement. Bioethics, 30
(3), 188–194.
Driver, J. (2014). Moral expertise: Judgement, practice, and analysis. Social Philosophy and Policy, 30 (1–
2), 280–296.
Gloor, L. (2016). Suffering-focused AI safety: Why “fail-safe” measures might be our top intervention. In
Foundational Research Institute, Report FRI -16-1.https://foundational-research.org/ﬁles/suffering-
focused-ai-safety.pdf .
Greene, J. D., Sommerville, R. B., Nystrom, L., Darley, J., & Cohen, J. (2001). An fMRI investigation of
emotional engagement in moral judgment. Science, 293 (5537), 2105–2108. doi: 10.1126/science.
1062872 .
Jones, K., & Schroeter, F. (2012). Moral expertise. Analyse & Kritik, 34 (2), 217–230.
Lockhart, T. (2000). Moral uncertainty and its consequences . Oxford: Oxford University Press. ISBN
978-0195126105.
MacAskill, W. (2013). The infectiousness of nihilism. Ethics, 123 (3), 508–520.
MacAskill, W. (2014). Normative uncertainty .http://commonsenseatheism.com/wp-content/uploads/
2014/03/MacAskill-Normative-Uncertainty.pdf .
MacAskill, W. (2016). Normative uncertainty as a voting problem. Mind, 125 (500), 967–1004. doi: 10.
1093/mind/fzv169 .
Nissan-Rozen, I. (2015). Against moral hedging. Economics and Philosophy, 3, 1–21. doi: 10.1017/
S0266267115000206 .
Oesterheld, C. (2015). Formalizing preference utilitarianism in physical world models. Synthese . doi: 10.
1007/s11229-015-0883-1 .
Oesterheld, C. (2016). Backup utility functions as a fail-safe AI technique. In Foundational Research
Institute, Report FRI -16-2.https://foundational-research.org/ﬁles/backup-utility-functions.pdf .
PhilPapers Foundation (2009). Preliminary survey results .http://philpapers.org/surveys/results.pl .
Schultz, E., Cokely, E. T., & Feltz, A. (2011). Persistent bias in expert judgments about free will and
moral responsibility: A test of the expertise defense. Consciousness and Cognition, 20 (4), 1722–
1731.
Schwitzgebel, E., & Cushman, F. (2012). Expertise in moral reasoning? Order effects on moral judgment
in professional philosophers and non-philosophers. Mind and Language, 27 (2), 135–153.
Shulman, C., Tarleton, N., & Jonsson, H. (2009). Which consequentialism? Machine ethics and moral
divergence. In AP-CAP 2009: The 5th Asia -Paciﬁc computing and philosophy conference .https://
intelligence.org/ﬁles/WhichConsequentialism.pdf .
Williams, E. (2015). The possibility of an ongoing moral catastrophe. Ethical Theory and Moral Practice,
18(5), 971–982. doi: 10.1007/s10677-015-9567-7 .Implementation of Moral Uncertainty in Intelligent Machines
123
Wiltshire, T. J. (2015). A prospective framework for the design of ideal artiﬁcial moral agents: Insights
from the science of heroism in humans. Minds and Machines . doi: 10.1007/s11023-015-9361-2 .
Z˙uradzki, T. (2016). Meta-reasoning in making moral decisions under normative uncertainty. In
Argumentation and reasoned action: Proceedings of the 1st European conference on argumentation,Lisbon, 2015 (Vol. 2, pp. 1093–1104).K. Bogosian
123


Synthese
DOI 10.1007/s11229-015-0883-1
Formalizing preference utilitarianism in physical world
models
Caspar Oesterheld1
Received: 23 April 2015 / Accepted: 27 August 2015
© The Author(s) 2015. This article is published with open access at Springerlink.com
Abstract Most ethical work is done at a low level of formality. This makes practical
moral questions inaccessible to formal and natural sciences and can lead to misunder-
standings in ethical discussion. In this paper, we use Bayesian inference to introduce a
formalization of preference utilitarianism in physical world models, speciﬁcally cel-
lular automata. Even though our formalization is not immediately applicable, it is a
ﬁrst step in providing ethics and ultimately the question of how to “make the world
better” with a formal basis.
Keywords Preference utilitarianism ·Formalization ·Artiﬁcial life ·
(Machine) ethics
1 Introduction
Usually, ethical imperatives are not formulated with sufﬁcient precision to study them
and their realization mathematically. ( McLaren 2011 , p. 297; Gips 2011 , p. 251) In
particular, it is impossible to implement them on an intelligent machine to make it
behave benevolently in our universe, which is the subject of a ﬁeld known as Friendly
AI(e.g. see Y udkowsky 2001 ,p .2 )o r machine ethics (e.g. see Anderson and Anderson
2011 , p. 1). Whereas existing formalizations of utilitarian ethics have been successfully
applied to economics, they are incomplete due to the nature of their dualistic world
model in which agents are assumed to be ontologically fundamental.
B Caspar Oesterheld
caspar.oesterheld@uni-bremen.de
1University of Bremen, Bremen, Germany
123
Synthese
In this paper however, we take the following steps towards a workable and simple
formalization of preference utilitarianism1in physical world models:
– We describe the problem of informality in ethics and the shortcomings of previous
dualist approaches to formalizing utilitarian ethics (Sect. 2).
– We justify cellular automata as a world model, use Bayes’ theorem to extract utility
functions from a given space-time embedded agent and introduce a formalization
of preference utilitarianism (Sect. 3).
– We compare our approach with existing work in ethics, game theory and artiﬁcial
intelligence (Sect. 4). Our formalization is novel but nevertheless relates to a
growing movement to treat agents as embedded into the environment.
2 The problem of formalizing ethics in physical systems
Discussion on informally speciﬁed moral imperatives can be difﬁcult due to differentinterpretations of the texts describing the imperative. Thus, formalizing moral imper-
atives could augment informal ethical discussion. ( Gips 2011 , p. 251; Anderson 2011 ;
Dennett 2006 ;Moor 2011 , p. 19)
Furthermore, science and engineering answer formally described questions and
solve well-speciﬁed tasks, but are not immediately applicable to the informal questionof how to make the world “better”.
This problem has been identiﬁed in economics and game theory, which has led to
some very useful formalizations of utilitarianism (e.g. Harsanyi 1982 ).
However, their formalization relies on consciousness-matter dualism: The agents
are not part of the physical world or embedded into it, so that their thoughts or com-
putations can not be inﬂuenced by physical laws. Also, agents’ utility functions areassumed to not depend on the agents (or their physical conﬁgurations) themselves.
These are typical assumptions in game theory. After all, game theory is about games,
in which players are not actually inside the game, nor can they decide themselves whatgoals to pursue. This classic (multi-)agent-environment model is depicted in Fig. 1.
Our world, however, is (usually presumed to be) a purely physical system: ethically
relevant entities (animals etc.) are embedded in the environment. For example, our
brains behave according to the same laws of physics as the rest of the world. Also,
happiness and preferences are not given by predetermined utility functions or rewardsfrom the environment, but are the result of physical processes in our bodies. Therefore,
dualist descriptions and formalizations leave questions unanswered: (compare Orseau
and Ring 2012 )
– What objects are ethically relevant? (What are the agents of our non-dualist world?)
– What is a space-time embedded agent’s or, more generally, an object’s utility
function?
Thus, even though classic formalizations of (preferentist) utilitarianism in the agent-
environment-model can formalize the vague notions of goals and preferences with
1For introductions to and ethical discussions of the underlying notion of preference utilitarianism see
Tomasik (2015a ,b).
123
Synthese
Fig. 1 The classic agent-environment-model
utility functions, these formalizations are incomplete, at least in our physical, non-
dualist world.
3 A Bayesian approach to formalizing preference utilitarianism in
physical systems
3.1 Cellular automata as non-dualist world models
To overcome the described problems of dualist approaches to utilitarianism, we ﬁrst
have to choose a new, physical setting for our ethical imperative. Instead of employing
string theory and other contemporary theoretical frameworks, we choose a model thatis much more simple to handle formally: cellular automata. These have sometimes
even been pointed out to be candidates for modeling our own universe, ( Wolfram
2002 , ch. 9; Schmidhuber 1999 ;Zuse 1967 ,1969 ) but even if physics will prove
cellular automata to be a wrong model, they may still be of instrumental value for
the purpose of this paper. (compare Downey 2012 , pp. 70f., 77–79; Hawking and
Mlodinow 2010 , ch. 8)
For detailed introductions to classic cellular automata with neighbor-based rules,
see Wolfram (2002 ) or Shiffman ( 2012 , ch. 7) for a purely informal and Wolfram
(1983 ) for a slightly more technical treatment that focuses on one-dimensional cel-
lular automata. In Sect. 3.1.1 , we will consider a generalized and relatively simple
formalism, which is not limited to rules that only depend on neighbors of a cell.
In CA, it is immediately clear that for a (preference) utilitarian morality we have
to answer the questions that are avoided by assuming a set of agents and their utility
functions to be known from the beginning. It also frees us from many ethical intuitionsthat we build up speciﬁcally for our own living situations and reduces moral intuition
to its very fundamentals.
Figure 2shows a state of a cellular automaton illustrating the problem of deﬁning
utilitarianism or any other ethical imperative in physical models. Clearly, many intu-
itions are very difﬁcult (if not impossible) to formulate universally and precisely in CA.
Thereby, the required formality helps in choosing and deﬁning an ethical imperative.
123
Synthese
Fig. 2 A state of a two-dimensional cellular automaton. It is very unclear, what agents are and which prefer-
ences they have. Adapted from http://en.wikipedia.org/wiki/Conway%27s\_Game\_of\_Life#mediaviewer/
File:Conways\_game\_of\_life\_breeder
3.1.1 A formal introduction to cellular systems
We now introduce some very basic notation and terminology of cellular systems,
a generalization of classic cellular automata, thus setting the scene for our ethical
imperative.
For given sets Aand B,l e t ABdenote the set of functions from BtoA.Acellular
system is a triple (C,S,d)of a countable set of cells C, a ﬁnite set of cell states Sand
a function d:SC→SCthat maps a world state s :C→Sonto its successor. So
basically a world consists of a set of cells that can have different values and a function
that models deterministic state-transitions.2
Cells of cellular systems do not necessarily have to be on a regular grid and com-
puting new states does not have to be done via neighbor-based lookup tables. This
makes formalization much easier.
But before anything else, we have to deﬁne structures which represent objects in
our cellular systems. A space Spc ⊆Cin a cellular system (C,S,d)is a ﬁnite subset
of the set of cells C.Astructure str on a space Spc is a function str:Spc→Sthat
maps the cells of the space onto cell values.
2The choice of deterministic systems was made primarily to simplify the formalization. It appears to
be unproblematic to transfer formal preference utilitarianism to non-deterministic systems, but deﬁning
non-deterministic cellular automata themselves is a little more difﬁcult.
123
Synthese
Ahistory is a function h:N→SCthat maps natural numbers as time steps onto
states of the system. For example, the history hsof an initial state scan then be deﬁned
recursively by hs(n)=d(hs(n−1))forn≥1 with the base case hs(0)=s.
3.2 Posterior probabilities and the priority of a (given) goal to a given agent
Before extracting preferences from a given structure, we have to decide on a model
of preferences. Preferences themselves are mere orderings of alternative outcomes orlotteries over these outcomes with the outcomes being entire histories h∈(S
C)Nin
our case. The problem is that this makes it difﬁcult to compare two outcomes when the
preferences of multiple individuals are involved. To be able to make such comparisons,we move from orderings to utility functions u:(S
C)N→ Rthat map histories of
the world onto their (cardinal) utilities.3This will make it possible to just add up the
utilities of different individuals and then compare the sum among outcomes. This byno means “solves” the problem of interpersonal comparison. Rather, it makes it more
explicit. For example, a given set of preferences is represented equally well by uand
2·u,b u t ceteris paribus 2·uwill make the preferences more signiﬁcant in summation.
Different approaches to the problem have been proposed. ( Hammond 1989 )I nt h i s
paper we will ignore the problem (or hope that the fair treatment in determining all
individuals’ utility functions induces moral permissibility). Now we ask the question:
Does a particular structure strwant to maximize some utility function u?
It is fruitful to think about how one would approach such questions in our world,
when encountering some very odd organism. At least one possible approach would be
to put it into different situations or environments and see what it does to them. If the
structure increases some potential utility function in different environments, it seemsas if this utility function represents an aspect of the structure’s preferences.
4
However, for some utility functions it is not very special that their values are
increased and then it might just be coincidence that the structure in question alsodoes so. For example, it is usually not considered a structure’s preference to increase
entropy even if entropy increases in environments including this structure, because an
increase in entropy is extremely common with or without the structure.
Also, we feel that some utility functions are less likely than others by themselves,
e.g. because they are very complex or speciﬁc.
But how can we formally capture these notions?Since uncertainty is involved, we interpret the degree to which a utility function uis
important to some structure strthat exists at time step ias the posterior probability of
that utility function given the structure, a probability we denote by P(u|str@i), where
str@idenotes the event that str exists in time step i.
5Here, the utility function is
3Other codomains of utility functions seem possible as long as they are subsets of a totally ordered vector
space over R. Intervals like [0,1]seem speciﬁcally suitable, because they avoid problems of inﬁnite utility
and allow for normalization. ( Isbell 1959 )
4Alternatively, one can try to avoid this hypothetical experiment by predicting the organism’s behavior. For
example, one could try to ask the organism what it would do or infer its typical behavior from its internals.
5Including iinto the data is important, because otherwise identical structures at different points in time
would have identical utility functions. This is a problem, when the utility function uis applied to the whole
123
Synthese
interpreted as a hypothesis about the structure’s “true intentions”6. In a purely physical,
non-dualist world there is nothing but the structure itself, of course. Therefore, the “true
intentions” do not really exist, which makes it still hard to know what P(u|str@i)
is supposed to mean. To ﬁnally overcome this problem, we will equate intention and
purpose, i.e. we equate the following interpretations of uas a hypothesis explaining
the data str@i: (compare Dennett 1989 , pp. 289ff., 299f., 318, 320f.)
– The utility function uis the goal of structure str.
– Maximizing uwas the goal of an entity that chose str.
The second interpretation is more useful, because it describes a data-generating process
and thus comes closer to typical statistical models.
Thus, we have to ﬁnd the posterior probability of some model (a utility function)
given some data (a structure). For this problem Bayes’ theorem suggests itself, becauseit provides an equation for posterior probabilities. In our case, Bayes’ theorem can be
used to infer the likelihood that some utility function was a goal when a structure was
chosen from some priors and the likelihood of choosing the structure given that thegoal is to maximize the utility function. Speciﬁcally, Bayes’ theorem gives us
P(u|str@i)=P(str@i|u)·P(u)
P(str@i), (1)
where P(u)and P(str@i)are prior probability distributions of utility functions and
structures, respectively, and P(str@i|u)is the probability of (some hypothetical entity
choosing) str at time step iwhen uis to be maximized. Whereas P(u|str@i)is
very hard to grasp intuitively, it is more clear what the probability distributions on
the right hand side of the equation mean. Nevertheless, they do not correspond tomeasurable probability distributions like the results from rolling a dice. Indeed, P(u)
and P(str@i)are ultimately subjective (e.g. see Olshausen 2004 , pp. 1f.; Robert
1994 , p. 9) and P(str@i|u)depends on what exactly the hypothesis uis supposed to
express, thus leaving our ethical imperative parametrized by these distributions.
Footnote 5 continued
history, because then structures cannot have preferences about themselves (“personal happiness”) without
also having preferences about all other identical structures (at the same place). An alternative would be to
apply utility functions only to the part of the history from the point of the existence of the structure onwards,
so that identical structures at different points in time have equal utility functions that are applied differently.
However, it seems like this neglects that the past can depend on the action of an agent in the present, asillustrated in Newcomb’s paradox by Nozick (1969 ).
6Intuitively, some structures can have more than one utility function, while others have no utility function
at all. One way to model this would be to understand different utility functions as events in separate samplespaces. So, the sum/summationtext
uP(u|str )could vary among different structures str. A similar scenario is the
inference of multiple diseases from a set of symptoms. ( Charniak 1983 ) While some individuals may have
no diseases or preferences at all, others may be thought of as having more than one disease or utility
function. In more technical terms, for each utility function there would be a sample space of having that
utility function and not having that utility function.In this paper however, we will assume mutual exclusivity and collective exhaustiveness of utility functions.
All utility functions live in the same sample space and thus/summationtext
uP(u|str )=1 for all structures str.T h i s
does not mean that all structures have equal moral standing: The idea is that “meaningless” structures str
have high P(u|str )only for constant utility functions u, i.e. for “don’t care”-utility functions, which are
irrelevant for decision making.
123
Synthese
Nonetheless, there seem to be canonical approaches. P(str@i|u)should be under-
stood as the probability that stris chosen at time step iby an approximately rational
agent that wants to maximize u. So, structures that are better at maximizing or more
suitable for ushould receive higher P(str@i|u)values. This corresponds to the
assumption of (approximate) rationality in Dennett’s intentional stance. ( Dennett 1989 ,
pp. 21, 49f.) Unfortunately, the debate about causal and evidential decision theory (e.g.Peterson 2009 , ch. 9) shows that formalizing the notion of rational choice is difﬁcult.
The prior of utility functions P(u)on the other hand should denote the “intrinsic
plausibility” of a goal u. That does not have to mean deﬁning and excluding “evil”
or “banal” utility functions. In the preference extraction context, utility functions are
models or hypotheses that explain the behavior of a structure. And Solomonoff’s
formalization of Occam’s razor is often cited as a universal prior distribution of
hypotheses. ( Legg 1997 ) It assumes complicated hypotheses (utility functions), i.e.
ones that require more symbols to be described in some programming language, to beless likely than simpler ones.
If utility functions are conceived of as competing hypotheses (see footnote 6), then
P(str@i)=/summationdisplay
uP(str@i|u)·P(u).
Otherwise, P(str@i)could potentially be chosen more freely.
Finally, note how Bayes’ theorem catches our intuitions from above, especially
when assuming probability distributions similar to the suggested ones: When some
structure str maximizes some utility function uvery well, then P(str@i|u)and
thereby the relevance of the utility function to the object would increase. On theother hand, if many other structures are comparably good, then the probability for
each one to be chosen when given the utility function is smaller (due to the sum of
the probabilities of all possible structures on a given space and time step being 1) andthe probability of the utility function being a real preference would decrease with it.
Finally, multiplying by P(u)catches abstruse utility functions, e.g. utility functions
that are speciﬁcally suited to be fulﬁlled by the structure in question.
3.3 An individual structure’s welfare function
Having introduced a way of determining how likely it is that some utility function is
the utility function of some object, we deﬁne the welfare U
str@iof a structure strthat
exists at some step iof a history h, as the weighted sum over all utility functions
Ustr@i=/summationdisplay
uP(u|str@i)u(h), (2)
where his the history and the sum is over all theoretically possible utility functions
u:(SC)N→ R.
We call this term expected utility , because this expression is generally used for
adding utilities based on their likelihood, which is a common concept. However, the
term usually suggests that there is also an actual utility . In our case of ascribing
123
Synthese
preferences to physical objects however, no such thing exists. We only imagine there
to be some real utility or welfare functions and that we use Bayesian inference to ﬁnd
them. But in fact, the structure itself is all there exists and thus the expected utility isas actual as possible.
The sum in the term for expected utility is over an uncountably inﬁnite set, which
can only converge when only countably many summands are non-zero.
7Some other
concerns are described in footnote 9and addressed in footnote 10.
3.4 Summing over all agents
The utilitarian imperative is to maximize a global welfare function that is the sum of
all individuals’ welfare functions. We already deﬁned the welfare function of singlestructures. So next we have to deﬁne what the set of all agents is and how to sum over it.
As foreshadowed before, we will consider all possible structures of a cellular automa-
ton using Eq. 2and rely on (intuitively) irrelevant ones to receive high P(u|str@i)
values only for constant and therefore irrelevant utility functions u(see footnote 6).
To sum the utility over all agents, we not only have to sum over all structures in a
particular state, but ﬁrst over all (discrete) time steps of the history of the cellularautomaton world and only then over all structures in every state. This way, we sum
the welfare of all agents ever coming into existence. For the summands, we can insert
the term obtained in Eq. 2
/summationdisplay
i/summationdisplay
str@iUstr@i=/summationdisplay
i/summationdisplay
str@i/summationdisplay
uP(u|str@i)u(h), (3)
where Ustr@idenotes the welfare or utility of the structure str that exists at time
step i, the ﬁrst sum is over all integers functioning as time steps, the second is over all
structures in h(i)and the third over all possible utility functions.8So, our formalization
of the main imperative of preference utilitarianism turns out to be nothing more than
maximizing global, all-time expected utility (of every space-time embedded agent that
ever comes into existence). In general, the value of the series depends on the order of
7If Solomonoff’s prior is chosen for P(u), all incomputable utility functions have zero probability. Since
the set of computable functions is countable, only countably many summands could possibly be non-zero.
8More precisely, but less elegantly, one could write
∞/summationdisplay
i=0/summationdisplay
Spc∈Fin (C)/summationdisplay
u:(SC)N→Ru(h)P(u|(h(i)|Spc )@i),
where Fin (C):= { A⊆C||A|∈ N}is the set of ﬁnite subsets of Candh(i)|Spc:Spc→S:c/mapsto→h(i)(c)
is the restriction of the state h(i)to the space Spc and therefore the structure on that space.
123
Synthese
these inﬁnite sums.9Also, the series can diverge. Nonetheless it may still be usable
for comparing histories in many cases.10
4 Related work
Preferentist utilitarianism has become a common form of utilitarianism in the sec-ond half of the 20th century, with the best known proponents being Hare and Singer.
However, the intuitions underlying the presented formalization are different from themost common ethical intuitions in preference utilitarianism. Since our formal prefer-
ence utilitarianism is not meant to describe a decision procedure for humans (or, more
generally and in Hare’s ( 1981 , pp. 44f.) terminology, non-“archangels”), we do not
consider an application-oriented utilitarianism like Hare’s two-level consequential-
ism. ( Hare 1981 , p. 25ff.) Also, most preference utilitarians ascribe preferences only
to humans (or abstract agents) and do not contain prioritization among individuals,(Harsanyi 1982 , p. 46) or they use a low number of classes of moral standing. ( Singer
1993 , pp. 101ff., 283f.) Whereas some have pointed out that a variety of behavior and
even trivial systems can be viewed from an “intentional stance”, ( Dennett 1971 ,1989 ,
especially pp. 29f.; compare Hofstadter 2007 , pp. 52ff.) only relatively recent arti-
cles in preference utilitarianism have discussed the connection between goal-directedbehavior and ethically relevant preferences and with the universality of the former
pointed out the potential universality of the latter. ( Tomasik 2015b , ch. 7; Tomasik
2015a , ch. 4, 6; Tomasik 2015c ) This idea is an important step when formalizing
preference utilitarianism because otherwise one would have to deﬁne moral standing
depending on other, usually binary, notions: being alive, the ability to suffer ( Ben-
tham 1823 , ch. 17 note 122) personhood ( Gruen 2014 , ch. 1), free will, sentience and
(self-)consciousness ( Singer 1993 , pp. 101ff.) or the ability of moral judgment. How-
ever, all of them seem to be very difﬁcult to deﬁne (universally) in physical systems
in the intended binary sense.
11Also, continuous deﬁnitions of these terms are often
connected with goal-directed behavior. ( Tomasik 2015a , ch. 4; Wolfram 2002 , p. 1136)
9Speciﬁcally, the Riemann series theorem states that any conditionally convergent series can be reordered
to have arbitrary values.
10It is very important to differentiate the series from its value. Otherwise, one may identify the series
with positive or negative inﬁnity or as being undeﬁned. Two inﬁnite values of the series would then not be
comparable anymore, which Bostrom (2011 ) identiﬁed as a problem for (consequentialist) ethics. But this
problem can sometimes be eliminated by comparing the series itself to another. In this particular case, ahistory his better than another history h/prime,i f
/summationdisplay
i/summationdisplay
Spc/summationdisplay
uu(h)P(u|(h(i)|Spc )@i)−u(h/prime)P(u|(h/prime(i)|Spc )@i)> 0,
where h(i)|Spc:Spc→S:c/mapsto→h(i)(c)denotes the restriction of h(i)toSpc,i . e .t h es t r u c t u r eo n Spc
in the state h(i). If no such relation can be established then the two histories are arguably incomparable
or may be called approximately equally good. Again, the ordering could be important in some cases, see
footnote 9.
11For example, Wolfram ( 2002 , pp. 823–825, 1178–1180) and Emmeche (1997 ) discuss the property
of life, Hofstadter ( 2007 , pp. 9–24, 51–54) discusses consciousness and Arneson (1998 , p. 5) discusses
personhood.
123
Synthese
Fig. 3 Comparison between
formalizations of utilitarianism.The ﬁrst row shows the
formalization of this paper, the
second row is adapted from
Harsanyi ( 1982 ,p .4 6 ) ,a n dt h e
third row from Gips ( 2011 ,
p. 245). The utility of an agent n
is denoted by U
nand its weight
bywnsum over all agents utility of an agent
istr@i uP(u|str@i)u(h)
agent n∈NUn
agent n∈Nwn·Un
Whereas most ethical work is conducted informally, ( McLaren 2011 , p. 297; Gips
2011 , p. 251) there has been some formal work at the intersection of (utilitarian) ethics,
game theory and economics, most notably by Harsanyi (1982 ). Some formalization has
also been conducted in the realm of machine ethics. ( Anderson et al. 2004 ;Gips 2011 ,
pp. 245ff.) However, inﬂuenced by game theory and dualist traditions in philosophy,
they are based on the classic agent-environment-model as displayed in Fig. 1and
assume utility functions (or even the utilities in different trajectories themselves) asgiven by the world model. Nonetheless, there is at least one parallel: all models of
utilitarianism contain the notion of summing the utility over all agents. As shown in
Fig. 3, both the deﬁnition of all agents and how to obtain the utility or welfare of an
agent differ among formalizations.
In Artiﬁcial Intelligence, the idea of learning preferences has become more popular,
e.g. see Fürnkranz and Hüllermeier (2010 ) and Nielsen and Jensen (2004 ) for technical
treatments or Bostrom ( 2014 , pp. 192ff.) for an introduction in the context of making an
AI do what the engineers value. However, most of the time, the agent is still presumed
to be separated from the environment.
Nonetheless, the idea of evaluating space-time-embedded intelligence is beginning
to be established in artiﬁcial (general) intelligence, ( Orseau and Ring 2012 ) which is
closely related to the probability distribution P(str@i|u).
5 Conclusion
By reversing Dennett’s intentional stance with Bayes’ theorem, we were able to ascribe
preferences to physical objects and thus formalize preference utilitarianism in cellu-
lar automata. Theoretically, such formalizations can function as a speciﬁcation foran artiﬁcial intelligence or more generally as a basis for “paradise engineering” (e.g.
seeEttinger 2009 , p. 124). However, there are several potential problems that require
further work before such practical applications of our formalization or improved vari-ations of it can be approached:
– Through sums over all structures, possible utility functions and states and the
application of incomputable concepts like Solomonoff’s prior in P(u), our formal-
ization is incomputable in theory and practice. So even in simulations of cellular
automata our formalization is not immediately applicable.
123
Synthese
– Computing our global welfare function in the real world is even more difﬁcult,
because it requires full information about the world on particle level. Also, the
formalization must ﬁrst be translated into the physical laws of our universe.
– The difﬁculty to apply our formalization is by no means only relevant to actu-
ally using it as a moral imperative. Instead, it is also relevant to discussing our
formalization from a normative standpoint: Even though the derivation of our for-malization is plausible, it may still differ signiﬁcantly from intuition. There could
be some kind of trivial agents with trivial preferences that dominate comparison of
different histories. Because the formalization’s incomputability makes it difﬁcultto assess whether such problems are present, further work on its potential ﬂaws
is necessary. Based on such discussion, our formalization may be revised or even
discarded. In any case, we could learn a lot from its shortcomings especially due
to the formalization’s simplicity and plausible derivation.
– We outlined how P(str@i|u)andP(u)could be determined in principle. However,
they need to be speciﬁed more formally, which in the case of P(str@i|u)seems
to require a solution to the problem of normative decision theory. Some problems
of our formalization could inspire additional reﬁnements of these distributions.
Acknowledgments I am grateful to Brian Tomasik for giving me important comments that led me to
systematize my formalization. I also thank Adrian Hutter for an interesting discussion on the formalization,
as well as Alina Mendt, Duncan Murray, Henry Heinemann, Juliane Kraft and Nils Weller for reading and
commenting on earlier versions of the paper. I owe thanks to the two anonymous reviewers whose commentsand suggestions helped improve and clarify this manuscript.
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 Interna-
tional License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution,
and reproduction in any medium, provided you give appropriate credit to the original author(s) and the
source, provide a link to the Creative Commons license, and indicate if changes were made.
References
Anderson, S. L. (2011). How machines might help us achieve breakthroughs in ethical theory and inspire us
to behave better. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 524–530). Cambridge:
Cambridge University Press.
Anderson, M., & Anderson, S. L. (Eds.). (2011). Machine ethics . Cambridge: Cam-
bridge University Press. http://www.cambridge.org/cr/academic/subjects/computer-science/
artiﬁcial-intelligence-and-natural-language-processing/machine-ethics .
Anderson, M., Anderson, S. L. & Armen, C. (2004). Towards machine ethics. In: AAAI-04 Workshop on
agent organizations: Theory and practice . American Association for Artiﬁcial Intelligence, Menlo
Park. http://www.researchgate.net/publication/259656154\_Towards\_Machine\_Ethics .
Arneson, R. J. (1998). What, if anything, renders all humans morally equal? http://philosophyfaculty.ucsd.
edu/faculty/rarneson/singer.pdf .
Bentham, J. (1823). An introduction to the principles of morals and legislation . Oxford: Clarendon Press.
http://www.econlib.org/library/Bentham/bnthPMLCover.html .
Bostrom, N. (2011). Inﬁnite ethics. Analysis and Metaphysics ,10, 9–59.
Bostrom, N. (2014). Superintelligence. Paths, dangers, strategies (1st ed.). Oxford: Oxford University Press.
Charniak, E. (1983). The Bayesian basis of common sense medical diagnosis. In: AAAI-83 Proceedings ,
Washington, DC.
Dennett, D. (1971). Intentional systems. The Journal of Philosophy ,68(4), 87–106. http://www.jstor.org/
stable/2025382 .
Dennett, D. (1989). The intentional stance . Cambridge: mit press.
123
Synthese
Dennett, D. (2006). Computers as prostheses for the imagination .I n : Talk at The International Computers
and Philosophy Conference. Laval, France.
Downey, A. B. (2012). Think complexity . Sebastopol: O’Reilly.
Emmeche, C. (1997). Center for the Philosophy of Nature and Science Studies Niels Bohr Institute. http://
www.nbi.dk/~emmeche/cePubl/97e.defLife.v3f.html .
Ettinger, R. C. W. (2009). Youniverse: Toward a self-centered philosophy of immortalism and cryonics .
Boca Raton: Universal-Publishers
Fürnkranz, J., & Hüllermeier, E. (Eds.). (2010). Preference learning . Berlin: Springer.
Gips, J. (2011). Towards the ethical robot. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (1st
ed., pp. 244–253). Cambridge: Cambridge University Press.
Gruen, L. (2014). In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy .http://plato.stanford.edu/
archives/fall2014/entries/moral-animal .
Hammond, P . J. (1989). Interpersonal comparison of utility: Why and how they are and should be made .
http://homepages.warwick.ac.uk/~ecsgaj/icuSurvey.pdf .
Hare, R. M. (1981). Moral thinking. Its levels method, and point. Two-level consequentialism . Oxford:
Clarendon Press.
Harsanyi, J. C. (1982). Morality and the theory of rational behaviour. In A. Sen & B. Williams (Eds.),
Utilitarianism and beyond, Chap. 2 (pp. 39–62). Cambridge: Cambridge University Press. http://
ebooks.cambridge.org/chapter.jsf?bid=CBO9780511611964&cid=CBO9780511611964A009&
tabName=Chapter .
Hawking, S. & Mlodinow, L. (2010). The grand design . New Y ork: Bantam B.
Hofstadter, D. (2007). I am a strange loop . New Y ork: Basic Books.
Isbell, J. R. (1959). Absolute games. In A. W. Tucker & R. D. Luce (Eds.), Contributions to the theory of
games (V ol. 4, pp. 357–396). Princeton: Princeton University Press.
Legg, S. (1997). Solomonoff induction. MA Thesis. University of Auckland. http://www.vetta.org/
documents/legg-1996-solomonoff-induction.pdf .
McLaren, B. N. (2011). Computation models of ethical reasoning. Challenges, initial steps, and future
directions. In M. Anderson & S. L. Anderson (Eds.), Machine Ethics (1st ed., pp. 297–315). Cambridge:
cambridge University Press.
Moor, J. H. (2011). The nature, importance, and difﬁculty of machine ethics. In M. Anderson & S. L.
Anderson (Eds.), Machine ethics, Chap. 1 (pp. 13–20). Cambridge: Cambridge University Press.
Nielsen, T. D. & Jensen, F. V . (2004). Learning a decision maker’s utility function from (possibly) incon-
sistent behavior. In: Artiﬁcial Intelligence ,160(1–2), 53–78. http://www.sciencedirect.com/science/
article/pii/S0004370204001328 .
Nozick, R. (1969). Newcomb’s problem and two principles of choice. In: N. Rescher et al. (Ed.), Essays
in honor of Carl G. Hempel (pp. 114–146). Berlin: Springer. http://faculty.arts.ubc.ca/rjohns/nozick\_
newcomb.pdf .
Olshausen, B. A. (2004). Bayesian probability theory . Retrieved from http://redwood.berkeley.edu/bruno/
npb163/bayes.pdf
Orseau, L. & Ring, M. (2012). Space-time embedded intelligence. In: J. Bach, B. Goertzel & M. Iklé (Eds.),
Artiﬁcial general intelligence, Vol. 5 (p. 391). http://agi-conference.org/2012/wp-content/uploads/
2012/12/paper\_76.pdf .
Peterson, M. (2009). An introduction to decision theory . Cambridge: Cambridge University Press.
Robert, C. P . (1994). The Bayesian choice. A decision-theoretic motivation (1st ed.). Berlin: Springer.
Schmidhuber, J. (1999). A computer scientist’s view of life, the universe, and everything .http://arxiv.org/
pdf/quant-ph/9904050v1.pdf .
Shiffman, D. (2012). In S. Fry (Ed.), The nature of code .http://natureofcode.com/book/ .
Singer, P . (1993). Practical ethics (2nd ed.). Cambridge: Cambridge University Press.
Tomasik, B. (2015a). Do video-game characters matter morally? http://www.webcitation.org/6X7w4AJnb .
Tomasik, B. (2015b). Hedonistic vs. preference utilitarianism .http://www.webcitation.org/6X7vepyJP .
Tomasik, B. (2015c). Is there suffering in fundamental physics? http://www.webcitation.org/6X7vte3XP .
Wolfram, S. (1983). Cellular automata. In: Los Alamos Science ,9, 2–21. http://www.stephenwolfram.com/
publications/academic/cellular-automata.pdf .
Wolfram, S. (2002). A new kind of science . Champaign: Wolfram Media. http://www.wolframscience.com/
nksonline/toc.html .
123
Synthese
Y udkowsky, E. (2001). Creating friendly AI 1.0: The analysis and design of benevolent goal architectures .
Machine Intelligence Research Institute. http://intelligence.org/ﬁles/CFAI.pdf .
Zuse, K. (1967). Rechnender Raum. In: Elektronische Datenverarbeitung ,8, 336–344. ftp://ftp.idsia.ch/
pub/juergen/zuse67scan.pdf .
Zuse, K. (1969). Rechnender Raum . Brunswick: Vieweg & Sohn.
123