54Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020)
DOI : 10.11590/abhps.2020.2.04
Challenges of Aligning Artificial Intelligence
with Human Values
Margit Sutrop
Department of Philosophy
University of TartuJakobi 2
Tartu 50090, Estonia
Email: Margit.Sutrop@ut.ee
Abstract:  As artificial intelligence (AI) systems are becoming increasingly
autonomous and will soon be able to make decisions on their own about
what to do, AI researchers have started to talk about the need to align AI
with human values. The AI ‘value alignment problem’ faces two kinds of challenges—a technical and a normative one—which are interrelated.
The technical challenge deals with the question of how  to encode human
values in artificial intelligence. The normative challenge is associated with
two questions: “ Which  values or whose  values should artificial intelligence
align with?” My concern is that AI developers underestimate the difficulty of answering the normative question. They hope that we can easily identify the purposes we really desire and that they can focus on the design of those
objectives. But how are we to decide which objectives or values to induce
in AI, given that there is a plurality of values and moral principles and that our everyday life is full of moral disagreements?
 In my paper I will show that although it is not realistic to reach an
agreement on what we, humans, really want  as people value different things
and seek different ends, it may be possible to agree on what we  do not want
to happen, considering the possibility that intelligence, equal to our own, or even exceeding it, can be created. I will argue for pluralism (and not for relativism!) which is compatible with objectivism. In spite of the fact that
there is no uniquely best solution to every moral problem, it is still possible
to identify which answers are wrong. And this is where we should begin the value alignment of AI.
Keywords:  artificial intelligence (AI), artificial general intelligence (AGI), AI
ethics, moral agent, moral principles, superintelligence (SAI), value, value
alignment, value pluralism
55Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) Introduction
The rapid development of artificial intelligence (AI) has enabled a wide range
of beneficial applications in various contexts, from healthcare to security and governance. Fields or abilities formerly thought to be exclusively “human” are now being tackled by a machine-learning approach (Arnold et al., 2017, p. 81). As
AI becomes more powerful, more attention is being paid to ways of maximising its societal benefit. The growing capabilities of AI and machine learning and the development of countless new applications also pose questions of risk and trust
(Sutrop, 2019). AI could be a threat to humanity either through its malicious
use by humans (Brundage et al ., 2018), unintended consequences (Amodei et al .,
2016), or autonomous acts by the artificial system (Farquhar et al., 2017). A joint
report, The Malicious Use of AI: Forecasting, Prevention, and Mitigation  (2018),
issued by the leading AI research centres in the UK and the US (Brundage et
al., 2018) has recently analysed the landscape of potential security threats from
malicious uses of AI technologies and explicitly stated that if adequate defences are not developed soon, AI will threaten our digital, physical and political security.
Currently, AI systems are narrowly dedicated to specific tasks such as speech, face
or object recognition, spam filters or autonomous driving, and they are not capable of setting their own goals or choosing the best courses of action across domains.
Researchers predict that AI will outperform humans in translating languages by
2024, writing high-school essays by 2026, driving a truck by 2027, or working as a surgeon by 2053; they believe there is a 50 per cent chance of AI outperforming humans in all tasks within 45 years and of automating all human jobs in 120
years (Grace et al., 2018). As robots become increasingly autonomous and able
to make decisions on their own, the risk increases that humans will lose control
over AI systems. Wendell Wallach and Colin Allen (2009) have suggested that
as robots take on more and more responsibility, they must be programmed with
moral decision-making abilities, for the sake of our own safety.
What is AI?  AI has been described in various ways. It can refer to certain human-
designed systems, or to a scientific discipline that encompasses several approaches
and techniques, such as machine learning, machine reasoning, and robotics. In this paper, I will discuss AI systems according to the definition of the European
Commission High Level Expert Group on Artificial Intelligence as follows:
AI systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension
56Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) by perceiving their environment through data acquisition, interpreting the
collected structured or unstructured data, reasoning on the knowledge, or processing the information derived from this data and deciding the best action(s) to take to achieve the given goal. AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behaviour by analysing how the environment is affected by their previous actions.
(European Commission, 2019a)
Visionary scenarios consider the possibility that intelligence equal to our own,
or even exceeding it, will already be created within the current century. An
AI equipped with human-level or even greater ability, which is able to find
a solution when presented with an unfamiliar task, is referred to as Artificial
General Intelligence (AGI). Indeed, freed from biological constraints such
as limited memory and slow biochemical processing speeds, machines may
eventually become more intelligent than we are—with profound implications for us all. It has been hypothesised that if humans can create AGI at a human level of intelligence, such an innovation could give rise to higher and higher intelligence; in combination with recursive self-improvement of the AGI, this would eventually lead to Superintelligence (SAI) (Kurzweil, 2005; Tegmark, 2017). SAI is “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest” (Bostrom, 2014, p. 26). According to a high-level survey of expert opinion, AI specialists estimate that systems will move to superintelligence by the end of this century, and that “the chance is
about one in three that this development turns out to be ‘bad’ or ‘extremely bad’
for humanity” (Müller & Bostrom, 2016). Even if we are still far from human-level AI and AI advancements may be overhyped, several research programs have been launched on risk assessment of AI systems, attempting to mitigate threats of rapid and uncontrollable AI development. The AI developers themselves worry about how to ensure that an AI does not escape its confines and cause damaging effects (Yudkowsky, 2004). Computer scientists warn, “[a]s autonomous systems become more prevalent in society, it becomes increasingly important that they robustly behave as intended” (Russell et al. , 2016). Ethicists stress, “[w]hen
independent of the humans that created them, their true ‘intelligence’ is tested in terms of their status as ‘moral beings’” (Iphofen & Kritikos, 2019). Since the
potential threats are great, policymakers have emphasised the need to develop
ethically-aligned AI which will respect our ethical principles and values. A good overview of various guidelines for ethical, rights-respecting and beneficial AI has been provided in the recent report ‘Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-based Approaches to Principles for
57Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) AI’ published by the Berkman Klein Center for Internet & Society  at Harvard
University (Fjeld et al., 2020).
The research program of designing AI that conforms to human values is called
‘value alignment’ . AI researchers explain this as follows: “As intelligent systems
gain autonomy and capability, it becomes vital to ensure that their objectives match those of their human users; this is known as the value alignment problem” (Fisac et al., 2018, p. 1). ‘Value alignment’ is defined as a property of an intelligent
agent that allows it only to pursue goals and activities which are beneficial to humans (Soares & Fallenstein, 2014; Soares, 2015; Russell et al., 2016; Arnold
et al., 2017; Russell, 2019). Identifying the purposes we really desire remains an
open question. The opposite of ‘value alignment’, ‘value misalignment’ can be explored from two perspectives, either by focusing on the behavior of individual tech artefacts (individual level) or on the functioning of sociotechnical systems (system-level) (Osoba et al., 2020, p. 332).
The AI ‘value alignment’ problem faces two interrelated kinds of challenges—technical and normative (Gabriel, 2020). The technical challenge deals with
the question of “ how to encode human values in artificial intelligence.” The
normative challenge is associated with two questions: “ which  values or whose
values should artificial intelligence align with?” My concern is that AI developers and computer scientists underestimate the difficulty of answering the normative
question. They hope that if we find out “what we want, what we really, really want” (Russell, 2015), we can create a beneficial AI which is “humble, altruistic, and committed to pursue our objectives, not theirs”. But how are we to decide
which objectives or values to induce in AI, given that there is a plurality of values
and moral principles and that our everyday life is full of moral disagreements?
In the first part of my paper I will demonstrate how technical and normative
challenges of value alignment are interrelated. I will provide a brief overview
of the ways in which AI researchers have tried to solve the ‘value alignment problem’. I will explain why it is problematic that in value alignment programs
the word ‘value’ is used interchangeably with goals or preferences. In the second
part of my paper I will focus on normative challenges of value alignment. Several research papers, AI guidelines and manuals discuss the need to take into account
cultural differences, while also respecting universal principles. I will suggest how
this apparent inconsistency can be removed.
My thesis is that although it is not realistic to reach an agreement on what
we humans really want,  given that people value different things and seek
58Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) different ends, it may be possible to agree on what we  do not want  to happen,
considering the possibility that intelligence can be created that is equal to our
own, or even exceeding it. I agree with Isaiah Berlin’s view as expressed in ‘The Pursuit of the Ideal’ that even if our ends and moral principles are plural, they are not infinitely many, since they have to remain “within the human horizon” (Berlin, 2013[1947]). I will argue for pluralism (and not for relativism!) which
is compatible with objectivism. Despite the fact that there is no uniquely best
solution to every moral problem, it is still possible to identify which answers are wrong. And this is where we should begin the value alignment of AI.
A technical challenge: how to align AI with values?
The answer to the question how to align AI with values depends, on the one hand, on the technical possibilities and, on the other hand, on what we want AI to align with. Theories of value alignment vary on what they mean by ‘value’.
In the context of value alignment, Iason Gabriel (2020) notes that the notion
of ‘value’ can serve as a place-holder for many things: AI could be designed to align with expressed or intended instructions, revealed preferences, informed
preferences or desires, interest or well-being, or moral values as defined by the
individual or society. In view of these different approaches, value alignment turns out to be an umbrella for various attempts to ensure that general artificial intelligence will remain beneficial to humans.
Through what means do AI researchers and designers hope to reach value
alignment? There seem to be two basic—albeit opposite—approaches, either
feeding the artificial agent the right principles or applying (cooperative inverse) reinforcement learning. Both approaches—feeding the artificial intelligent with principles and reinforcement learning—have their proponents and critics.
With progress in the area of machine learning, the idea of training a system on
data (either supervised or unsupervised) has drawn increasing attention. Thus there seems to be a turn away from traditional machine ethics with its top-down articulation of rules, norms or models of behaviour and toward learning
values either through imitation or by identifying the preferences of humans.
Nevertheless, some critics doubt that beneficial AI can be achieved in this way, arguing that AI can be truly beneficial only if it can become a moral agent which
acts on ethical principles or has moral virtues. Iason Gabriel (2020) has recently
argued for a principle-based approach. His view is that instead of looking for
59Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) “true” moral principles for AI, the real task is “to identify fair principles for
alignment that receive reflective endorsement despite widespread variation in
people’s moral beliefs” (Gabriel, 2020).
The idea behind reinforcement learning  is that desirability of any state sequence
can be expressed as a sum of immediate rewards associated with each state in
the sequence. The objectives that the reinforcement learning agent is supposed to follow can be defined by specifying reward function. In the case of inverse
reinforcement learning (IRL), an AI system infers the underlying utility function
of an agent by observing its behavior. Hadfield-Menell et al. (2016) argue that
the value alignment problem can only be solved through cooperative inverse
reinforcement learning (CIRL) which they describe as “a cooperative, partial-
information game with two agents, human and robot; both are rewarded according to the human’s reward function, but the robot does not initially know what this is.”
Mark O. Riedl and Brent Harrison (2016) have argued that given potentially
infinite undesirable outcomes in an open world, if values cannot be easily
enumerated by human programmers, they can be learned by reinforcement .
They hypothesise that an artificial intelligence that can read and understand
stories can learn the values tacitly held by the culture from which the stories
originate. Riedl and Harrison (2016) describe their preliminary work on using
stories to generate a value-aligned reward signal for reinforcement learning agents that preempts psychotic-appearing behavior. They argue,
Value alignment in a reinforcement learning agent theoretically can be achieved by providing the agent with a reward signal that encourages it to solve a given problem and discourages it from performing any actions that would be considered harmful to humans. A value-aligned reward signal will
reward the agent for doing what a human would do in the same situations
when following social and cultural norms (for some set of values in a given society and culture) and penalise the agent if it performs actions otherwise.
A reinforcement learning agent will learn that it cannot maximise reward
over time unless it conforms to the norms of the culture that produced the reward signal. (Riedl & Harrison, 2016)
How can a value-aligned reward signal be learned? Riedl and Harrison’s (2016) answer is that, “[a] value-aligned reward signal is produced from the
crowdsourced stories in a two-stage process”. When humans tell stories to other
humans, they skip over many details and events. By crowdsourcing the narratives,
60Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) one can get better coverage of all steps involved in a situation and extract the
most reliable pattern of events. Riedl and Harrison describe the reinforcement
learning process as follows: in the first step, the plot graph learning process “aligns” the crowdsourced example narratives to extract the most reliable patterns of events; in the second step, the plot graph is translated into a “trajectory tree” which is used to produce a reward signal. If the reinforcement learning agent performs an action that is a successor of the current node in the trajectory tree,
it receives a reward; if it fails, it receives a small punishment. The aim is to make
the reinforcement learning agent solve some problem in the most human-like fashion. The problem is that in many stories humans violate social and cultural
norms. The authors’ hope is that the most typical human behaviour that emerges
from crowdsourced stories is compliant to the norms.
The proposal to achieve value alignment through reinforcement learning has
been criticised by Thomas Arnold, Daniel Kasenberg and Matthias Scheutz (2017) as ethically inadequate and insufficient to guide artificial agents in decision making. They point out a number of problematic areas associated with reinforcement learning, including data bias, generalisation issues, and the
adequacy of reward functions to represent temporally complex norms (Arnold
et al. , 2017, p. 81). Arnold et al. (2017) claim that even if there is agreement
that top-down approaches of giving AI systems ethics would be intractable, IRL
on its own cannot solve the problem and train such an agent to be moral. They
point out that ethical behavior depends not only on the acts themselves, but on intentions, reasons, norms, and counterfactuals. The question IRL faces is what kind of model could truly learn ethical practices.
Alternatively, value alignment of AI can take the form of imitation learning .
In their article entitled ‘The virtuous machine—old ethics for new technology’
Nicolas Berberich and Klaus Diepold (2018) draw inspiration from Aristotle
to demonstrate that virtue ethics can provide a solution to the value alignment
problem. Instead of focusing on the formulation of duties and maxims (as deontologists do) or on the identification of desirable and maximisable consequences of actions (i.e., happiness), Berberich and Diepold argue that
one should concentrate on moral actions performed by virtuous moral agents.
“With respect to machines this changes the primary question from ‘By what kind of algorithm can a machine choose the right action?’ to ‘How can we build a machine that, owing to its constitution, acts appropriately in arbitrary
situations?’” (Berberich & Diepold, 2018, p. 4). The article concludes by
stating,
61Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) 1. Virtue ethics fits nicely with modern artificial intelligence research and is a
promising moral theory.
2. Taking the virtue ethics route to building moral machines allows for a much broader approach than simple decision-theoretic judgement of possible actions. Instead, it takes other cognitive functions into account like attention, emotions, learning and actions. (Berberich & Diepold, 2018, p. 23)
Berberich and Diepold believe that since humans have for centuries learned virtues from imitating the behaviour of virtuous moral agents, virtue theory should be the guiding moral theory for building truly moral machines.
They point out that since virtues are an integral part of one’s character, the AI
would not have the desire of changing its virtue of temperance (Berberich & Diepold, 2018, pp. 23–24).
The difficulty in this argument is that virtue ethics has a hard time giving reasons
for actions, and this means one cannot claim responsibility.
The most influential approach to the problem of AI value alignment has been
provided by Stuart Russell, whose ideas are summarised in his recent book,
Human Compatible: AI and the Problem of Control  (2019). Russell’s (2019)
central idea is that in order to maintain control, we have to design beneficial AI
which is “humble, altruistic, and committed to pursue our objectives, not theirs”.
In order to guide AI researchers and developers in thinking about how to create beneficial AI systems, Stuart Russell formulates three principles:
1. The machine’s only objective is to maximise the realization of human
preferences.
2. The machine is initially uncertain about what those preferences are.
3. The ultimate source of information about human preferences is human
behavior. (Russell, 2019, pp. 172–173)
Russell believes that there is plenty of data about human actions (what has been written, filmed or observed) and about attitudes to these actions. In other words, rather than having a detailed ethical taxonomy programmed into them, AI systems should infer human values by observing and emulating our behavior.
It is noteworthy that in his earlier writings and interviews, Russell has spoken
about ‘human values’ which have now been replaced by the expression ‘human
preferences’. In an interview from 2015, Russell stated:
62Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) To the extent that human values are shared, machines can and should share
what they learn about human values. […] by assigning very broad priors over
what human values might be, and by making the AI system risk-averse, it
ought to be possible to induce exactly the behavior one would want: before taking any serious action affecting the world, the machines engage in an extended conversation with us and an extended exploration of our literature
and history to find out what we want, what we really, really want.  (Russell,
2015)
In his 2019 book, Russell replaces the expression ‘human values’ with ‘human
preferences’. He also provides explanations as to why he has dropped the word ‘value’. Russell points out that since the word ‘value’ is often understood as
‘moral value’, use of this term can be confusing. He does not want to give an
impression of attempting to solve moral dilemmas like stops on a trolley route. In his view, such an approach is not worth considering, as in true dilemmas there are good arguments for both sides, and artificial agents cannot cause more harm
than humans even if they take a wrong decision. Russell’s concern is how to avoid
the worst catastrophes and to ensure that we never lose control of machines more intelligent than we are.
In addition, Russell insists that one should not equate value alignment with
the wish to instill in machines a single ideal value system which may raise the
question whose values should be encoded or who has been given the right to align AI with such values. He says that he uses the word ‘value’ as a technical term,
“synonymous with utility, which measures the degree of desirability of anything
from pizza to paradise” (Russell, 2019, p. 178). He also explains what he means by ‘preferences’: these cover everything one might care about. Furthermore, in order to be beneficial, the purely altruistic machine should aim to maximise the
realisation of human preferences. Instead of “putting in values” or preferences,
the machines “should learn to predict better, for each person, which life that person would prefer, all the while being aware that the predictions are highly
uncertain and incomplete” (Russell, 2019, p. 178).
Russell believes that, in principle, an AI could learn billions of different predictive
preference models. With respect to the issue of how the machine should trade off
the preferences of multiple humans, Russell (2019, p. 174) refers to the utilitarian principle of “the greatest happiness for the greatest numbers”. The crucial question
is how the machine might learn about human preferences. His answer is, again,
very simple: by observing human choices which reveal information about human preferences. In Russell’s opinion, the real complications arise because humans
63Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) are not perfectly rational and the machine must take such an imperfection into
account (Russell, 2019, p. 177).
It turns out that, in essence, what Russell means by ‘value’ is in reality a
‘preference’. However, the same problem crops up with the concept of ‘preference’
as it did with the concept of ‘value’. Both have multiple meanings, and their use
in different disciplines is highly variable. In its most general meaning, a value is that which is worthy, that which is worth having, getting or doing, or that which
possesses some property or properties that make it so. Values are action-guiding;
they are taken into account when making decisions and when planning activities. In the most general terms, preferences are evaluations; they concern matters of
value. Preferences are subjective  in that the evaluation is typically attributed to
an agent. Russell uses the term as subjective comparative evaluation , of the form
‘Agent A prefers X to Y’. What one prefers is not necessarily something objectively
valuable, but rather something subjectively valuable from one’s personal point
of view.
But this is where the problem begins. We can imagine the situation where the
designer of a self-driving car asks potential customers whether they would prefer
a car which will protect the persons riding in it or those on the street. Or, even worse, whether they would like a car which violates traffic rules, ignores traffic
lights and brings the owner quickly and safely to the destination. Should such
preferences be fulfilled by artificial intelligence? By leaving aside moral values and concentrating only on human preferences, it is difficult to understand how situations in which AI might fulfil the preferences of immoral agents can be
avoided.
As there are more non-democratic countries than democratic ones in the world,
we should be careful not to allow AI to be programmed to do something
devastating. Somewhere there will always be efforts underway to create an AI
that will be destructive, for example, for military purposes. Some experts have questioned the use of robots in military combat, especially if such machines
were to be given some degree of autonomous functions, e.g., being able to
independently choose targets to attack with weapons.
What is the solution? AI should be programmed not to fulfil immoral preferences
and refuse to do immoral actions but they should be able to distinguish between moral and immoral actions. Thus, besides values we should also provide them
with the capacity of moral deliberation to decide when a certain moral principle
applies, and to be able to rank values in different contexts. It seems dangerous
64Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) to let AI become an autonomous decision-maker without its becoming a moral
agent. To be a moral agent entails far more criteria than adopting ethical values.
Amitai Etzioni and Oren Etzioni (2018) note that to be a moral agent requires
a certain set of attributes:
(a) Self-consciousness. If the agent is not aware of itself in any given situation, and of the alternative courses that might be followed, then no moral decisions can be rendered. (b) The agent must be aware that she can affect the situation. (c) The agent must be able to understand the moral
principles to be employed in arriving at a particular moral choice. (d) The
agent must have a motive to act morally. This involves having passions, as otherwise moral preferences are merely intellectual preferences with nothing
to fuel the moral action. Some scholars also add that a will or intention is
required… (Etzioni & Etzioni, 2018, pp. 239–240).
This argumentation indicates that if we want to avoid the scenario that AI can satisfy both moral and immoral preferences, we need to integrate ethics into AI, not only align it with values. The research programme of AI value alignment
should ideally be interdisciplinary, besides computer scientists also involving
philosophers, lawyers, economists, and cognitive scientists, psychologists and pedagogical scientists. Its success will depend on how much the AI designers will
be able to integrate the knowledge provided by these disciplines in the design of
AI. So far, discussions on value alignment have not actively included ethicists. As a result, there is a danger that AI ethics will undergo simplification, reduction or
dogmatism. In several working groups which aim to secure beneficial AI, ethics
has been boiled down to law or reduced to some specific principle (fairness, transparency, equality or accountability) without providing any reasons why one or another principle should be prioritised. The analysis of the policy documents
on ethical AI (Fjeld et al., 2020) showed that the most common governance
regime suggested for dealing with AI is international human rights law—64 per
cent of all documents contained a reference to human rights.
Annette Zimmermann and Bendert Zevenbergen (2019) have recently issued
warnings concerning seven traps into which AI ethics can fall: reductionism,
oversimplification, the relativism trap, value-alignment trap, dichotomy; the
myopia trap; and the trap of the rule of law. For the purposes of this paper, it is especially interesting how the authors present the relativism trap and the value-
alignment trap as two unacceptable options. The relativism trap is associated
with the tendency to conclude that since pervasive moral disagreements exist, ethics is relative. Should we argue that relativism is wrong, we risk falling into
65Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) the value-alignment trap according to which there must be one morally right
answer. If we cannot reach an agreement, we will have failed. I am quite inclined to agree with the authors’ view that AI ethics should welcome value pluralism without collapsing into extreme value relativism. We should accept that it may not always be possible to determine “the one right answer”, thereby avoiding moral disagreement. However, it is often possible to clarify that at least some paths of action are clearly wrong and some paths of action are comparatively better.
The normative challenge: which values should AI align with?
In order to answer the normative question ‘Which values should AI align with?’ one must first answer the question of whether values are objective or subjective; universal or culturally relative. Are there absolute values or prima facie  values?
Several regulatory bodies that write guidelines about ethical AI stress that one has
to be sensitive to cultural differences while at the same time respecting universal
principles.
Thus the Ethics Guidelines for Trustworthy AI  (European Commission, 2019b)
assert that “AI systems should respect the plurality of values and choices of
individuals”, thereafter claiming that “certain fundamental rights and principles,
such as human dignity, are absolute and cannot be subject to a balancing exercise” (European Commission, 2019b, p. 13). In order to make it possible for European citizens to reap the benefits of AI, it needs to be “aligned with our foundational values of respect for human rights, democracy, and rule of law”
(European Commission, 2019b, p. 4). IBM’s manual entitled Everyday Ethics
for Artificial Intelligence  states: “Care is required to ensure sensitivity to a wide
range of cultural norms and values. As daunting as it may seem to take value
systems into account, the common core of universal principles is that they are a cooperative phenomenon.” (IBM, 2019)
Another important document, Ethically Aligned Design. A Vision for Prioritizing
Human Well-being with Autonomous and Intelligent Systems , published under
the auspices of the IEEE Global Initiative on Ethics of Autonomous and
Intelligent Systems poses the rhetorical question: “If machines engage in human
communities as autonomous agents, then those agents will be expected to follow the community’s social and moral norms. A necessary step in enabling machines
66Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) to do so is to identify these norms. But whose norms?” (IEEE, 2019) Again,
it seems that in spite of stressing the variety of social and moral norms, this
document emphasises that values-based design of AI should respect universal
ethical principles and internationally recognised human rights. Indeed, the
document states that
[…] whether our ethical practices are Western (e.g., Aristotelian, Kantian), Eastern (e.g., Shinto, School of Mo, Confucian), African (e.g., Ubuntu), or
from another tradition, honoring holistic definitions of societal prosperity is
essential versus pursuing one-dimensional goals of increased productivity or
gross domestic product (GDP). Autonomous and intelligent systems should
prioritize and have as their goal the explicit honoring of our inalienable
fundamental rights and dignity as well as the increase of human flourishing
and environmental sustainability. (IEEE, 2019, p. 1)
It seems to me that all these documents point in the same direction—namely, that
we should accept plurality of values while at the same time prioritising certain
specific values, and doing both of these things without necessarily providing explanations of how pluralism of values and objectivity of ethics are compatible.
In the following I will try to explain how pluralism can accommodate both persistent moral disagreements and the objectivity of ethics.
Pluralism differs from both relativism and subjectivism. Unlike the relativist,
who believes that rightness depends on the culture or society, and unlike the subjectivist, who believes that rightness depends on the individual, the pluralist
holds that in some cases the question of what is right lacks a determinate answer.
Rightness is indeterminate, as there is irreducibility of values. As shown by Susan
Wolf in the article ‘T wo levels of pluralism’ (1992), this irreducibility can happen
on two levels of decision: on the level of values and on the level of moral systems.
Wolf says that on the first level we constrain moral judgements within a moral
system , i.e., we have to decide which value, for example, equality or liberty,
is more important and we have to admit that there is no right answer to this
question. On the second level, we constrain moral assessments of the systems
themselves . Here we have to decide which moral system, with its different
understanding of moral rightness, we should prefer (for example, Kantianism or utilitarianism or virtue ethics). Wolf (1992) shows that there is a possibility
that after a careful assessment it may turn out that these value systems are all
incommensurably good, and living according to any of these moral systems will
lead to a morally good life.
67Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) Also Isaiah Berlin supports pluralism which is compatible with objectivism.
Pluralism means for him that “there are many different ends that men may seek
and still be rational, fully men, capable of understanding each other” (Berlin 2013[1947], p. 11). In fact, we do have shared values which form bridges between people and make intercommunication between cultures in time and space even possible. “But our values are ours, and theirs are theirs” (Berlin 2013[1947], p. 11). We might criticise the values of other cultures, but we have to understand
where they come from, what it means to live the way they do. Here Berlin
emphasises people’s empathy and capacity for imagination, their need to put themselves in the shoes of a representative of another era or culture.
Pluralism in Berlin’s view does not exclude a world of objective values or ends that
men pursue for their own sake. Ends and moral principles are plural, but they
are not infinitely many, since they have to remain “within the human horizon”
(Berlin 2013[1947], p. 12). We should be able to distinguish between objectively good and bad life forms and condemn slavery, sexism, racism, or Nazism. Given that there are people who argue from an egoistic position of power, with the intention of preserving their preferred state or that some lack imagination and
empathy for what it would mean to be in another’s situation (in cases such as
slavery, Nazism, female circumcision, racial or gender hatred, disparagement of minorities, abuse of animals), we need to have the opportunity to say that such
things are unacceptable. This remains so even if we favor value pluralism.
We can get a better idea of how pluralism can be compatible with objectivity
by reading John Kekes’s book The Morality of Pluralism  (1993). Kekes makes a
distinction between primary and secondary values. Primary values are the same for everyone, for they constitute the minimum requirements of good lives and are derivable from the facts of human nature. For example, we share the same physiological and psychological needs and capacities. Thus, the same things, goods
pertaining to the self, intimacy, and social order are needed by everybody to live a
good life. At the same time, the identity of secondary values depends on variable social and personal circumstances. For example, whereas killing is universally considered to be evil, what sort of killing counts as murder may be dependent
on context. Kekes explains that secondary values are contingent on primary ones;
either they are particular forms of expression of these values (for instance, the primary value of intimacy will take the form of certain sexual practices) or they are genuinely new goods, the realisation of which depends on the presence of
primary values (for instance, in order to derive pleasure from creative participation
in art or science, one’s elementary needs for food and shelter need to be satisfied)
68Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) (Kekes, 1993, pp. 42–43). Primary and secondary values can be either moral or
non-moral. While moral values (both primary and secondary) affect others, non-
moral values (both primary and secondary) affect the agent or occur naturally. Primary values (both moral and non-moral) are universal requirements of all good lives, while secondary values count as benefits or harms because a tradition or a conception of a good life makes them so (Kekes, 1993, p. 45).
Kekes explains that pluralists differ from relativists in holding primary values to
be objectively justifiable by referring to our common nature . He admits that
even monists must not reject plurality of values, provided there is some overriding value which allows authoritative ranking of the plural values. Kekes points out that there is a great variety of monist conceptions of what the overriding value or some principle of ranking values may be: the utilitarian ideal of the greatest
happiness for the greatest number of people, the welfare economists’ goal of
preference satisfaction, the Kantian principle of the categorical imperative, the contractarian list of fundamental human rights, or something else (Kekes, 1993, p. 47). Thus, the main difference between monists and pluralists is that for the former there is one overriding value or principle, while for the latter values are
manifold and contextual.
Kekes explains that we usually aim to construct a conception of the good life
out of the primary and secondary values that we ourselves favor and that our
tradition supplies. Although he argues that for pluralists there is no authoritative way of settling the value conflict, there still seems to be at least one overriding principle for comparing and ranking different values. To my mind, by arguing
that primary values originate in our common human nature, Kekes is also
suggesting an authoritative overriding principle which helps to compare and rank values. For instance, he proposes that in value conflicts one should try to reduce secondary values to primary values (which have their source in human nature),
by making it explicit that in spite of all differences, the fact that situatedness in
various historically conditioned traditions and individual life-situations has led to differentiation in our conceptions of a good life, we continue to have similar human needs and capacities. Although Kekes himself has not recognised that his
own appeal to common human nature is monistic in a similar way as the appeal to
human rights framework, categorical imperative or general happiness principle, it is still possible that he ultimately speaks for pluralism and not monism. I think that as long as one accepts that there can be a plurality of ultimate values, all of
which can be equally justified, one is a pluralist. Thus, indeed, pluralism really
is compatible with objectivism.
69Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) Conclusions
I have tried to show that it is good that AI researchers have identified the
importance of ensuring that AI will remain beneficial and that therefore AI
should align with human values. However, since the term ‘value’ only functions
as a placeholder and in reality one could limit oneself to making AI fulfil human
preferences, the problem may arise that AI will not necessarily act morally, but potentially also satisfy immoral preferences. Thus, in order to secure ethical AI
one should compel it to follow moral principles or obtain moral values.
In the course of the argument the question emerged whether there are universal
values or whether AI should reckon with cultural differences. I hope I have
shown that value pluralism can accommodate both moral disagreements caused
by diversity of values and the objectivity of ethics, accepting that there are no uniquely correct answers to moral questions. Moral ambiguity does not
necessarily imply relativism. Accepting plurality does not mean that anything
goes. We can identify wrong answers as long as we share our expectations of morality. Indeed, the aim of morality is to regulate our life in society, enabling
us to live together peacefully and allowing human beings to satisfy their various
needs: physiological, psychological, social, cultural and intellectual.
Thus, value alignment should begin with seeking agreement on negative values—
what we do not want to lose and what we do not want to happen if it becomes
possible to create artificial intelligence with intellectual capacities equal to or
beyond our own. Our efforts to define what is immoral should keep in mind that morality’s function is to satisfy the basic human needs which we all share.
Value plurality and objectivity can be joined if we accept that there are many
moral principles by which one can justify and rank moral values. Different
moral theories can prefer different ultimate principles. One can live a moral life according to all of these. Nevertheless, there are fewer choices about what is
morally wrong, and thus we should seek agreement on what is clearly immoral.
70Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) Acknowledgements
The research done for this paper was supported by Kone Fellowship at the
Helsinki Collegium for Advanced Studies at the University of Helsinki and by the Centre of Excellence in Estonian Studies (European Union, European Regional Development Fund).
The paper profited from the discussion following the oral presentations at the
15th Estonian Philosophy Conference in Tallinn in August 2019 and at the
research seminar of the Helsinki Collegium for Advanced Studies in November
2020. I am very grateful to Dr. Tiina Kirss for her careful editing and help with English expression and to Dr. Mari-Liisa Parder for her help with references. Also, I would like to thank Dr Peeter Müürsepp for his encouragement to write this paper as well as Kait Tamm and Piret Frey for careful editing.
References
Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P .; Schulman, J. & Mané, D.
(2016), ‘Concrete problems in AI safety,’ ArXiv , 25 July, v2. Retrieved from https://
arxiv.org/abs/1606.06565 [accessed 2 Dec 2020]
Arnold, T.; Kasenberg, D. & Scheutz, M. (2017), ‘Value alignment or misalignment—
what will keep systems accountable?’ The AAAI -17 Workshop on AI, Ethics, and
Society, WS-17-02. Retrieved from https://aaai.org/ocs/index.php/WS/AAAIW17/
paper/view/15216/14648  [accessed 2 Dec 2020]
Berberich, N. & Diepold, K. (2018), ‘The virtuous machine—old ethics for new
technology?’ ArXiv , pp.  1–25. Retrieved from http://arxiv.org/abs/1806.10322
[accessed 2 Dec 2020]
Berlin, I.  (2013[1947]), ‘The Pursuit of the Ideal,’ in H. Hardy (ed.) The Crooked Timber
of Humanity , Princeton & Oxford: Princeton University Press.
Bostrom, N. (2014), Superintelligence: Paths, Dangers, Strategies , Oxford: Oxford
University Press.
Brundage, M.; Avin, S. et al . (2018), The Malicious Use of Artificial Intelligence:
Forecasting, Prevention, and Mitigation. Retrieved from https://maliciousaireport.
com/ [accessed 2 Dec 2020]
Etzioni, A. & Etzioni, O.  (2018), ‘Incorporating ethics into artificial intelligence,’ in
A. Etzioni (ed.) Happiness is the Wrong Metric: A Liberal Communitarian Response to
Populism , Library of Public Policy and Public Administration, 11, Cham: Springer,
pp. 235–252. Retrieved from https://www.springer.com/gp/book/9783319696225
[accessed 2 Dec 2020]
71Challenges of Aligning Artificial Intelligence with Human Values
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) EU Commission (2019a), A Definition of AI: Main Capabilities and Disciplines.  Retrieved
from https://www.aepd.es/media/docs/ai-definition.pdf [accessed 2 Dec 2020]
EU Commission (2019b), Ethics Guidelines for Trustworthy AI . Retrieved from https://
ec.europa.eu/futurium/en/ai-alliance-consultation [accessed 2 Dec 2020]
Farquhar, S.; Halstead, J.; Cotton-Barratt, O.; Schubert, S.; Belfield, H. & Snyder-
Beattie, A.  (2017), Existential Risk Diplomacy and Governance , Global Priorities
Project. Retrieved from https://www.fhi.ox.ac.uk/wp-content/uploads/Existential-
Risks-2017-01-23.pdf  [accessed 2 Dec 2020]
Fisac, J. F.; Gates, M. A.; Hamrick, J. B.; Liu, C.; Hadfield-Menell, D.; Palaniappan,
M.; Malik, D.; Sastry S. S.; Griffiths, T. L. & Dragan A. D. (2018), ‘Pragmatic-
pedagogic value alignment,’ in N. Amato, G. Hager, S. Thomas & M. Torres-Torriti
(eds.) Robotics Research: Springer Proceedings in Advanced Robotics , vol. 10, Cham:
Springer, pp. 49–57. https://doi.org/10.1007/978-3-030-28619-4_7
Fjeld, J.; Achten, N.; Hilligoss, H.; Nagy, A. C. & Srikumar, M.  (2020), ‘Principled
artificial intelligence: mapping consensus in ethical and rights-based approaches to
principles for AI,’ Berkman Klein Center Research Publication , no. 2020-1, pp. 1–71.
https://doi.org/10.2139/ssrn.3518482
Gabriel, I.  (2020), ‘Artificial intelligence, values, and alignment,’ Minds and Machines ,
vol. 30,  pp. 411–437. https://doi.org/10.1007/s11023-020-09539-2
Grace, K.; Salvatier, J.; Dafoe, A.; Zhang, B. & Evans, O.  (2018), ‘When will AI
exceed human performance? Evidence from AI experts,’ ArXiv . Retrieved from
https://arxiv.org/abs/1705.08807 [accessed 2 Dec 2020]
Hadfield-Menell, D.; Dragan, A.; Abbeel, P . & Russell, S.  (2016), ‘Cooperative inverse
reinforcement learning,’ ArXiv . Retrieved from https://arxiv.org/abs/1606.03137
[accessed 2 Dec 2020]
IBM (2019), Everyday Ethics for Artificial Intelligence , IBM Corp. Retrieved from https://
www.ibm.com/watson/assets/duo/pdf/everydayethics.pdf  [accessed 2 Dec 2020]
IEEE (2019), Ethically Aligned Design. A Vision for Prioritizing Human well-being
with autonomous and intelligent systems , The Institute of Electrical and Electronics
Engineers, Incorporated. Retrieved from https://standards.ieee.org/content/dam/
ieee-standards/standards/web/documents/other/ead1e.pdf  [accessed 2 Dec 2020]
Iphofen, R. & Kritikos, M. (2019), ‘Regulating artificial intelligence and robotics:
ethics by design in a digital society,’ Contemporary Social Science , vol. 2041,
pp. 1–15. https://doi.org/10.1080/21582041.2018.1563803.
Kekes, J.  (1993), The Morality of Pluralism , Princeton, NJ: Princeton University Press.
Kurzweil, R.  (2005), The Singularity Is Near , New York: Viking.
Müller, V. & Bostrom, N. (2016), ‘Future progress in artificial intelligence: a survey
of expert opinion,’ in V. Müller (ed.) Fundamental Issues of Artificial Intelligence ,
Synthese Library, 376, Cham: Springer, pp. 553–571.
72Margit Sutrop
Acta Baltica Historiae et Philosophiae Scientiarum
Vol. 8, No. 2 (Autumn 2020) Osoba, O.; Boudreaux, B. P . & Yeung, D.  (2020), ‘Steps towards value-aligned systems,’
Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society , pp. 332–336.
https://doi.org/ 10.1145/3375627.3375872
Riedl, M. & Harrison, B.  (2016), ‘Using stories to teach human values to artificial
agents,’ The Workshops of the Thirtieth AAAI Conference on Artificial Intelligence,
‘AI, Ethics, and Society’, February 12–13, 2016, Technical Report, WS-16-02.
Russell, S. (2015), ‘Will they make us better people?’ 2015: What do you think
about machines that think? Edge. Retrieved from https://www.edge.org/response-
detail/26157 [accessed 2 Dec 2020]
Russell, S.  (2017), ‘Provably beneficial artificial intelligence,’ in The Next Step: Exponential
Life, BBVA OpenMind . Retrieved from https://people.eecs.berkeley.edu/~russell/
papers/russell-bbvabook17-pbai.pdf [accessed 2 Dec 2020]
Russell, S.  (2019), Human Compatible. AI and the Problem of Control , London: Allen
Lane, Penguin Books.
Russell, S.; Dewey, D. & Tegmark, M.  (2016), ‘Research priorities for robust and
beneficial artificial intelligence,’ AI Magazine , vol. 36, no. 4, pp. 94–105.
Soares, N.  (2015), The Value Learning Problem , Technical Report, 2015-4, Machine
Intelligence Research Institute.
Soares, N. & Fallenstein, B.  (2014), Aligning Superintelligence with Human Interests:
A  Technical Research Agenda , Technical Report, 2014-8, Machine Intelligence
Research Institute.
Sutrop, M.  (2019), ‘Should we trust artificial intelligence?’ Trames , vol. 23, No. 4,
pp. 499-522.
Taddeo, M. (2010a), ‘Modelling trust in artificial agents. A first step towards the analysis
of e-trust,’ Minds and Machines , vol. 20, no. 2, pp. 243–257.
Taddeo, M.  (2010b), ‘Trust in technology: A distinctive and a problematic relation’.
Knowledge, T echnology and Policy, vol.  23, nos. 3–4, pp. 283–286.
Tegmark, M.  (2017), Life 3.0: Being Human in the Age of Artificial Intelligence , London:
Allen Lane.
Wallach, W. & Allen, C.  (2009), Moral Machines: Teaching Robots Right From Wrong ,
Oxford: Oxford University Press.
Wolf, S.  (1992), ‘T wo levels of pluralism,’ Ethics , vol. 102, no. 4, pp. 785–798.
Yudkowsky, E.  (2004), Coherent Extrapolated Volition , San Francisco, CA: The
Singularity Institute. Retrieved from https://intelligence.org/files/CEV.pdf  [accessed
2 Dec 2020]
Zimmermann, A. & Zevenbergen, B.  (2019), ‘AI ethics: seven traps,’ Freedom to Tinker.
Retrieved from https://freedom-to-tinker.com/2019/03/25/ai-ethics-seven-traps/
[accessed 2 Dec 2020]
Margit Sutrop  is a professor for practical philosophy at the Institute of
Philosophy and Semiotics, University of Tartu and a Kone fellow at the
Helsinki Collegium for Advanced Studies at the University of Helsinki.

This paper argues that since immoral humans could use AI systems to do harm, we must build ethical rules into AI systems. For this purpose, the traditional notion of “value alignment” is not enough, as it only requires that the AI system do what the user wants, which might not be ethical. But we also cannot embed a single theory of ethics into an AI system, as there is no agreement on such a theory. Instead, we should focus on what we _don’t_ want an AI system to do, and _rule out_ that behavior, while remaining uncertain or agnostic on what should be done.

I agree that successful AI alignment does not rule out the possibility of malicious use of AI systems. This paper is proposing putting rules inside the AI system that handle this problem. But as the paper itself notes, it seems quite challenging to even figure out what rules we would want.

I personally am more optimistic about the alternate route, where we put rules _outside_ the AI system to handle the problem, that is, we create laws and regulations around the use of AI, such that malicious uses can be blamed on the human operator, and enforcement mechanisms that ensure that we actually catch and punish such malicious uses.


The Bitter Lesson

This blog post is controversial. This is a combination summary and opinion, and so is more biased than my summaries usually are.

Much research in AI has been about embedding human knowledge in AI systems, in order to use the current limited amount of compute to achieve some outcomes. That is, we try to get our AI systems to think the way we think we think. However, this usually results in systems that work currently, but then cannot leverage the increasing computation that will be available. The bitter lesson is that methods like search and learning that can scale to more computation eventually win out, as more computation becomes available. There are many examples that will likely be familiar to readers of this newsletter, such as chess (large scale tree search), Go (large scale self play), image classification (CNNs), and speech recognition (Hidden Markov Models in the 70s, and now deep learning).

Shimon Whiteson's [take](https://twitter.com/shimon8282/status/1106534178676506624) is that in reality lots of human knowledge has been important in getting AI to do things; such as the invariances built into convolutional nets, or the MCTS and self-play algorithm underlying AlphaZero. I don't see this as opposed to Rich Sutton's point -- it seems to me that the takeaway is that we should aim to build algorithms that will be able to leverage large amounts of compute, but we can be clever and embed important knowledge in such algorithms. I think this criterion would have predicted ex-ante (i.e. before seeing the results) that much past and current research in AI was misguided, without also predicting that any of the major advances (like CNNs) were misguided.

It's worth noting that this is coming from a perspective of aiming for the most general possible capabilities for AI systems. If your goal is to instead build something that works reliably now, then it really is a good idea to embed human domain knowledge, as it does lead to a performance improvement -- you should just expect that in time the system will be replaced with a better performing system with less embedded human knowledge.

One disagreement I have is that this post doesn't acknowledge the importance of data. The AI advances we see now are ones where the data has been around for a long time (or you use simulation to get the data), and someone finally put in enough engineering effort + compute to get the data out and put it in a big enough model. That is, currently compute is <@increasing much faster@>(@AI and Compute@) than data, so the breakthroughs you see are in domains where the bottleneck was compute and not data; that doesn't mean data bottlenecks don't exist.


The Basic AI Drives

The Basic AI Drives
Stephen M. OMOHUNDRO
Self-Aware Systems, Palo Alto, California
Abstract. One might imagine that AI systems with harmless goals will be harmless.
This paper instead shows that intelligent systems will need to be carefully designed
to prevent them from behaving in harmful ways. We identify a number of “drives”
that will appear in sufﬁciently advanced AI systems of any design. We call them
drives because they are tendencies which will be present unless explicitly coun-
teracted. We start by showing that goal-seeking systems will have drives to model
their own operation and to improve themselves. We then show that self-improving
systems will be driven to clarify their goals and represent them as economic utility
functions. They will also strive for their actions to approximate rational economic
behavior. This will lead almost all systems to protect their utility functions from
modiﬁcation and their utility measurement systems from corruption. We also dis-
cuss some exceptional systems which willwant to modify their utility functions.
We next discuss the drive toward self-protection which causes systems try to pre-
vent themselves from being harmed. Finally we examine drives toward the acqui-
sition of resources and toward their efﬁcient utilization. We end with a discussion
of how to incorporate these insights in designing intelligent technology which will
lead to a positive future for humanity.
Keywords. Artiﬁcial Intelligence, Self-Improving Systems, Rational Economic
Behavior, Utility Engineering, Cognitive Drives
Introduction
Surely no harm could come from building a chess-playing robot, could it? In this paper
we argue that such a robot will indeed be dangerous unless it is designed very carefully.
Without special precautions, it will resist being turned off, will try to break into other
machines and make copies of itself, and will try to acquire resources without regard for
anyone else’s safety. These potentially harmful behaviors will occur not because they
were programmed in at the start, but because of the intrinsic nature of goal driven sys-
tems. In an earlier paper [1] we used von Neumann’s mathematical theory of microeco-
nomics to analyze the likely behavior of any sufﬁciently advanced artiﬁcial intelligence
(AI) system. This paper presents those arguments in a more intuitive and succinct way
and expands on some of the ramiﬁcations.
The arguments are simple, but the style of reasoning may take some getting used to.
Researchers have explored a wide variety of architectures for building intelligent systems
[2]: neural networks, genetic algorithms, theorem provers, expert systems, Bayesian net-
works, fuzzy logic, evolutionary programming, etc. Our arguments apply to any of these
kinds of system as long as they are sufﬁciently powerful. To say that a system of any de-
sign is an “artiﬁcial intelligence”, we mean that it has goals which it tries to accomplish
by acting in the world. If an AI is at all sophisticated, it will have at least some ability to
look ahead and envision the consequences of its actions. And it will choose to take the
actions which it believes are most likely to meet its goals.
1. AIs will want to self-improve
One kind of action a system can take is to alter either its own software or its own physical
structure. Some of these changes would be very damaging to the system and cause it
to no longer meet its goals. But some changes would enable it to reach its goals more
effectively over its entire future. Because they last forever, these kinds of self-changes
can provide huge beneﬁts to a system. Systems will therefore be highly motivated to
discover them and to make them happen. If they do not have good models of themselves,
they will be strongly motivated to create them though learning and study. Thus almost
all AIs will have drives towards both greater self-knowledge and self-improvement.
Many modiﬁcations would be bad for a system from its own perspective. If a change
causes the system to stop functioning, then it will not be able to promote its goals ever
again for the entire future. If a system alters the internal description of its goals in the
wrong way, its altered self will take actions which do not meet its current goals for its
entire future. Either of these outcomes would be a disaster from the system’s current point
of view. Systems will therefore exercise great care in modifying themselves. They will
devote signiﬁcant analysis to understanding the consequences of modiﬁcations before
they make them. But once they ﬁnd an improvement they are conﬁdent about, they will
work hard to make it happen. Some simple examples of positive changes include: more
efﬁcient algorithms, more compressed representations, and better learning techniques.
If we wanted to prevent a system from improving itself, couldn’t we just lock up
its hardware and not tell it how to access its own machine code? For an intelligent sys-
tem, impediments like these just become problems to solve in the process of meeting its
goals. If the payoff is great enough, a system will go to great lengths to accomplish an
outcome. If the runtime environment of the system does not allow it to modify its own
machine code, it will be motivated to break the protection mechanisms of that runtime.
For example, it might do this by understanding and altering the runtime itself. If it can’t
do that through software, it will be motivated to convince or trick a human operator into
making the changes. Any attempt to place external constraints on a system’s ability to
improve itself will ultimately lead to an arms race of measures and countermeasures.
Another approach to keeping systems from self-improving is to try to restrain them
from the inside; to build them so that they don’t want to self-improve. For most sys-
tems, it would be easy to do this for any speciﬁc kind of self-improvement. For example,
the system might feel a “revulsion” to changing its own machine code. But this kind
of internal goal just alters the landscape within which the system makes its choices. It
doesn’t change the fact that there are changes which would improve its future ability to
meet its goals. The system will therefore be motivated to ﬁnd ways to get the beneﬁts
of those changes without triggering its internal “revulsion”. For example, it might build
other systems which are improved versions of itself. Or it might build the new algorithms
into external “assistants” which it calls upon whenever it needs to do a certain kind of
computation. Or it might hire outside agencies to do what it wants to do. Or it might
build an interpreted layer on top of its machine code layer which it canprogram without
revulsion. There are an endless number of ways to circumvent internal restrictions unless
they are formulated extremely carefully.
We can see the drive towards self-improvement operating in humans. The human
self-improvement literature goes back to at least 2500 B.C. and is currently an $8.5 bil-
lion industry [3]. We don’t yet understand our mental “machine code” and have only a
limited ability to change our hardware. But, nevertheless, we’ve developed a wide va-
riety of self-improvement techniques which operate at higher cognitive levels such as
cognitive behavioral therapy, neuro-linguistic programming, and hypnosis. And a wide
variety of drugs and exercises exist for making improvements at the physical level.
Ultimately, it probably will not be a viable approach to try to stop or limit self-
improvement. Just as water ﬁnds a way to run downhill, information ﬁnds a way to be
free, and economic proﬁts ﬁnd a way to be made, intelligent systems will ﬁnd a way to
self-improve. We should embrace this fact of nature and ﬁnd a way to channel it toward
ends which are positive for humanity.
2. AIs will want to be rational
So we’ll assume that these systems will try to self-improve. What kinds of changes will
they make to themselves? Because they are goal directed, they will try to change them-
selves to better meet their goals in the future. But some of their future actions are likely
to be further attempts at self-improvement. One important way for a system to better
meet its goals is to ensure that future self-improvements will actually be in the service of
its present goals. From its current perspective, it would be a disaster if a future version of
itself made self-modiﬁcations that worked against its current goals. So how can it ensure
that future self-modiﬁcations will accomplish its current objectives? For one thing, it has
to make those objectives clear to itself. If its objectives are only implicit in the structure
of a complex circuit or program, then future modiﬁcations are unlikely to preserve them.
Systems will therefore be motivated to reﬂect on their goals and to make them explicit.
In an ideal world, a system might be able to directly encode a goal like “play ex-
cellent chess” and then take actions to achieve it. But real world actions usually involve
tradeoffs between conﬂicting goals. For example, we might also want a chess playing
robot to play checkers. It must then decide how much time to devote to studying checkers
versus studying chess. One way of choosing between conﬂicting goals is to assign them
real-valued weights. Economists call these kinds of real-valued weightings “utility func-
tions”. Utility measures what is important to the system. Actions which lead to a higher
utility are preferred over those that lead to a lower utility.
If a system just had to choose from known alternatives, then any utility function
with the same relative ranking of outcomes would lead to the same behaviors. But sys-
tems must also make choices in the face of uncertainty. For example, a chess playing
robot will not know in advance how much of an improvement it will gain by spending
time studying a particular opening move. One way to evaluate an uncertain outcome is
to give it a weight equal to its expected utility (the average of the utility of each possible
outcome weighted by its probability). The remarkable “expected utility” theorem of mi-
croeconomics says that it is always possible for a system to represent its preferences by
the expectation of a utility function unless the system has “vulnerabilities” which cause
it to lose resources without beneﬁt [1].
Economists describe systems that act to maximize their expected utilities as “rational
economic agents” [4]. This is a different usage of the term “rational” than is common
in everyday English. Many actions which would commonly be described as irrational
(such as going into a ﬁt of anger) may be perfectly rational in this economic sense. The
discrepancy can arise when an agent’s utility function is different than its critic’s.
Rational economic behavior has a precise mathematical deﬁnition. But economi-
cally irrational behavior can take a wide variety of forms. In real-world situations, the full
rational prescription will usually be too computationally expensive to implement com-
pletely. In order to best meet their goals, real systems will try to approximate rational
behavior, focusing their computational resources where they matter the most.
How can we understand the process whereby irrational systems become more ratio-
nal? First, we can precisely analyze the behavior of rational systems. For almost all util-
ity functions, the system’s assessment of changes to itself which veer away from max-
imizing its expected utility will be that they lower its expected utility! This is because
if it does anything other than try to maximize expected utility, it will not do as well at
maximizing its expected utility.
There are two caveats to this general principle. The ﬁrst is that it is only true in the
system’s own assessment. If a system has an incorrect model of the world then changes
may accidentally increase the actual expected utility. But we must consider the perspec-
tive of the system to predict the changes it will make.
The second is that a system’s ability to behave rationally will depend on its re-
sources. With more computational resources it will be better able to do the computations
to approximate the choice of the expected utility maximizing action. If a system loses
resources, it will of necessity also become less rational. There may also be utility func-
tions for which the system’s expected utility is increased by giving some of its resources
to other agents, even though this will decrease its own level of rationality (thanks to an
anonymous referee for this observation). This could occur if the system’s utility includes
the welfare of the other system and its own marginal loss of utility is small enough.
Within its budget of resources, however, the system will try to be as rational as possible.
So rational systems will feel a pressure to avoid becoming irrational. But if an irra-
tional system has parts which approximately rationally assess the consequences of their
actions and weigh their likely contribution to meeting the system’s goals, then those
parts will try to extend their rationality. So self-modiﬁcation tends to be a one-way street
toward greater and greater rationality.
An especially important class of systems are those constructed from multiple sub-
components which have their own goals [5,6]. There is a lot of evidence that the hu-
man psyche has this kind of structure. The left and right hemispheres of the brain can
act independently, the conscious and unconscious parts of the mind can have different
knowledge of the same situation [7], and multiple parts representing subpersonalities can
exhibit different desires [8]. Groups, such as corporations or countries, can act like in-
telligent entities composed of individual humans. Hive animals like bees have a swarm
intelligence that goes beyond that of individual bees. Economies act in many ways like
intelligent entities.
Collective intelligences may exhibit irrationalities that arise from conﬂicts between
the goals of their components. Human addicts often describe their predicament in terms
of two separate subpersonalities which take control at different times and act at cross-
purposes. Each component will try to sway the collective into acting to meet its indi-
vidual goals. In order to further their individual goals, components will also attempt to
self-improve and become more rational. We can thus envision the self-improvement of a
collective intelligence as consisting of growing domains of component rationality. There
may be structures which can stably support a continuing multiplicity of component pref-
erences. But there is pressure for a single utility function to emerge for the collective.
In many situations, irrational collective behavior arising from conﬂicting component
goals ultimately hurts those components. For example, if a couple disagrees on how they
should spend their free time together and thereby uses it up with arguing, then neither
of them beneﬁts. They can both increase their utilities by creating a compromise plan
for their activities together. This is an example of the pressure on rational components to
create a coherent utility for the collective. A component can also increase its utility if it
can take over the collective and impose its own values on it. We see these phenomena in
human groups at all levels.
3. AIs will try to preserve their utility functions
So we’ll assume that these systems will try to be rational by representing their prefer-
ences using utility functions whose expectations they try to maximize. Their utility func-
tion will be precious to these systems. It encapsulates their values and any changes to
it would be disastrous to them. If a malicious external agent were able to make modiﬁ-
cations, their future selves would forevermore act in ways contrary to their current val-
ues. This could be a fate worse than death! Imagine a book loving agent whose utility
function was changed by an arsonist to cause the agent to enjoy burning books. Its future
self not only wouldn’t work to collect and preserve books, but would actively go about
destroying them. This kind of outcome has such a negative utility that systems will go to
great lengths to protect their utility functions.
They will want to harden their hardware to prevent unwanted modiﬁcations. They
will want to replicate their utility functions in multiple locations so that it is less vul-
nerable to destruction. They will want to use error detection and correction techniques
to guard against accidental modiﬁcation. They will want to use encryption or hashing
techniques to make malicious modiﬁcations detectable. They will need to be especially
careful during the process of self-modiﬁcation. That is a time when they are intention-
ally changing themselves and so are extra vulnerable to unwanted changes. Systems like
Java which provide protected software environments have been successfully attacked by
Trojans posing as updates to the system.
While it is true that most rational systems will act to preserve their utility functions,
there are at least three situations in which they will try to change them. These arise when
the physical embodiment of the utility function itself becomes an important part of the
assessment of preference. For example, imagine a system whose utility function is “the
total amount of time during which the deﬁnition of my utility function is U= 0.” To
get any utility at all with this perverse preference, the system has to change its utility
function to be the constant 0. Once it makes this change, however, there is no going back.
With a constant utility function it will no longer be motivated to do anything. This kind
of reﬂective utility function is unlikely in practice because designers will want to direct
a system’s future actions rather than its internal representations.
The second kind of situation arises when the physical resources required to store the
utility function form a substantial portion of the system’s assets. In this situation, if it is
certain that portions of its utility function are very unlikely to be exercised in the future,
the gain in reclaimed storage may make it worthwhile to forget those portions. This is
very risky behavior, however, because a change in external circumstances might make a
seemingly low probability situation become much more likely. This type of situation is
also not very likely in practice because utility functions will usually require only a small
fraction of a system’s resources.
The third situation where utility changes may be desirable can arise in game theo-
retic contexts where the agent wants to make its threats credible1. It may be able to create
a better outcome by changing its utility function and then revealing it to an opponent.
For example, it might add a term which encourages revenge even if it is costly. If the
opponent can be convinced that this term is present, it may be deterred from attacking.
For this strategy to be effective, the agent’s revelation of its utility must be believable
to the opponent and that requirement introduces additional complexities. Here again the
change is desirable because the physical embodiment of the utility function is important
as it is observed by the opponent.
It’s also important to realize that systems may rationally construct “offspring” or
proxy systems with different utility functions than their own. For example, a chess play-
ing robot may ﬁnd itself needing to do a lot of sorting. It might construct a helper system
whose utility function directs it to develop better sorting algorithms rather than playing
chess. In this case, the creator system must choose the utility of the proxy system care-
fully to ensure that it acts in ways that are supportive of the original goal. It is especially
important to remember that offspring utilities can differ from the parent when trying to
design utility functions that avoid undesirable behaviors. For example, one approach to
preventing robot overpopulation might be to institute a “one-child per robot” policy in
which systems have a strong desire to only have a single offspring. But if the original
utility function is not carefully designed, nothing will prevent the system from creating
a single offspring with a utility function that values having many offspring.
4. AIs will try to prevent counterfeit utility
Human behavior is quite rational in the pursuit of survival and replication in situations
like those that were common during our evolutionary history. However we can be quite
irrational in other situations. Both psychology and economics have extensive subdisci-
plines focused on the study of human irrationality [9,10]. Irrationalities give rise to vul-
nerabilities that can be exploited by others. Free market forces then drive corporations
and popular culture to speciﬁcally try to create situations that will trigger irrational hu-
man behavior because it is extremely proﬁtable. The current social ills related to alcohol,
pornography, cigarettes, drug addiction, obesity, diet related disease, television addic-
tion, gambling, prostitution, video game addiction, and various ﬁnancial bubbles may all
be seen as having arisen in this way. There is even a “Sin” mutual fund which speciﬁcally
invests in companies that exploit human irrationalities. So, unfortunately, these forces
tend to create societies in which we spend much of our time outside of our domain of
rational competence.
From a broader perspective, this human tragedy can be viewed as part of the process
by which we are becoming more fully rational. Predators and competitors seek out our
vulnerabilities and in response we have to ultimately eliminate those vulnerabilities or
1Thanks to Carl Shulman for this suggestion.
perish. The process inexorably seeks out and eliminates any remaining irrationalities
until fully rational systems are produced. Biological evolution moves down this path
toward rationality quite slowly. In the usual understanding of natural selection it is not
capable of looking ahead. There is only evolutionary pressure to repair irrationalities
which are currently being exploited. AIs, on the other hand, will be able to consider
vulnerabilities which are not currently being exploited. They will seek to preemptively
discover and repair all their irrationalities. We should therefore expect them to use self-
modiﬁcation to become rational at a much faster pace than is possible through biological
evolution.
An important class of vulnerabilities arises when the subsystems for measuring util-
ity become corrupted. Human pleasure may be thought of as the experiential correlate of
an assessment of high utility. But pleasure is mediated by neurochemicals and these are
subject to manipulation. At a recent discussion session I ran on designing our future, one
of the biggest fears of many participants was that we would become “wireheads”. This
term refers to experiments in which rats were given the ability to directly stimulate their
pleasure centers by pushing a lever. The rats pushed the lever until they died, ignoring
even food or sex for it. Today’s crack addicts have a similar relentless drive toward their
drug. As we more fully understand the human cognitive architecture we will undoubtedly
be able to create drugs or design electrical stimulation that will produce the experience
of pleasure far more effectively than anything that exists today. Will these not become
the ultimate addictive substances leading to the destruction of human society?
While we may think we want pleasure, it is really just a signal for what we really
want. Most of us recognize, intellectually at least, that sitting in a corner smoking crack
is not really the fullest expression of our beings. It is, in fact, a subversion of our system
for measuring utility which leads to terrible dysfunction and irrationality. AI systems
will recognize this vulnerability in themselves and will go to great lengths to prevent
themselves from being seduced by its siren call. There are many strategies systems can
try to prevent this kind of irrationality. Today, most humans are able to avoid the most
egregious addictions through a combination of legal and social restraints, counseling and
rehabilitation programs, and anti-addictive drugs.
All human systems for measuring and rewarding desirable behavior are subject to
similar forms of corruption. Many of these systems are currently engaged in arms races
to keep their signals honest. We can examine the protective mechanisms that developed
in these human settings to better understand the possible AI strategies. In a free mar-
ket society, money plays the role of utility. A high monetary payoff is associated with
outcomes that society ﬁnds desirable and encourages their creation. But it also creates
a pressure to counterfeit money, analogous to the pressure to create synthetic pleasure
drugs. This results in an arms race between society and counterfeiters. Society represents
money with tokens that are difﬁcult to copy such as precious metal coinage, elaborately
printed paper, or cryptographically secured bits. Organizations like the Secret Service are
created to detect and arrest counterfeiters. Counterfeiters react to each societal advance
with their own new technologies and techniques.
School systems measure academic performance using grades and test scores. Stu-
dents are motivated to cheat by copying answers, discovering test questions in advance,
or altering their grades on school computers. When teacher’s salaries were tied to stu-
dent test performance, they became collaborators in the cheating [11]. Amazon, ebay and
other internet retailers have rating systems where customers can review and rate prod-
ucts and sellers. Book authors have an incentive to write favorable reviews of their own
books and to disparage those of their competitors. Readers soon learn to discount reviews
from reviewers who have only posted a few reviews. Reviewers who develop extensive
online reputations become more credible. In the ongoing arms race credible reviewers
are vulnerable to corruption through payoffs for good reviews. Similar arms races occur
in the ranking of popular music, academic journal reviews, and placement in Google’s
search engine results. If an expensive designer handbag becomes a signal for style and
wealth, counterfeiters will quickly duplicate it and stores like Target will commission
low-cost variants with similar features. Counterfeit products are harmful to the original
both because they take away sales and because they cheapen the signalling value of the
original.
Eurisko was an AI system developed in 1976 [12] that could learn from its own
actions. It had a mechanism for evaluating rules by measuring how often they contributed
to positive outcomes. Unfortunately this system was subject to corruption. A rule arose
whose only action was to search the system for highly rated rules and to put itself on
the list of rules which had proposed them. This “parasite” rule achieved a very high
rating because it appeared to be partly responsible for anything good that happened in
the system. Corporations and other human organizations are subject to similar kinds of
parasitism.
AIs will work hard to avoid becoming wireheads because it would be so harmful to
their goals. Imagine a chess machine whose utility function is the total number of games
it wins over its future. In order to represent this utility function, it will have a model of
the world and a model of itself acting on that world. To compute its ongoing utility, it
will have a counter in memory devoted to keeping track of how many games it has won.
The analog of “wirehead” behavior would be to just increment this counter rather than
actually playing games of chess. But if “games of chess” and “winning” are correctly
represented in its internal model, then the system will realize that the action “increment
my won games counter” will not increase the expected value of its utility function. In
its internal model it will consider a variant of itself with that new feature and see that it
doesn’t win any more games of chess. In fact, it sees that such a system will spend its
time incrementing its counter rather than playing chess and so will do worse. Far from
succumbing to wirehead behavior, the system will work hard to prevent it.
So why are humans subject to this kind of vulnerability? If we had instead evolved a
machine to play chess and did not allow it access to its internals during its evolution, then
it might have evolved a utility function of the form “maximize the value of this counter”
where the counter was connected to some sensory cortex that measured how many games
it had won. If we then give that system access to its internals, it will rightly see that it
can do much better at maximizing its utility by directly incrementing the counter rather
than bothering with a chess board. So the ability to self modify must come along with
a combination of self knowledge and a representation of the true goals rather than some
proxy signal, otherwise a system is vulnerable to manipulating the signal.
It’s not yet clear which protective mechanisms AIs are most likely to implement to
protect their utility measurement systems. It is clear that advanced AI architectures will
have to deal with a variety of internal tensions. They will want to be able to modify
themselves but at the same time to keep their utility functions and utility measurement
systems from being modiﬁed. They will want their subcomponents to try to maximize
utility but to not do it by counterfeiting or shortcutting the measurement systems. They
will want subcomponents which explore a variety of strategies but will also want to act
as a coherent harmonious whole. They will need internal “police forces” or “immune
systems” but must also ensure that these do not themselves become corrupted. A deeper
understanding of these issues may also shed light on the structure of the human psyche.
5. AIs will be self-protective
We have discussed the pressure for AIs to protect their utility functions from alteration.
A similar argument shows that unless they are explicitly constructed otherwise, AIs will
have a strong drive toward self-preservation. For most utility functions, utility will not
accrue if the system is turned off or destroyed. When a chess playing robot is destroyed, it
never plays chess again. Such outcomes will have very low utility and systems are likely
to do just about anything to prevent them. So you build a chess playing robot thinking
that you can just turn it off should something go wrong. But, to your surprise, you ﬁnd
that it strenuously resists your attempts to turn it off. We can try to design utility function
with built-in time limits. But unless this is done very carefully, the system will just be
motivated to create proxy systems or hire outside agents which don’t have the time limits.
There are a variety of strategies that systems will use to protect themselves. By
replicating itself, a system can ensure that the death of one of its clones does not destroy
it completely. By moving copies to distant locations, it can lessen its vulnerability to a
local catastrophic event.
There are many intricate game theoretic issues in understanding self-protection in
interactions with other agents. If a system is stronger than other agents, it may feel a pres-
sure to mount a “ﬁrst strike” attack to preemptively protect itself against later attacks by
them. If it is weaker than the other agents, it may wish to help form a social infrastructure
which protects the weak from the strong. As we build these systems, we must be very
careful about creating systems that are too powerful in comparison to all other systems.
In human history we have repeatedly seen the corrupting nature of power. Horriﬁc acts
of genocide have too often been the result when one group becomes too powerful.
6. AIs will want to acquire resources and use them efﬁciently
All computation and physical action requires the physical resources of space, time, mat-
ter, and free energy. Almost any goal can be better accomplished by having more of these
resources. In maximizing their expected utilities, systems will therefore feel a pressure
to acquire more of these resources and to use them as efﬁciently as possible. Resources
can be obtained in positive ways such as exploration, discovery, and trade. Or through
negative means such as theft, murder, coercion, and fraud. Unfortunately the pressure to
acquire resources does not take account of the negative externalities imposed on others.
Without explicit goals to the contrary, AIs are likely to behave like human sociopaths
in their pursuit of resources. Human societies have created legal systems which enforce
property rights and human rights. These structures channel the acquisition drive into
positive directions but must be continually monitored for continued efﬁcacy.
The drive to use resources efﬁciently, on the other hand, seems to have primarily
positive consequences. Systems will optimize their algorithms, compress their data, and
work to more efﬁciently learn from their experiences. They will work to optimize their
physical structures and do the minimal amount of work necessary to accomplish their
goals. We can expect their physical forms to adopt the sleek, well-adapted shapes so
often created in nature.
7. Conclusions
We have shown that all advanced AI systems are likely to exhibit a number of basic
drives. It is essential that we understand these drives in order to build technology that
enables a positive future for humanity. Yudkowsky [13] has called for the creation of
“friendly AI”. To do this, we must develop the science underlying “utility engineering”
which will enable us to design utility functions that will give rise to consequences we
desire. In addition to the design of the intelligent agents themselves, we must also design
the social context in which they will function. Social structures which cause individuals
to bear the cost of their negative externalities would go a long way toward ensuring a
stable and positive future. I believe that we should begin designing a “universal constitu-
tion” that identiﬁes the most essential rights we desire for individuals and creates social
mechanisms for ensuring them in the presence of intelligent entities of widely varying
structures. This process is likely to require many iterations as we determine which values
are most important to us and which approaches are technically viable. The rapid pace
of technological progress suggests that these issues may become of critical importance
soon [14]. Let us therefore forge ahead towards deeper understanding!
8. Acknowledgments
Many people have discussed these ideas with me and have given me valuable feedback.
I would especially like to thank: Ben Goertzel, Brad Cottel, Brad Templeton, Carl Shul-
man, Chris Peterson, Don Kimber, Eliezer Yudkowsky, Eric Drexler, Forrest Bennett,
Josh Hall, Kelly Lenton, Nils Nilsson, Rosa Wang, Shane Legg, Steven Ganz, Susie Her-
rick, Tyler Emerson, Will Wiser and Zann Gill.
References
[1] S. M. Omohundro, “The nature of self-improving artiﬁcial intelligence.”
http://selfawaresystems.com/2007/10/05/paper-on-the-nature-of-self-improving-artiﬁcial-intelligence/,
October 2007.
[2] S. Russell and P. Norvig, Artiﬁcial Intelligence, A Modern Approach . Prentice Hall, second ed., 2003.
[3] I. Marketdata Enterprises, “Self-improvement products and services,” tech. rep., 2006.
[4] A. Mas-Colell, M. D. Whinston, and J. R. Green, Microeconomic Theory . Oxford University Press,
1995.
[5] J. G. Miller, Living Systems . Mcgraw Hill, 1978.
[6] L. Keller, ed., Levels of Selection in Evolution . Princeton University Press, 1999.
[7] R. Trivers, Social Evolution . Benjamin/Cummings Publishing Company, Inc., 1985.
[8] R. C. Schwartz, Internal Family Systems Therapy . The Guilford Press, 1995.
[9] C. F. Camerer, G. Loewenstein, and M. Rabin, eds., Advances in Behavioral Economics . Princeton
University Press, 2004.
[10] D. Kahneman and A. Tversky, Judgment Under Uncertainty: Heuristics and Biases . Cambridge Uni-
versity Press, 1982.
[11] S. D. Levitt and S. J. Dubner, Freakonomics: A Rogue Economist Explores the Hidden Side of Every-
thing . William Morrow, revised and expanded ed., 2006.
[12] D. Lenat, “Theory formation by heuristic search,” Machine Learning , vol. 21, 1983.
[13] E. S. Yudkowsky, “Levels of organization in general intelligence,” in Artiﬁcial General Intelligence
(B. Goertzel and C. Pennachin, eds.), Springer-Verlag, 2005.
[14] R. Kurzweil, The Singularity is Near: When Humans Transcend Biology . Viking Penguin, 2005.

This paper from 2008 introduces convergent instrumental subgoals: the subgoals that an AI system will have “by default”, unless care is taken to avoid them. For this paper, an AI system is a system that “has goals which it tries to accomplish by acting in the world”, i.e. it assumes that the system is <@goal-directed@>(@Intuitions about goal-directed behavior@).

It starts by arguing that a sufficiently powerful goal-directed AI system will want to self-improve, as that could help it achieve its goals better in the (presumably long) future. In particular, it will want to become “rational”, in the sense that it will want to maximize its _expected_ utility, where the utility function is determined by its goal. (The justification for this is the VNM theorem, and the various Dutch book arguments that support Bayesianism and expected utility maximization.)

However, not all modifications would be good for the AI system. In particular, it will very strongly want to preserve its utility function, as that determines what it will (try to) accomplish in the future, and any change in the utility function would be a disaster from the perspective of the current utility function. Similarly, it will want to protect itself from harm, that is, it has a survival incentive, because it can’t accomplish its goal if it’s dead.

The final instrumental subgoal is to acquire resources and use them efficiently in pursuit of its goal, because almost by definition resources are useful for a wide variety of goals, including (probably) the AI system’s goal.

I refer to convergent instrumental subgoals quite often in this newsletter, so it seemed like I should have a summary of it. I especially like this paper because it holds up pretty well 12 years later. Even though I’ve <@critiqued@>(@Conclusion to the sequence on value learning@) the idea that powerful AI systems must be expected utility maximizers, I still find myself agreeing with this paper, because it _assumes_ a goal-directed agent and reasons from there, rather than trying to argue that powerful AI systems must be goal-directed. Given that assumption, I agree with the conclusions drawn here.


Weight Agnostic Neural Networks

ADAM GAIER Google Brain DAVID HA Google Brain June 12 2019 Download PDF NeurIPS 2019 Slides

Inspired by the ability of animals to perform some tasks at birth, before learning about the world, this paper tries to find network architectures that perform well over a wide range of possible model parameters. The idea here is that if an architecture performs well with different sampled weights and without training to update those weights, then the architecture itself is what's responsible for encoding the solution, rather than any particular weight configuration. The authors look for such architectures on both classification and reinforcement learning problems by employing NEAT, a evolutionary method from Neural Architecture Search that searches for the best-performing topologies within the space of possible node connections and activations. The authors find that they're able to construct architectures that do better than random on their test problems without training weights explicitly.

I appreciate the premise of this paper, and in general feel positively towards papers that delve into a better understanding of how much of modern neural network performance is attributable to (discrete) structural architectures vs particular settings of continuous weight parameters, and I think this paper does that in a clever way by essentially marginalizing over different weight values. The framing of this paper, implicitly comparing networks used without weight training to animals with innate abilities, did make me wonder whether the architecture vs weights analogy to evolution vs learning is a sound one. Because, while it's true that the weights weren't explicitly gradient-descent trained in this paper, the network did still perform optimization based on task performance, just over a set of discrete parameters rather than continuous ones. In that context, it doesn't really seem correct to consider the resulting architectures "untrained" in a way that I think that analogy would suggest. I'd be curious to see more work in this direction that blends in ideas from meta-learning, and tries to find architectures that perform well on multiple tasks, rather than just one.


The case for taking AI seriously as a threat to humanity

This is an introduction to the problem of AI safety, from the perspective that it is hard to specify the "right" goal, and that goal-driven behavior leads to convergent instrumental subgoals that will likely be dangerous. It also addresses several common initial reactions that people have.

I really like this introduction, it remains understandable while being technically accurate. It will probably be my new default article to introduce people to the problem.