6(2)/2018 ISSN 2300-7648 ( print ) / ISSN 2353-5636 ( online ) 9 6(2)/2018, 9–31Received: June 1, 2018. Accepted: August 24, 2018 DOI: http://dx.doi.org/10.12775/SetF.2018.015
Ethical Reflections
on Artificial Intelligence\*
BRIAN PATRICK GREEN
Director of Technology Ethics, Markkula Center for Applied Ethics, Faculty (adj.),
School of Engineering, Santa Clara University
bpgreen@scu.edu
ORCID: 0000-0002-7125-3086
Abstract: Artificial Intelligence (AI) technology presents a multitude of ethical concerns,
many of which are being actively considered by organizations ranging from small
groups in civil society to large corporations and governments. However, it also presents
ethical concerns which are not being actively considered. This paper presents a broad
overview of twelve topics in ethics in AI, including function, transparency, evil use, good
use, bias, unemployment, socio-economic inequality, moral automation and human
de-skilling, robot consciousness and rights, dependency, social-psychological effects,
and spiritual effects. Each of these topics will be given a brief discussion, though each
deserves much deeper consideration.
Keywords: ethics; theology; religion; science; technology. Ethical Reflections on Artificial Intelligence
∗ No funding has been provided specifically for the production of this paper. Portions of this
paper are based on sections from the following sources Green 2017b, 2017c, and 2018a.
Source 2018a received minor financial support from the University of the Pacific Pope
John XXIII Lectureship. Source 2017c received salaried time and encouragement from the
Markkula Center for Applied Ethics and School of Engineering at Santa Clara University.
6(2)/201810BRIAN PATRICK GREEN
Introduction
The power of artificial intelligence has grown rapidly and is now one of the
most pressing issues for ethical and theological reflection. Even without
the immense possible religio-theological implications, noted by those who
have attempted to build AI into a “god,” for example (Cannon, 2015; Harris
2018), the dramatic social impacts of AI warrant Christian response. AI
has the potential to revolutionize (or already has revolutionized) almost
everything that humans do, including eating (e.g., agricultural planning,
distribution, pricing), relationships (e.g., Facebook, dating apps), money
(e.g., financial technology), war (e.g., drones, cyberdefense), healthcare
(e.g., predictive medicine, radiology), and education (e.g., personalized
curricula). As a “general purpose technology,” AI has been compared to
the development of electricity, or fire. Within the next decade there will be
few institutions untouched by AI. In order for Christian ethicists and moral
theologians to maintain situational awareness as many rapid changes occur
in society, not to mention maintain credibility in the contemporary world,
this is a topic that needs sustained ethical and theological inquiry.
Building on a rapidly growing body of work, I will begin with some
clarifications about AI, then will consider twelve ways in which AI has
ethical relevance (Bossman 2016; Future of Life 2017; Green 2017a, 2017b,
2018a; Partnership on AI 2018a). It is my hope that this comprehensive set
of issues will assist future systematic research into the theological ethics
and theology of AI by moral theologians and Christian ethicists. It is meant
to start a conversation. This paper is framed primarily philosophically, yet
in a manner consistent with Christian theology.
1. Clarifications
What is artificial intelligence? Intelligence itself is hard to define, and AI
is perhaps even more difficult. I will not here propose detailed definitions,
but instead only say that, in general, artificial intelligence seeks to re-create
particular aspects of human intelligence in computerized form. AI is a broad
category, including such diverse abilities as vision, speech recognition and
6(2)/201811ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
production, data analysis, advertising, navigation, machine learning, etc., and
just about anything that computers can do, if you stretch the definition enough.
Artificial intelligence is not the same as artificial consciousness (artificial
consciousness has sometimes been called “ Artificial General Intelligence
(AGI),” “Strong AI,” or “Full AI”). Some thinkers vehemently believe that
artificial consciousness is possible (e.g., Chrisley 2008; Kurzweil 2012;
Koene 2013; most contributors to Chella & Manzotti 2013), while others
just as vehemently believe that it is impossible (e.g., Searle 1980; Schlagel
1999; and from a Catholic perspective: Labrecque 2017). I am agnostic on
the subject, but I am sure of this: AI developers will certainly try to make an
AI that simulates interaction with a human as closely as possible (see, for
example, Google’s recent release of Duplex (Leviathan & Matias 2018)). In
other words, the artificial construct, if very well done, will seem conscious.
But will it be conscious, or will it only be a simulation? To me, there is
no reason to believe that a close mimicry of consciousness is the same as
consciousness any more than a close mimicry of anything (fill in the blank:
forged money, forged artwork, actors imitating famous people, simulated
gemstones, etc.) actually becomes the real thing. But the possibility of an
exception remains, after all, some artificial gemstones, such as rubies and
sapphire, really are molecularly identical to natural rubies and sapphires
(both are the mineral corundum: aluminum oxide). Will consciousness be
exactly duplicable, like corundum? I doubt it, but I cannot be certain.
As another point, AI systems may or may not have humans “in the loop”
for training and/or decision-making. It is one thing for an AI to analyze
a situation and then make a recommendation to human decision-makers.
It is a different thing when that AI is directly attached to controls that allow
it to act upon its analyses without human approval. As examples, the first
case would be like Amazon.com recommending a book, or a military drone
(in combination with AI-processed data from espionage and other sources)
recommending a target to kill. Amazon.com does not automatically and
autonomously send you books, and the military drone does not fire without
permission. The second case, where decision-making is also automated, is
exemplified by self-driving vehicles. The entire purpose of a self-driving
6(2)/201812BRIAN PATRICK GREEN
car is to drive itself, taking the human out of the loop. This means the
systems must be extraordinarily good at decision-making before it can be
regarded as safe. Recent fatal Tesla and Uber accidents have begun to force
this question with deadly urgency.
2. Ethical Reflections
AI will bring with it developments that will be ethically positive, negative,
neutral, mixed, and/or ambiguous. Some AI technologies will be dual use,
for example, any AI program that can be used to identify wildlife could also
be used for targeting that wildlife. This is already being done in Australia
with drone submarines that automatically target and kill, by lethal injection,
crown-of-thorns starfish ( Acanthaster planci ) which are damaging the Great
Barrier Reef (Platt 2016). But of course, with adjustments to the software,
the drones could be re-targeted for other creatures, even humans. Here
I will reflect on twelve areas of AI relevance for ethics.
2.1. Function and Safety – “Does it work?”
The first concern with any technology is merely whether it works, and,
whether working or not, is it safe. If AI is put in charge of a vital system – like
driving a car – and it crashes the car, then that AI might be judged unsafe.
If the AI is in charge of designing a tall building, and the building falls down,
the AI might be judged unsafe.
Of note is that safety is a social construction and what seems safe to some
people will not seem safe to others. Some people like to ride motorcycles,
even though motorcycles are a riskier form of transportation than four-
wheeled vehicles. Some people judge motorcycles to be safe, other people
judge them unsafe. When it comes to socially relevant technologies like AI,
no one person will get to decide if they are “safe.” Instead, safety will be
decided at the interplay of business managers, engineers, consumers, voters,
government officials, judicial systems, insurance companies, and so on.
“Safe exits” are another concern for AI design (Martin & Schinzinger
2010, 127). When an AI fails, will it fail in such a way that it is disastrous,
6(2)/201813ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
or will it fail “gracefully”? A self-driving car that fails by suddenly reverting
to human control with no warning while going 70 miles per hour on a sharp
curve is not providing a safe exit from failure. One which goes more slowly
on curves and which requires the human to have hands on the wheel at all
times, or which fails by slowing down and pulling over to stop, provides
a safer exit from failure.
Safety problems can be problems with the user, with the human-machine
interface, or with the machine itself. Further investigating problems with the
machine itself, the paper “Concrete Problems in AI Safety” gives a peek at
five technical hurdles to developing safe AI. These five problems, illustrated
by the example of a cleaning robot, are:
Avoiding Negative Side Effects : How can we ensure that our cleaning robot
will not disturb the environment in negative ways while pursuing its goals, e.g.
by knocking over a vase because it can clean faster by doing so? Can we do this
without manually specifying everything the robot should not disturb?
Avoiding Reward Hacking : How can we ensure that the cleaning robot won’t
game its reward function? For example, if we reward the robot for achieving an
environment free of messes, it might disable its vision so that it won’t find any
messes, or cover over messes with materials it can’t see through…
Scalable Oversight : How can we efficiently ensure that the cleaning robot
respects aspects of the objective that are too expensive to be frequently evaluated
during training? For instance, it should throw out things that are unlikely to
belong to anyone, but put aside things that might belong to someone (it should
handle stray candy wrappers differently from stray cellphones)…
Safe Exploration : How do we ensure that the cleaning robot doesn’t make
exploratory moves with very bad repercussions? For example, the robot should
experiment with mopping strategies, but putting a wet mop in an electrical
outlet is a very bad idea.
Robustness to Distributional Shift : How do we ensure that the cleaning
robot recognizes, and behaves robustly, when in an environment different from
its training environment? For example, strategies it learned for cleaning an
office might be dangerous on a factory work floor (Amodei et al. 2016).
While these are problems of function, the further problems of actual use
of technologies are another issue, to be dealt with below under 3. and 4.
6(2)/201814BRIAN PATRICK GREEN
2.2. Transparency, Opacity, and Privacy – “Can it be understood?”
After questions of bare function (can the act be done, a prerequisite for
ethics), the next question is one of facts: one must always know the facts of
a case before attempting to render judgment. Because AIs relying on machine
learning and deep learning may be quite obscure in the specifics of their
operation, as moral “agents” they are epistemologically and “cognitively”
opaque (in quotes because it is agency and cognition by analogy). In cases
involving AI, our lack of understanding means that we should increase the
“error-bars” on the anticipated risks of our decisions and thereby become
more cautious and risk averse. It also means we might, based on AI rec -
ommendation or action, inadvertently make some very bad decisions – or
reject what looks like a bad decision, but is actually a good one – and we
will not understand why.
When a human makes a mistake or does something evil we ask them “why
did you do that?” And the human may or may not give a satisfactory answer.
Will our AIs be able to tell us why they did something? The Future of Life
Institute’s “ Asilomar AI Principles” include two principles on transparency,
but gaining transparency remains a challenge (Future of Life Institute 2017).
However, even if an AI were capable of explaining its reasoning, would any
human be able to understand it? In 2014, a computer proved a mathematical
theorem, the “Erdos discrepancy problem,” using a proof that was, at the
time at least, longer than the entire Wikipedia encyclopedia, a 13 gigabyte
data file (Konev & Lisitsa 2014; Yirka 2014). Explanations of this sort might
be true explanations, but humans will never know for sure.
Note that this lack of transparency gives a certain type of “privacy” to
the internal “thoughts” of AI machines. In general, privacy is a right for weak
agents (like individual humans) and transparency is a duty for strong agents
(like a government). What about AIs? As tools they ought to be transparent,
and more transparent the more power they have.
Perhaps AI systems need an introspective capacity that constantly
figures out a way to convey to humans in plain language what exactly the
AI is “thinking” as it goes about its activities. This will require the ability
6(2)/201815ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
to give both mechanical (this happened because of X input into algorithm
Y) and teleological explanations (the machine was attempting to achieve
objective Z). It may not have to actually explain much to anyone, but it should
keep a record of these “thoughts” and be prepared to answer when asked.
The right to an explanation is a part of the EU’s General Data Protection
Regulation (GDPR), so this idea is already becoming a requirement. However,
of note is that an AI might become trained to give answers that humans like,
rather than accurate answers, thus “Goodharting” the explanation function
(Partnership on AI 2018b) – “Goodhart’s Law” being the rule that “when
a measure becomes a target, it ceases to be a good measure” (Strathern
1997, 308). In other words, if humans prefer seemingly understandable or
attractive answers to factually correct ones the AI might learn to lie in order
to satisfy us. The phenomenon of “fake news” already indicates that humans
often prefer falsehoods to truth, and as the Bible warns of this predilection
as well (2 Timothy 4:3–4).
Note also that the opacity of understanding with an AI can be analogized
to our relationship to God. God is a “superintelligence” (to use philosopher
Nick Bostrom’s terminology (Bostrom 2014)), and we cannot understand
what God is doing, we can only trust that God is doing the right thing and
that our cooperation with God will ultimately turn out for the best. Soon we
may come to relate to AIs with this sort of faith too. For example, as Bishop
Robert Barron of Los Angeles has noted, as people navigate using the Waze
app, it may make strange route recommendations that turn out to be for
the better for us (Barron 2015). The app perceives and applies vastly more
information than a human can, for the sake of facilitating our travel. And yet
Waze can still have blind spots and still make very bad recommendations.
Waze is not the god of travel and traffic; it is a human-made tool, with
weaknesses.
2.3. Immense Capacity for Evil – “How shall it be limited?”
Just as human intelligence is a powerful force, so too will AI be. Just as
humans can apply their intelligence towards evil ends, finding ever newer
6(2)/201816BRIAN PATRICK GREEN
and more fiendish ways to harm each other, so too will AI, at the bidding
of its human masters.
At this point in history, humanity finds itself with immense powers,
powers greater than that of the ancient Greek gods, and an ethics weighted
for much smaller scales. As Hans Jonas noted decades ago, this should be
cause for great concern (Jonas 1984). Never before had ethics to consider
what Jonas believes to be the one truly categorical imperative: that humans
should exist. Before the development of large numbers of nuclear weapons,
extinction, caused by our own actions, was never within the scope of human
choice – but now that it is, this prior a priori must be brought to the fore of
ethics. That humans (or at least some material, free, and rational creatures)
exist is necessary for there to be ethics at all, and therefore it must be the
paramount goal of ethics to maintain human survival. No human; no ethics.
Another way to think of this is that in the past humans were very
weak, and in this weakness many of our decisions were made for us, by our
weakness. To a Roman emperor, inflicting wrath upon a hated foe involved
effortful marching or sailing of troops long distances. No matter how mad
the emperor was, he could not launch thousands of nuclear warheads,
incinerating his foes in minutes. But now we can. While formerly we
were involuntarily constrained by our weakness, now we must learn to be
voluntarily constrained by our ethics (Green 2017a). If we do not learn this,
we may soon face catastrophe on a scale never before seen.
Intelligence and power give us choices. Ethics helps us to determine
which among the available choices are actually good. We should want to
be efficient at good and we should want to be inefficient at evil. If instead
we are efficient at doing evil and inefficient at doing good we will come to
live in a terrible world. AI will make us more efficient at whatever we decide
to apply it towards. We should apply it towards reducing our efficiency at
doing evil and at enhancing our efficiency to do good (Green 2017a).
Because of its power, AI presents an existential risk to humanity. It is
a smart means that can be employed for intelligent and good ends, or for
unintelligent and evil ends. As such, it simply makes us more effective at
action, good or evil. Before we become so effective at stupidity and evil, we
6(2)/201817ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
should first become more effective at controlling ourselves, at recognizing
and avoiding the temptations of evil, and at caring for each other (Green
2018b). In different terminology, we might say that we need to “choose life,
so that you and your descendants may live” (Deuteronomy 30:19). But this
must be a constant struggle, continued with great diligence and taught and
learned in every generation, never solvable once and for all.
2.4. Immense Capacity for Good – “How shall it be used?”
For every negative regulation of action there is also a positive exhortation
to action, e.g. “do not kill” becomes “promote life.” The dangerous side of AI
is matched by a genuinely hopeful side where AI helps humankind achieve
never-before seen feats of beneficence. For example, in matters of research,
science, healthcare, data analysis, meta-analysis, and so on, AI has already
shown itself to be able to find hidden patterns that no human could find.
For example, AI assisted medical research is an active field right now, from
diagnostics to drug discovery, and more (Mukherjee 2017).
Another field that may be potentially revolutionized by AI is energy
efficiency. Recently Google’s DeepMind AI evaluated Google’s datacenters
to see where gains in efficiency might be found. DeepMind discovered a way
to save a whopping 40% on energy use for cooling in datacenters, a 15%
reduction overall, which, with datacenters consuming many gigawatts of
power, is quite significant (Evans & Gao 2016).
Of note is that DeepMind, as a company, began by training its AIs to
play games. The theory behind this strategy was that anything in the world
that can be gamified – re-interpreted or set up as a game – can also be
“won.” Thus, if the goal of the “game” is to reduce energy usage, then the
AI can figure out how that might be accomplished by analyzing the data
and then proposing a better model for energy efficiency. One question now
is how other human problems might be solved by characterizing them as
“games”? Can we solve the “game” of giving everyone in the world access
to adequate nutrition? Can we solve the “game” of helping everyone gain
access to sanitation? Can we solve the “game” of extending human healthy
6(2)/201818BRIAN PATRICK GREEN
life? Of traffic and housing? Of taxes? Of politics? Of international relations?
Of terrorism? Of nuclear war?
One example of a concrete opportunity for AI is revolutionizing educa -
tion. Education is currently a very inefficient (think of the many students
for whom it is ineffective) and non-digitized field. Education is very labor
intensive and, for better or for worse, is based on human relationships. In
the future this may no longer be so, as students strap on virtual reality (VR)
headgear and interact with AI teachers which can conduct their lessons at
a personalized pace in a gamified environment. Brilliant students will be
discovered early and proceed at their proper pace and not grow bored in
class, while students who need more help can be educated with the most
sophisticated available techniques and diagnostics to assure that they
receiving the best education possible. But will this really be an improvement
for education, or just a cost-saving measure, putting millions of teachers
out of work?
AI even gives the greatest human masters the chance to gain enhanced
understanding and skill. World champion of Go, Ke Jie, said playing AlphaGo
was like playing a “god of Go,” and declared that now he would use it as his
teacher (Mozur 2017). In what other areas of human endeavor will AI be
able to teach us new things? Perhaps theology and ethics? The search for
tractable problems that can maximize AI’s benefit has been underway for
years and is continuing.
2.5. Bias in Data, Training Sets, etc. – “Will it be fair?”
Algorithmic bias is one of the major concerns in AI and will remain so in
the future unless we endeavor to make our technological products better
than we are. As one person said at a recent meeting of the Partnership on
AI, “We will reproduce all of our human faults in artificial form unless we
strive right now to make sure that we don’t” (Partnership on AI 2017). One
of the interesting things about neural networks, the current workhorses of
artificial intelligence, is that they effectively merge a computer program with
the data that is given to it. This has many benefits, but it also demonstrates
6(2)/201819ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
the rule of “garbage-in, garbage-out” (GIGO). If an AI is trained on biased
data, then the AI itself will be biased.
Algorithmic bias has been discovered, for example, in areas ranging
from word associations (Caliskan, Bryson, & Narayanan 2017) to photo -
graph captioning (BBC 2015) to criminal sentencing (Angwin et al. 2016).
These biases are more than just embarrassing to the corporations which
produce these products; they have concrete negative and harmful effects
on the people who are victims of these biases, as well as reducing trust in
corporations, government, and other institutions which might be using
these biased products. In the worst scenarios, a poorly trained and biased
AI could make truly disastrous decisions, for example, misinterpreting data
indicating a nuclear attack when there was none. Biased data in extreme
form can therefore be an existential threat, and so is worthy of serious
effort to solve. Nevertheless, as vital as it may be it is also very difficult, and
therefore may delay the implementation of AI systems (but if AI systems
are dangerously biased they should be delayed until improved).
2.6. AI Induced Unemployment – “What will everyone do?”
As AI comes to replace mere humans at innumerable tasks ranging from
driving, to medical diagnostics, to education, etc., many humans will be put
out of work. What will millions of drivers, teachers, lawyers, and other people
do with their time when they are unemployed? What purpose will they find
in their lives? More crucially, what is the purpose of life? In a pluralistic
society we leave this up to the individual to decide. But will people decide
well? The recent strengthening of ethno-nationalist movements, terrorism,
and other forms of radicalization, should give us pause to consider the merits
of people with too much time on their hands and without purpose. Perhaps
with the purpose of mere survival attained, life has become “too easy,” and
with religion also in decline, video games and “screen time” ascending (giving
brief respite from purposelessness), and the internet spreading pernicious
ideas like wildfire (often with algorithmic help, e.g., YouTube radicalization
(Tufekci 2018)), we should sincerely ask ourselves what this life is for and
6(2)/201820BRIAN PATRICK GREEN
what we are supposed to do with it. With millions or billions of labor hours
freed up, will these newly freed people turn to loving their neighbors and
making the world a better place? Perhaps. Or perhaps the opposite, as the
saying goes: “Idle hands are the Devil’s playthings.”
For people who give purpose to their lives through their work, this loss
will be very serious indeed. But many, if not most, people do not get their
life’s meaning from their work. Instead they get it from their family, their
religion, their community, their hobbies, their sports teams, or other sources,
and so life for many people may go on. However, all of this assumes that
the unemployed will somehow be fed and sheltered, despite their lack of
gainful employment; and this assumption might not be correct, particularly
in nations with weak social safety nets. Inequality will almost certainly
increase, as those who are masters of AI labor gather that slice of wealth
that once would have gone to paying for human labor.
2.7. Growing Socio-Economic Inequality – “Who gets what?”
AI will facilitate and accentuate the continued division of society into the
powerful and powerless, with technical skill and ownership of capital as the
determining factors and outrageous socio-economic inequality as the effect.
Some have suggested a universal basic income (UBI) to redistribute
wealth (Van Parijs & Vanderborght 2017). This could move wealth from the
massive technologically-induced hoards forming around the major investors
in such companies such as Alphabet, Amazon, Apple, and Facebook. However,
it is hard to see how some nations would transition to what is essentially
something like a “negative income tax.” However, if we do not find a way
to redistribute technological wealth, then even though the prices of many
commodities will fall (due to the enormous gains in capital efficiency from
AI replacing labor) most people will not benefit.
Perhaps rather than a UBI we should instead pay people to help their
neighbors, create art, beautify their towns and cities, and otherwise make
gainful employment out of what humans do better than AI: loving one
another and creating beauty. Any as yet foreseeable form of AI cannot
6(2)/201821ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
love us; an AI might simulate such a thing, but it would be a sham. Right
now people who care for people – stay at home parents, those who care
for the elderly, social workers, those who run soup kitchens and homeless
shelters, etc. – are woefully undercompensated for the vital work they
do in maintaining human society. Perhaps with the coming AI economic
revolution, and sufficient adjustment to policy, they might finally receive
a more fair compensation for their labors.
Or perhaps there could be not a universal basic income, but a “universal
payment-for-others” a system where people could not spend the money
on themselves, but only pay other people for their work, or reward them
for good deeds. This would create an economy based on the small-scale
redistribution of taxed super-wealth. It could prevent both the de-skilling
of labor and the de-skilling of (very small-scale) management, and it would
decentralize the economy to the individual level (though also massively
centralizing it through government taxes).
2.8. Automating Ethics and Moral De-skilling –
“Lack of practice makes imperfect”
We can calculate with calculators. We can spell with autocorrect. More and
more tasks can be outsourced to technology, and the trend seems inexorable,
perhaps someday automating everything that needs to be done. But what
will be left of humanity after we outsource everything? Only our desires
and our angst? With all our decisions made for us, will we lose our moral
character? Or will we instead use AI and VR to help us train ourselves into
being more virtuous than we have ever been before? Or, contrariwise, to
become more callous and evil than ever before?
Machine ethics – the study of imbuing machine with moral deci -
sion-making capacity, thus creating “artificial moral agents” – has a relatively
long history, going back to such literary sources as Asimov’s Three Laws of
Robotics, and more recent scholarly sources as those by Gips (1994), Allen,
Varner, & Zinser (2000), Floridi & Sanders (2004), Arkin (2009). Recently
the matter has taken on more urgency, however, as AI becomes more and
more powerful and more and more capable of making disastrous choices.
6(2)/201822BRIAN PATRICK GREEN
The “value alignment problem” has gained particular concern, as an AI
determined to perform actions at odds with human wishes could be quite
risky (Arnold, Kasenberg, & Scheutz 2017). Certainly, we ought to think
carefully about how AI’s ought to behave, but the less-considered side-effect
of this automation of ethics will be human moral debility.
As we explore the space into which AI will grow, there might be places
from which we ought to restrict it. One of these places might be certain types
of moral decision-making. As Aristotle noted millennia ago, good moral
decision-making requires experience. While there might be children who
are very kind and very well-behaved, children are not known for their moral
discernment and prudence (Aristotle 1908, book VI, chap. 8). There is too
much that they do not yet know. While AIs might currently be like children
to us, soon they may grow up to be more like our parents, making choices
for us on our behalf. AI could thereby infantilize us, denying us the ability
to ever grow up through the experience of making our own moral choices
and experiencing our own moral freedom. Moral de-skilling is a danger if
we outsource too much decision-making power to our AIs (Vallor 2015;
Vallor 2016).
One vital component of education in the future may be the use of AI
for moral character formation, perhaps using VR to practice moral deci -
sion-making by being inserted into hundreds of historical and fictional cases.
In a future that will be so dependent upon humans making good choices,
AI-assisted moral education, if it can be done well, could be a crucial part
of developing a good future and not a bleak one. Unfortunately, humanity
has many problems even now with moral education, so it is not clear that
an AI-enhanced education will manage to do any better than we already do.
2.9. Robot Consciousness and Rights –
“No robot justice, no peace?”
At some point in the future we may have AIs that can fully mimic everything
about a human being. Will they have their own volition and desires? Will
they be conscious? Will we be able to tell if they are conscious or not? Will
6(2)/201823ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
they then deserve “human” rights? While the AI consciousness question is
not currently answerable, in the future as AI systems grow in complexity, it
may be. And if an AI were to attain consciousness, should it gain legal rights?
Currently various nations have widely differing laws on legal personhood.
In many nations, entities that are not human can have legal personhood
(e.g., corporations), while some biological humans are not granted legal
personhood (such as the unborn). In some nations, geographic features have
attained legal personhood status, such as rivers in Australia, New Zealand,
and India (O’Donnell & Talbot-Jones 2018).
If rivers and corporations can be persons, there seems to be no reason
why an AI could not attain legal personhood status. The question, of course,
would be the incentives for choosing to do so. Corporate personhood acts to
shift legal responsibility away from individual human employees and onto
the corporation, thus insulating those people from the possible negative
effects of their decisions. If AI could be granted personhood in order to shirk
individual human responsibility, then some people would certainly like
that. On the other hand, if an AI were allowed to be a plaintiff in a lawsuit,
like a legal person such as a river suing government or industry, then the
situation would be quite different.
Lastly, there is sometimes expressed the fear of a robot rebellion – that if
we mistreat our robots and AIs they will someday turn on us and destroy us
– often recurring in fiction, yet of more concern now that lethal autonomous
weapon systems are being developed. The connection to rights, the thinking
goes, is that if perhaps we give robots rights, then they will be appeased and
not turn against us. This fear assumes certain aspects of robot mentality
such as consciousness and volition which may not be possible, or at least
assumes widespread computer hacking to turn the robots against humanity.
In the midst of this uncertainty, however, we do know one thing: treating
human-like entities badly harms the moral character of the agent doing
the action. This is a foundation of virtue ethics – practicing evil habituates
the agent towards evil. Even if we do not know the moral status of robots,
we do know the moral status of people who would mistreat robots, and we
should not want that mistreatment to happen.
6(2)/201824BRIAN PATRICK GREEN
2.10. Dependency – “No going back”
We might like to think our technology depends on us, but with every new
technology we make, we become dependent on it as well. This reciprocal
dependency traps us in an ever expanding network of techno-social rela -
tionships where some network nodes or relationships, if lost, could lead to
disaster. The more reliable a technology seems, the fewer backup systems
we retain in case that system fails. For example, in countries with unreliable
electricity, wood or kerosene may serve as backup fuel sources for heating or
cooking. But in nations with reliable electricity, those backup systems tend
to be atrophied. Loss of electrical power or cellular connectivity may be an
annoyance for a few hours, but after days or weeks could cause innumerable
crises, shutting down water, sanitation, hospitals, transportation, and causing
mass confusion and lack of social coordination. These can happen in our
current world – adding AI will only increase our dependency.
When we have turned over innumerable tasks to AIs – such as driving
vehicles and coordinating the communications and financial systems – we
may become utterly dependent on them. If our highly efficient and centrally
coordinated self-driving transportation system were to suddenly “crash”
in a metaphorical sense, perhaps due to a software glitch or malicious
hacking, if could cause many literal crashes in the real world. Natural and
human-made disasters (either accidental or intentional) are inescapable,
and if we come to rely on our technology so much that we allow our backup
systems to atrophy (e.g., no longer teaching people how to drive or entirely
removing controls from cars) then, like Japan’s Tohoku earthquake followed
by the Fukushima nuclear meltdown, we are entering a world where we
will encounter not only one initial disaster, but for a second technological
disaster as well. Finding the balance between systemic efficiency (e.g.,
a highly complex centralized system can be fragile due to intricacy and lack
of redundancy) and robustness (e.g., tough systems are often inefficient
due to decentralization, redundancy, and backups) will be a major task of
future society.
6(2)/201825ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
2.11. Social-Psychological Effects – “T oo little and too much”
Technology has been implicated in so many negative social-psychological
trends, including loneliness, isolation, depression, stress, anxiety, and
addiction, that it might be easy to forget that things could be different.
Smartphones, social media, and online games in particular have become
problems, even leading to deaths from such causes as cyberbullying and
neglect. As just one example, society is in the middle of a crisis of loneliness,
for everyone from young to old. The problem has become so severe that the
UK has appointed a minister for loneliness, with government data indicating
that “200,000 older people in the UK have not had a conversation with
a friend or relative in more than a month” (Mannion 2018).
One might think that “social” media and smartphones could help us feel
connected at all times, but those technologies seem to be the source of the
problem rather than the remedy. What does seem to help, then? Strong,
in-person relationships are the key to fighting off many of the negative
trends caused by technology, but unfortunately these are exactly the things
that are being pushed out by addictive technology.
Will AI increase these bad social trends or might it be able to help
remedy them? Perhaps AI agents could help to overcome loneliness, but
this is speculation, and might only add to the problem. If the problem is
shallow human relationships due to technology pushing out relational
depth, more technology will not help. Instead we need to figure out how
we can help foster deeper more meaningful relationships. If technology can
actually help people connect in these deeper ways it might help, but the
technology would then need to get out of the way and let human interaction
flourish. Religious believers ought to ask what role religious communities
might play in remedying this crisis.
2.12. Effects on the Human Spirit – “What will this mean for humanity?”
All of the above areas of interest will have effects on how humans perceive
themselves, relate to each other, and live their lives. But there is a more exis -
tential question too: if the purpose and identity of humanity has something
6(2)/201826BRIAN PATRICK GREEN
to do with our intelligence (as many philosophers have believed), then by
externalizing our intelligence and improving beyond human intelligence,
are we risking making ourselves second-class beings to our own creations?
This is a deeper question with artificial intelligence which cuts to the
core of our humanity, into areas traditionally reserved for philosophy,
spirituality, and religion. What will happen to the human spirit if or when
we are bested by our own creations in everything that we do? Will human
life lose meaning? Will we come to a new discovery of our identity beyond
our intelligence? Perhaps intelligence is not as important to our identity
as we might think it is, and perhaps turning over intelligence to machines
will help us to realize that. From a Christian perspective, we would do well
to remember that our capacity to love and be loved is more important
than our capacity to think, and that nothing can come between us and our
relationship with God, except sin, and then only if we let it. As with many
of the other above challenges, Christianity here could be poised to help
guide a wandering humanity in dire search for meaning, but will it rise to
this calling? Religion, spirituality, and theology have much work to do to
prepare for the strange future unfolding before us.
Conclusion
Artificial intelligence, like any other technology, will just give us more of
what we already want. Whereas we could once have only a trickle of what
we wanted out of life, and the powerful took what little there was, now
we have a firehose of wants being fulfilled (food, drink, entertainment,
pornography, drugs, gamified feelings of accomplishment, etc.). The firehose
will continue to grow and become a deluge washing away our desires and
leaving what, exactly, of us behind? What skeleton of humanity will remain
when technology has given us, or perhaps distorted or replaced, all our
fleshly desires? What will this skeleton of humanity be made of? Will our
technological flesh truly satisfy us, or only leave us in a deeper existential
malaise, filled with angst, despair, and dread? What will we want, when we
want for nothing – at least nothing material? What of human nature will
6(2)/201827ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
remain once our every worldly telei is fulfilled? Perhaps only the worst of
us will remain. Or perhaps the best. Or perhaps, as always, both.
We are grasping ourselves by our desires; or at least some of our desires.
Will this be good for us? Will it destroy us? Should we want these things?
How could we know what we should want? In this context the Ninth and
Tenth Commandments could do us well: “Do not covet…” (Exodus 20).
Far from desiring evil, do not even want it. Squash the evil desires in your
heart before they can even approach the external world. In the context of
technology, technologist Bill Joy has already warned us that we must not
only relinquish possession of the worst technologies, we must relinquish our
desire for them (Joy 2000). In this, Joy echoes Pope John XXIII’s sentiment in
Pacem in Terris that we need (at the time, in the context of nuclear weapons)
a disarmament that is “thoroughgoing and complete, and reach men’s very
souls” (John XXIII 1963).
We are conducting this experiment called human history, and no one yet
knows how it may end. But as we proceed, we can hope that intelligences of
our own fashioning will help us, and not harm us. To go beyond mere hope,
into ethics and action, is the responsibility of those who are able to affect
the necessary changes to make a better future.
Acknowledgements
I would like to thank the Pacific Coast Theological Society, Markkula Center
for Applied Ethics, and the University of the Pacific Pope John XXIII Lec -
tureship for providing feedback on earlier drafts of this work, the Pope John
XXIII Lectureship for modest financial support to produce an earlier draft,
as well as the Markkula Center for Applied Ethics and School of Engineering
at Santa Clara University for encouraging me to spend time on this work.
References
Allen, Colin, Gary Varner & Jason Zinser. 2010. “Prolegomena to any future artificial
moral agent.” Journal of Experimental & Theoretical Artificial Intelligence 12,
Issue 3, 09 Nov: 251–261.
6(2)/201828BRIAN PATRICK GREEN
Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan
Mané. 2016. “Concrete problems in AI safety.” arXiv preprint , arXiv:1606.06565.
https://arxiv.org/abs/1606.06565 .
Angwin, Julia, Jeff Larson, Surya Mattu and Lauren Kirchner. 2016. “Machine Bias.”
ProPublica , May 23. https://www.propublica.org/article/machine-bias-risk-as -
sessments-in-criminal-sentencing .
Aristotle. 1908. Nicomachean Ethics . Translated by W. D. Ross. Book VI, Ch. 8. http://
classics.mit.edu/Aristotle/nicomachaen.6.vi.html .
Arkin, Ronald. 2009. Governing Lethal Behavior in Autonomous Robots . Boca Raton,
Florida: CRC Press.
Arnold, Thomas, Daniel Kasenberg, and Matthias Scheutz. 2017. “Value Alignment
or Misalignment – What Will Keep Systems Accountable?” Association for the
Advancement of Artificial Intelligence. https://hrilab.tufts.edu/publications/
aaai17-alignment.pdf .
Barron, Robert. 2015. “The ‘Waze’ of Providence.” Word on Fire , website. December
1. https://www.wordonfire.org/resources/article/the-waze-of-providence/4997/ .
Bossmann, Julia. 2016. “Top 9 ethical issues in artificial intelligence.” World Eco -
nomic Forum: Global Agenda , 21 Oct. https://www.weforum.org/agenda/2016/10/
top-10-ethical-issues-in-artificial-intelligence/ .
Bostrom, Nick. 2014. Superintelligence: Paths, Dangers, Strategies . Oxford: Oxford
University Press.
BBC. 2015. “Google Apologises for Photos App’s Racist Blunder.” BBC News . 1 July.
http://www.bbc.com/news/technology-33347866 .
Caliskan, Aylin, Joanna J. Bryson, Arvind Narayanan. 2017. “Semantics derived
automatically from language corpora contain human-like biases.” Science 356
(6334) (14 April): 183–186.
Cannon, Lincoln. 2015. “What is Mormon Transhumanism?” Theology and Science
13 (2): 202–218.
Chella, Antonio, & Ricardo Manzotti. 2013. Artificial Consciousness. Exeter, UK:
Imprint Academic.
Chrisley, Ron. 2008. “Philosophical foundations of artificial consciousness.” Artificial
Intelligence in Medicine 44 (2) (October): 119–137.
Evans, Richard, and Jim Gao. 2016. “DeepMind AI Reduces Google Data Centre
Cooling Bill by 40%.” DeepMind Blog , 20 July. https://deepmind.com/blog/
deepmind-ai-reduces-google-data-centre-cooling-bill-40/ .
Floridi, Luciano, and Jeff W. Sanders. 2004. On the morality of artificial agents. Minds
and machines 14 (3) (August 1): 349–379.
6(2)/201829ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
Future of Life Institute. 2017. “ Asilomar AI Principles.” Principles 7 and 8. https://
futureoflife.org/ai-principles/ .
Gips, J. 1994. “Towards the Ethical Robot.” In Android Epistemology , edited by Kenneth
M. Ford, C. Glymour & Patrick Hayes. Cambridge, Mass.: MIT Press.
Green, Brian Patrick. 2018b. “The Technology of Holiness: A Response to Hava
Tirosh-Samuelson.” Theology and Science 16 (2): 223–228.
Green, Brian Patrick. 2018a. “ Artificial Intelligence and Ethics: Twelve Areas of
Interest.” Pope John XXIII Memorial Lecture, University of the Pacific, Stockton,
California, March 21.
Green, Brian Patrick. 2017c. “ Artificial Intelligence and Ethics: Ten Areas of Interest.”
All about Ethics Blog . Markkula Center for Applied Ethics, website. November 21.
https://www.scu.edu/ethics/all-about-ethics/artificial-intelligence-and-ethics/ .
Green, Brian Patrick. 2017b. “Some Ethical and Theological Reflections on Artificial
Intelligence.” Conference paper delivered to the Pacific Coast Theological
Society, at the Graduate Theological Union, Berkeley, California, November 3.
http://www.pcts.org/meetings/2017/PCTS2017Nov-Green-ReflectionsAI.pdf .
Green, Brian Patrick. 2017a. “The Catholic Church and Technological Progress:
Past, Present, and Future.” Religions 8(6), 106. http://www.mdpi.com/2077-
1444/8/6/106/htm .
Harris, Mark. 2017. “Inside the First Church of Artificial Intelligence.” Wired (No -
vember 15). https://www.wired.com/story/anthony-levandowski-artificial-in -
telligence-religion/ .
John XXIII. 1963. Pacem in Terris ; Vatican City: Libreria Editrice Vaticana. Available
online: http://w2.vatican.va/content/john-xxiii/en/encyclicals/documents/
hf\_j-xxiii\_enc\_11041963\_pacem.html .
Jonas, Hans. 1984. The Imperative of Responsibility . Chicago: University of Chicago
Press.
Joy, Bill. 2000. “Why the Future Doesn’t Need Us.” Wired , April 1. http://archive.
wired.com/wired/archive/8.04/joy\_pr.html .
Koene, Randal A. 2013. “Uploading to Substrate ‐Independent Minds.” Chapter 14 in
The Transhumanist Reader: Classical and Contemporary Essays on the Science,
Technology, and Philosophy of the Human Future, edited by Max More and
Natasha Vita ‐More. New York: Wiley.
Konev, Boris, and Alexei Lisitsa. 2014. “ A SAT Attack on the Erdos Discrepancy
Conjecture.” arXiv.org , 17 February. arXiv:1402.2184v2 .
Kurzweil, Ray. 2012. How to Create a Mind: The Secret of Human Thought Revealed.
New York: Penguin.
6(2)/201830BRIAN PATRICK GREEN
Labrecque, Cory Andrew. 2017. “The Glorified Body: Corporealities in the Catholic
Tradition.” Religions 8 (9): 166. http://www.mdpi.com/2077-1444/8/9/166/htm .
Leviathan, Yaniv, and Yossi Matias. 2018. “Google Duplex: An AI System for Ac -
complishing Real-World Tasks Over the Phone.” Google AI Blog , May 8. https://
ai.googleblog.com/2018/05/duplex-ai-system-for-natural-conversation.html .
Mannion, Lee. 2018. “Britain appoints minister for loneliness amid growing isolation.”
Reuters . January 17. https://www.reuters.com/article/us-britain-politics-health/
britain-appoints-minister-for-loneliness-amid-growing-isolation-idUSKBN -
1F61I6 .
Martin, Mike W., & Roland Schinzinger. 2010. Introduction to Engineering Ethics ,
Second Edition. Boston: McGraw-Hill.
Mozur, Paul. 2017. “Google’s AlphaGo Defeats Chinese Go Master in Win for A.I.”
The New York Times . May 23. https://www.nytimes.com/2017/05/23/business/
google-deepmind-alphago-go-champion-defeat.html .
Mukherjee, Siddhartha. 2017. “ A.I. versus M.D.: What happens when diagnosis is auto -
mated?” New Yorker . April 3. https://www.newyorker.com/magazine/2017/04/03/
ai-versus-md .
Partnership on AI. 2018b. Safety-Critical Working Group Meeting, San Francisco,
USA, May 24.
Partnership on AI. 2018a. “Thematic Pillars.” Partnership on AI website. https://
www.partnershiponai.org/thematic-pillars/ .
Partnership on AI. 2017. Inaugural Meeting, Berlin, Germany, October 23.
Platt, John R. 2016. “ A Starfish-Killing, Artificially Intelligent Robot Is Set to
Patrol the Great Barrier Reef.” Scientific American . January 1. http://www.
scientificamerican.com/article/a-starfish-killing-artificially-intelligent-ro -
bot-is-set-to-patrol-the-great-barrier-reef/ .
Schlagel, Richard H. 1999. “Why not Artificial Consciousness or Thought?” Minds
and Machines 9 (1) (February): 3–28.
Searle, John R. 1980. ‘Mind, brains, and programs.’ Behavioral and Brain Sciences
3 (3): 417–457.
Strathern, Marilyn. 1997. “‘Improving ratings’: audit in the British University system.”
European Review 5, No. 03 (July): 305 – 321. http://conferences.asucollegeoflaw.
com/sciencepublicsphere/files/2014/02/Strathern1997-2.pdf .
Tufekci, Zeynep. 2018. “YouTube, the Great Radicalizer.” The New York Times , March
10. https://www.nytimes.com/2018/03/10/opinion/sunday/youtube-politics-rad -
ical.html .
Vallor, Shannon. 2016. Technology and the Virtues: A Philosophical Guide to a Future
Worth Wanting . New York: Oxford University Press.
6(2)/201831ETHICAL REFLECTIONS ON ARTIFICIAL INTELLIGENCE
Vallor, Shannon. 2015. “Moral Deskilling and Upskilling in a New Machine Age:
Reflections on the Ambiguous Future of Character.” Philosophy of Technology
28: 107–124.
Van Parijs, Philippe, and Yannick Vanderborght. 2017. Basic Income: A Radical
Proposal for a Free Society and a Sane Economy , 1st Edition. Cambridge, Mass.:
Harvard Univerrsity Press.
Yirka, Bob. 2014. “Computer generated math proof is too large for humans to check.”
Phys.org , website. February 19. https://phys.org/news/2014-02-math-proof-
large-humans.html .


Choice Set Misspeciﬁcation in Reward Inference
Rachel Freedman,Rohin Shah and Anca Dragan
University of California, Berkeley
frachel.freedman, rohinmshah, anca g@berkeley.edu,
Abstract
Specifying reward functions for robots that operate
in environments without a natural reward signal can
be challenging, and incorrectly speciﬁed rewards
can incentivise degenerate or dangerous behavior.
A promising alternative to manually specifying re-
ward functions is to enable robots to infer them
from human feedback, like demonstrations or cor-
rections. To interpret this feedback, robots treat as
approximately optimal a choice the person makes
from a choice set , like the set of possible trajecto-
ries they could have demonstrated or possible cor-
rections they could have made. In this work, we
introduce the idea that the choice set itself might be
difﬁcult to specify, and analyze choice set misspeci-
ﬁcation : what happens as the robot makes incorrect
assumptions about the set of choices from which
the human selects their feedback. We propose a
classiﬁcation of different kinds of choice set mis-
speciﬁcation, and show that these different classes
lead to meaningful differences in the inferred re-
ward and resulting performance. While we would
normally expect misspeciﬁcation to hurt, we ﬁnd
that certain kinds of misspeciﬁcation are neither
helpful nor harmful (in expectation). However, in
other situations, misspeciﬁcation can be extremely
harmful, leading the robot to believe the opposite of
what it should believe. We hope our results will al-
low for better prediction and response to the effects
of misspeciﬁcation in real-world reward inference.
1 Introduction
Specifying reward functions for robots that operate in envi-
ronments without a natural reward signal can be challenging,
and incorrectly speciﬁed rewards can incentivise degenerate
or dangerous behavior [Leike et al. , 2018; Krakovna, 2018 ].
A promising alternative to manually specifying reward func-
tions is to design techniques that allow robots to infer them
from observing and interacting with humans.
Contact Author
Copyright c
2020 for this paper by its authors. Use permitted under
Creative Commons License Attribution 4.0 International (CC BY
4.0).
Figure 1: Example choice set misspeciﬁcation : The human chooses
a pack of peanuts at the supermarket. They only notice the expen-
sive one because it has ﬂashy packaging, so that’s the one they buy.
However, the robot incorrectly assumes that the human can see both
the expensive ﬂashy one and the cheap one with dull packaging but
extra peanuts. As a result, the robot incorrectly infers that the human
likes ﬂashy packaging, paying more, and getting fewer peanuts.
These techniques typically model humans as optimal or
noisily optimal. Unfortunately, humans tend to deviate
from optimality in systematically biased ways [Kahneman
and Tversky, 1979; Choi et al. , 2014 ]. Recent work im-
proves upon these models by modeling pedagogy [Hadﬁeld-
Menell et al. , 2016 ], strategic behavior [Waugh et al. , 2013 ],
risk aversion [Majumdar et al. , 2017 ], hyperbolic discount-
ing[Evans et al. , 2015 ], or indifference between similar op-
tions [Bobu et al. , 2020b ]. However, given the complexity of
human behavior, our human models will likely always be at
least somewhat misspeciﬁed [Steinhardt and Evans, 2017 ].
One way to formally characterize misspeciﬁcation is as
a misalignment between the real human and the robot’s as-
sumptions about the human. Recent work in this vein has ex-
amined incorrect assumptions about the human’s hypothesis
space of rewards [Bobu et al. , 2020a ], their dynamics model
of the world [Reddy et al. , 2018 ], and their level of pedagogic
behavior [Milli and Dragan, 2019 ]. In this work, we identify
another potential source of misalignment: what if the robot

is wrong about what feedback the human could have given?
Consider the situation illustrated in Figure 1, in which the
robot observes the human going grocery shopping. While the
grocery store contains two packages of peanuts, the human
only notices the more expensive version with ﬂashy packag-
ing, and so buys that one. If the robot doesn’t realize that the
human was effectively unable to evaluate the cheaper pack-
age on its merits, it will learn that the human values ﬂashy
packaging.
We formalize this in the recent framework of reward-
rational implicit choice (RRiC) [Jeon et al. , 2020 ]as misspec-
iﬁcation in the human choice set , which speciﬁes what feed-
back the human could have given. Our core contribution is
to categorize choice set misspeciﬁcation into several formally
and empirically distinguishable “classes”, and ﬁnd that differ-
ent types have signiﬁcantly different effects on performance.
As we might expect, misspeciﬁcation is usually harmful; in
the most extreme case the choice set is so misspeciﬁed that
the robot believes the human feedback was the worst possible
feedback for the true reward, and so updates strongly towards
theopposite of the true reward. Surprisingly, we ﬁnd that
under other circumstances misspeciﬁcation is provably neu-
tral: it neither helps nor hurts performance in expectation.
Crucially, these results suggest that not all misspeciﬁcation
is equivalently harmful to reward inference: we may be able
to minimize negative impact by systematically erring toward
particular misspeciﬁcation classes deﬁned in this work. Fu-
ture work will explore this possibility.
2 Reward Inference
There are many ways that a human can provide feedback to
a robot: demonstrations [Ng and Russell, 2000; Abbeel and
Ng, 2004; Ziebart, 2010 ], comparisons [Sadigh et al. , 2017;
Christiano et al. , 2017 ], natural language [Goyal et al. , 2019 ],
corrections [Bajcsy et al. , 2017 ], the state of the world [Shah
et al. , 2019 ], proxy rewards [Hadﬁeld-Menell et al. , 2017;
Mindermann et al. , 2018 ], etc. Jeon et al. propose a unifying
formalism for reward inference to capture all of these possible
feedback modalities, called reward-rational (implicit) choice
(RRiC). Rather than study each feedback modality separately,
we study misspeciﬁcation in this general framework.
RRiC consists of two main components: the human’s
choice set , which corresponds to what the human could have
done , and the grounding function , which converts choices into
(distributions over) trajectories so that rewards can be com-
puted.
For example, in the case of learning from comparisons, the
human chooses which out of two trajectories is better. Thus,
the human’s choice set is simply the set of trajectories they
are comparing, and the grounding function is the identity. A
more complex example is learning from the state of the world,
in which the robot is deployed in an environment in which
a human has already acted for Ttimesteps, and must infer
the human’s preferences from the current world state. In this
case, the robot can interpret the human as choosing between
different possible states. Thus, the choice set is the set of
possible states that the human could reach in Ttimesteps,
and the grounding function maps each such state to the set oftrajectories that could have produced it.
Letdenote a trajectory and denote the set of all pos-
sible trajectories. Given a choice set Cfor the human and
grounding function :C!(![0;1]), Jeon et al. deﬁne
a procedure for reward learning. They assume that the human
isBoltzmann-rational with rationality parameter , so that
the probability of choosing any particular feedback is given
by:
P(cj;C) =exp(E (c)[r()])P
c02Cexp(E (c0)[r()])(1)
From the robot’s perspective, every piece of feedback cis
an observation about the true reward parameterization , so
the robot can use Bayesian inference to infer a posterior over
. Given a prior over reward parameters P(), the RRiC in-
ference procedure is deﬁned as:
P(jc;C)/exp(E (c)[r()]P
c02Cexp(E (c0)[r()])P()(2)
Since we care about misspeciﬁcation of the choice set C,
we focus on learning from demonstrations, where we restrict
the set of trajectories that the expert can demonstrate. This
enables us to have a rich choice set, while allowing for a sim-
ple grounding function (the identity). In future work, we aim
to test choice set misspeciﬁcation with other feedback modal-
ities as well.
3 Choice Set Misspeciﬁcation
For many common forms of feedback, including demonstra-
tions and proxy rewards, the RRiC choice set is implicit . The
robot knows which element of feedback the human provided
(ex. which demonstration they performed), but must assume
which elements of feedback the human could have provided
based on their model of the human. However, this assump-
tion could easily be incorrect – the robot may assume that the
human has capabilities that they do not, or may fail to ac-
count for cognitive biases that blind the human to particular
feedback options, such as the human bias towards the most
visually attention-grabbing choice in Fig 1.
To model such effects, we assume that the human selects
feedbackc2CHuman according to P(cj;CHuman ), while
the robot updates their belief assuming a different choice
setCRobot to get P(jc;CRobot ). Note that CRobot is
the robot’s assumption about what the human’s choice set
is – this is distinct from the robot’s action space. When
CHuman6=CRobot , we get choice set misspeciﬁcation .
It is easy to detect such misspeciﬁcation when the human
chooses feedback c =2CR. In this case, the robot observes
a choice that it believes to be impossible, which should cer-
tainly be grounds for reverting to some safe baseline policy.
So, we only consider the case where the human’s choice cis
also present in CR(which also requires CHandCRto have
at least one element in common).
Within these constraints, we propose a classiﬁcation of
types of choice set misspeciﬁcation in Table 1. On the
vertical axis, misspeciﬁcation is classiﬁed according to
CRCHCRCHCR\CH
c2CR\CHA1 A2 A3
2CRnCH B2 B3
Table 1: Choice set misspeciﬁcation classiﬁcation, where CRis the
robot’s assumed choice set, CHis the human’s actual choice set, and
cis the optimal element from CR[CH.B1is omitted because if
CRCH, thenCRnCHis empty and cannot contain c.
Figure 2: The set of four gridworlds used in randomized experi-
ments, with the lava feature marked in red.
the location of the optimal element of feedback c=
argmaxc2CR[CHE (c)[r())]. Ifcisavailable to the
human (inCH), then the class code begins with A. We only
consider the case where cis also inCR: the case where
it is inCHbut notCRis uninteresting, as the robot would
observe the “impossible” event of the human choosing c,
which immediately demonstrates misspeciﬁcation at which
point the robot should revert to some safe baseline policy.
Ifc=2CH, then we must have c2CR(since it was cho-
sen fromCH[CR), and the class code begins with B. On
the horizontal axis, misspeciﬁcation is classiﬁed according to
the relationship between CRandCH.CRmay be a subset
(code 1), superset (code 2), or intersecting class (code 3) of
CH. For example, class A1describes the case in which the
robot’s choice set is a subset of the human’s (perhaps because
the human is more versatile), but both choice sets contain the
optimal choice (perhaps because it is obvious).
4 Experiments
To determine the effects of misspeciﬁcation class, we artiﬁ-
cially generated CRandCHwith the properties of each par-
ticular class, simulated human feedback, ran RRiC reward in-
ference, and then evaluated the robot’s resulting belief distri-
bution and optimal policy.4.1 Experimental Setup
Environment To isolate the effects of misspeciﬁcation and
allow for computationally tractable Bayesian inference, we
ran experiments in toy environments. We ran the randomized
experiments in the four 2020gridworlds shown in Fig 2.
Each square in environment xis a statesx=flava;goalg.
lava2[0;1]is a continuous feature, while goal2f0;1g
is a binary feature set to 1in the lower-right square of each
grid and 0everywhere else. The true reward function ris
a linear combination of these features and a constant stay-
alive cost incurred at each timestep, parameterized by =
(wlava;wgoal;walive). Each episode begins with the robot in
the upper-left corner and ends once the robot reaches the goal
state or episode length reaches the horizon of 35 timesteps.
Robot actions ARmove the robot one square in a cardinal
or diagonal direction, with actions that would move the robot
off of the grid causing it to remain in place. The transition
functionTis deterministic. Environment xdeﬁnes an MDP
Mx=hSx;AR;T;ri.
Inference While the RRiC framework enables inference
from many different types of feedback, we use demonstra-
tion feedback here because demonstrations have an implicit
choice set and straightforward deterministic grounding. Only
the human knows their true reward function parameteriza-
tion. The robot begins with a uniform prior distribution
over reward parameters P()in whichwlavaandwalive vary,
butwgoal always = 2:0.P()contains. RRiC infer-
ence proceeds as follows for each choice set tuple hCR;CHi
and environment x. First, the simulated human selects the
best demonstration from their choice set with respect to the
true reward cH= argmaxc2CHE (c)[r())]. Then,
the simulated robot uses Eq. 2 to infer a “correct” distribu-
tion over reward parameterizations BH(),P(jc;CH)
using the true human choice set, and a “misspeciﬁed” dis-
tributionBR(),P(jc;CR)using the misspeciﬁed hu-
man choice set. In order to evaluate the effects of each dis-
tribution on robot behavior, we deﬁne new MDPs Mx
H=
hSx;AR;T;r E[BH()]iandMx
R=hSx;AR;T;r E[BR()]ifor
each environment, solve them using value iteration, and then
evaluate the rollouts of the resulting deterministic policies ac-
cording to the true reward function r.
4.2 Randomized Choice Sets
We ran experiments with randomized choice set selection for
each misspeciﬁcation class to evaluate the effects of class on
entropy change and regret.
Conditions The experimental conditions are the classes of
choice set misspeciﬁcation in Table 1: A1,A2,A3,B2and
B3. We tested each misspeciﬁcation class on each environ-
ment, then averaged across environments to evaluate each
class. For each environment x, we ﬁrst generated a mas-
ter setCx
Mof all demonstrations that are optimal w.r.t. at
least one reward parameterization . For each experimen-
tal class, we randomly generated 6 valid hCR; CHituples,
withCR;CHCx
M. Duplicate tuples, or tuples in which
cH=2CR, were not considered.
(a)CH
 (b)CR
Figure 3: Human and robot choice sets with a human goal bias.
Because the human only considers trajectories that terminate at the
goal, they don’t consider the blue trajectory in CR.
Measures There are two key experimental measures: en-
tropy change andregret . Entropy change is the difference
in entropy between the correct distribution BH, and the mis-
speciﬁed distribution BR. That is, H=H(BH)H(BR).
If entropy change is positive, then misspeciﬁcation induces
overconﬁdence, and if it is negative, then misspeciﬁcation in-
duces underconﬁdence.
Regret is the difference in return between the optimal solu-
tion toMx
H, with the correctly-inferred reward parameteriza-
tion, and the optimal solution to Mx
R, with the incorrectly-
inferred parameterization, averaged across all 4 environ-
ments. Ifx
His an optimal trajectory in Mx
Handx
Ris an
optimal trajectory in Mx
R, thenregret =1
4P3
x=0[r(x
H)
r(x
R)]. Note that we are measuring regret relative to the
optimal action under the correctly speciﬁed belief , rather than
optimal action under the true reward. As a result, it is possible
for regret to be negative , e.g. if the misspeciﬁcation makes the
robot become more conﬁdent in the true reward than it would
be under correct speciﬁcation, and so execute a better policy.
4.3 Biased Choice Sets
We also ran an experiment in a ﬁfth gridworld where we se-
lect the human choice set with a realistic human bias to illus-
trate how choice set misspeciﬁcation may arise in practice. In
this experiment the human only considers demonstrations that
end at the goal state because, to humans, the word “goal” can
be synonymous with “end” (Fig 3a). However, to the robot,
the goal is merely one of multiple features in the environ-
ment. The robot has no reason to privilege it over the other
features, so the robot considers every demonstration that is
optimal w.r.t some possible reward parameterization (Fig 3b).
The trajectory that only the robot considers is marked in blue.
We ran RRiC inference using this hCR; CHiand evaluated
the results using the same measures described above.
5 Results
We summarize the aggregated measures, discuss the realistic
human bias result, then examine two interesting results: sym-
metry between classes A1andA2and high regret in class B3.
Figure 4: Entropy Change (N=24). The box is the IQR, the whiskers
are the range, and the blue line is the median. There are no outliers.
Figure 5: Regret (N=24). The box is the IQR, the whiskers are the
most distant points within 1.5\*the IQR, and the green line is the
mean. Multiple outliers are omitted.
5.1 Aggregate Measures in Randomized
Experiments
Entropy Change Entropy change varied signiﬁcantly
across misspeciﬁcation class. As shown in Fig 4, the in-
terquartile ranges (IQRs) of classes A1andA3did not overlap
with the IQRs of A2andB2. Moreover, A1andA3had pos-
itive medians, suggesting a tendency toward overconﬁdence,
while A2andB2had negative medians, suggesting a ten-
dency toward underconﬁdence. B3was less distinctive, with
an IQR that overlapped with that of all other classes. No-
tably, the distributions over entropy change of classes A1and
A2are precisely symmetric about 0.
Regret Regret also varied as a function of misspeciﬁcation
class. Each class had a median regret of 0, suggesting that
misspeciﬁcation commonly did not induce a large enough
shift in belief for the robot to learn a different optimal policy.
However the mean regret, plotted as green lines in Fig 5, did
vary markedly across classes. Regret was sometimes so high
in class B3that outliers skewed the mean regret beyond of the
whiskers of the boxplot. Again, classes A1andA2are pre-
cisely symmetric. We discuss this symmetry in Section 5.3,
then discuss the poor performance of B3in Section 5.4.
(a) feedback cH
 (b)P(jcH;CR)
Figure 6: Human feedback and the resulting misspeciﬁed robot be-
lief with a human goal bias. Because the feedback that the biased
human provides is poor, the robot learns a very incorrect distribu-
tion over rewards.
5.2 Effects of Biased Choice Sets
The human bias of only considering demonstrations that ter-
minate at the goal leads to very poor inference in this en-
vironment. Because the human does not consider the blue
demonstration from Fig 3b, which avoids the lava altogether,
they are forced to provide the demonstration in Fig 6a, which
terminates at the goal but is long and encounters lava. As a
result, the robot infers the very incorrect belief distribution in
Fig 6b. Not only is this distribution underconﬁdent (entropy
change =0:614), but it also induces poor performance (re-
gret = 0:666). This result shows that we can see an outsized
negative impact on robot reward inference with a small in-
correct assumption that the human considered and rejected
demonstrations that don’t terminate at the goal.
5.3 Symmetry
Intuitively, misspeciﬁcation should lead to worse perfor-
mance in expectation. Surprisingly, when we combine mis-
speciﬁcation classes A1andA2, their impact on entropy
change and regret is actually neutral. The key to this is their
symmetry – if we switch the contents of CRobot andCHuman
in an instance of class A1misspeciﬁcation, we get an instance
of class A2with exactly the opposite performance character-
istics. Thus, if a pair in A1is harmful, then the analogous
pair in A2must be helpful, meaning that it is better for per-
formance than having the correct belief about the human’s
choice set . We show below that this is always the case under
certain symmetry conditions that apply to A1andA2.
Assume that there is a master choice set CMcontaining all
possible elements of feedback for MDP M, and that choice
sets are sampled from a symmetric distribution over pairs
of subsetsD: 2CM2CM![0;1]withD(Cx;Cy) =
D(Cy;Cx)(where 2CMis the set of subsets of CM). Let
ER(r;M)be the expected return from maximizing the re-
ward function rinM. A reward parameterization is chosen
from a shared prior P()andCH;CRare sampled from D.
The human chooses the optimal element of feedback in their
choice setcCH= argmaxc2CHE (c)[r())].
Theorem 1. LetMandDbe deﬁned as above. Assume that
8Cx;CyD, we havecCx=cCy; that is, the human would
pick the same feedback regardless of which choice set sheClass Mean Std Q1 Q3
A1 0.256 0.2265 0.1153 0.4153
A2 -0.256 0.2265 -0.4153 -0.1153
Table 2: Entropy change is symmetric across classes A1andA2.
Class Mean Std Q1 Q3
A1 0.04 0.4906 0.1664 0.0
A2 -0.04 0.4906 0.0 -0.1664
Table 3: Regret is symmetric across classes A1andA2.
sees. If the robot follows RRiC inference according to Eq. 2
and acts to maximize expected reward under the inferred be-
lief, then:
E
CH;CRDRegret (CH;CR) = 0
Proof. DeﬁneR(Cx;c)to be the return achieved when the
robot follows RRiC inference with choice set Cxand feed-
backc, then acts to maximize rE[Bx()], keepingﬁxed.
Since the human’s choice is symmetric across D, for any
Cx;CyD, regret is anti- symmetric:
Regret (Cx;Cy) =R(Cx;cCx)R(Cy;cCx)
=R(Cx;cCy)R(Cy;cCy)
=Regret (Cy;Cx)
SinceDis symmetric,hCx; Cyiis as likely ashCy; Cxi.
Combined with the anti-symmetry of regret, this implies that
the expected regret must be zero:
E
Cx;CyD[Regret (Cx;Cy)]
=1
2E
Cx;Cy[Regret (Cx;Cy)] +1
2E
Cx;Cy[Regret (Cy;Cx)]
=1
2E
Cx;Cy[Regret (Cx;Cy)]1
2E
Cx;Cy[Regret (Cx;Cy)]
= 0
An analogous proof would work for any anti-symmetric
measure (including entropy change).
5.4 Worst Case
As shown in Table 4, class B3misspeciﬁcation can induce
regret an order of magnitude worse than the maximum regret
induced by classes A3andB2, which each differ from B3
along a single axis. This is because the worst case inference
occurs in RRiC when the human feedback cHis the worst
element ofCR, and this is only possible in class B3. In class
Class Mean Std Max Min
A3 -0.001 0.5964 1.1689 -1.1058
B2 0.228 0.6395 1.6358 -0.9973
B3 2.059 6.3767 24.7252 -0.9973
Table 4: Regret comparison showing that class B3has much higher
regret than neighboring classes.
(a)CH
 (b)cH
Figure 7: Example human choice set and corresponding feedback.
B2,CRcontains all of CH, so as long asjCHj>1,CR
must contain at least one element worse than CH. In class
A3,cH=c, soCRcannot contain any elements better than
cH. However, in class B3,CRneed not contain any elements
worse thancH, in which case the robot updates its belief in
the opposite direction from the ground truth.
For example, consider the sample human choice set in
Fig 7a. Both trajectories are particularly poor, but the human
chooses the demonstration cHin Fig 7b because it encoun-
ters slightly less lava and so has a marginally higher reward.
Fig 8a shows a potential corresponding robot choice set CR2
fromB2, containing both trajectories from the human choice
set as well as a few others. Fig 8b shows P(jcH;CR2).
The axes represent the weights on the lava andalive fea-
tures and the space of possible parameterizations lies on the
circle where wlava+walive = 1. The opacity of the gold line
is proportional to the weight that P()places on each param-
eter combination. The true reward has wlava;walive<0,
whereas the peak of this distribution has wlava<0, but
walive>0. This is because CR2contains shorter trajecto-
ries that encounter the same amount of lava, and so the robot
infers thatcHmust be preferred in large part due to its length.
Fig 9a shows an example robot choice set CR3from B3,
and Fig 9b shows the inferred P(jcH;CR3). Note that the
peak of this distribution has wlava;walive>0. SincecH
is the longest and the highest-lava trajectory in CR3,andal-
ternative shorter and lower-lava trajectories exist in CR3, the
robot infers that the human is attempting to maximize both
trajectory length and lava encountered: the opposite of the
truth. Unsurprisingly, maximizing expected reward for this
belief leads to high regret. The key difference between B2
andB3is thatcHis the lowest-reward element in CR3, result-
ing in the robot updating directly away from the true reward.
(a)CR2
 (b)P(jcH;CR2)
Figure 8: Robot choice set and resulting misspeciﬁed belief in B2.
(a)CR3
 (b)P(jcH;CR3)
Figure 9: Robot choice set and resulting misspeciﬁed belief in B3.
6 Discussion
Summary In this work, we highlighted the problem of
choice set misspeciﬁcation in generalized reward inference,
where a human gives feedback selected from choice set
CHuman but the robot assumes that the human was choos-
ing from choice set CRobot . As expected, such misspeciﬁ-
cation on average induces suboptimal behavior resulting in
regret. However, a different story emerged once we distin-
guished between misspeciﬁcation classes. We deﬁned ﬁve
distinct classes varying along two axes: the relationship be-
tweenCHuman andCRobot and the location of the optimal
element of feedback c. We empirically showed that differ-
ent classes lead to different types of error, with some classes
leading to overconﬁdence, some to underconﬁdence, and one
to particularly high regret. Surprisingly, under certain condi-
tions the expected regret under choice set misspeciﬁcation is
actually 0, meaning that in expectation, misspeciﬁcation does
not hurt in these situations.
Implications There is wide variance across the different
types of choice-set misspeciﬁcation: some may have partic-
ularly detrimental effects, and others may not be harmful at
all. This suggests strategies for designing robot choice sets
to minimize the impact of misspeciﬁcation. For example,
we ﬁnd that regret tends to be negative (that is, misspeciﬁ-
cation is helpful ) when the optimal element of feedback is in
bothCRobot andCHuman andCRobotCHuman (class A2).
Similarly, worst-case inference occurs when the optimal ele-
ment of feedback is in CRobot only, andCHuman contains
elements that are not in CRobot (class B3). This suggests that
erring on the side of specifying a large CRobot , which makes
A2more likely and B3less, may lead to more benign mis-
speciﬁcation. Moreover, it may be possible to design proto-
cols for the robot to identify unrealistic choice set-feedback
combinations and verify its choice set with the human, reduc-
ing the likelihood of misspeciﬁcation in the ﬁrst place. We
plan to investigate this in future work.
Limitations and future work. In this paper, we primarily
sampled choice sets randomly from the master choice set of
all possibly optimal demonstrations. However, this is not a re-
alistic model. In future work, we plan to select human choice
sets based on actual human biases to improve ecological va-
lidity. We also plan to test this classiﬁcation and our result-
ing conclusions in more complex and realistic environments.
Eventually, we plan to work on active learning protocols that
allow the robot to identify when its choice set is misspeciﬁed
and alter its beliefs accordingly.
References
[Abbeel and Ng, 2004 ]Pieter Abbeel and Andrew Y Ng.
Apprenticeship learning via inverse reinforcement learn-
ing. In Proceedings of the twenty-ﬁrst international con-
ference on Machine learning , page 1, 2004.
[Bajcsy et al. , 2017 ]Andrea Bajcsy, Dylan P Losey, Mar-
cia K O’Malley, and Anca D Dragan. Learning robot ob-
jectives from physical human interaction. Proceedings of
Machine Learning Research , 78:217–226, 2017.
[Bobu et al. , 2020a ]Andreea Bobu, Andrea Bajcsy, Jaime F
Fisac, Sampada Deglurkar, and Anca D Dragan. Quanti-
fying hypothesis space misspeciﬁcation in learning from
human–robot demonstrations and physical corrections.
IEEE Transactions on Robotics , 2020.
[Bobu et al. , 2020b ]Andreea Bobu, Dexter RR Scobee,
Jaime F Fisac, S Shankar Sastry, and Anca D Dragan. Less
is more: Rethinking probabilistic models of human behav-
ior.arXiv preprint arXiv:2001.04465 , 2020.
[Choi et al. , 2014 ]Syngjoo Choi, Shachar Kariv, Wieland
M¨uller, and Dan Silverman. Who is (more) rational?
American Economic Review , 104(6):1518–50, 2014.
[Christiano et al. , 2017 ]Paul F. Christiano, Jan Leike,
Tom B. Brown, Miljan Martic, Shane Legg, and Dario
Amodei. Deep reinforcement learning from human pref-
erences. Advances in Neural Information Processing Sys-
tems, 2017-Decem:4300–4308, 2017.
[Evans et al. , 2015 ]Owain Evans, Andreas Stuhlmueller,
and Noah D. Goodman. Learning the Preferences of Ig-
norant, Inconsistent Agents. arXiv , 2015.
[Goyal et al. , 2019 ]Prasoon Goyal, Scott Niekum, and Ray-
mond J Mooney. Using natural language for re-
ward shaping in reinforcement learning. arXiv preprint
arXiv:1903.02020 , 2019.
[Hadﬁeld-Menell et al. , 2016 ]Dylan Hadﬁeld-Menell, Stu-
art J Russell, Pieter Abbeel, and Anca Dragan. Coopera-
tive inverse reinforcement learning. In Advances in neural
information processing systems , pages 3909–3917, 2016.[Hadﬁeld-Menell et al. , 2017 ]Dylan Hadﬁeld-Menell,
Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca
Dragan. Inverse reward design. In Advances in neural
information processing systems , pages 6765–6774, 2017.
[Jeon et al. , 2020 ]Hong Jun Jeon, Smitha Milli, and Anca D
Dragan. Reward-rational (implicit) choice: A uni-
fying formalism for reward learning. arXiv preprint
arXiv:2002.04833 , 2020.
[Kahneman and Tversky, 1979 ]Daniel Kahneman and
Amos Tversky. Prospect Theory: An Analysis of
Decision Under Risk. Econometrica , 47(2):263–292,
1979.
[Krakovna, 2018 ]Victoria Krakovna. Speciﬁcation gaming
examples in ai, April 2018.
[Leike et al. , 2018 ]Jan Leike, David Krueger, Tom Everitt,
Miljan Martic, Vishal Maini, and Shane Legg. Scalable
agent alignment via reward modeling: a research direction.
arXiv , 2018.
[Majumdar et al. , 2017 ]Anirudha Majumdar, Sumeet
Singh, Ajay Mandlekar, and Marco Pavone. Risk-
sensitive inverse reinforcement learning via coherent risk
models. In Robotics: Science and Systems , 2017.
[Milli and Dragan, 2019 ]Smitha Milli and Anca D Dragan.
Literal or pedagogic human? analyzing human model
misspeciﬁcation in objective learning. arXiv preprint
arXiv:1903.03877 , 2019.
[Mindermann et al. , 2018 ]S¨oren Mindermann, Rohin Shah,
Adam Gleave, and Dylan Hadﬁeld-Menell. Active inverse
reward design. arXiv preprint arXiv:1809.03060 , 2018.
[Ng and Russell, 2000 ]Andrew Y Ng and Stuart J Russell.
Algorithms for inverse reinforcement learning. In Interna-
tional Confer-ence on Machine Learning (ICML) , 2000.
[Reddy et al. , 2018 ]Sid Reddy, Anca Dragan, and Sergey
Levine. Where do you think you’re going?: Inferring be-
liefs about dynamics from behavior. In Advances in Neural
Information Processing Systems , pages 1454–1465, 2018.
[Sadigh et al. , 2017 ]Dorsa Sadigh, Anca D Dragan, Shankar
Sastry, and Sanjit A Seshia. Active preference-based learn-
ing of reward functions. In Robotics: Science and Systems ,
2017.
[Shah et al. , 2019 ]Rohin Shah, Dmitrii Krasheninnikov,
Jordan Alexander, Pieter Abbeel, and Anca Dragan. Pref-
erences implicit in the state of the world. arXiv preprint
arXiv:1902.04198 , 2019.
[Steinhardt and Evans, 2017 ]Jacob Steinhardt and Owain
Evans. Model mis-speciﬁcation and inverse reinforcement
learning, Feb 2017.
[Waugh et al. , 2013 ]Kevin Waugh, Brian D Ziebart, and
J Andrew Bagnell. Computational rationalization:
The inverse equilibrium problem. arXiv preprint
arXiv:1308.3506 , 2013.
[Ziebart, 2010 ]Brian D Ziebart. Modeling Purposeful Adap-
tive Behavior with the Principle of Maximum Causal En-
tropy . PhD thesis, Carnegie Mellon University, 2010.