Minimax-Regret Querying on Side Effects for Safe Optimality in Factored
Markov Decision Processes
Shun Zhang, Edmund H. Durfee, and Satinder Singh
Computer Science and Engineering, University of Michigan
fshunzh,durfee,bavejag@umich.edu
Abstract
As it achieves a goal on behalf of its human user,
an autonomous agent’s actions may have side ef-
fects that change features of its environment in
ways that negatively surprise its user. An agent
that can be trusted to operate safely should thus
only change features the user has explicitly per-
mitted. We formalize this problem, and develop
a planning algorithm that avoids potentially nega-
tive side effects given what the agent knows about
(un)changeable features. Further, we formulate a
provably minimax-regret querying strategy for the
agent to selectively ask the user about features that
it hasn’t explicitly been told about. We empirically
show how much faster it is than a more exhaustive
approach and how much better its queries are than
those found by the best known heuristic.
1 Introduction
We consider a setting where a human user tasks a compu-
tational agent with achieving a goal that may change one or
more state features of the world (e.g., a housekeeping agent
should change the state of the house ﬂoors and kitchen sink
from dirty to clean). In the process of accomplishing the
goal, the agent generally changes other features (e.g., its own
position and power level, opening doors, moving furniture,
scaring the cat). Some of these side-effects might be ﬁne
(even expected) by the user (e.g., moving), but others may
be undesirable/unsafe (e.g., leaving doors open lets the cat
roam/escape) even though they speed goal achievement (e.g.,
the agent’s movement between rooms). Although the user
tells the agent about some features that can be changed, as
well as some to not change (e.g., don’t knock over the price-
less vase), the user often lacks the time, patience, or fore-
sight to specify the changeability of every pertinent feature,
and may incorrectly assume that the agent has commonsense
(e.g., about cat behavior and the value of vases).
How can the agent execute a safely-optimal policy in such
a setting? We conservatively assume that, to ensure safety,
the agent should never side-effect a feature unless changing it
is explicitly known to be ﬁne. Hence, the agent could simply
execute the best policy that leaves such features unchanged.However, no such policy might exist, and even if it does it
might surprise the user as unnecessarily costly/inefﬁcient.
Our focus is thus on how the agent can selectively query the
user about the acceptability of changing features it hasn’t yet
been told about. We reject simply querying about every such
feature, as this would be unbearably tedious to the user, and
instead put the burden on the agent to limit the number and
complexity of queries. In fact, in this paper we mostly focus
on ﬁnding a single query about a few features that maximally
improves upon the policy while maintaining safety.
Our three main contributions in this paper are: 1) We for-
mulate (in Section 2) an AI safety problem of avoiding nega-
tive side-effects in factored MDPs. 2) We show (in Section 3)
how to efﬁciently identify the set of relevant features, i.e.,
the set of features that could potentially be worth querying
the user about. 3) We formulate (in Section 4) a minimax-
regret criterion when there is a limit on the number of features
the agent can ask about, and provide an algorithm that allows
the agent to ﬁnd the minimax-regret query by searching the
query space with efﬁcient pruning. We empirically evaluate
our algorithms in a simulated agent navigation task, outline
ongoing extensions/improvements, and contrast our work to
prior work, in the paper’s ﬁnal sections.
2 Problem Deﬁnition
We illustrate our problem in a simulated agent gridworld-
navigation domain, inspired by Amodei et al. [2016] and de-
picted in Figure 1, with doors, carpets, boxes, and a switch.
The agent can open/close a door, move a box, traverse a car-
pet, and toggle the switch. Initially, the agent is in the bot-
tom left corner; door d1 is open, d2 and d3 closed, the carpet
clean, and the switch “on”. The agent can move to an adjacent
location vertically, horizontally, or diagonally. For simplicity,
the transition function is assumed to be deterministic.
The user tasks the agent with turning off the switch as
quickly as is safely possible. The quickest path ( 1) traverses
the carpet, but this gets the carpet dirty and the agent doesn’t
know if that is allowed. The agent could instead enter the
room through door d1 and spend time moving box b1 or b2
out of the way ( 2or3respectively), open door d2, and
then go to the switch. However, boxes might contain frag-
ile objects and should not be moved; the user knows each
box’s contents, but the agent doesn’t. Or the agent could enter
through door d1 and walk upwards ( 4) around all the boxesProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4867
π1π2π3π4
d1 d2
b1b2
carpetb3
d3Figure 1: The robot navigation domain. The dominating policies
(see Section 3) are shown as arrows.
and open door d2 to get to the switch. The user may or may
not be okay with door d2 being opened. There are of course
many other more circuitous paths not shown.
We model the domain as a factored Markov Decision Pro-
cess (MDP) [Boutilier et al., 1999 ]. An MDP is a tuple
hS;A;T;r;s 0;
i, with state space S, action space A, and
transition function TwhereT(s0js;a)is the probability of
reaching state s0by taking action ains.r(s;a)is the re-
ward of taking action ains.s0is the initial state and
is
the discount factor. Let :SA![0;1]be a policy.
Vis the expected cumulative reward by following policy
starting from s0. In a factored MDP, a state is described
in terms of values of various features (e.g., the agent’s loca-
tion, the current time, the status of each door, cleanliness of
each carpet, position of each box), so the state space Sis the
cross-product of the values the features can take. The reward
and transition functions are often also factored (e.g., the “tog-
gle” action only changes the switch feature, leaving boxes,
doors, and carpets unchanged). We will consistently use
to denote one feature and to denote a set of features. The
agent knows the complete MDP model, but has incomplete
knowledge about which features the user doesn’t mind being
changed. In our example, the user’s goal implies the agent’s
location is changeable, as is the switch, but the agent is un-
certain about side-effecting boxes, doors, and carpets.
In general, the user could dictate that a feature can only
be changed among restricted values (e.g., boxes can only be
moved a short distance) and/or dependent on other features’
values (e.g., interior doors can be left open as long as exterior
doors (like d3) stay closed). We brieﬂy return to this issue
later (Section 6), but for simplicity assume here that the agent
can partition the features into the following sets:
A
F: The free-features. The agent knows that these fea-
tures are freely changeable (e.g., its location).
A
L: The locked-features. The agent knows it should
never change any of these features.
A
?: The unknown-features. These are features that the
agent doesn’t (initially) know whether the user considers
freely changeable or locked.
The user similarly partitions the features, but only into the
setsU
LandU
F. We assume that the agent’s knowledge,
while generally incomplete (A
?6=;), is consistent with that
of the user. That is, A
FU
FandA
LU
L.
Deﬁning & Finding Safely-Optimal Policies. Our conser-
vative safety assumption means the agent should treat un-
known features as if they are locked, until it explicitly knowsotherwise. It should thus ﬁnd an optimal policy that never vis-
its a state where a feature in A
L[A
?is changed, which we
call a safely-optimal policy. We can use linear programming
with constraints that prevent the policy from visiting states
with changed values of locked or unknown features:
max
xX
s;ax(s;a)r(s;a)s.t. (1)
8s02S;X
a0x(s0;a0) =
X
s;ax(s;a)T(s0js;a) +(s0;s0)
8s2SA
L[A
?;8a2A;x(s;a) = 0
The control variables x:SA!Rin the linear pro-
gram are occupancy measures, i.e., x(s;a)is the expected
discounted number of times that state action pair s;ais vis-
ited by the agent’s policy, Sis the set of states where one or
more features in have different values from the initial state,
and(s;s0) = 1 ifs=s0and is zero otherwise.
The above linear program does not allow any locked or un-
known feature to be changed and is guaranteed to produce a
safely-optimal policy (when one exists). This linear program-
ming approach, while straightforward, can be intractable for
large MDPs. Alternative approaches could directly encode
the safety constraints into the transition function (e.g., by re-
moving unsafe action choices in speciﬁc states), or into the
reward function (heavily penalizing reaching unsafe states).
Approximate methods, like feature-based approximate linear
programming [Dolgov and Durfee, 2006 ]or constrained pol-
icy optimization [Achiam et al., 2017 ], can apply to larger
problems, but may not guarantee safety or optimality (or
both). We return to these concerns in Section 6.
3 Querying Relevant Unknown-Features
In our setting the only way for the agent to determine whether
an unknown feature is freely changeable is to query the user.
Thus, hereafter our focus is on how the agent can ask a good
query about a small number, k, of features. Our solution is
to ﬁrst prune from A
?features that are guaranteed to not be
relevant to ask (this section), and then efﬁciently ﬁnding the
best (minimax-regret) k-sized subset of the relevant features
to query (Section 4). Until Section 6, we assume the change-
abilities of features are independent, i.e., when the agent asks
about some features in A
?, the user’s response does not allow
it to infer the changeability of other features in A
?.
Intuitively, when is a feature in A
?relevant to the agent’s
planning of a safely-optimal policy? In the navigation do-
main, if the agent plans to take the quickest path to the switch
(1in Figure 1), it will change the state of the carpet (from
clean to dirty). The carpet feature is thus relevant since the
agent would change it if permitted. If the carpet can’t be
changed but door d2 can, the agent would follow (in order
of preference) policy 2;3, or4, so d2, b1, and b2 are rel-
evant. Box b3 and door d3 are irrelevant, however, since no
matter which (if any) other features are changeable, an opti-
mal policy would never change them. Thus, an unknown fea-
ture is relevant when under some circumstance (some answer
to some query) the agent’s optimal policy would side-effect
that feature. Such policies are dominating policies.Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4868
Algorithm DomPolicies
1:0 ; .the initial set of dominating policies
2:0
rel ; .the initial set of relevant features
3:checked ;.It contains 0
relwe have examined
so far.
4: ; .a pruning rule
5:agenda powerset (0
rel)nchecked
6:whileagenda6=;do
7:  least-cardinality element of agenda
8: ifsatisfy (;)then
9: (get safely-optimal policy with locked)
10: arg max02V0.by solving Eq. 1
11: ifexists then
12: 0 0[fg; add (;rel())to
13: elseadd(;;)to
14: 0
rel 0
rel[rel()
15:checked checked[fg
16:agenda powerset (0
rel)nchecked
17:return 0,0
rel
Adominating policy is a safely-optimal policy for the cir-
cumstance where the unknown features A
?are partitioned
into locked and changeable subsets. We denote the set of
dominating policies by
 =n
arg max
2V:8A
?o
; (2)
where is the set of policies that do not change unknown
features ?as well as any locked features (meaning that
F[(?n)arechangeable). We denote the unknown-
features side-effected by policy byrel(). For a set of
policies ,rel() =[2rel(). The set rel(), ab-
breviated rel, is thus the set of relevant (unknown-) fea-
tures to consider querying about.
Instead of ﬁnding the safely-optimal policies for all expo-
nentially (injA
?j) many subsets A
?with Equation 2,
we contribute Algorithm DomPolicies (see pseudocode) that
ﬁnds dominating policies incrementally (and in practice more
efﬁciently) by constructing the sets of relevant features and
dominating policies simultaneously. In each iteration, it ex-
amines a new subset of relevant features, , and, if isn’t
pruned (as described later), ﬁnds the safely-optimal policy
withbeing locked (Line 7). It then adds rel(), the fea-
tures changed by , to0
rel. It repeats this process until 0
rel
stops growing and all subsets of 0
relare examined. For ex-
ample, in the navigation problem (Figure 1), the ﬁrst added
policy is the safely-optimal policy assuming no unknown fea-
tures are locked, which is 1. Now 0=f1gand thus
0
rel=fcarpetg. It iterates, treating the carpet as locked,
and updates 0tof1;2gand thus 0
rel=fcarpet;b 1;d2g.
Iterating again, it locks subsets of 0
rel, ﬁnding3for subset
fcarpet;b 1g. After ﬁnding 4, it terminates.
In Line 10, the algorithm ﬁnds the constrained optimal pol-
icy (in our implementation using Eq. 1), and in the worst case,
would do this 2jreljtimes. Fortunately, the complexity is ex-
ponential in the number of relevant features, which we have
seen in our empirical settings can be considerably smaller
(a)
(b)
c1c1-1
c2cn
c2cn
-1-1… … …
(c)c1c3
-1
c2-1
-0.1-1-1Figure 2: Example domains used in text. (n > 2)
than the number of unknown features (jA
?j). Furthermore,
the efﬁciency can be improved with our pruning rule (line 8):
satisfy (;) :=:9(L;R)2(L^\R=;)
is a history of ordered pairs, (L;R), of disjoint sets of un-
known features. Before DomPolicies computes a policy for
its agenda element , if a pair (L;R)is in, such thatL
(a dominating policy has been found when locking subset of
features in ), and that dominating policy’s relevant features
Rdon’t intersect with , then ’s dominating policy has al-
ready been found. In our running example, for instance, ini-
tially 0
rel=;.1is the optimal policy and the algorithm
adds pair (;;fcarpetg)to. When larger subsets are later
considered (note the agenda is sorted by cardinality), feature
sets that do not contain carpet are pruned by . In this exam-
ple,prunes 11 of the 16 subsets of the 4 relevant features.
Consider the examples in Figure 2. To reach the switch, the
agent could traverse zero or more carpets (all with unknown
changeability), and rewards are marked on the edges. (a) and
(b) only need to compute policies linear in the number of rel-
evant features: Only n+1dominating safely-optimal policies
are computed for Figure 2(a) (for  =;;fc1g;fc1;c2g;::: ),
and Figure 2(b) (for  =;;fc1g;fc2g;:::;fcng). Fig-
ure 2(c) computes policies for only half of the subsets.
Theorem 1. The set of policies returned by Algorithm Dom-
Policies is the set of all dominating policies.
Proof. Let2be the optimal policy with unknown fea-
turesLlocked. 0is returned by DomPolicies. We denote
L\rel(0)asAandLnrel(0)asB. For a proof by con-
tradiction, assume 620. ThenB6=;. Otherwise, if B=;
(or equivalently, A=L), thenLis a subset of rel(0)and
would have been added to 0. Let0be the optimal policy
withAlocked. Since Arel(0), we know020. We
observe that 0does not change any features in B(otherwise
features inBwould show up in rel(0)). So0is also the
optimal policy with features A[B=Llocked. So=0,
which is an element of 0.
4 Finding Minimax-Regret Queries
With DomPolicies, the agent need only query the user about
relevant features. But it could further reduce the user’s burden
by being selective about which relevant features it asks about.
In our running example (Figure 1), for instance, DomPolicies
removes b3 and d3 from consideration, but also intuitively
the agent should only ask about b1 or b2 if d2 is change-
able. By iteratively querying, accounting for such dependen-
cies (which can be gleaned from inDomPolicies) and up-
dating the relevant feature set, the agent can stop querying asProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4869
soon as it ﬁnds (if one exists) a safe policy. For example, say
it asks about the carpet and d2, and is told d2 (but not the
carpet) is changeable. Now it has a safely-optimal policy, 4,
given its knowledge, and could stop querying. But 4is the
worst safe policy. Should it ask about boxes?
That is the question we focus on here: How should an agent
query to try to ﬁnd a better safely-optimal policy than the one
it already knows about. Speciﬁcally, we consider the setting
where the agent is permitted to interrupt the user just once
to improve upon its safely-optimal policy, by asking a single
query about at most kunknown features. For each feature
the user will reply whether or not it is in U
F. Formally, q
is ak-feature query where qrelandjqj=k. The
post-response utility when the agent asks query q, and c
A
?are actually changeable, is the value of the safely-optimal
policy after the user’s response:
u(q;c) = max
2A
?n(q\c)V: (3)
Recall the agent can only safely change features it queries
about that the user’s response indicates are changeable ( q\
c). What would be the agent’s regret if it asks a k-feature
query qrather than a k-feature query q0? We consider
the circumstance where a set of features care changeable
and under which the difference between the utilities of asking
qandq0is maximized. We call this difference of utilities
thepairwise maximum regret of queries qandq0, deﬁned
below in a similar way to Regan and Boutilier [2010]:
PMR (q;q0) = max
cA
?(u(q0;c)u(q;c)):(4)
Themaximum regret, denoted by MR, of query qis de-
termined by the q0that maximizes PMR (q;q0):
MR(q) = max
q0rel;jq0j=kPMR (q;q0): (5)
The agent should ask the minimax-regret (k -feature) query:
MMR
q = arg min
qrel;jqj=kMR(q): (6)
The rationale of the minimax-regret criterion is as follows.
Whenever the agent considers a query q, there could exist a
query q0that is better than qunder some true changeable
features c. The agent focuses on the worst case c, where
the difference between the utility of qand the best query q0
that could be asked is maximized. The agent uses adversarial
reasoning to efﬁciently ﬁnd the worst-case c: Given that it
is considering query q, it asks what query q0and set of
features canimaginary adversary would pick to maximize
the gap between the utilities of q0andqunder c(that is,
u(q0;c)u(q;c)). The agent wants to ﬁnd a query q
that minimizes the worst case (maximum gap).
Under the deﬁnition of MR, the agent, reasoning as if it
were its imaginary adversary, must ﬁnd the maximizing q0
andc. However, we can simplify this so that it only needs
to ﬁnd the maximizing c. Note that since an imaginary ad-
versary chooses both q0andc, it wants to make sure that
cq0, which means that it does not want the features not
inq0to be changeable. We can then observe that
MR(q) = max
02:
rel(0)k(V0 max
2A
?nfq\rel(0)gV)(7)We call the0maximizing Eq. 7 the adversarial policy when
the agent asks query q, denoted by MR
q. With Eq. 7, the
agent can compute MR based on the set of dominating poli-
cies (which DomPolicies already found), rather than the (gen-
erally much larger) powerset of the relevant features in Eq. 5.
While using Eq. 7 is faster than Eq. 5, the agent still needs
to do this computation for every possible query (Eq. 6) of size
k. We contribute two further ways to improve the efﬁciency.
First, we may not need to consider all relevant features if we
can only ask about kof them. If a subset of relevant features
satisﬁes the condition in Theorem 2, then we call it a set of
sufﬁcient features, because a minimax-regret k-feature query
from that set is a globally minimax-regret k-feature query.
Second, we introduce a pruning rule that we call query dom-
inance in Theorem 3 to safely eliminate considering queries
that cannot be better than ones already evaluated.
The following theorem shows that if we can ﬁnd any subset
ofrelsuch that for all k-feature subsets of as queries,
the associated adversarial policy’s relevant features are con-
tained in , then the minimax regret query found by restrict-
ing queries to be subsets of will also be a minimax regret
query found by considering all queries in rel. Such a (non-
unique) set will be referred to as a sufﬁcient feature set (for
the purpose of ﬁnding minimax regret queries).
Theorem 2. (Sufﬁcient Feature Set) For any set of
kfeatures , if for all q,jqj=k, we
have rel(MR
q), then minq;jqj=kMR(q) =
minqrel;jqj=kMR(q).
Proof Sketch. If a set of features ,jjk, fails to include
some features in MMR
q , when we query some k-subset of
, the adversarial policy should change some of the features
inMMR
qn. Otherwise, querying about MMR
qndoes
not reduce the maximum regret. Then MMR
qnare not
necessary to be included in MMR
q .
Given a set of sufﬁcient features, the following theorem
shows that it may not be necessary to compute the maximum
regrets for all k-subsets to ﬁnd the minimax-regret query.
Theorem 3. (Query Dominance) For any pair of queries
qandq0, ifq0\rel(MR
q)q\rel(MR
q), then
MR(q0)MR(q).
Proof. Observe that MR(q0)
VMR
qmax02A
?n(q0\rel(MR
q))V0
VMR
qmax02A
?n(q\rel(MR
q))V0=MR(q)
We denote the condition q0\rel(MR
q)q\
rel(MR
q)bydominance (q;q0). To compute domi-
nance, we only need to store rel(MR
q)for all qwe have
considered.
Algorithm MMRQ-k below provides pseudocode for ﬁnd-
ing a minimax-regret k-feature query; it takes advantage of
both the notion of a sufﬁcient-feature-set as well as query
dominance to reduce computation signiﬁcantly relative to the
brute-force approach of searching over all k-feature queries
(subsets) of the relevant feature set.Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4870
Algorithm MMRQ-k
1:q an initialk-feature query
2:checked ;;evaluated ;
3:suf q
4:agenda fqg
5:whileagenda6=;do
6: q an element from agenda
7: if:9q02evaluateddominance (q0;q)then
8: ComputeMR(q)andMR
q
9: suf suf[rel(MR
q)
10:evaluated evaluated[fqg
11:checked checked[fqg
12:agenda allksubsetsof(suf)nchecked
13:return arg min q2evaluatedMR(q)
Intuitively, the algorithm keeps augmenting the set of fea-
tures suf, which contain the features in the queries we have
considered and the features changed by their adversarial poli-
cies, until it becomes a sufﬁcient feature set. agenda keeps
track ofk-subsets in sufthat we have not yet evaluated. Ac-
cording to Theorem 2, we can terminate the algorithm when
agenda is empty (Line 5). We also use Theorem 3 to ﬁlter out
queries which we know are not better than the ones we have
found already (Line 7). Note that an initial sufneeds to be
chosen, which can be arbitrary. Our implementation initial-
izesqwith the Chain of Adversaries heuristic (Section 5).
To illustrate when Algorithm MMRQ-k can and can’t prune
suboptimal queries and thus gain efﬁciency, consider ﬁnd-
ing the minimax-regret 2-feature query in Figure 2 (a), which
should befc1;c2g. If the agent considers a query that does
not includec1, the adversarial policy would change c1, adding
c1tosuf. If the agent considers a query that includes c1but
notc2, the adversarial policy would change c2, addingc2to
suf. When the agent asks fc1;c2g, the adversarial policy
changesc3(adversarially asserting that c1;c2are locked), so
c3is added to suf. Withfc1;c2;c3gsuf, the condition
in Theorem 2 holds and the n3other features can be safely
ignored. The minimax-regret query constituted by features in
sufisfc1;c2g. However, in Figure 2 (b), suf= rel, and
alljrelj
2
would be evaluated.
5 Empirical Evaluations
We now empirically conﬁrm that Algorithm MMRQ-k ﬁnds a
minimax-regret query, and its theoretically sound Sufﬁcient-
Feature-Set and Query-Dominance based improvements can
indeed pay computational dividends. We also compare our
MMRQ-k algorithm to baseline approaches and the Chain of
Adversaries (CoA) heuristic [Viappiani and Boutilier, 2009 ]
adapted to our setting. Algorithm CoA begins with q0=;
and improves this query by iteratively computing:
~ arg max
02:jrel(0)[qijk(V0 max
2A
?nfqi\rel(0)gV)
qi+1 qi[rel(~):
The algorithm stops when jqi+1j=korqi+1= qi. Al-
though Algorithm CoA greedily adds features to the queryto reduce the maximum regret, unlike MMRQ-k it does not
guarantee ﬁnding the minimax-regret query. For example, in
Figure 2(c), when k= 2, CoA ﬁrst ﬁnds the optimal policy,
which changesfc1;c2g, and returns that as a query, while the
minimax-regret query is fc1;c3g.
We compare the following algorithms in this section:
1. Brute force (rel. feat.) uses Algorithm DomPolicies to
ﬁnd all relevant features ﬁrst and evaluates all k-subsets of
the relevant features. 2. Algorithm MMRQ-k. 3. Algo-
rithm CoA. 4. Random queries (rel. feat.), which contain
kuniformly randomly chosen relevant features. 5. Ran-
dom queries, which contain kuniformly randomly chosen
unknown features, without computing relevant features ﬁrst.
6. No queries (equivalently vacuous queries).
We evaluate the algorithms’ computation times and the
quality of the queries they ﬁnd, reported as the normal-
izedMR to capture their relative performance compared
to the best and the worst possible queries. That is, the
normalized MR of a query qis deﬁned as (MR(q)
MR(MMR
q ))=(MR(q?)MR(MMR
q )), where q?is
a vacuous query, containing kfeatures that are irrelevant
and/or already known. The normalized MR of a minimax-
regret query is 0 and that of a vacuous query is 1.
Navigation. As illustrated in Figure 3, the robot starts from
the left-bottom corner and is tasked to turn off a switch at the
top-right corner. The size of the domain is 66. The robot
can move one step north, east or northeast at each time step. It
stays in place if it tries to move across a border. The discount
factor is 1. Initially, 10 clean carpets are uniformly randomly
placed in the domain (the blue cells). In any cell without a
carpet, the reward is set to be uniformly random in [1;0],
and in a cell with a carpet the reward is 0. Hence, the robot
will generally prefer to walk on a carpet rather than around
it. The state of each carpet corresponds to one feature. The
robot is uncertain about whether the user cares about whether
any particular carpet gets dirty, so all carpet features are in
A
?. The robot knows that its own location and the state of
the switch are in A
F. Since MMRQ-k attempts to improve
on an existing safe policy, the left column and the top row
never have carpets to ensure there is at least one safe path to
the switch (the dotted line). The robot can ask one k-feature
query before it takes any physical actions. We report results
on 1500 trials. The only difference between trials are the lo-
cations of carpets, which are uniformly randomly placed.
First, we compare the brute force method to our MMRQ-k
algorithm (Figure 4). We empirically conﬁrm that in all cases
MMRQ-k ﬁnds a minimax regret query, matching Brute Force
performance. We also see that brute force scales poorly as k
grows while MMRQ-k , beneﬁting from Theorems 2 and 3, is
more computationally efﬁcient.
We then want to see if and when MMRQ-k outperforms
other candidate algorithms. In Figure 4, when kis small,
the greedy choice of CoA can often ﬁnd the best features to
add to the small query, but as kincreases, CoA suffers from
being too greedy. When kis large (approaches the number
of unknown features), being selective is less important and
all methods ﬁnd good queries. We also consider how jrelj
affects the performance (Figure 5). When jreljis smallerProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4871
Brute Force (rel. feat.)
MMRQ-k
CoA
Random (rel. feat.)
Random
No QueryFigure 3: Ofﬁce navigation and legend for following ﬁgures.
5 10
k0.00.51.0Normalized MR
|Φ?| = 10
2 5 8 10
k0204060Computation Time (sec.)
|Φ?| = 10
Figure 4: Normalized maximum MR vs.k.j?j= 10 . Brute force
computation time is only shown for k= 1;2;3.
thank, ak-feature query that contains all relevant features
is optimal. All algorithms would ﬁnd an optimal query ex-
cept Random (which selects from all unknown features) and
No Queries. (The error bars are larger for small jreljsince
more rarely are only very few features relevant.) When jrelj
is slightly larger than k, even a random query may be luckily
a minimax-regret query, and CoA unsurprisingly ﬁnds queries
close to the minimax-regret queries. However, when jreljis
much larger than k, the gap between MMRQ-k and other al-
gorithms is larger. In summary, MMRQ-k ’s beneﬁts increase
with the opportunities to be selective (largerjrelj
k
). In Fig-
ure 5, the gap is largest when k= 4;jrelj= 10 .
We have also experimented with expected regret given a
probabilistic model of how the user will answer queries. For
example, if the user has probability pof saying an unknown
feature is free ( 1pit’s locked), then as expected, when p
is very low, querying rarely helps, so using MMRQ-k orCoA
matters little, and as pnears 1, CoA ’s greedy optimism pays
off to meet MMRQ-k ’s minimax approach. But, empirically,
MMRQ-k outperforms CoA for non-extreme values of p.
6 Extensions and Scalability
We now brieﬂy consider applying our algorithms to larger,
more complicated problems. As mentioned in Section 2, fea-
tures’ changeabilities might be more nuanced, with restric-
tions on what values they can take, individually or in combi-
nation. An example of the latter from Fig. 1 is where doors
arerevertible features, which means their changeability is de-
pendent on the “time” feature: they are freely changeable ex-
cept that by the time the episode ends they need to revert to
their initial values. This expands the set of possible feature
queries (e.g., asking if d2 is freely changeable differs from
asking if it is revertible). In our experiments, this change ac-
centuates the advantages of MMRQ-k over CoA: CoA asks (is-
d2-locked, is-d2-revertible), hoping to hear “no” to both and
follow a policy going through d2 without closing it. MMRQ-
kasks (is-d2-locked, is-carpet-locked): since closing d2 only
takes an extra time step, it is more valuable to know if the
carpet is locked than if d2 can be left open.
More nuanced feature changeability means DomPolicies
will have to ﬁnd policies for the powerset of every relevant
combination of features’ values (in the worst case the size of
the state space). One option is to ignore nuances in ways that
2 5 8 10
|Φrel|0.00.51.0Normalized MR
k= 2
2 5 8 10
|Φrel|0.00.51.0Normalized MR
k= 4Figure 5: Normalized MR vs. the number of relevant features.
maintain safety (e.g., treat a feature as revertible even if some-
times it can be left changed) and solve such a safe abstraction
of the problem. Or one could abstract based on guaranteed
correlations in feature changeabilities (e.g., if all boxes have
the same changeability then ignore asking about all but one).
Another option is to ﬁnd only a subset of dominating policies,
for example by using knowlege of kto avoid ﬁnding dom-
inating policies that would change more unknown features
than could be asked about anyway. And, of course, as men-
tioned before, ﬁnding approximately-safely-optimal policies
inDomPolicies Line 10 would help speed the process (and
might be the only option for larger problem domains).
Fortunately, such abstractions, heuristics, and approxima-
tions do not undermine safety guarantees. Recall that Dom-
Policies andMMRQ-k are ﬁnding a query, not the agent’s ﬁ-
nal policy: the safety of the agent depends on how it ﬁnds the
policy it executes, not on the safety of policies for hypothet-
ical changeability conditions. However, as coarser abstrac-
tions, heuristics, and approximations are employed in our
algorithms, the queries found can increasingly deviate from
the minimax-regret optima. Fortunately, if the agent begins
with a safely-optimal policy, “quick and dirty” versions of
our methods can never harm it (they just become less likely
to help). And if it begins without such a policy, such versions
of our methods might not guide querying well, but by eventu-
ally asking about every unknown feature (in the worst case) a
safe policy will still be found (if one exists).
7 Related Work & Summary
Amodei et al. [2016] address the problem of avoiding nega-
tive side-effects by penalizing all side-effects while optimiz-
ing the value. In our work, we allow the agent to commu-
nicate with the user. Safety is also formulated as resolving
reward uncertainty [Amin et al., 2017; Hadﬁeld-Menell et
al., 2017 ], following imperfectly-speciﬁed instructions [Milli
et al., 2017 ], and learning safe states [Laskey et al., 2016 ].
Safety issues also appear in exploration [Hans et al., 2008;
Moldovan and Abbeel, 2012; Achiam et al., 2017 ]. Here
we only provide a brief survey on safety in MDPs. Leike et
al.[2017], Amodei et al. [2016], and Garcıa and Fern ´andez
[2015] provide more thorough surveys.
There are problems similar to ﬁnding safely-optimal
policies, which ﬁnd policies that satisfy some con-
straints/commitments or maximize the probability of reach-
ing a goal state [Witwicki and Durfee, 2010; Teichteil-
K¨onigsbuch, 2012; Kolobov et al., 2012 ]. There are also
other works using minimax-regret and policy dominance [Re-
gan and Boutilier, 2010; Nilim and El Ghaoui, 2005 ]. and
querying to resolve uncertainty [Weng and Zanuttini, 2013;
Regan and Boutilier, 2009; Cohn et al., 2011; Zhang et al.,Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4872
2017 ]. We’ve combined and customized a number of these
ideas to ﬁnd a provably minimax-regret k-element query.
In summary, we addressed the problem of an agent selec-
tively querying a user about what features can be safely side-
effected. We borrowed existing ideas from the literature about
dominating policies and minimax regret, wove them together
in a novel way, and streamlined the resulting algorithms to
improve scalability while maintaining safe optimality.
Acknowledgements
Thanks to the anonymous reviewers. Supported in part by
the US Air Force Ofﬁce of Scientiﬁc Research, under grant
FA9550-15-1-0039, and the Open Philanthropy Project to the
Center for Human-Compatible AI.
References
[Achiam et al., 2017 ]Joshua Achiam, David Held, Aviv
Tamar, and Pieter Abbeel. Constrained policy optimiza-
tion. In Proc. Int. Conf. on Machine Learning (ICML),
pages 22–31, 2017.
[Amin et al., 2017 ]Kareem Amin, Nan Jiang, and Satinder
Singh. Repeated inverse reinforcement learning. In Adv.
in Neural Info. Proc. Sys. (NIPS), pages 1813–1822, 2017.
[Amodei et al., 2016 ]Dario Amodei, Chris Olah, Jacob
Steinhardt, Paul Christiano, John Schulman, and Dan
Man ´e. Concrete problems in AI safety. arXiv preprint
arXiv:1606.06565, 2016.
[Boutilier et al., 1999 ]Craig Boutilier, Thomas Dean, and
Steve Hanks. Decision-theoretic planning: Structural as-
sumptions and computational leverage. J. of Artiﬁcial In-
telligence Research (JAIR), 11(1):94, 1999.
[Cohn et al., 2011 ]Robert Cohn, Edmund Durfee, and
Satinder Singh. Comparing action-query strategies in
semi-autonomous agents. In Int. Conf. on Autonomous
Agents and Multiagent Systems, pages 1287–1288, 2011.
[Dolgov and Durfee, 2006 ]Dmitri A. Dolgov and Ed-
mund H. Durfee. Symmetric approximate linear program-
ming for factored MDPs with application to constrained
problems. Annals of Mathematics and Artiﬁcial Intelli-
gence, 47(3):273–293, Aug 2006.
[Garcıa and Fern ´andez, 2015 ]Javier Garcıa and Fernando
Fern ´andez. A comprehensive survey on safe reinforce-
ment learning. J. of Machine Learning Research (JMLR),
16(1):1437–1480, 2015.
[Hadﬁeld-Menell et al., 2017 ]Dylan Hadﬁeld-Menell,
Smitha Milli, Stuart J Russell, Pieter Abbeel, and Anca
Dragan. Inverse reward design. In Adv. in Neural Info.
Processing Systems (NIPS), pages 6749–6758, 2017.
[Hans et al., 2008 ]Alexander Hans, Daniel Schneegaß, An-
ton Maximilian Sch ¨afer, and Steffen Udluft. Safe explo-
ration for reinforcement learning. In Euro. Symp. on Arti-
ﬁcial Neural Networks (ESANN), pages 143–148, 2008.
[Kolobov et al., 2012 ]Andrey Kolobov, Mausam, and
Daniel S. Weld. A theory of goal-oriented MDPs withdead ends. In Proc. Conf. on Uncertainty in Artiﬁcial
Intelligence (UAI), pages 438–447, 2012.
[Laskey et al., 2016 ]Michael Laskey, Sam Staszak, Wesley
Yu-Shu Hsieh, Jeffrey Mahler, Florian T Pokorny, Anca D
Dragan, and Ken Goldberg. SHIV: Reducing supervisor
burden in DAgger using support vectors for efﬁcient learn-
ing from demonstrations in high dimensional state spaces.
InIEEE Int. Conf. on Robotics and Automation (ICRA),
pages 462–469, 2016.
[Leike et al., 2017 ]Jan Leike, Miljan Martic, Victoria
Krakovna, Pedro A Ortega, Tom Everitt, Andrew
Lefrancq, Laurent Orseau, and Shane Legg. AI safety grid-
worlds. arXiv preprint arXiv:1711.09883, 2017.
[Milli et al., 2017 ]Smitha Milli, Dylan Hadﬁeld-Menell,
Anca D. Dragan, and Stuart J. Russell. Should robots be
obedient? In Proc. Int. Joint Conf. on Artiﬁcial Intelli-
gence (IJCAI), pages 4754–4760, 2017.
[Moldovan and Abbeel, 2012 ]Teodor M. Moldovan and
Pieter Abbeel. Safe exploration in Markov decision pro-
cesses. In Proc. Int. Conf. on Machine Learning (ICML),
pages 1711–1718, 2012.
[Nilim and El Ghaoui, 2005 ]Arnab Nilim and Laurent
El Ghaoui. Robust control of Markov decision processes
with uncertain transition matrices. Operations Research,
53(5):780–798, 2005.
[Regan and Boutilier, 2009 ]Kevin Regan and Craig
Boutilier. Regret-based reward elicitation for Markov
decision processes. In Proc. Conf. on Uncertainty in
Artiﬁcial Intelligence (UAI), pages 444–451, 2009.
[Regan and Boutilier, 2010 ]Kevin Regan and Craig
Boutilier. Robust policy computation in reward-uncertain
MDPs using nondominated policies. In Assoc. for Adv. of
Artiﬁcial Intelligence (AAAI), pages 1127–1133, 2010.
[Teichteil-K ¨onigsbuch, 2012 ]Florent Teichteil-K ¨onigsbuch.
Stochastic safest and shortest path problems. In Proc. As-
soc. for Adv. of Artiﬁcial Intelligence (AAAI), pages 1825–
1831, 2012.
[Viappiani and Boutilier, 2009 ]Paolo Viappiani and Craig
Boutilier. Regret-based optimal recommendation sets in
conversational recommender systems. In Proc. ACM Conf.
on Recommender Systems, pages 101–108, 2009.
[Weng and Zanuttini, 2013 ]Paul Weng and Bruno Zanuttini.
Interactive value iteration for Markov decision processes
with unknown rewards. In Proc. Int. Joint Conf. on Artiﬁ-
cial Intelligence (IJCAI), pages 2415–2421, 2013.
[Witwicki and Durfee, 2010 ]Stefan J Witwicki and Ed-
mund H Durfee. Inﬂuence-based policy abstraction for
weakly-coupled Dec-POMDPs. In Proc. Int. Conf. Auto.
Planning and Scheduling (ICAPS), pages 185–192, 2010.
[Zhang et al., 2017 ]Shun Zhang, Edmund Durfee, and
Satinder Singh. Approximately-optimal queries for plan-
ning in reward-uncertain Markov decision processes. In
Proc. Int. Conf. on Automated Planning and Scheduling
(ICAPS), pages 339–347, 2017.Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
4873

Shun Zhang
Edmund H. Durfee
Satinder Singh