Multitasking: Efﬁcient Optimal Planning for Bandit Superprocesses
Dylan Hadﬁeld-Menell and Stuart Russell
Abstract
A bandit superprocess is a decision problem
composed from multiple independent Markov
decision processes (MDPs), coupled only by the
constraint that, at each time step, the agent may
act in only one of the MDPs. Multitasking prob-
lems of this kind are ubiquitous in the real world,
yet very little is known about them from a com-
putational viewpoint, beyond the observation that
optimal policies for the superprocess may pre-
scribe actions that would be suboptimal for an
MDP considered in isolation. (This observation
implies that many applications of sequential de-
cision analysis in practice are technically incor-
rect, since the decision problem being solved
is often part of a larger, unstated bandit super-
process.) The paper summarizes the state-of-the-
art in the theory of bandit superprocesses and
contributes a novel upper bound on the global
value function of a bandit superprocess, deﬁned
in terms of a direct relaxation of the arms. The
bound is equivalent to an existing bound (the
Whittle integral), but is deﬁned constructively,
as the value of a related multi-armed bandit. We
provide a new method to compute this bound and
derive the ﬁrst practical algorithm to select opti-
mal actions in bandit superprocesses. The algo-
rithm operates by repeatedly establishing domi-
nance relations between actions using upper and
lower bounds on action values. Experiments indi-
cate that the algorithm’s run-time compares very
favorably to other possible algorithms designed
for more general factored MDPs.
1 INTRODUCTION
Multitasking is no doubt an activity familiar to the reader:
one faces several decision problems but can act on only
one (or perhaps a bounded number) at a time. Such prob-lems are ubiquitous for individuals, corporations, armies,
and governments.
Multitasking problems are expressed by the class of ban-
dit superprocesses [Nash, 1973] or BSPs. A k-armed BSP
Mconsists ofkindependent Markov decision processes
M1;:::;Mk; the MDPs are coupled by a common discount
factor and by the constraint that at each time step the agent
can act in only one MDP.
Since the MDPs are independent of each other, one might
imagine that the optimal policy for Mis obtained by
solving each MDP—turning the BSP into a multi-armed
bandit—and then using Gittins indices to choose a se-
quence of arms. In fact, the optimal policy for a BSP may
include actions that are suboptimal from the point of view
of the constituent MDP in which they are taken . The reason
for this is that the availability of other MDPs in which to
act changes the balance between short-term and long-term
rewards in a component MDP; in fact, it tends to lead to
greedier behavior in each MDP because aiming for long-
term reward in one MDP would delay rewards in all the
other MDPs. The globally and locally optimal policies nec-
essarily coincide only when the discount factor is 1.
Hence, in addition to their general importance in the real
world, a second reason to study multitasking problems is
that they undermine an assumption implicit in practical ap-
plications of sequential decision analysis: the assumption
that an optimal solution for the user’s decision problem is
optimal for a user who faces multiple decision problems.
Despite these considerations, there has been remarkably lit-
tle research on bandit superprocesses and almost none in
AI. Section 2 summarizes what is known. Obviously, there
are connections to multi-armed bandits (MABs), which are
a special case of BSPs in which each arm only allows
one choice of action rather than several. Unfortunately,
the index theorems that simplify the computation of opti-
mal MAB policies are not valid for BSPs. There are also
strong connections—hitherto unexplored—between BSPs
andsums of games as studied in combinatorial game the-
ory [Conway, 2000]. The principal question we address in
this paper is whether an algorithm exists that is substan-
tially more efﬁcient than applying a standard MDP solver
to the “cross-product” MDP obtained by combining the
states spaces of the constituent MDPs.
The sole known case where planning for a BSP is provably
more efﬁcient is the case where a dominating policy exists:
if a single policy is optimal across the family of retirement
processes associated with each arm then a BSP can be re-
duced to an equivalent MAB [Whittle, 1980]. (Retirement
processes provide a measure of how the optimal policy de-
pends on context and are deﬁned in Section 3.) However,
this condition is seldom satisﬁed in practice.
This paper provides three contributions. First, we give a
concise survey of the bandit superprocess literature tailored
to the AI community. Second, we provide a novel upper
bound on the global value function of a BSP. For a BSP,
M, with armsfM1;:::;Mkgwe relax each arm to obtain
an MABM0=fM0
1;:::;M0
kgby adding actions to en-
sure that dominating policies exists. We show that this up-
per bound is equivalent to the Whittle integral , an existing
upper bound for BSPs [Brown and Smith, 2013]. Because
our bound is deﬁned in terms of an explicit relaxation, it
provides insight into the nature of the bound and opens
an avenue to extend this work to more general MDPs. Fi-
nally, we describe a practical computational approach for
solving BSPs: we derive a simple method for computing
the Whittle integral upper bound that can use an arbitrary
MDP solver as a black box; then we combine this upper
bound with a lower bound to derive an efﬁcient algorithm
for-optimal decision making in a BSP. We present em-
pirical results to show that it substantially improves over
more general optimal factored MDP algorithms; we use it
to compute provably optimal actions for problems with up-
wards of 1030states in the full state space.
2 RELATED WORK
Robbins [1952] provided the ﬁrst formulation of multi-
armed bandits (MABs) in their modern form. The famous
Gittins index theorem [Gittins, 1979] showed that optimal
MAB policies are obtained by ranking the arms according
to an index function deﬁned on each seperately. An imme-
diate corollary is that optimal decision making in an MAB
islinear in the number of arms. Problems with this prop-
erty are said to be indexable .
Bandit superprocesses (BSPs) were introduced by Nash
[1973] as a generalization of multi-armed bandits to study
allocation of resources among research projects. Whittle
[1980] provided an alternate proof of the Gittins index the-
orem that extends to bandit superprocesses with dominat-
ing policies. His proof utilized a construction called the
Whittle integral , which allows one to compute the value
of a composite state in an MAB. Glazebrook [1982] pro-
vides a proof that bandit superprocesses are notindexablein general. Glazebrook [1993] considers bandit superpro-
cesses where the arms can exert limited inﬂuence on each
other and shows a result analogous to Whittle’s.
Brown and Smith [2013] were the ﬁrst to identify the sig-
niﬁgance of the Whittle integral for sequential decision
making. They developed a version of policy iteration to
compute it, and recognized that it upper bounds the value
function of a BSP,1but did not derive an algorithm for solv-
ing BSPs. Our work provides two further contributions re-
garding the Whittle integral: a short proof that it is an upper
bound and a simpler algorithm to compute it.
Within the AI community, there has been limited study
of loosely coupled Markov decision processes. Singh and
Cohn [1998] consider optimal solutions to simultaneous
MDPs , where an agent can take actions in a number of
MDPs. Their formulation is more general than ours, as it
is possible to act in multiple MDPs at once. They derive
bounds for this problem and give an algorithm that com-
bines a form of real-time dynamic programming with prun-
ing steps to remove provably suboptimal actions. However,
their bounds are substantially looser than ours, and our ex-
periments show that the corresponding algorithm can be
impractical for simple BSPs with many arms.
Meuleau et al. [1998] examine MDPs that are coupled by
constraints on the use of shared resources. BSPs ﬁt within
this class if we view the restriction to a single MDP at a
time as a constraint on the agent’s attention. Interestingly,
their heuristics are also deﬁned in terms of a parameter-
ized value function for the component MDPs—it would be
interesting to attempt to generalize the approaches consid-
ered here to their problem domain. In this work, we lever-
age the particular structure of our resource to compute op-
timal solutions; Meuleau et al. construct a heuristic policy.
3 TECHNICAL BACKGROUND
MARKOV DECISION PROCESSES
Deﬁnition 1. (Markov Decision Process [Puterman,
2009]) A (ﬁnite-state, discounted) MDP , M, is a tuple
M=hS;A;T;R;
i.Sis a set of states.Ais a set of
actions.T:SAS! [0;1]is a function that assigns
probability to state transitions for each state–action pair.
Ris a (bounded) reward function that maps state–action
pairs to (positive) rewards R:SA! R+.
2[0;1)is
a discount factor.
A solution to Mis a policy,, that maps states to actions.
The value of a state, s, underis the sum of expected dis-
counted rewards received by starting in sand selecting ac-
1This result ﬁrst appears in the literature as an intermediate
step in a larger proof from Whittle [1980]. It went unnoticed or
unappreciated in the intervening 30+ years.
tions according to :
V(s) =E"1X
t=0
tR(st)js0=s;#
:
The optimal policy, , maximizes this value. In the above
deﬁnition, and the ones that follow, we use superscripts to
indicate dependence on the agent’s policy. To simplify no-
tation, we will omit these superscripts when the policy re-
ferred to is the optimal policy (e.g., V(s) =V(s)). The
Q-function for the state–action pair, (s;a), is the value of
takingainsand selecting future actions according to .
RETIREMENT PROCESSES
Given an MDP, Whittle deﬁnes a family of optimal stop-
ping problems:
Deﬁnition 2. (Retirement Process [Whittle, 1980]) Let M
be an MDP . For 0, the retirement process for Mwith
retirement reward, , is an MDP , M, with a single addi-
tional state, sR, and action, aR.aRtransitions determinis-
tically tosRand receives reward .sRis a sink state that
accrues zero reward.
We refer to a decision to select aRas a decision to retire.
We denote the retirement process value function as a func-
tion of a state and retirement reward, V(s;). We let the op-
timal policy for retirement reward be
. We write the set
of states where the policy, , retires as
. We use()
and+()to denote the interval of retirement rewards so
thatis optimal:
02[+();+()])V(s;) =V(s;):
We adopt the convention from the MAB/BSP literature
and abuse notation to denote the (random) number of
steps prior to retirement as
(s). Fors02
we let
Pretire (s0js;; )be the probability that s0is the ﬁrst state
in
the agent will reach given that it is in state sand
executes policy . We denote the expected discounted re-
ward accrued prior to retirement, starting in s, asR
(s).
In regions of retirement reward where the optimal policy
and stopping rule do not change,@V
@(s;)is deﬁned and
is equal to the expected value of the discount parameter at
retirement. This allows us to write the following expression
for the retirement process value function:
V(s;) =R(s) +E[
(s)]: (1)
V(s;)is piecewise linear in . Figure 1 shows V(s;)for
the example BSP in Ex. 1. A policy that is optimal for every
setting ofis called a dominating policy .
Deﬁnition 3. (Dominating Policy) Let be a policy for an
MDP ,M.is adominating policy iff
808s =2(s) =
(s):MULTI-ARMED BANDITS AND BANDIT
SUPERPROCESSES
Multi-armed bandits (MAB) are a restricted class of MDPs
that have received extensive study. Of particular interest are
the form of the optimal policy and its utility in modelling
“exploration vs exploitation” trade-offs.
An MAB consists of a set of Markov reward processes
(MRPs), where an MRP is an MDP with a single action.
Each MRP is referred to as an armof the problem. At each
time-step, the agent selects an arm, that arm transitions ac-
cording to its transition distribution, and the agent receives
the corresponding reward. We adopt a summation notation
to indicate the combination of several MRPs into an MAB.
For example, if XandYare MRPs and Zis the MAB with
armsXandY, we will write Z=X+Y.
A famous result is that the optimal policy for any MAB
is an index policy [Gittins, 1979]. Each state, si, in the
individual arms is assigned an index, Isi. In joint state
s=fs1;:::;skg, the optimal action is to select
argmaxiIsi. For MAB arm Miin statesi, the index is de-
ﬁned as the value of retirement reward such that the agent
is indifferent between immediate retirement and following
the optimal stopping policy [Whittle, 1980]:
Isi= min
f;Vi(si;) =g:
This means that, in a sense, context for a multi-armed ban-
dit can always be summarized by a single number.
To model a multi-tasking problem, we consider a gen-
eralization of multi-armed bandits: bandit superprocesses
(BSPs). Bandit superprocesses allow arms that are arbitrary
MDPs (and so are a generalization of MABs); at each time
step, the agent selects both an arm and an action to take
within that arm.
Deﬁnition 4. (Bandit Superprocess [Nash, 1973]) Given k
MDPs,fMi=hSi;Ai;Ti;Ri;
ig, we deﬁne
M=X
iMi=hS;A; T; R;
i
to be the bandit superprocess with arms fMig.S=
iSi
andA=[
iAi:The transition distribution is station-
ary for arms that are not selected and follows the identical
reward and transition distributions for the selected arm.
Naturally, this makes planning more difﬁcult. To see how,
consider the following proposal:
Conjecture 1. LetX=hSX;AX;TX;RX;
ibe an
MDP . Deﬁne Ysimilarly and let Z=X+Ybe the
bandit superprocess that is their sum. Let a2 AX,
sX2 SXbe a state–action pair from X. Suppose that
this transition is suboptimal in every retirement process:
80; VX(sX; )> QX((sX; a); ):Thenais
suboptimal for any state in ZwithsXas one of the compo-
nents:
8sY2SY;VZ(fsX;sYg)>QZ(fsX;sYg;a):
This conjecture essentially proposes that state–action pairs
can be safely ignored if they are suboptimal for all settings
of a constant alternative. Unfortunately, as the following
example illustrates, this is not the case.
Example 1. DeﬁneXto be a deterministic reward chain
andYto be an initial choice between three reward chains
(y0;y1;y2) deﬁned as follows:
X= 28;28;28;28;28;28;0;0:::
Y=8
<
:y0= 100;100;100;0;0;0:::
y1= 99;99;99;1:4;1:4;1:4:::
y3= 28;28;28;:::
The retirement process value function for each reward
chain can be computed by simulating the Gittins index pol-
icy:2
V(y0;) = max(
100 2X
t=0
t!
+
3;)
V(y1;) = max(
99 2X
t=0
t!
+
3max1:4
1
;
;)
V(y2;) = max28
1
;
:
If we let
=:9, then we have
8280>0 maxfV(y0; ); V(y2; )g> V (y1; ):
Thus, any policy for a retirement process retirement process
derived from Ythat initially selects y1is suboptimal. We
can apply the same strategy to compute the value of each
combination of reward streams:
V(y0+X) = 100 2X
t=0
t!
+
3
285X
t0=0
t0!
V(y1+X) = 99 2X
t=0
t!
+
3
285X
t0=0
t0+
61:4
1
!
V(y2+X) =28
1
From this we can see that the optimal policy for the BSP
that combines these MDPs, Z=X+Y, initially collects
reward from y1. This contradicts Conjecture 1.
2Here we use the fact that the Gittins index of a non-increasing
reward sequence is equal to the instantaneous reward.4 AN UPPER BOUND FOR BANDIT
SUPERPROCESSES
In this section we show a bound on the value function of
a bandit superprocess. We derive this bound by adding ac-
tions to a BSP so that it is equivalent to an MAB. The value
function of our relaxed BSP is equivalent to the Whittle in-
tegral bound derived in Brown and Smith [2013] and so it
yields insight into that computation. We begin by deﬁning
the Whittle integral and show some basic results in order to
motivate our relaxation.
Deﬁnition 5. (Whittle Integral [Brown and Smith, 2013])
LetMbe a BSP . Let iindex the arms of M. For any state,
s=fsig, and0, the Whittle integral of sis deﬁned as
^V(s;) =IZI
x=dxY
i@Vi
@(si;x): (2)
WhereImaxiIsi.
When the arms of a BSP admit a dominating policy the
Whittle integral is equal to the value function:
Theorem 1. (Whittle Condition [Whittle, 1980]) Let Mbe
ak-armed BSP with components fMigand state spaceS.
If eachMihas a dominating policy, then
8s2S;80;^V(s;) =V(s;):
MRPs have a single action per state, so the Whittle con-
dition is trivially satisﬁed for all multi-armed bandits.
V(s;0) =V(s), so^V(s;0)provides an efﬁcient method to
compute the value of an MAB. For BSPs an arm that sat-
isﬁes the Whittle condition can be replaced with an MRP
that selects actions according to .
The formula in Eq. 2 lends itself to a straightforward im-
plementation, but is very challenging to interpret. We show
below that this is equivalent to evaluating the retirement
process value function for a single arm with a set of re-
wards determined by the other arms3. We refer to this set
of rewards as the critical points of those arms.
Deﬁnition 6. (Critical Points of an MDP) Let Mbe
a Markov decision process. The critical points of M,
C(M) =fig, are the values of retirement reward such
that the optimal stopping rule or policy changes.
These are points where there is a discontinuity in@V
@. We
let
M(s;) = lim
!0@VM
@(s;+)@VM
@(s;)
be the size of the corresponding discontinuities. This is
equivalent to the expected increase in E[
]under the new
stopping rule. Theorem 2 shows that andCcharacterize
the interaction between arms of an MAB.
3Our result is a small extension on a related result in Brown
and Smith [2013]; it is primarily included to provide intuition.
Theorem 2. LetX;Y be Markov reward processes. Let
Z=X+Ybe the 2-armed bandit. 8s=fsX;sYg2SZ;
VZ(s) =X
2C(Y)VX(sX;)Y(sY;): (3)
Proof. (sketch) As a ﬁrst step, we follow the steps in Whit-
tle [1980] and integrate Equation 2 by parts. This expresses
^Vas the following integral:
^VZ(s) =ZIsX
0=VX(sX;0)@2VY(sY;0)
@2d0:
VYis piecewise linear with respect to so@2VY
@2is a
weighted sum of delta functions centered at VY’s kinks.
C(Y)andYrespectively characterize these kinks and
weights. Thus, ^VZ(s)is equal to the rhs of Eq. 3. XandY
are MRPs, so an appeal to Theorem 1 shows the result.
A similar property holds for arbitrary k-armed bandits. This
shows that we can summarize the context for an arm as
a collection of weighted retirement rewards. Turning to
BSPs, this lends insight into the approximation the Whit-
tle integral introduces.
To see how, we reconsider Ex. 1. Recall that this example
consists of the BSP Z=X+Yand thatYis a choice
between three reward chains, fy0;y1;y2g. Theorem 2 al-
lows us to write the gap between the Whittle integral upper
bound and the value of selecting y1as
X
2C(X)
max
iVyi(si;)Vy1(s1;)
X(sX;)
Figure 1 shows the retirement process value functions for
this example. While the retirement process value of y1is
always less than that of either y0ory2, it is close enough
to their maximum that choosing y1essentially achieves
the upper bound. The Whittle integral for Zis a weighted
combination of distinct retirement process value functions
(Vy0;Vy2) but, in reality, the agent will be forced to pick a
single policy of the two.
DOMINATED RELAXATION OF AN MDP
In this section, we introduce our primary theoretical result:
a relaxation for the arms of a BSP so that a dominating pol-
icy exists. This reduces the BSP to an MAB whose value
upper bounds the value of states in the BSP. We can show
that the Whittle integral computes the value of states in this
MAB and arrive at a straightforward proof that the Whit-
tle integral is an upper bound. We call the result the dom-
inated relaxation of an MDP. Before providing the details,
we illustrate the main ideas with an example. The relaxed
MDP is actually a Semi-MDP (SMDP): a generalization of
an MDP where each action, a, has a duration, (a)2R+.Example 2. Let MDPMbe an initial choice between
MRPs:
M=15;0;0;0:::
10;10;10;3;3;3;::::
LetT(B)be the top (bottom) MRP . Let T(B)be the
policy that selects T(B)ins0. We can relax Mwith the ad-
dition of a single durative action, a0, that transitions from
the sink state in Tto the sink state B. We set(a0) = 2 .
We takesuch thatVT(s0;) =VB(s0;)and set the
rewards associated with a0to be
RB(s0)RT(s0) = 102X
0
t15 = 12:1:
With this change, at low settings of retirement reward, the
agent is indifferent between a policy that opts for the top
chain then selects a0and the bottom reward chain. For high
settings of retirement reward, the optimal policy retires im-
mediately or retires after collecting the reward of 15. This
SMDP satisﬁes the Whittle condition and can be replaced
by an MRP in any bandit superprocess. Fig. 1 (c) shows an
illustration of the state space and the introduced action.
In this example, we connected the state where Tretires to
the state where Bretires. This lets the agent collect short-
term and long-term rewards with the same policy. To do
this in general, we introduce multiple copies of the state
space, one for each policy that is optimal for some .
Deﬁnition 7. (Dominated Relaxation of an MDP) Let M
be an MDP with discount factor
and state spaceS. Let
sbe a state in M. The dominated relaxation of Mfors,
MD(s), is a semi-Markov decision process that ﬁxes sas
an initial state. Let figbe the policies that are optimal for
some:f
j2C(M)g. This sequence is ordered so that
(i)is increasing in i4.
For eachi, we introduce a copy of the state space, Si, where
the agent is restricted to following i. Lets0
ibe the ana-
logue ofs0inSi. Fors0
i2(i), we introduce a single
durative action, ai, that takes the agent from SitoSi1and
characterize it as follows:
R(s0
i) =R+(i1)(s)R(i)(s)
T(s0
i;ai;s00
i1) =Pretire (s00ji1;+(i1);s)
(ai) = log
Eh

i1
+(s)i
log
Eh

i(s)i
For eachi, we introduce an action that transitions from sto
siwith= 0. LetVD(s)represent the value of sinMD(s).
Theorem 3. LetMbe an MDP with state space S. The
following statements are true for s2S and0:
4Recall that [();+()]is the interval of retirement re-
wards where is optimal.
1.MD(s)satisﬁes the Whittle condition.
2.VD(s;) =^V(s;):
3.^V(s;)V(s;)
Proof. See supplementary materials
5 AN -OPTIMAL ALGORITHM FOR
BANDIT SUPERPROCESSES
In this section, we present two algorithms related to BSPs.
The ﬁrst is a novel algorithm to compute V(s;)that can
use any method to solve the underlying MDP. The second is
an efﬁcient algorithm to compute optimal actions for a BSP.
Our approach, Branch-and-Bound Value Iteration (BBVI),
uses Whittle integrals to compute upper and lower bounds
on value. Then, we apply a modiﬁed Branch-and-Bound
search to ﬁnd provably optimal actions.
COMPUTING A RETIREMENT PROCESS V ALUE
FUNCTION
Before we present BBVI, we give a simple algorithm to
compute a retirement process value function that uses an
(arbitrary) MDP solver as a black box. Brown and Smith
[2013] describe an algorithm to compute retirement pro-
cess value functions, but their approach requires custom
implementation of a modiﬁed simplex algorithm. Our ap-
proach is simple and based on an algorithm that initially
appeared in the solutions manual for Russell and Norvig
[2010].
Our goal is to identify each component of V(;), so our
output will be a list of critical points and the associated
slopes. First, we will need the following result.
Lemma 1. LetMbe an MDP with state space S. Let
0<  1. Consider a retirement reward in the interior of
this interval, 2(0;1). If,8s2S
V(s;) =V(s; 0) +V(s; 1)V(s; 0)
10(0);(4)
then there is at least one policy and stopping rule that is op-
timal for every 02[0;1]. The expected value of the dis-
count factor at retirement is the slope between those points:
Eh

(s)i
=V(s; 1)V(s; 0)
10: (5)
Proof. This follows from the form of Equation 1 and the
fact thatV(s;)is piecewise linear, increasing, and convex.
(a)
(b)
15101010
03
r= 12.1t=2
(c)
Figure 1: (a-b) Retirement process value functions for
the reward chains in Ex. 1. The slope of the lines is the
expected value of the discount parameter at retirement:
E[
]. Flat sections indicate regions of retirement rewards
where retirement is always suboptimal. The kink in the
green curve at = 280 indicates that it has become op-
timal to retire immediately. The kink in the black curve at
= 14 increases the slope to
6<1; for>14, it is op-
timal to retire after collecting a preﬁx of the reward stream
fromy2. (c) Dominated relaxation of the MDP in Ex. 2. We
add additional durative actions to the state space to ensure
that a dominating policy exists. Note that the original MDP
does not admit a dominating policy.
This test allows us to implement a binary search over re-
tirement rewards. Our approach relies on a given MDP so-
lution method, SOLVE , that returns the vector of values for
a given MDP. We initialize an interval that contains all criti-
cal points and make a call to SOLVE to compute the retire-
ment process value at each endpoint. Then we call SOLVE
to compute the retirement process value for the midpoint
of our interval and apply the test in Eq. 4. If it succeeds,
we return the lower endpoint and the corresponding slope.
If this does not, we sub-divide our interval and recurse. We
can concatenate the results to get the breakpoints and slopes
for the retirement process value function over this interval.
Algorithm 1 shows pseudocode for this algorithm.
Algorithm 1 Computing an MDP’s Critical Points
Deﬁne: CRITICAL POINTS (M[;;+;V;V+])
Input: MDP,M; Interval of retirement values, ;+;
Values at interval endpoints, V;V+
if;+are not set then
 0
+ maxReward (M)
1
V SOLVE (M)
V+ +## Retirement is initially optimal
end if
 +
2
V SOLVE (M)
ifjV
V+ ()V+V
+
j<then
return [];h
V+V
+i
else
pts;slope CRITICAL POINTS (M;;;V;V)
pts+;slope+ CRITICAL POINTS (M;;+;V;V+)
## Merge adjacent intervals with the same slope
ifslope[1] =slope+[0]then
delpts+[0];delslope+[0]
end if
returnpts[pts+;slopes[slopes+
end if
BRANCH-AND-BOUND V ALUE ITERATION
Now, we present a practical algorithm to compute optimal
actions in a bandit superprocess. While a BSP is not index-
able, we would like to be able to plan efﬁciently when the
arms are ‘close’ to indexable—when there are only a few
states where the optimal policy changes in response to the
opportunity cost of delayed rewards in other arms.
Our approach is based on envelope dynamic programming
methods to solve MDPs: we compute value estimates for
a given initial state by solving a dynamic program deﬁned
over a reachable subset of the state space [Gardiol, Natalia
H and Kaelbling, Leslie P, 2003, Hansen and Zilberstein,
2001]. The primary difference between our approach and
standard envelope methods is that we use dynamic pro-gramming on our envelope to update an upper bound and
a lower bound on the value. This is useful in a BSP be-
cause the Whittle integral allows efﬁcient calculation of
both bounds.
Our goal is to compute the optimal action for a given state
sin a BSP,M=hM1;:::;MKi. We can obtain an upper
bound onV(s)with a Whittle integral. We use a Whittle
integral for the MAB that solves each arm independently to
compute a lower bound. Our algorithm maintains upper and
lower bounds on the Q-function for each action that could
be executed from s. We write these bounds as Q+(s;a)
andQ(s;a)respectively. If there is a pair of actions a;a0
such thatQ+(s;a0)<Q(s;a) +, then we can conclude
thata0is at least-suboptimal. If this test removes all but
a single action, then we have found a -optimal action and
return it.
In the event that more than one action remains, we do a par-
tial expansion of the state space around s. We keep track of
a set of expanded states, E, and a set of boundary states, B.
States in the boundary set, B, are states that some expanded
state can transition to but have not yet been added to E.
We can use these states to update the bounds on Q(s;)by
solving a related MDP. This is formalized in the following
theorem.
Theorem 4. LetMbe an MDP with state space Sand
action spaceA. LetS0S whereS0=B [ E and
8s2 E ,T(s;a;s0)6= 0)s02 S0. LetM+be
an MDP with state space S0[fg.M+’s transition distri-
bution and rewards are identical for states in E(expanded
states). Each state s2B transitions deterministically to
and receives a reward that is an upper bound on VM(s).
is a sink state that accrues 0 reward. Then the value of a
state inM+is an upper bound on its value in M:
VM+(s)VM(s):
Proof. See supplementary materials.
It is straightforward to show that an updated lower bound
can be computed by ﬁxing boundary states to have value
equal to a lower bound. Our approach alternates between
pruning actions based on dominance relations, adding
states toE, and computing the value of an MDP deﬁned
overE. We interleave value iteration with a branch-and-
bound search, so we call it Branch-and-Bound Value Itera-
tion (BBVI). Algorithm 2 shows pseudocode to implement
this method. At termination, the action we return is guaran-
teed to be at most -suboptimal.
Theorem 5. LetMbe a BSP and let sbe a state in M. Let
abe the action returned by BBVI (M;s; ).
V(s)Q(s;a)<:
Proof. See supplementary materials.
Large regions of the state space are irrelevant to our ob-
jective so we use a heuristic to guide node expansion. Our
heuristic is a measure of the sensitivity of values at the root
to change in the upper and lower bounds of unexpanded
nodes. Similar heuristics have been applied in many set-
tings [Rivest, 1987, Smith and Simmons, 2004]. In BBVI,
these values are the expected discounted visitation rates in
the bounding MDPs and are computed as the dual variables
from a value iteration LP. The backup procedure is slower
than node expansions, so we perform backups in batches. In
between backups and heuristic computations, we approxi-
mate the state space as a tree and push new states onto the
agenda with their parent’s heuristic value weighted by the
transition probability.
Algorithm 2 Branch-and-Bound Value Iteration
Deﬁne: BBVI (hM0;:::;MKi;s0;)
Input: BSP arms,Mk; Initial state, s0; Tolerance,
# Lower bound Mby ﬁxing a policy for each arm
LBk toMRP (Mk)
Compute critical points for Mk;LBk
Compute bounds on Q(s0;)with Whittle integrals for
MandLB.
# Keep track of expanded region of state space
E ;
# Keep track of states at boundary of M+;M
B fs0g
a argmax
aQ(s0;a)
while9a06=as:tQ+(s0;a0)Q(s0;a)do
s pop(B)
E E[fsg
fora2A do
fors02successors (s)do
Compute upper and lower bounds for Q(s0;)
B B[fs0g
end for
end for
Q+ SOLVE (BOUNDMDP (E;B;Q+))
Q SOLVE (BOUNDMDP (E;B;Q))
a argmax
aQ(s0;a)
end while
return-optimal action a
6 EMPRIRICAL EV ALUATION
We implemented BBVI in Python and use the linear pro-
gramming solver Gurobi to compute value functions. Our
experiments were run on an Intel i7 with 16 GB of RAM. In
our ﬁrst experiments, we compare the scalability of BBVI
with that of standard MDP algorithms. In particular we
compare the following algorithms:
SPUDD: the baseline factored MDP algorithm in the
2011 IPPC [Hoey et al., 1999].
(a) State Space for R & D BSP
(b) Runtime vs. Problem Size
(c) Solution Quality vs. Problem Size
Figure 2: (a) the state space for an arm of a R & D BSP.
All states accrue 0 reward except for the ‘Working Prod-
uct’ state, which collects a ﬁxed reward per time step. (b)
shows runtime (log-scale) vs problem size for BBVI and
several alternatives on an R & D BSP. SPUDD [Hoey et al.,
1999] and LRTDP (with the heuristic from Singh and Cohn
[1998]) scale poorly compared with algorithms that lever-
age the Whittle integral. BBVI’s additional improvement
over LRTDP stems from the use of an efﬁcient, exact lower
bound to check convergence. (c) BBVI’s bound on sub-
optimality after 10,000 node expasions (measured in units
of discounted reward). Changing the number of rewards
in each arm allows us to measure BBVI’s performance as
arms become more sensitive to context.
LRTDP(Merge): Labeled Real-Time Dynamic Pro-
gramming [Bonet and Geffner, 2003] with the upper
bound from Singh and Cohn [1998] as a heuristic.
LRTDP(WI): LRTDP with the Whittle integral as an
upper bound.
BBVI: Branch-and-Bound Value Iteration.
We evaluate performance on a simple BSP designed to
model allocation of research resources to product research
and development. Each arm in this problem corresponds to
a potential product that our agent could sell in the market.
However, before we sell a product we must devote effort to
R&D. This can be done in one of two ways, a safe research
approach that is slow but reliable, and a risky approach that
is fast, but runs a risk of producing a defective product.
After at least one product has been researched, the agent
can opt to spend a round to produce and sell that product or
continue research on a different project. Because taking a
product to market does not change the state of the BSP, it is
straightforward to show that any stationary policy (includ-
ing) will only ever produce a single product. We deﬁne
a distribution over these problems by selecting a random
market value for each product uniformly from [0, 1] and
random durations for the safe and risky research strategies
(although we ensure that safe research is at least 3x as slow
as risky research). Figure 2 (a) shows the state space for an
example arm with typical parameters.
Figure 2 (b) shows the results of our comparison. We used
a timeout of 1000s and set a memory limit of 4 GB. We ran
our experiments in an online setting and running times re-
ﬂect the amount of time to select an optimal action from
scratch. While this is a natural setting for LRTDP and
BBVI, it is admittedly a little unfair to SPUDD (which
computes a full policy). This is mitigated by the fact that
BBVI’s improvement over SPUDD, which is a little under
5 orders of magnitude with 108states, is such that an agent
would need to be planning over an immense horizon before
SPUDD presents a reasonable alternative.
The factored MDP bounds from Singh and Cohn [1998]
are quite ill-suited to this problem. This is because its up-
per bound (the sum of the independent value for each arm)
is very loose and it essentially forces LRTDP to enumer-
ate the state space. In contrast, LRTDP performs quite well
when it has good heuristic information. However, its per-
formance degrades faster than that of BBVI, because it does
not effectively leverage a lower bound on value. Even for
very large problem instances, BBVI computes optimal so-
lutions in well under a second.
BBVI’s performance in the R&D domain is largely ex-
plained by the structure of the arms and the short horizon
within each arm. Although there may be a large number
of arms, BBVI reduces each individual arm to an equiva-
lent MRP after a small number of node expansions. Thismeans that running time is essentially linear in the number
of arms. Note that, in this domain, the solution that solves
each MDP independently and executes the corresponding
index policy will just opt for conservative research on the
most valuable option. This policy is essentially always sub-
optimal: as long as there are reasonable alternatives it is
usually a good idea to try some risky projects.
Our next experiment uses a synthetic domain to explore the
performance of BBVI as the structure and number of arms
changes. In this BSP, each arm is a 20x20 grid world. The
actions in this world correspond to moving in the 4 car-
dinal directions. Rewards are 0 everywhere except at ran-
domly chosen locations and after receiving a reward, the
agent transitions to a random location.
In this experiment we vary the number of rewards that are
available in each arm. Adding rewards will typically cause
the optimal policy for an arm to be more sensitive to con-
text. This means that the dominated relaxation will produce
a looser upper bound, because it will need to add more ac-
tions. We evaluate this difﬁcultly by measuring the con-
vergence criterion of BBVI after 10,000 node expansions.
Figure 2 (c) shows the results of this experiment. We can
see that BBVI’s solution quality decreases with the number
of states, but the primary variation comes from changes in
the properties of the arms.
7 CONCLUSION AND FUTURE WORK
In summary, we presented a model of highly decoupled de-
cision making: bandit superprocesses. A BSP presents an
agent with the opportunity to act in one of several Markov
decision processes. The key constraint is that the agent can
only act in a single process at a time. We provided a sum-
mary of BSP research, including an upper bound for the
value function of a BSP in the form of the Whittle integral.
We derived an equivalent relaxation and thus gave a novel
proof that the Whittle integral is an upper bound. Finally
we presented and evaluated algorithmic solutions for this
class of decision problems.
In future work, we plan to extend these ideas in three direc-
tions. The ﬁrst is algorithmic improvements for BSPs. A
potential direction here is to leverage factored dynamics in
the bounding MDPs themselves. A search over sequences
of MDPs that only consider states where the current pol-
icy stops could be more efﬁcient. Next, also plan to explore
generalizations of the bounds used for BSPs to more gen-
eral cases of global resource constraints (e.g., those con-
sidered in Meuleau et al. [1998]). Finally, the similarity to
sums of games studied in Conway [2000] suggests an unex-
plored connection between BSPs and combinatorial game
theory.
References
Blai Bonet and Hector Geffner. Labeled RTDP: Improving
the convergence of real-time dynamic programming. In
IEEE Conference on Automated Planning and Schedul-
ing, pages 12–21, 2003.
David B. Brown and James E. Smith. Optimal sequential
exploration: Bandits, clairvoyants, and wildcats. Opera-
tions Research , 61(3):644–665, 2013.
John H. Conway. On Numbers and Games . CRC Press,
2000.
Gardiol, Natalia H and Kaelbling, Leslie P. Envelope-
based planning in relational MDPs. In Advances in Neu-
ral Information Processing Systems , page None, 2003.
John C. Gittins. Bandit processes and dynamic allocation
indices. Journal of the Royal Statistical Society. Series
B (Methodological) , pages 148–177, 1979.
Kevin D. Glazebrook. On a sufﬁcient condition for super-
processes due to Whittle. Journal of Applied Probability ,
pages 99–110, 1982.
Kevin D. Glazebrook. Indices for families of competing
Markov decision processes with inﬂuence. The Annals
of Applied Probability , pages 1013–1032, 1993.
Eric A Hansen and Shlomo Zilberstein. Lao: A heuristic
search algorithm that ﬁnds solutions with loops. Artiﬁ-
cial Intelligence , 129(1):35–62, 2001.
Jesse Hoey, Robert St-Aubin, Alan Hu, and Craig Boutilier.
SPUDD: Stochastic planning using decision diagrams.
InFifteenth Conference on Uncertainty in Artiﬁcial In-
telligence , pages 279–288. Morgan Kaufmann Publish-
ers Inc., 1999.
Nicolas Meuleau, Milos Hauskrecht, Kee-Eung Kim,
Leonid Peshkin, Leslie Pack Kaelbling, Thomas L Dean,
and Craig Boutilier. Solving very large weakly coupled
Markov decision processes. In Fifteenth National Con-
ference on Artiﬁcial Intelligence , pages 165–172, 1998.
Peter Nash. Optimal allocation of Resources Between Re-
search Projects. PhD thesis, University of Cambridge,
1973.
Martin L. Puterman. Markov Decision Processes: Discrete
Stochastic Dynamic Programming . John Wiley & Sons,
2009.
Ronald L. Rivest. Game tree searching by min/max ap-
proximation. Artiﬁcial Intelligence , 34(1):77–96, 1987.
Herbert Robbins. Some aspects of the sequential design of
experiments. In Herbert Robbins Selected Papers , pages
169–177. Springer, 1952.
Stuart Russell and Peter Norvig. Artiﬁcial Intelligence: A
Modern Approach . 3 edition, 2010. Excercise 17.5.
Satinder Singh and David Cohn. How to dynamically
merge Markov decision processes. Advances in Neural
Information Processing Systems , (10):1057–1063, 1998.Trey Smith and Reid Simmons. Heuristic search value it-
eration for POMDPs. In Proceedings of the 20th Con-
ference on Uncertainty in Artiﬁcial Intelligence , pages
520–527. AUAI Press, 2004.
Peter Whittle. Multi-armed bandits and the Gittins in-
dex. Journal of the Royal Statistical Society. Series B
(Methodological) , pages 143–149, 1980.


The Liability Problem for Autonomous Artificial Agents
Asaro, Peter M.
School of Media Studies, The New School
Center for Information Technology Policy, Princeton University
Center for Internet and Society, Stanford Law School
asaro@newschool.edu
Abstract
This paper describes and frames a central ethical issue–the
liability problem–facing the regulation of artificial computational
agents, including artificial intelligence (AI) and robotic systems,
as they become increasingly autonomous, and supersede current
capabilities. While it frames the issue in legal terms of liability
and culpability, these terms are deeply imbued and
interconnected with their ethical and moral correlate–
responsibility. In order for society to benefit from advances in AI
technology, it will be necessary to develop regulatory policies
which manage the risk and liability of deploying systems with
increasingly autonomous capabilities. However, current
approaches to liability have difficulties when it comes to dealing
with autonomous artificial agents because their behavior may be
unpredictable to those who create and deploy them, and they will
not be proper legal or moral agents. This problem is the
motivation for a research project that will explore the
fundamental concepts of autonomy, agency and liability; clarify
the different varieties of agency that artificial systems might
realize, including causal, legal and moral; and the illuminate the
relationships between these. The paper will frame the problem of
liability in autonomous agents, sketch out its relation to
fundamental concepts in human legal and moral agency–
including autonomy, agency, causation, intention, responsibility
and culpability–and their applicability or inapplicability to
autonomous artificial agents.
 Introduction 1
There is a growing sense of concern over the
development of increasingly autonomous non-human
agents–in the public and the media, as well as among
policy makers and researchers. Such concerns are not
unprecedented, yet there is something different about this
next wave of technological innovation and change. While
the impacts of the adoption of any technology are in some
sense uncertain, and result in many and various

Copyright © 2015, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
1
Research for this paper was supported by a grant from the Future of Life Institute.unintended consequences, there seems to be something
particularly unsettling and deeply uncertain about
increasingly autonomous technologies. I believe that this
sense of concern stems from the recognition that
autonomous systems will not only be unpredictable in
terms of their unintended actions and general effects on
society, but that they may also be out of control, in the
sense that these effects will occur beyond the scope of
human responsibility.
Previous technological innovations have not raised this
concern in quite the same way, or to the same degree,
because it was possible to rely upon human morality, as
well as law, to regulate the use of new technologies.2
While there were recognized risks to using technologies
like guns, cars, airplanes, steam engines, and many more,
those risks were circumscribed by limits on their
immediate effects and the proximate responsibility of
individual humans and human institutions for those
effects. In most cases, these risks were also limited in their
complexity and longevity, in the sense that those effects
could be anticipated, managed, and reduced over time.3 As
long as there is a human hand on the controls, there is a
margin of assurance that the technology cannot go too far
out of control before it is reigned in by human agency or
regulatory policy. Thus, human control presents itself as a
2There are some notable exceptions. There have been debates over the morality of research and development of technologies with potentially catastrophic results or applications (and sometimes the dissemination of basic research in these areas) particularly the weaponization of nuclear physics and biological and chemical agents. There have been similar debates over the risks of basic research itself in the case of genetic engineering and nanotechnology, where the experiments themselves might go “out of control” and cause irrevocable or catastrophic harm. Many of these concerns are related to the potential causal “chain reactions” of positive feedback systems and exponential growth.
3Notable exceptions here are certain tragedies of the commons and the problem of many hands (Nissenbaum 1996), in which proximate responsibility is small and requires many participants or instances for the harm to be realized. Environmental degradation is a good example where the participation of many separate responsible agents leads to a diffusion of perceived responsibility. Such harms are difficult to regulate largely because it is more difficult to establish the causal link to the responsible agents and rule out intervening or contributory causes.
“kill switch” to technology going completely out of
control, while human responsibility further acts as “policy
lever” for policy to enact regulation over material
technologies.
Non-human agents, including robotics and software
agents, and especially those using advanced artificial
intelligence (AI), are becoming increasingly autonomous
in terms of the complexity of tasks they can perform, their
potential casual impacts on the world that are unmitigated
by human agents, and the diminishing ability of human
agents to understand, predict or control how they operate.
This increasing autonomy of non-human agents seriously
challenges human control, and thus challenges both the
“kill switch” and the “policy lever” functions of human
control. Some have argued that one factor or other is more
troubling, or that one challenge or the other is more dire.
But, I believe that the growing concern over increasingly
autonomous agents stems from the combination of these
factors and the challenges they impart, and they must be
addressed comprehensively and systematically. Central to
addressing these challenges will be finding a way to
manage the liability problem–how to continue to hold
people legally liable for increasingly autonomous agents.
This is closely related to the responsibility problem–how
to ensure that human agents take moral responsibility for
the technologies they make and use as these become
increasingly autonomous.
How should we regulate artificial computational agents,
including AI and robotic systems, as they become
increasingly autonomous, and supersede current
capabilities? In order to take a systematic approach to this
question, we must integrate an analysis of fundamental
concepts of autonomy and agency, with a review of
current processes of systems design and development, and
chart a course for future regulatory policy. This will
include investigating 1) the conceptual foundations of
autonomy and agency; 2) survey the existing legal and
moral theories of liability and responsibility and their
applicability to artificial agents; and 3) explore how
implementing an enhanced theory of liability and agency
for autonomous artificial agents could shape regulatory
policy going forward. In this paper I will focus on
sketching out 1) the conceptual foundations of autonomy
and agency, through an examination of the liability
problem.
One of the central problems facing the development of
autonomous artificial agents as technological capabilities
continue to increase, is the uncertain status of liability for
the effects caused by artificial agents–the liability
problem. Resolving the liability problem will require
untangling a set of theoretical and philosophical issues
surrounding causation, intention, agency, responsibility,
culpability and compensation. The primary objective of
the proposed research is to address this problem through a
conceptual analysis that is informed by existing legal and
moral theories, and grounded in technological capabilities
of current and potential artificial agents.The Liability Problem for Autonomous Artificial
Agents
In order for society to enjoy many of the benefits of
advanced AI and robotics, it will be necessary to be able to
deal with situations that arise in which autonomous
artificial agents violate laws or cause harm. If we want to
allow AIs and robots to roam the internet and the physical
world and take actions that are unsupervised by humans–
as might be necessary for, e.g. personal shopping
assistants, self-driving cars, and host of other
applications–we must be able to manage the liability for
the harms they might cause to individuals and property.
Already, there have been automated shopping software
bots that have purchased illegal items on the DarkNet,
albeit as an art project (Kasperkevic 2015). There have
also been serious questions raised as to who could be held
accountable for the deaths caused by autonomous weapons
in war (Human Rights Watch 2015).
Traditional approaches to handling liability are
inadequate for dealing with autonomous artificial agents
due to a combination of two factors–unpredictability, and
causal agency without legal agency.
First, unlike traditional engineering and design, the
actual functioning of an autonomous artificial agent is not
necessarily predictable in the same way as most
engineered systems. Some artificial agents may be
unpredictable in principle, and many will be unpredictable
in practice. Predictability is critical to current legal
approaches to liability. In traditional product liability, the
manufacturer is responsible for the product working as
designed, and foreseeing likely problems or harms it may
cause. Determining what is “foreseeable” often falls upon
courts to decide, but the legal standards used are whether
the manufacturer had knowledge of the potential problem,
or whether a reasonable person should have foreseen it, or
whether there is an industry standard of practice that
would have revealed it. While there is a degree of
unpredictability in the performance of any engineered
product, due to failures or unforeseen circumstances of
use, there are shared expectations regarding its
performance, testing for the limits of that performance and
likelihood of failure, and management of foreseeable risks.
In the case of advanced AI, a system that learns from
environmental data may act in ways that its designers have
no feasible way to foresee (Asaro 2008). In a limited
sense, this is already the case with current machine
learning techniques, which use large sets of training data
and produce novel problem solutions. Currently, it is
possible to analyze and test a learned function and
determine its behavior, as with traditional engineering. But
when AI systems are allowed to continue modifying their
functions and learn after they are deployed, their behavior
will become dependent on novel input data, which
designers and users cannot predict or control. As a result,
the behavior of the learned functions will, to various
degrees, also be unpredictable. A truly robust AI program
capable of open-ended learning could learn functions that
its manufacturer could not foresee, perhaps in principle.
Insofar as autonomous artificial agents utilize learning and
open-ended learning, their behavior will also be
unpredictable. Unpredictability by itself is not an
insurmountable problem for liability, insofar as the agents
who introduce that unpredictability could be themselves
held liable, or the risks from unpredictability could be
managed.
However, the second factor challenging traditional
approaches to liability is that autonomous artificial agents
may “act” on their own, yet are not accountable or liable
in a legal sense. With most engineered products, the
predictability is bounded by the actions of other agents–
consumers, users, service technicians, etc.–and how they
maintain and use a given product.4 In those cases, there is
a clear legal agent or user who may have used the product
inappropriately and is thus liable for the consequences of
the use in that instance (at least partially, if not
completely).
Autonomous artificial agents can act in the world
independently of their designers or operators. This makes
it difficult to identify the user or operator, who would
normally be liable. In the case of learning systems, the
causal influence resulting in the unpredictability of a
system stems from the datasets used for learning, not a
legally responsible agent (unless someone deliberately
seeks to influence the datasets and learning process).
These cases are somewhat analogous to the way that
parents may be held liable for the actions of small
children, but they are not usually held liable for the actions
of their adult children. As adults, the children have learned
enough about the world to become their own legal agents.
Something similar applies in cases of employer-employee
liability. Usually the employer is responsible for any
damages their employee causes in the course of doing
their job (such as damage from a delivery truck hitting a
parked car while making routine deliveries), but if the
employee is off on a frolic (and took the delivery truck to
visit a friend), then the employer is not responsible for the
damages and the employee is (Asaro 2011).
Approaches to the Problem
Taken together these two factors result in a number of
problems with applying traditional theories of legal
liability to autonomous artificial agents. There are two
basic liability frameworks in the law, criminal and civil
(primarily tort) liability. In both domains, it is difficult to
hold the artificial agent legally liable for its actions, as
they are not legal persons, and treating them as legally
fictitious persons (like corporations) does not really solve
4 For example, a hammer is designed to drive nails, but can also be used as a weapon–with the liability falling on the person who uses it as a weapon. Similarly, the car owner is responsible for changing the brake pads of a car when they are worn, and the mechanic is responsible for doing it properly, while the manufacturer’s liability is limited to properly informing car owners of the service needs for the car.the problem (Asaro 2011). It is possible in some criminal
situations to hold the people who operate the artificial
systems liable, provided you can show intent to commit a
crime, or foreseeable risk of a harm rising to the level of
criminal negligence. To the extent that artificial agents
become increasingly complex, those who build or deploy
advanced AIs and robotics will not necessarily have intent
or foresight of the actions those systems may take. This is
especially true for systems that can substantially change
their operations through advanced learning techniques, or
those future systems that might even become genuinely
autonomous in generating their own goals and purposes,
or even intentions.
At some point in the evolution of autonomous artificial
agents, they might become legal and moral agents, and
society will be faced with the question of whether to grant
them some or all of the legal rights bestowed on persons
or corporations. At that point, some or all of current
liability law might apply to those AIs and robots that
qualify as legal persons, though it might not be clear what
the exact boundaries of a particular entity might be, or
how to punish it (Asaro 2011). For example, whether a
particular instantiation of a program is the legal subject, or
all copies of a program are part of the same legal subject
(as copyright law might suggest). It will also not be clear
how to appropriately punish them, or otherwise correct
their future actions and provide retributive compensation
to those that have been harmed. Despite these challenges,
being able to treat autonomous artificial agents as legal
persons is likely to be the easier problem.
In the near term, the hard liability problem for
autonomous artificial agents will lie in devising a system
of liability that promotes beneficial innovation while
offering just and adequate compensation to those harmed.
Justice requires that those who will be held liable should
be able to understand the scope and extent of the risks and
liability they are assuming in deploying an autonomous
artificial agent, and have some means of managing that
risk through controls over the system. Adequate
compensation means that there needs to be sufficient
means to compensate those harmed, monetary or
otherwise–whether this entails holding large corporations
accountable, or providing proper risk-pooling insurance.
For insurance to work, of course, it will be necessary for
actuaries to be able to assess the risks posed by various
artificial agents. Assessing those risks will require both
expertise, and probably means for imposing or assuring
predictability in the systems themselves. And even then,
there may be catastrophic risks involved, which may not
be manageable under these frameworks. Ultimately there
will be questions that society must address as to the
perceived benefits and risks of such technologies, and who
in society should receive those benefits and should
shoulder the burden of those risks.
There are a few liability policy options already in use in
tort law, such as joint and several liability, strict liability,
and risk-pooling insurance that might be deployed. But
there are reasons to doubt that these will scale adequately
or address all the problems posed by autonomous artificial
agents.
Existing forms of joint and several liability permit those
harmed to seek monetary damages from the deepest
pockets among those parties sharing some portion of the
liability. This works well where there is a large
corporation or government that is able to pay damages.
While the amount they pay is disproportionate to their
contribution to the damages, those harmed are more likely
to be adequately compensated for their damages. One
result of this is that those manufacturers likely to bear the
liability burden would seek to limit the ability of
consumers and users to modify, adapt or customize their
advanced AI and robotics products in order to retain
greater control over how they are used. This would stifle
innovation coming from the hacking, open-source and
DIY communities. The hacking, open-source and DIY
communities, while a powerful source of innovation, will
have limited means of compensating those who might be
harmed from their products—not just those who chose to
use them.
Strict liability goes further and designates or stipulates a
party, usually the manufacturer or owner, as strictly liable
for any damages caused. This model of liability applies to
such things as the keeping of wild animals. It is expected
that tigers will harm people if they get free, so as the
keeper of a tiger you are strictly liable for any and all
damages the tiger may cause. We apply normal property
liability to domesticated animals, however, which we
expect to not harm people under normal circumstances. It
has been suggested that we could apply this to robotics
(Schaerer, Kelley, and Nicolescu 2009), and perhaps
designate certain advanced AIs as essentially wild
animals, and others as domesticated. But then the question
arises: How does one determine whether an advanced AI
or robot is appropriately domesticated, and stays that way?
A designated liable person may not know the extent to
which the autonomous system is capable of changing itself
once it is activated. More problematic, however, is that a
system of strict liability might result in the slow adoption
of beneficial AI technologies, as those who are strictly
liable would have a large and uncertain risk, and be less
likely to produce or use the technology, or soon go out of
business.
There are other proposals that aim to design
mechanisms of accountability into complex socio-
technical systems (van den Hoven, Robichaud, and
Santoni de Sio 2015). But such approaches, much like
strict liability, do not really address the fundamental issue
of autonomous agency–how to define it or to regulate it.
They merely try to manage it in an ad hoc effort to
maintain traditional concepts rather than find a simpler
and more powerful solution.
We thus have a few general options when it comes to
regulating autonomous artificial agents. We could
preclude or prohibit autonomous artificial agents because
of their risks and uncertainties. We might call this the
precautionary approach. To the extent that it works, it would also impede the development and deployment of
many beneficial advanced AIs and robots because of their
problematic autonomy. We could permit the development
and deployment of autonomous artificial agents, and
accept the risks and costs at a social level, without
developing a better framework for regulating autonomy.
We might call this the permissive approach.5 This would
allow many beneficial applications of advanced AI and
robots, but also many harmful ones, including many harms
for which no one might be liable and those harmed would
not be compensated. As a secondary effect, there would
likely be a general backlash against advanced AI and
robotics as the technology comes to be seen as harmful
and offering few options for restitution of those harms.
Or, we could pursue one of the heavy-handed liability
schemes, such as strict liability, that would regulate the
industry to some extent, but also limit innovation to those
areas where there are sufficient profits to motivate large
capital companies to enter the market and accept the risks.
We might see this work in applications such as financial
trading algorithms, self-driving cars and medical-care
robots, but many other beneficial applications of AI may
not have such clear profit margins. Even in clearly
profitable areas, strict liability is likely to stifle innovation
for a range of beneficial applications of autonomous
artificial agents.
Alternatively, we could seek a better solution to the
liability problem than current models afford. We could
attempt to reconceptualize how we think about agency,
causality, liability responsibility, culpability and autonomy
for the new age of artificial autonomous agents. While it is
not yet clear how this might work, a clear framing of the
problem, as this paper has presented, is the first step in
that larger project. The next steps involve clarifying the
distinctions between the purposes of algorithms and
machines from the purposes of the persons who use them
(Asaro forthcoming), and distinguishing different types of
agency accordingly.
References
Asaro, Peter (forthcoming) “Roberto Cordeschi on Cybernetics
and Autonomous Weapons: Reflections and Responses,”
Paradigmi: Rivista di critica filosofica , 2016.
Asaro, Peter (2008) “From Mechanisms of Adaptation to Intelligence Amplifiers: The Philosophy of W. Ross Ashby,” in Michael Wheeler, Philip Husbands and Owen Holland (eds.) The Mechanical Mind in History , Cambridge, MA: MIT Press, pp. 149-184.
Asaro, Peter (2011) “A Body to Kick, But Still No Soul to Damn: Legal Perspectives on Robotics,” in Patrick Lin, Keith Abney, and George Bekey (eds.) Robot Ethics: The Ethical and Social Implications of Robotics. Cambridge, MA: MIT Press, pp. 169-186.
Chopra, Samir and Laurence F. White (2011) A Legal Theory for Autonomous Artificial Agents , University of Michigan Press.
5This is the approach taken by Chopra and White (2011).
Human Rights Watch (2015) “Mind the Gap: The Lack of
Accountability for Killer Robots,” HRW Report, April 9, 2015,
38pp. Downloaded from: http://www.hrw.org/node/133918
Jeroen van den Hoven, Jeron, Phil Robichaud, and FillipoSantoni
de Sio (2015) “Why the Future Needs us Today: Moral
Responsibility and Engineering Autonomous Weapon Systems,”
Address to the United Nations Convention on Certain
Conventional Weapons Expert Meeting on Lethal Autonomous
Weapons. Geneva, Switzerland, April 16, 2015. Downloaded
from: http://www.unog.ch/80256EDD006B8954/
(httpAssets)/89116E298AE593C2C1257E2A00413D61/$file/20
15\_LAWS\_MX\_VanDenHoven.pdf
Kasperkevic, Jana (2015) “Swiss police release robot that bought
ecstasy online,” The Guardian, April 22, 2015. Downloaded
from: http://www.theguardian.com/world/2015/apr/22/swiss-
police-release-robot-random-darknet-shopper-ecstasy-deep-web
Nissenbaum, Helen (1996) Accountability in a Computerized
Society,” Science and Engineering Ethics , March 1996, V olume
2, Issue 1, pp 25-42 . Downloaded from:
http://www.nyu.edu/projects/nissenbaum/papers/accountability.p
df
Schaerer, Enrique, Richard Kelley, and Monica Nicolescu (2009) “Robots as animals: A framework for liability and responsibility in human-robot interactions,” Proceedings of RO-MAN 2009: The 18th IEEE International Symposium on Robot and Human Interactive Communication , Toyama, Japan, Sept. 27–Oct. 2, 72–77.