Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints.

Autonomous Robots
https://doi.org/10.1007/s10514-021-10022-9
Dynamic multi-robot task allocation under uncertainty and temporal
constraints
Shushman Choudhury1·Jayesh K. Gupta1·Mykel J. Kochenderfer1·Dorsa Sadigh1·Jeannette Bohg1
Received: 14 January 2021 / Accepted: 27 September 2021
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2021
Abstract
We consider the problem of dynamically allocating tasks to multiple agents under time window constraints and task completionuncertainty. Our objective is to minimize the number of unsuccessful tasks at the end of the operation horizon. We present
a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under
uncertainty and multi-agent coordination, and addresses them in a hierarchical manner. The lower layer computes policies forindividual agents using dynamic programming with tree search, and the upper layer resolves conﬂicts in individual plans to
obtain a valid multi-agent allocation. Our algorithm, Stochastic Conﬂict-Based Allocation (SCoBA), is optimal in expectation
and complete under some reasonable assumptions. In practice, SCoBA is computationally efﬁcient enough to interleaveplanning and execution online. On the metric of successful task completion, SCoBA consistently outperforms a number ofbaseline methods and shows strong competitive performance against an oracle with complete lookahead. It also scales well
with the number of tasks and agents. We validate our results over a wide range of simulations on two distinct domains:
multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
Keywords Task allocation ·Multi-robot systems ·Hierarchical planning ·Conﬂict-based search
1 Introduction
Efﬁcient and high-quality task allocation is crucial for
modern cooperative multi-robot applications (Gerkey andMataric 2004 ). For warehouse logistics, teams of mobile
robots carry goods between assigned locations (Yan et al.
2012 ). Industrial and manufacturing operations involve
manipulators collaborating on assembly lines (Johannsmeierand Haddadin 2016 ). On-demand ridesharing and delivery
services dispatch agents to incoming requests (Hyland and
Mahmassani 2018 ). Multi-robot task allocation needs to be
computationally efﬁcient and produce high-quality solutions
under the challenges of real-world robotics: the uncertainty
of executing tasks successfully, temporal constraints such asordering and time windows, and tasks dynamically appearing
online. For instance, in one of our simulation domains, a team
This is one of the several papers published in Autonomous Robots
comprising the Special Issue on Robotics: Science and Systems 2020.
B Shushman Choudhuryshushman@cs.stanford.edu
1Department of Computer Science, Stanford University,Stanford, CA 94305, USAof robot arms pick objects that appear on a conveyor belt from
an external loading process and place them in bins (Fig. 1a).
With time window constraints induced by workspace limits
and uncertainty due to imperfect grasping, the arms attemptto pick-and-place as many objects as possible.
Multi-robot task allocation is a difﬁcult problem; it inher-
its the combinatorial optimization challenges of classicalallocation as well as the uncertainty and dynamic onlineenvironments of robotics settings. Time-extended tasks and
time window constraints further require algorithms to plan
over horizons rather than instantaneously, and account forspatio-temporal relationships among tasks (Gini 2017 ). The
robotics community has worked on multi-agent task alloca-
tion with Markov Decision Processes (Campbell et al. 2013 )
and robust task matching (Liu and Shell 2011 ). The classic
multi-robot task allocation problem has been studied exten-
sively (Gerkey and Mataric 2004 ) and extended to account
for uncertainty (Mataric et al. 2003 ), temporal and ordering
constraints (Gini 2017 ), and dynamic task arrivals (Cordeau
and Laporte 2007 ). The operations research community has
developed methods that plan under task execution uncer-tainty (Timotheou 2010 ; Rahmani and Heydari 2014 ) and
can efﬁciently recompute schedules online (O’Donovan et
123
Autonomous Robots
(a)
(b)
Fig. 1 The above domains motivate our multi-robot task allocation
approach. We allocate robots (arms or drones) to tasks (pick-and-placeor delivery) that arrive online. Task completion is subject to uncertainty(grasping or ﬂight time) and time window constraints
al.1999 ). However, they have simpliﬁed agent models (ﬂow
shops, job shops) that are unable to represent complex spatialrelationships between tasks.
The algorithmic challenges for our allocation setting
aresequential planning under uncertainty and coordinat-
ing multi-agent decisions . The prior works above typically
attempt the computationally intractable joint problem. Thus,
they require simplifying approximations or heuristics foreither multi-agent planning or coordination, such as ignor-
ing uncertainty or imposing arbitrary priority orderings on
agents.
Our key idea is to decouple the algorithmic challenges
and address them hierarchically in an efﬁcient two-layer
approach. The lower layer plans for individual agents, using
dynamic programming on a policy tree to reason aboutthe uncertainty over task completion. The upper layer uses
optimal conﬂict resolution logic from the path planning
community to coordinate the multi-agent allocation deci-sions (Sharon et al. 2012 ). Our overall algorithm, Stochastic
Conﬂict-Based Allocation (SCoBA), yields allocation poli-
cies that minimize the expected cumulative penalty forunsuccessful tasks. Due to its computational efﬁciency andtree structure in both layers, SCoBA can also seamlessly
interleave planning and execution online for new tasks.
The following are the contributions of our work:
1. We propose a general formulation for multi-robot allo-
cation under task uncertainty and temporal constraints.
2. We present a hierarchical algorithm, SCoBA, which uses
multi-agent conﬂict resolution with single-agent policy
tree search. We prove that SCoBA is both optimal in
expectation and complete under mild assumptions.
3. We demonstrate SCoBA ’s strong competitive perfor-
mance against an oracle with complete lookahead, and
its advantage over four baseline methods for successfuly
executing tasks. Our results also show that SCoBA scaleswell with increasing numbers of agents and tasks. We run
simulations on two distinct robotics domains (Fig. 1); a
team of robot arms picking and placing objects from aconveyor belt and on-demand delivery of packages by a
team of drones in a city-scale area.
This paper builds upon an earlier version that appeared
in Robotics: Science and Systems 2020 (Choudhury et al.2020a ). It adds detailed proofs for both theoretical proper-
ties and elaborates on the other algorithmic features. It also
augments the experiments section with an expanded descrip-
tion of how we generate tasks in our ﬁrst simulation domain,an additional sets of results for each of the two domains with a
different task execution probability model, and more detailed
comparisons on computation time.
2 Background and related work
We brieﬂy discuss three background areas: algorithms forassignment and scheduling, multi-agent decision-making,
and relevant work in multi-robot task allocation.
2.1 Assignment and scheduling
A major topic in discrete optimization is assigning resourcesto tasks (Burkard et al. 2009 ), for which the Hungarian algo-
rithm is a fundamental one (Munkres 1957 ). In temporal
tasks, we use the scheduling model, where the objective is a
function of completed jobs (Pinedo 2012 ). Scheduling prob-
lems with multiple resources and tasks are computationally
hard, even when deterministic (Lenstra et al. 1977 ). In online
scheduling, each task is only observed when made avail-
able (Albers 1999 ). Hard real-time tasks have a time window
constraint for completion (Dertouzos and Mok 1989 ).
Approaches for scheduling under uncertainty address
problems where task execution is not fully determinis-
tic (Chaari et al. 2014 ). These approaches are either proactive
in anticipating future disruptions (Lin et al. 2004 ), reactive
123
Autonomous Robots
to changes (Szelke and Kerr 1994 ; Raman et al. 2015 ), or
hybrid (Church and Uzsoy 1992 ). For scenarios with order-
ing constraints, such as assembly lines, additional models
like job shops (Al-Hinai and ElMekkawy 2011 ) and ﬂow
shops (González-Neira et al. 2017 ) are useful, particularly
real-time ﬂow shop scheduling (Rahmani and Heydari 2014 ;
Framinan et al. 2019 ). This extensive body of work provides
valuable insights but does not address task conﬁgurations
more complex than an ordered sequence on an assembly line(such as delivery requests in a geographical area) and the
uncertainty of travel time to reach these tasks.
2.2 Multi-agent sequential decision-making
The Markov Decision Process (MDP) is a mathemati-
cal model for our setting of sequential decision makingunder uncertainty (Kochenderfer 2015 ). Different solution
techniques exist for MDPs, depending on available infor-
mation; dynamic programming (Bertsekas 2005 ) when the
explicit transition model is known, sample-based online plan-
ning (Péret and Garcia 2013 ) when only a generative model
exists, and reinforcement learning (Sutton and Barto 2018 )
when no model is available. Our problem is a multi-agentMDP (MMDP), where agents coordinate to achieve a single
shared objective; planning for MMDPs is generally computa-
tionally intractable in practice due to the exponentially largedecision space (Boutilier 1996 ).
Reinforcement learning techniques are often employed to
alleviate tractability issues (Littman 1994 ) by learning val-
ues of different states and actions, but model-free methods
like Q-Learning face exploration challenges (Lanctot et al.
2017 ). Recently, graph neural network representation tech-
niques have been used to learn good heuristics and reduce theoverall combinatorial complexity of such multi-robot alloca-
tion policies (Wang and Gombolay 2020 ; Zhang et al. 2020 ).
Online tree search methods can fare better than these ofﬂineapproaches by focusing on relevant states that are reachable
from the current one (V odopivec et al. 2017 ), and a recent
multi-robot allocation algorithm is based on online MonteCarlo Tree Search (Kartal et al. 2016 ). None of these methods
have any guarantee on solution quality or completeness, due
to the computational intractability of Multi-Agent MDPs.
2.3 Multi-robot task allocation
We outline a number of domain-agnostic and domain-speciﬁcworks on multi-robot task allocation. MDP solvers have
been used to generate a sequential greedy strategy, but with-
out accounting for completion uncertainty (Campbell et al.2013 ). The probability of task failure has been considered
by two-stage Stochastic Integer Programs and network ﬂow
algorithms, which are exhaustive combinatorial approachesunsuitable for tasks that are streaming in online (Ahmed andGarcia 2003 ; Timotheou 2010 ,2011 ). A sensitivity analysis
approach to optimal assignment under uncertainty providessome insights on robustness but has no notion of temporal
constraints (Liu and Shell 2011 ).
Existing taxonomies for multi-robot task allocation help
us understand our problem difﬁculty (Gerkey and Mataric2004 ; Nunes et al. 2017 ). Among the early works reviewed
in these taxonomies, two fundamental ones study the effects
of uncertain environments and dynamic tasks (Mataric etal.2003 ;L e r m a ne ta l . 2006 ). However, they both con-
sider distributed multi-robot systems, with simple alloca-
tion strategies based on local information that lack anyglobal guarantees. More recent work has developed an
auction-based method for distributed mobile robot teams
that plans time-constrained pickup and delivery schedulesonline (Coltin and V eloso 2014 ). On the centralized side,
another recent paper presents a comprehensive approach for
task assignment and scheduling that handles tightly cou-
pled spatial and temporal constraints (Gombolay et al. 2018 ).
However, neither of them explicitly models or addresses task
execution uncertainty and how it interacts with time con-
straints to yield task failure probabilities.
Previous work on assembly lines includes hierarchical
planning frameworks, constraint programming, and robust
scheduling for robotic ﬂowshop systems (Johannsmeier andHaddadin 2016 ; Behrens et al. 2019 ; Che et al. 2017 ). How-
ever, they all simplify one or more of the key complexities
such as task completion uncertainty or multi-agent conﬁgura-
tion models. Dynamic vehicle dispatch problems have beenexplored in work on vehicle routing algorithms with time
windows and trip assignment algorithms for ridesharing (Lau
et al. 2003a ; Alonso-Mora et al. 2017 ). However, they make
restrictive assumptions on the uncertainty and environment
dynamics. Driver-task assignment with uncertain durations
and task windows do solve for a similar setting as ours butassume some knowledge of future requests (Cheung et al.2005 ).
3 Problem formulation
We base our formulation on previous work for multi-robot
task allocation with temporal constraints (Gini 2017 ). There
is a set of Nagents , denoted as [N]and Ktasks, denoted
as[K]; the problem horizon is Ttime-steps. For each agent
n∈[N]and task k∈[K], the service time window is
Wnk=/parenleftbig
tl
nk,tu
nk/parenrightbig
, where land uare respectively the lower
and upper time limits within which ncan attempt k. There
may also be an additional so-called downtime if the agent
executes the task successfully, e.g., the time for a robot armto transfer the object to the bin. We represent task dur ation
123
Autonomous Robots
uncert ainty as
τnk(t)=Prob/bracketleftbig
ncompletes kwithin time t/bracketrightbig
. (1)
We assume knowledge of this cumulative distribution as
part of the problem speciﬁcation, typical for task scheduling
under uncertainty (Chaari et al. 2014 ); the particular model is
domain-dependent. By deﬁnition, the conjunction of Wand
τimposes an upper bound on task completion prob ability ,
i.e.,
Prob/bracketleftbig
ncompletes k/bracketrightbig
≤τnk/parenleftBig
tu
nk−tl
nk/parenrightBig
. (2)
For all unsuccessful tasks, the system incurs a penaltyof/summationtext
kJ(k)units. An agent can attempt only one task at a time.
We seek an allocation policy that minimizes the expected
cumulative penalty due to unsuccessful tasks . An allocation
policy πis a mapping from the agents to the tasks and their
respective attempt times, i.e. π:[N]→[ K]×[ T]. Since
there is uncertainty about task completion, a single-shot allo-
cation is insufﬁcient. Of course, the attempt times for futuretasks depend on when the earlier tasks are actually executed
(successfully or unsuccessfully). Our optimization problem
is
argmin
π∈ΠE⎡
⎣/summationdisplay
k∈[K]1[k]·J(k)|π⎤
⎦
s.t. t∈Wnk∀(k,t)∈π(n),(3)
where the indicator function 1[k]= 1i ft h et a s k kremains
incomplete at the end of the horizon, and Πis the set of
all possible allocation policies. The constraint enforces thatan agent attempts a task within the valid time window. The
expectation is over the task execution success distribution for
the allocation policy. For the rest of the discussion, we willassume that J(k)=1, i.e., all tasks are equally important;
this objective is the unweighted tardy jobs penalty (Pinedo
2012 ).
Multi-robot task allocation algorithms have been devel-
oped for various different objective functions (Gini 2017 ).
Our choice to minimize the number of missed tasks is astandard one for allocation problems with time windowconstraints (Lau et al. 2003b ). This objective is entirely inde-
pendent of the the underlying task, which makes it more
domain-agnostic than some other ones. Its additive natureis also useful for how we design and prove the algorithmic
properties of our proposed approach.
The discrete-time rolling horizon formulation described
above is fairly expressive and useful. We are concerned
with high-level allocation rather than the underlying task
execution, so we avoid the added complexity of continuous-time representations. The underlying tasks typically involve
Fig. 2 The illustrated conveyor belt has N=3a r m sa n d K=5
objects. The belt is of unit length, and each arm’s workspace spans 0 .3
units (dashed lines are the limits). Given the belt speed, the agent-tasktime window for any arm-object pair is at most 5 s
time-constrained trajectory planning, for which there are
well-established models and methods (Laumond 1998 ). Fur-
thermore, we can interleave planning and execution suitably
and recompute an allocation policy when new tasks appearonline (and we do so in practice).
3.1 Motivating examples
We describe two distinct robotics settings to instantiate ourformulation. First, consider the previously introduced exam-ple of robot arms along a conveyor belt (see Fig. 2). Each arm
has an associated collection bin for objects picked up from
the belt. The objects appear on the belt through an externalprocess. The arms take varying amounts of time for pick-ing, depending on the quality of the grasp strategy or gripper
attributes. Arms have ﬁnite reach, and an object may not be
picked up before it goes out of reach. Objects missed byall arms must be sorted by hand afterwards. The goal is to
successfully pick-and-place as many objects, or equivalently,
miss as few objects as possible.
Second, consider on-demand multi-drone dispatch for
package delivery in a city (note the underlying similarities
to the previous example). Delivery tasks arise through anexternal process of customer requests. Drones take vary-ing amounts of time to travel from the product depot to the
package delivery location, depending on ﬂight conditions.
Requests arrive with time windows, such that drones mustwait until the window starts to deliver the product to the
customer, and late deliveries are penalized. Over a certain
time horizon, our objective is to minimize the number of latedeliveries.
3.2 Computational challenges
To motivate our approach, we brieﬂy discuss the problemcomplexity. By the multi-robot task allocation taxonomyof Gerkey and Mataric ( 2004 ), the deterministic version of
our problem is ST-SR-TA, i.e. a single robot (SR) executes
a single task (ST) at a time, where tasks are time-extended(TA) rather than instantaneous. The ST-SR-TA is an instance
123
Autonomous Robots
of an NP-Hard scheduling problem, speciﬁcally multi-agent
scheduling with resource constraints (Garey and Johnson1975 ). The uncertainty of task execution success due to time
windows exacerbates this difﬁculty (Gini 2017 ). Finally, new
tasks streaming in require our approach to interleave plan-ning and execution effectively, e.g., by replanning at taskarrivals (Cordeau and Laporte 2007 ).
4 Hierarchical multi-robot task allocation
Our key algorithmic challenges are sequential planning
under uncertainty of task completion and multi-agent coordi-
nation of allocations. The joint multi-agent planning problem
is computationally prohibitive for large settings (Boutilier1996 ); most closely related previous works either use sim-
plifying approximations for planning and optimization (Tim-
otheou 2011 ; Hyland and Mahmassani 2018 )o rs i m p l e
coordination heuristics (Kartal et al. 2016 ; Mataric et al.
2003 ).
In contrast, we address the challenges hierarchically in a
two-layer approach called Stochastic Conﬂict-Based Alloca-
tion (SCoBA). At the low level, we independently determine
the optimal task attempt sequence for each individual agent,
ignoring other agents. At the high level, we resolve potentialconﬂicts in assigned tasks across multiple agents to obtain a
valid multi-robot allocation. In this section, we will elabo-
rate on the two layers and how they come together in SCoBA.We will then discuss brieﬂy how we interleave planning andexecution online and how SCoBA can exploit sparse agent
interactions using coordination graphs.
4.1 Low-level: single agent policy
We ﬁrst consider the perspective of an individual agent, inde-
pendent of the other ones. From the deﬁnition in Sect. 3,w e
are given the set of current tasks, corresponding time win-
dows, and task completion uncertainty distribution, and wewant a task attempt policy tree for the agent. Since task exe-
cution is stochastic, the ﬁrst time an agent can attempt a task
depends on what the agent has attempted before it. We makea simplifying approximation to compute the policy tree: the
agent attempts a task as soon as possible and observes the
outcome at the end of the window . This approximation col-
lapses the temporal dimension by treating tasks as discreteevents rather than extended ones.
We illustrate the policy tree search process for a single
robot n
1and three tasks (objects) k1,k2,k3in Fig. 3. First,
we sort tasks in increasing order of the start of their time
window. Then, we sweep along the time axis and update the
tree at every event point , i.e., the start or ﬁnish of the window
(and the end of the downtime in case the task is successful).The updates to the policy tree depends on the event point
(start/ﬁnish/downtime).
For the start of a time window, we introduce two new
decision nodes (ovals) to attempt ( ↔)o rl e a v e( /notarrowboth)t h et a s k
respectively. At the end of a time window and the down-time, we introduce outcome nodes (rectangles) respectively
for failure or success, where the outcome probability p
depends on the minimum feasible start time for the attempt,
which in turn depends on the speciﬁc branch of the tree. Forinstance, notice in Fig. 3the three copies of the decision
node ( n
1↔k2), with different probabilities, depending on
whether it was attempted after the failure, success, or non-attempt of task k
1. This difference is due to the time left to
complete the task, e.g., a non-attempt of k1leaves the most
time and highest probability to complete k2.
The leaves of the binary policy tree are annotated with the
cumulative penalty along their branches, e.g., a penalty of 1
for each unsuccessful task. We then use dynamic program-
ming to propagate values upwards from the leaves to the root.For a pair of outcome node siblings, we set the parent’s value
(denoted as V) to the expected value of its children,
V(parent ):=p·V(Fail)+(1−p)·V(Succ). (4)
For a pair of decision node siblings, the parent’s value is
the minimum of the children’s, i.e.,
V(parent ):=min{V(child1 ),V(child2 )}. (5)
In the running example of Fig. 3,w eh a v e V(root)=
min{V(n
1↔k1),V(n1/notarrowbothk1)}. The resulting tree encodes
the policy that minimizes the agent’s expected penalty for all
tasks up to the planning horizon, and V(root)is the value of
this expected penalty. We obtain the next task assigned to theagent by following child nodes of minimum value until the
ﬁrst attempt node (e.g., n
1↔k1).
4.2 High-level: multi-agent coordination
The policy tree determines the approximately optimal task
attempt sequence for an individual agent (approximate dueto the temporal simpliﬁcation mentioned earlier). The treesearches are independent of each other, so two agents may
have conﬂicting allocations. Since our objective function
depends on all agents, breaking ties naïvely could yield
arbitrarily poor global allocations . Multi-agent pathﬁnd-
ing algorithms face a similar challenge and have to resolve
inter-agent conﬂicts between shortest paths (Felner et al.2017 ). Conﬂict-Based Search is an effective strategy for this
problem (Sharon et al. 2012 ); by decoupling single-agent
path planning and inter-path conﬂict resolution, it is efﬁ-cient in practice without losing optimality. It has even been
123
Autonomous Robots
Fig. 3 The low-level routine of SCoBA generates the policy tree over
valid tasks for an individual agent, speciﬁcally, by sweeping along thetime axis and branching on the start or ﬁnish of a task’s time window. Atthe start of a window, two new decision nodes (ovals) are introduced: to
attempt (↔)o rto leave (/notarrowboth) the task respectively. At the end of a time
window and the downtime, the outcome nodes (rectangles) depict failure
or success. After the tree generation, dynamic programming propagatesthe values from the leaves to the root. The probability values p=0.03,
p
/prime=0.1,p/prime/prime=0.9 are just hypothetical values that illustrate how the
same attempt node (n1↔k2)has three different copies, with different
outcome probabilities (depending on the branch of the tree). Since weinterleave planning and execution, we can ignore task k
3as its time
window begins after the end of every task time window before it
used to solve a joint task assignment and path ﬁnding prob-
lem (Hönig et al. 2018 ).
We leverage the idea of inter-agent conﬂict resolution
from Conﬂict-Based Search. The high level of our algo-
rithm, SCoBA, searches a binary constraint tree (Fig. 4)
generated from conﬂicts between solutions for individualagents obtained from the low level, i.e., the policy tree
search. Two agents n
1andn2are in conﬂict if they are allo-
cated the same task kin overlapping time windows, i.e., if
(k,t1)∈π(n1),(k,t2)∈π(n2)and either t2∈Wn1,kor
t1∈Wn2,k.Aconstraint for an agent is a task excluded from
consideration by the tree search for that agent.
Each node in the constraint tree maintains (i) a set of con-
straints, i.e., tasks to ignore, for each agent, (ii) a multi-agent
allocation that respects all constraints, and (iii) the cost of
the allocation. For SCoBA, the cost of the allocation is thesum of expected penalties for each agent, where the expected
penalty for each agent is the value of the root node of its pol-
icy tree. The allocation cost is used as the criteria for best-ﬁrst
search on the constraint tree ; this best ﬁrst search continues
until it ﬁnds a conﬂict-free allocation.
4.3 Stochastic conflict-based allocation (SCoBA)
Algorithm 1 describes SCoBA, using the tree search of Sect. 4.1
as the PlanTree subroutine. Its structure is similar to an
earlier presentation of Conﬂict-Based Search by Felner et al.Fig. 4 A constraint tree node with a conﬂict in the allocation generates
two children with corresponding constraints on the conﬂicting agents(n
1andn3)and task ( k1). Best-ﬁrst search on the constraint tree returns
the ﬁrst high-level node with a conﬂict-free allocation. The solutionnode of the constraint tree is coloured green and the others are colouredred
(2017 ). The constraint tree is initialized with the root node,
which has an empty constraint set and the allocation from run-
ning PlanTree for each individual agent (lines 2–6). When a
high-level node is expanded, the corresponding allocation is
checked for validity (line 9). If there is no conﬂict, we return
this allocation as the solution. Otherwise, for every conﬂictbetween two or more agents, new child nodes are added,
where constraints are imposed on the agents involved (line
15). A child constraint tree node inherits its parent’s con-straints and adds one more constraint for a particular agent.
123
Autonomous Robots
Consider the simple illustrative example in Fig. 4.T h e
root node has agents n1and n3both assigned to task k1.
This conﬂict yields two possible constraints, one inherited
by each of the two child nodes. The ﬁrst constraint excludes
k1from the recomputed policy tree search for n1. The second
constraint does the same for n3. For each new (non-root)
node, the low level tree search is only re-run on the agent for
which the constraint is added (line 19). Both of the resulting
child nodes are conﬂict-free, but the left one, with a lowerallocation cost of 2 .6, is returned as the solution.
Our problem setting is both online and stochastic. However,
under some simplifying assumptions, we can establish opti-mality and completeness properties for SCoBA. We derive
them from the corresponding optimality and completeness
proofs of the Conﬂict-Based Search algorithm for multi-agent pathﬁnding (Sharon et al. 2012 ).
Algorithm 1 Stochastic Conﬂict-Based Allocation
1:procedure Main ([N],[K],T,W n,k,τ nk∀n, k )
2: Initialize A as the root3: A.soln ←PlanTree (n,[K],T,W ,τ )∀n
4: A.constr ←{ } ⊿Empty constraint set
5: A.cost ←SumOfIndividualCosts( A.solution )
6: Insert Ainto Open ⊿Open list of Constraint Tree
7: while Open not empty
8: S←PopBest( Open ) ⊿Min. Cost Allocation
9: ifS.soln is valid ⊿No conﬂicts
10: return S.soln
11: C←ﬁnd-conﬂicts( S) ⊿Inter-agent conﬂicts
12: for all conﬂicts ( n, k )∈C
13: A←GenerateNewChildPlanTree (S, n, k )
14: A.cost ←SumOfIndividualCosts( A.soln )
15: Insert Ainto Open
16: procedure GenerateNewChildPlanTree (S, n, k )
17: A.constr ←S.constr ∪k⊿ Task to exclude
18: A.soln
←S.soln
19: A.soln ←PlanTree (n,[K]\A.constr, T, W, τ )
20: return A
Proposition 1 If (i) no new tasks are added online, (ii) the tree
search is executed to the full horizon, and (iii) task completion
is determined at the end of the time window, then SCoBA isoptimal in expectation, i.e., SCoBA minimizes the expected
number of incomplete tasks at the end of the time horizon.
Proof Conﬂict-Based Search yields optimal multi-agent
solutions (paths) if two conditions hold: (a) the low-level
routine yields optimal solutions for individual agents and (b)
the overall multi-agent objective is the sum-of-costs of indi-vidual agent solutions. SCoBA uses the same multi-agent
conﬂict resolution logic as Conﬂict-Based Search and will
inherit its optimality property if it satisﬁes the two sufﬁcientconditions.
We ﬁrst show that (a) is true for SCoBA. Its low-level
single-agent routine uses dynamic programming with for-ward tree search to obtain a policy tree. By construction,this routine computes a policy that is optimal under expecta-
tion for any discrete, ﬁnite-horizon Markov Decision Process(MDP) if it conducts the tree search exhaustively, i.e., to the
full horizon . The expectation is over the uncertainty of the
action outcomes. Assumption (i) ensures we know all infor-mation of future tasks at the initial state of the agent andAssumption (ii) ensures the tree search is exhaustive.
Recall that in Sect. 4.1we treated time-windows as dis-
crete events, a simplifying approximation. Under Assump-tion (iii), this temporal approximation now becomes exact;
if we begin each task at the earliest possible time-step and
continue until the end of the window, each task attempt hasa single probability mass function over the two outcomes of
success and failure. Therefore, the single agent problem is
a discrete ﬁnite-horizon MDP and the low-level policy treesearch routine is optimal in expectation. That is, for eachagent n, the policy π
∗(n)obtained from PlanTree satisﬁes
π∗(n)=argmin
π(n)∈Π(n)E⎡
⎣/summationdisplay
k∈π(n)1[k]·J(k)|π(n)⎤
⎦, (6)
subject to the constraints in Eq. 3, where Π(n)is the set
of all possible policy trees for agent n. With some abuse of
notation, we use k∈π(n)to denote all the tasks allocated to
agent n.Thus, SCoBA satisﬁes condition (a) from above .
Now we examine condition (b). Our overall objective is
to minimize the expected total cost for tasks that the multi-agent allocation policy πfails to complete. Denote this cost
asJ(π).F r o mE q . 3,w eh a v e
J(π)=E⎡
⎣/summationdisplay
k∈[K]1[k]·J(k)|π⎤
⎦. (7)
By linearity of expectation, Eq. 7becomes
J(π)=/summationdisplay
k∈[K]E/bracketleftbig
1[k]·J(k)|π/bracketrightbig
. (8)
SCoBA resolves conﬂicts between single-agent allocation
policies to ensure that no two agents are allocated to the
same task. Therefore, we can split the summation over all
tasks based on their allocation to agents and rewrite Eq. 8as
J(π)=/summationdisplay
n/summationdisplay
k∈π(n)E/bracketleftbig
1[k]·J(k)|π(n)/bracketrightbig
. (9)
where, again, we use k∈π(n)to denote the tasks uniquely
allocated to agent n.
Consider the cost function for the single-agent policy
in Eq. 6. Recall from Sect. 4.1 that the minimizing value
of the single-agent cost is V (root n)where root nis the root
123
Autonomous Robots
of the single-agent optimal plan tree π∗(n), i.e.,
V(root n)= min
π(n)∈Π(n)E⎡
⎣/summationdisplay
k∈π(n)1[k]·J(k)|π(n)⎤
⎦.(10)
Using linearity of expectation, Eq. 10becomes
V(root n)= min
π(n)∈Π(n)/summationdisplay
k∈π(n)E/bracketleftbig
1[k]·J(k)|π(n)/bracketrightbig
. (11)
Finally, using Eqs. 9–11and that no two agents are allo-
cated to the same task, we have
min
π∈ΠJ(π)=min
π∈Π/summationdisplay
n/summationdisplay
k∈π(n)E[1[k]·J(k)|π(n)]
=/summationdisplay
nmin
π(n)∈Π(N)/summationdisplay
k∈π(n)E[1[k]·J(k)|π(n)]
=/summationdisplay
nV(root n),(12)
which is precisely the sum-of-costs of the individual agent
solutions. Thus, SCoBA satisﬁes condition (b) from above .
Therefore, SCoBA is optimal in expectation under the
given assumptions. /intersectionsq/unionsq
Proposition 2 Under the assumptions of Proposition 1,
SCoBA is complete. If a valid allocation exists, SCoBA
returns it.
Proof As with the proof for optimality, our proof for com-
pleteness follows that of the original Conﬂict-Based Search.
There, the authors showed completeness by establishing thatif a valid multi-agent path exists, the high-level constrainttree has a ﬁnite number of nodes, i.e., an upper bound on the
number of generated nodes. The upper bound arises because
a multi-agent path problem has a ﬁnite number of constraintsand Conﬂict-Based Search generates at least one new con-
straint per new high-level node. Since it executes best-ﬁrst
search with monotonically non-decreasing cost on the con-straint tree, it is guaranteed to ﬁnd a valid solution in ﬁnite
time if one exists.
We now apply the same reasoning to SCoBA. First, every
new high-level node Ain SCoBA ’s constraint tree must have
at least one more constraint than its predecessor A
/prime, derived
from a conﬂict in the solution that A/primerepresents. Second, the
maximum possible number of constraints is the number ofways Ktasks can be distributed across Nagents multiplied
by the time horizon, i.e.,
(K+N−1)!
K!(N−1)!·T.
The ﬁnite number of possible constraints and the addi-
tion of at least one new constraint per new node implies
a ﬁnite number of nodes in the constraint tree. SCoBA ’s
high-level routine uses systematic best-ﬁrst search over theconstraint tree, whose expanded nodes have monotonicallynon-decreasing cost. Therefore, a conﬂict-free allocation, if
it exists, must be found after expanding a ﬁnite number ofSCoBA constraint tree nodes. Thus, SCoBA is a complete
algorithm. /intersectionsq/unionsq
We reiterate that SCoBA exhibits the above properties
only under simplifying assumptions about the multi-robottask allocation problem. In practice, these assumptions do
not hold in our experimental domains, with the possible
exception of (ii), i.e., running the policy tree search to thefull horizon. As we describe in Sect. 3.2, our problem for-
mulation adds two distinct axes of difﬁculty (task execution
uncertainty and new tasks online) on top of an already NP-Hard problem. These assumptions are thus necessary to curbthe intractability of the original formulation and make claims
that are useful to know even in the restricted setting.
4.4 Interleaving planning and execution
We do not assume any particular model for new tasks arrivingonline as it would depend on the nature of tasks in a speciﬁc
domain; contrast our task generation parameters in Sect. 5.2
with those in Sect. 5.3. A standard technique for dealing
with online updates in allocation or scheduling problems is
tointerleave planning and execution , i.e., compute a solution
up to some horizon based on the current information, executethe ﬁrst part of that solution, and recompute a new solution in
either a periodic or event-driven fashion (Church and Uzsoy
1992 ).
SCoBA is concerned with high-level allocation, so execu-
tion in this context simply refers to simulating the attempt of
a task by the agent allocated to it (in practice this simulationdepends on the speciﬁc domain). For replanning, we use theevent-driven replanning strategy, where each event is the out-
come of the next task attempt by any agent . After each such
event, we have several updates to consider: the attemptedtask is either removed or retained in the set of available tasks
depending on whether the attempt was a success or failure;
all new tasks that arrived between the previous and currentevent are added; time windows of any existing but yet unat-
tempted tasks may have changed as well. After incorporating
all these updates, we re-run SCoBA from scratch on the cur-rent problem.
SCoBA ’s elegant representation makes interleaving plan-
ning and execution straightforward at both levels. For the
single agent policy tree search, we truncate the search horizonbased on domain knowledge or computation requirements.
In our implementation, we run the sweep until the ﬁrst task
whose time window begins after the downtime of all tasksbefore it ( k
3in Fig. 3). For the multi-agent coordination, we
can set a threshold on the number of high-level conﬂicts,
once again based on domain or computation requirements.If the threshold is exceeded, we return the current high-level
123
Autonomous Robots
solution. In this case, for robots allocated to the same task,
we break ties arbitrarily and keep the unassigned robots freefor the next event. In principle, we can thus plan for a vari-
able number of robots as well, because the effective number
of available robots at each event would change depending onthe current status.
4.5 Coordination graphs
SCoBA works with any arbitrary conﬁguration of agents,but we can use coordination graphs (CGs) from multi-agentdecision-making for more efﬁciency (Kok et al. 2003 ). In
CGs, each node represents an agent, and each edge encodes a
dependency between them, such that only connected agentsneed to coordinate allocations. The choice of coordination
graph for a problem is domain-dependent. For instance,
the arms are ordered along the conveyor belt and theirworkspaces are mutually exclusive, but we want to accountfor objects near a boundary that may enter the next workspace
soon (if unattempted). Thus, the coordination graph is a
directed chain from the ﬁrst arm to the last.
The CG structure impacts the high-level multi-agent coor-
dination stage of SCoBA. The absence of an edge between
two agents implies that their sets of possible tasks are dis-joint , i.e., they cannot have conﬂicting allocations. Therefore,
in practice, SCoBA need not consider dependencies between
all the agents. If the CG is directed, as in the conveyor belt,we run the tree search for agents along a topological order-ing of the CG. For any agent, we exclude the tasks already
assigned to its predecessors. By construction, we will obtain
a conﬂict-free allocation at the end, without any child nodesbeing generated in the high-level constraint tree). If the CG
is undirected, as in multi-drone delivery, such a topological
ordering is not feasible, and conﬂicts may be unavoidable.However, if the CG has multiple connected components, then
nodes (agents) in different components cannot conﬂict with
each other, so we can run SCoBA on each component inparallel.
5 Experiments and results
The primary metric for evaluating SCoBA is the accumulatedpenalty for unsuccessful tasks. We will also evaluate its scal-
ability to problem size. We ﬁrst outline the range of methods
we use to baseline SCoBA. We then present and discuss theresults for multiple performance metrics on simulations for
each of our two distinct robotics-inspired domains: conveyor
belt pick-and-place and on-demand multi-drone delivery dis-patch. We use the Julia programming language (Bezanson etal.2017 ) on a 16 GiB RAM machine and a 6-core 3 .7 GHz
CPU for all simulations.
1
5.1 Baselines and evaluation
We use multiple complementary algorithms as baselines:
1.EDD : The Earliest Due Date heuristic assigns each agent
to the task with the nearest time window deadline (Pinedo2012 ).
2.Hung arian: An unbalanced Hungarian algorithm, where
the edge weight for an agent-task pair is the probabil-ity of successful task completion (Munkres 1957 ). This
method is a special case of a general purpose network-
ﬂow approach that assigns one agent to multiple tasks ata time (Timotheou 2011 ).
3.MCTS : A recent Monte-Carlo Tree Search approach
speciﬁcally for multi-robot task allocation (Kartal et al.2016 ). The tree search is conceptually similar to ours
(albeit with Monte Carlo sampling of outcomes) but it
uses arbitrary priority orderings among agents to coordi-
nate decisions and control the tree branching factor. Forall scenarios, we used 100 Monte Carlo trials to compute
each action, an exploration constant of 0 .1, and a tree
depth of 20.
The baselines cover a range of approaches for multi-robot
allocation from scheduling to sequential decision-makingunder uncertainty. Both EDD and Hungarian are reactive,i.e., do not plan sequentially. The latter optimizes for multi-
ple agents, unlike the former. MCTS is a model-based online
method that plans sequentially by framing the allocationproblem as a Markov Decision Process. As with SCoBA,
we implemented all baselines in Julia. For MCTS, we used
the POMDPs.jl framework for modeling and solving MarkovDecision Processes (Egorov et al. 2017 ). The rollout policy
of MCTS used the Earliest Due Date heuristic.
We compare SCoBA to the baselines on both the met-
rics of total unsuccessful tasks and computation time. Thelatter comparison is not apples-to-apples, because the base-
lines have different input interfaces that impose different
restrictions on problem size, depending on the domain. Fur-thermore, computation time is not an optimizing metric but
rather a satisﬁcing one; our objective for SCoBA ’s compu-
tation time is to be reasonable for the requirements of therespective domains.
5.2 Conveyor belt domain
In this domain, three identical robot arms are arranged along amoving conveyor belt, picking objects from the belt and plac-
1The code is available at https://github.com/sisl/SCoBA.jl .
123
Autonomous Robots
Fig. 5 We generate tasks for the conveyor belt domain by reﬂecting the setup in space and time. When the objects placed on the virtual belt (shaded
with a gradient ﬁll) by the reﬂected arms cross the Y -axis, they appear as new tasks on the real belt (with solid grey ﬁll)
ing them in collection bins (Fig. 1a). We design an abstracted
simulation of the scenario (Fig. 2), scaled along an X-axis
of unit length. The arms have mutually exclusive adjacentworkspaces of 0 .3 units each, from x=0.05 to x=0.95.
New tasks arrive as new objects appearing at the head of the
belt.
5.2.1 Task generation and scenario parameters
To generate new tasks, we reﬂect the conveyor belt setup in
space and time and create a virtual assembly line (on the leftside of Fig. 5) where arms pick objects from bins and place
them on a virtual belt. When the virtual belt crosses x=0
(the Y -axis), the virtual objects appear as new real objects.As we explain shortly, this process allows us to generate tasksequences that are solvable under the assumption of no grasp
uncertainty.
Three scenario parameters instantiate a conveyor belt
problem and affect its difﬁculty:
(i)Belt speed: The speed of the belt determines the effective
time window for each arm-object pair, e.g., 5 s in Fig. 2.
If task execution is successful, each arm has a downtime
ofΔt=2s to deposit the object in the bin. We expect
performance to degrade as belt speed increases.
(ii)New object probability: At each timestep, a virtual arm
drops its object onto the virtual belt with some Bernoulli
probability (all virtual arms have the same such proba-
bility). The drop location is uniformly sampled withinthe virtual arm’s workspace. After a virtual arm drops
an object, it moves to the virtual bin to collect the next
one. We expect performance to degrade as new objectprobability increases.
(iii) Grasp success probability: We use two different mod-
els of uncertainty over task completion due to imperfectgrasping. The ﬁrst is a geometric distribution where pis the Bernoulli probability of a successful pick by each
arm (since arms are identical, they all have the same grasp
success probability); the corresponding cumulative prob-ability function from Eq. 1is
τ
nk(t)=1−(1−p)t,where t∈N. (13)
The second is a uniform distribution over the maximum
number of time-steps Tmaxthat an object can spend in an
arm’s workspace, where the total probability of successby each arm is p. The corresponding cumulative function
from Eq. 1is
τ
nk(t)=t
Tmax·p,where t,Tmax∈N. (14)
We expect performance to improve as pincreases, but
the second model yields much more adversarial scenarios
than the ﬁrst one.
5.2.2 Competitive performance against oracle
For online algorithms like SCoBA, a standard metric is the
competitive performance against an oracle with complete
information, i.e., full lookahead. Our task generation pro-
cess allows an ablation study for the effect of lookahead alone
(deconfounded from uncertainty). The process ensures thatwhen there is no grasp uncertainty, there exists at least one
allocation policy that successfully completes all tasks. One
such policy is that which reﬂects the virtual setup itself. Forinstance, it allocates the ﬁrst arm, i.e., the arm closest to the
Y -axis, to pick up the real objects whose corresponding vir-
tual forms were placed by the ﬁrst arm’s reﬂection (similarlyfor other arms). The attempt location for any arm-object pair
is the reﬂection, on the real belt, of the drop point of the
corresponding virtual object from the corresponding virtualarm.
123
Autonomous Robots
Table 1 The mean proportions of objects lost per trial by SCoBA when
grasping is perfect
Belt speed New object probability
(units/s) 0 .50 .75 1 .0
0.04 0 .00 .00 .0
0.07 2 .7×10−57.9×10−51.3×10−4
0.11 .7×10−43.7×10−45.2×10−4
We vary the other two parameters, that affect the rate of appearance of
new tasks. The negligible values demonstrate SCoBA ’s strong compet-itive performance relative to the oracle
To obtain the perfect allocation policy in practice (assum-
ing no uncertainty), we would need complete access to the
virtual generator for a problem instance. However, we do not
need to know the oracle itself, only its performance, which isan upper bound on the performance of any other method. If a
perfect allocation policy exists, an oracle with full lookahead
would have full success rate for any task sequence generatedby our process. We can thus evaluate the competitive per-
formance of SCoBA simply by evaluating the proportion of
tasks that it fails to complete. The smaller this number, the
better is SCoBA’s competitive performance .
We set p=1 with the geometric distribution of grasp suc-
cess probability for all arms and jointly varied the other two
parameters, belt speed and new object probability. For eachtrial in a setting, we simulated T=500 time-steps (seconds)
and evaluated the proportion of objects missed by SCoBA rel-
ative to the total number of objects. We compute the averageof this proportion-per-trial over 100 trials (standard error neg-
ligible) in Table 1. The low magnitudes demonstrate SCoBA ’s
robustness to insufﬁcient lookahead. With increasing valueof either parameter, performance degrades.
5.2.3 Unsuccessful task penalty
We varied all three scenario parameters independently and
compared the fraction of missed objects for SCoBA versus
the other baselines. Figure 6demonstrates the results, with
the upper panel for the geometric distribution of grasp prob-
ability and the lower panel for the uniform distribution.
We average all numbers over 100 trials with standard error
bars. For each subplot, only one parameter varies (the x-label), while the other two stay at their default values: grasp
probability p=0.75 for the geometric and p=0.8f o r
the uniform distributions, 0 .07 units/s for belt speed, and
0.75 for new object probability. We use a different range
of values for grasp probability in the lower panel (the left-
most subplot) because the uniform distribution case is muchmore adversarial than the geometric one. This increase in
difﬁculty reﬂects in the much higher absolute values over all
settings in the lower panel. Note that p=1.0i nt h el o w e r
left-most subplot does not mean that grasping is perfect inthose scenarios, just that the maximum probability of success
from Eq. 14is 1 (if the arm can attempt the object at its ﬁrst
point of entry into the workspace).
SCoBA considerably outperforms the other baselines
across all settings . Furthermore, its performance degrades
or improves as expected relative to the change in each prob-lem parameter (e.g., more objects missed with increasing
new object probability). Among the baselines, the reactive
Hungarian method has the best performance, likely becausesequential deliberation is not as crucial with non-overlapping
workspaces and small downtime (unlike in the next domain).
The MCTS baseline (Kartal et al. 2016 ) assumed a task
environment similar to our drone delivery domain. Thus, for
the conveyor belt domain, we had to make our own design
choices for the state and action space; we chose to discretizethe belt into uniform width slots, each with some number ofobjects at every time-step, for the system state. The action
space for each arm is the set of slots in its corresponding
workspace. We used 0 .02 units per slot for MCTS, which is
a resolution ﬁne enough to ensure that MCTS does not lose
any information compared to SCoBA. Too much ﬁner would
have led to a prohibitively large state space.
5.2.4 Scalability
In this domain, the Coordination Graph is a directed chain
(see Sect. 4), so the computational bottleneck for SCoBA
is the policy tree search (multi-agent coordination is triv-
ial). In Table 2we report average tree search computation
times for a single arm with an increasing number of objects
(scattered throughout the arm workspace). Empirically, we
observe that the computation time is roughly cubic in thenumber of objects, and the wall clock times are quite reason-able.
The table also reports the computation times for the Hun-
garian algorithm. It is much faster than SCoBA, but unlike
SCoBA, it does not plan sequentially and only matches each
arm to one object at a time. EDD is a simple heuristic that
does not plan jointly for all agents, so its computation time isnot useful. For MCTS, the action computation time is inde-
pendent of the number of the objects because for the action
space, we discretize the conveyor belt into 15 slots per agent.With 100 iterations and a search depth of 20, the averageMCTS action takes 0 .1 s to compute. Increasing the MCTS
computation time with more iterations or samples would not
have yielded different decisions and hence would not havechanged its performance on the task penalty.
5.3 Drone delivery domain
In the second domain, we dispatch drones from depots to
deliver packages around a city-scale area, subject to deliv-ery time window constraints. Each drone is associated with
123
Autonomous Robots
Fig. 6 Legend: EDD Hungarian MCTS SCoBA. All results
are averaged over 100 trials, with T=500 time-steps per trial (standard
error bars are negligible). Top row: geometric distribution of grasp suc-cess probability (Eq. 13). Bottom row: uniform distribution (Eq. 14).The latter yields much more adversarial scenarios than the former, which
is why the bars in the lower panel are much higher across the board.On the metric of the fraction of unsuccessful tasks, i.e., objects missed,SCoBA consistently outperforms all other baselines
Fig. 7 Legend: EDD Hungarian MCTS SCoBA. Results are
averaged over 100 trials, each of T=100 time-steps. The upper and
lower panels are respectively for the Epanechnikov (Eq. 15)a n dN o r m a l(Eq. 16) distributions over travel time. For the drone delivery domain,
on the primary metric of the fraction of late package deliveries, SCoBAoutperforms the baselines on all but one setting
123
Autonomous Robots
Table 2 The low computation times demonstrate that SCoBA ’s tree
search for an individual arm scales with the number of objects in thearm’s workspace
Objects SCoBA Hungarian
40 9 ×10−4s1 .7×105s
80 0 .004 s 3 .1×10−5s
120 0 .013 s 4 .3×10−5s
160 0 .029 s 5 .7×10−5s
200 0 .052 s 8 .3×10−5s
The Hungarian algorithm is faster as it solves a one-shot deterministic
problem rather than a sequential stochastic one
a speciﬁc depot and can carry only one package at a time;
therefore, each drone makes trips from its depot to a desti-nation and back to the same depot. Every package request
arrives with a speciﬁc time window within which a drone
from the team must deliver it.
Our setup is based on our recent work for multi-drone
delivery over ground transit (Choudhury et al. 2020b ). To
estimate drone travel time, we use the location-to-locationestimates from its North San Francisco scenario, which sim-
ulates deliveries over an area of 150 km
2(see Fig. 1b). We
pre-select geographic locations for up to 5 depots scatteredaround the city to ensure good coverage. Drones have a maxi-mum ﬂight range of 10 km, which restricts the set of possible
package deliveries for each drone. Two scenario parameters
affect the performance here:
(i)Drones and depots: The number of depots and the ratio
of drones to depots both impact the ability of the system
to dispatch agents to a given delivery location in time.
We distribute drones equally across depots. With bettercoverage, we expect performance to improve.
(ii)New request probability: A new delivery request arrives
per minute with some probability. Each delivery location
is sampled uniformly within a bounding box. We startwith a number of packages roughly 1 .5 times the num-
ber of drones in the scenario. With higher probability,
we expect performance to degrade. Each request has awindow duration sampled uniformly between 15 and 30
minutes.
Our reference framework gives us deterministic drone
travel-time estimates between a depot dand package loca-
tion p,s a y TT(d,p). Similar to the previous domain, we
use two different models for task completion uncertainty, i.e.,
travel time uncertainty based on TT(d,p): a ﬁnite-support
Epanechnikov distribution,
τ
n,k(t)∼Epan(μ=TT(d,p),r=TT(d,p)/3.0), (15)and an inﬁnite-support Normal distribution,
τn,k(t)∼N(μ=TT(d,p),σ=TT(d,p)/3.0). (16)
Unlike the previous domain, the uncertainty models here do
not have parameters that we vary to change the problem dif-
ﬁculty, and we do not expect either model to be much more
adversarial than the other. In each experiment for both mod-els, the true travel times between two locations are drawn
from Eqs. 15and 16respectively. Our synthetic distribution
choices are arbitrary but reasonable because a high meantravel time is likely to have higher variance, due to more
opportunities for delays or speedups.
5.3.1 Unsuccessful task penalty
We vary the scenario parameters and compare the fraction of
late package deliveries for SCoBA versus the other baselinesin Fig. 7. The upper panel is for the Epanechnikov travel time
distribution and the lower one is for the Normal travel time
distribution. We choose three sets of depot-and-drone num-
bers with complementary coverage properties, e.g., (3,18)
has fewer depots and a higher drone-depot ratio while (5,15)
has more depots but a smaller ratio. We vary the new request
probability, simulate T=100 time-steps (minutes) per trial,
and average results over 100 trials. SCoBA is generally the
best across all settings except in one (5 depots, 15 drones,
probability 1 .0) where MCTS is slightly better. Having more
drones per depot appears to be more inﬂuential than havingmore depots, e.g., the errors for (5,15)are higher than the
corresponding ones for (3,18).
The improvement in relative performance of the MCTS
baseline (Kartal et al. 2016 ) is not surprising. It is tailored for
vehicle dispatch problems, with search heuristics that exploit
the domain structure, e.g., agents have longer downtime toreturn to their depots, and the per-agent action space is a sub-
set of valid tasks rather than a discretization of a conveyor belt
into slots. The drone delivery domain also penalizes MCTSless for its inability to coordinate between agents, comparedto the conveyor belt. In drone delivery, each task has a single
time window for any agent to complete it in. In conveyor belt,
each object has effectively three time windows, one for eacharm; unlike MCTS, SCoBA can coordinate between arms
such that an earlier one delegates an object to a later arm to
improve overall system performance.
5.3.2 Scalability
High-level conﬂicts may occur in this domain, and SCoBA
will invoke its multi-agent coordination layer (on top of pol-
icy tree search) to compute a valid multi-agent allocation.
Therefore, in Table 3we report the mean and standard error
for the computation times of the full SCoBA algorithm (over
123
Autonomous Robots
Table 3 The mean and standard error (over 50 trials in each setting) for SCoBA ’s computation time on the multi-drone delivery domain, as well as
the mean times for Hungarian and MCTS
(Dep., Dr.,) 20 Requests 50 Requests 100 Requests
SCoBA Hung MCTS SCoBA Hung MCTS SCoBA Hung MCTS
(3,18)( 0.02,0.003) 0.00008 0 .007 (1.48,0.16) 0.0001 0 .008 (2.55,0.23) 0.0002 0 .009
(5,15)( 0.06,0.008) 0.00008 0 .006 (2.13,0.2) 0.0001 0 .007 (5.42,0.5) 0.0002 0 .009
(5,30)( 0.17,0.003) 0.0002 0 .01 (1.76,0.12) 0.0002 0 .013 (7.08,0.47) 0.0003 0 .016
All times are in seconds
50 different trials for each setting). We vary the number of
drones and depots and the number of tasks, i.e., the current
package delivery requests. The absolute wall clock valuesare reasonable considering the time-scale of operation of the
system in the real world is minutes and hours. Some sce-
narios have disproportionately high mean and variance dueto more high-level conﬂicts, a known behavioral property ofConﬂict-Based Search algorithms (Sharon et al. 2012 ).
Table 3also shows average times for the Hungarian and
MCTS baselines on the same settings; their standard errorsare negligible in comparison. As expected, Hungarian is
orders of magnitude faster than SCoBA. For MCTS the action
space is directly proportional to the number of tasks (unlike inthe previous domain) and the computation times vary slightly
across settings. The absolute values of MCTS are lower than
that of SCoBA; the former imposes an arbitrary orderingon agents and computes allocations for agents one-by-onebased on that order, thus circumventing all the complexity
of multi-agent coordination. As in the conveyor belt domain,
increasing the MCTS computation time with more iterationsor samples would not have changed its performance on the
task penalty.
6 Conclusion
We presented SCoBA, a hierarchical approach for multi-robot task allocation under uncertainty and time constraints.
In theory, SCoBA is optimal in expectation and completeunder mild technical assumptions. In practice, over two dis-
tinct domains, it has strong competitive performance against
an oracle, consistently outperforms a number of baselines,and is scalable in terms of computation time to both numberof agents and tasks.
6.1 Limitations and future work
We assume a known uncertainty model, which is typicalfor multi-robot task allocation algorithms. However, since
SCoBA is based on policy tree search, we could use it in-the-
loop with model-based reinforcement learning in case theuncertainty model needs to be estimated online. We focus onhigh-level allocation here, but we could integrate SCoBA in
a full pipeline for robotics applications.
SCoBA ’s computation time is sensitive to the number of
high-level conﬂicts; future work could run ablation stud-
ies to investigate how many conﬂicts arise in practice, how
many it resolves, and what the correlation is with compu-tation time. This analysis could motivate improvements tothe multi-agent conﬂict resolution level such as bounded
sub-optimal variants (Barer et al. 2014 ), better resolution
heuristics (Boyarski et al. 2015 ) and reasoning about the
stochasticity of conﬂicts.
Acknowledgements This work was supported by the Ford Motor
Company, National Science Foundation Grant Number 1941722 andNational Science Foundation Grant Number 1849952.
References
Ahmed, S., & Garcia, R. (2003). Dynamic capacity acquisition and
assignment under uncertainty. Annals of Operations Research,
124(1–4), 267–283.
Al-Hinai, N., & ElMekkawy, T. Y . (2011). Robust and stable ﬂexible job
shop scheduling with random machine breakdowns using a hybridgenetic algorithm. International Journal of Production Economics,
132(2), 279–291.
Albers, S. (1999). Better bounds for online scheduling. SIAM Journal
on Computing, 29 (2), 459–473.
Alonso-Mora, J., Samaranayake, S., Wallar, A., Frazzoli, E., & Rus,
D. (2017). On-demand high-capacity ride-sharing via dynamictrip-vehicle assignment. Proceedings of the National Academy of
Sciences, 114 (3), 462–467.
Barer, M., Sharon, G., Stern, R., & Felner, A. (2014). Suboptimal vari-
ants of the conﬂict-based search algorithm for the multi-agentpathﬁnding problem. In European conference on artiﬁcial intel-
ligence (ECAI) , pp. 961–962.
Behrens, J. K., Lange, R., & Mansouri, M. (2019). A constraint pro-
gramming approach to simultaneous task allocation and motionscheduling for industrial dual-arm manipulation tasks. In IEEE
international conference on robotics and automation (ICRA) ,
IEEE, pp. 8705–8711.
Bertsekas, D. P . (2005). Dynamic programming and optimal control .
Athena Scientiﬁc.
Bezanson, J., Edelman, A., Karpinski, S., & Shah, V . B. (2017). Julia:
A fresh approach to numerical computing. SIAM Review, 59 (1),
65–98.
Boutilier, C. (1996). Planning, learning and coordination in multia-
gent decision processes. In Conference on theoretical aspects of
123
Autonomous Robots
rationality and knowledge , Morgan Kaufmann Publishers Inc., pp.
195–210.
Boyarski, E., Felner, A., Stern, R., Sharon, G., Tolpin, D., Betzalel, O.,
& Shimony, S. E. (2015). ICBS: Improved conﬂict-based searchalgorithm for multi-agent pathﬁnding. In International joint con-
ference on artiﬁcial intelligence (IJCAI) , pp. 740–746.
Burkard, R. E., Dell’Amico, M., & Martello, S. (2009). Assignment
problems ,S I A M .
Campbell, T., Johnson, L., & How, J. P . (2013). Multiagent allocation
of Markov decision process tasks. In American control conference
(ACC) , IEEE, pp. 2356–2361.
Chaari, T., Chaabane, S., Aissani, N., & Trentesaux, D. (2014).
Scheduling under uncertainty: Survey and research directions.InInternational conference on advanced logistics and transport,
ICALT , pp. 229–234.
Che, A., Kats, V ., & Levner, E. (2017). An efﬁcient bicriteria algo-
rithm for stable robotic ﬂow shop scheduling. European Journal
of Operational Research, 260 (3), 964–971.
Cheung, R. K., Hang, D. D., & Shi, N. (2005). A labeling method
for dynamic driver-task assignment with uncertain task durations.Operations Research Letters, 33 (4), 411–420.
Choudhury, S., Gupta, J. K., Kochenderfer, M. J., Sadigh, D., & Bohg,
J. (2020a). Dynamic multi-robot task allocation under uncertainty
and temporal constraints. Robotics: Science and Systems Founda-
tion.
Choudhury, S., Solovey, K., Kochenderfer, M. J., & Pavone, M. (2020b).
Efﬁcient large-scale multi-drone delivery using transit networks.InIEEE international conference on robotics and automation
(ICRA) .
Church, L. K., & Uzsoy, R. (1992). Analysis of periodic and event-
driven rescheduling policies in dynamic shops. International
Journal of Computer Integrated Manufacturing, 5 (3), 153–163.
Coltin, B., & V eloso, M. M. (2014). Online pickup and delivery planning
with transfers for mobile robots. In IEEE international conference
on robotics and automation (ICRA) , IEEE, pp. 5786–5791.
Cordeau, J., & Laporte, G. (2007). The dial-a-ride problem: Models and
algorithms. Annals of Operations Research, 153 (1), 29–46.
Dertouzos, M. L., & Mok, A. K. (1989). Multiprocessor online
scheduling of hard-real-time tasks. IEEE Transactions on Software
Engineering, 15 (12), 1497–1506.
Egorov, M., Sunberg, Z. N., Balaban, E., Wheeler, T. A., Gupta, J.
K., & Kochenderfer, M. J. (2017). POMDPs.jl: A framework forsequential decision making under uncertainty. Journal of Machine
Learning Research (JMLR), 18 (26), 1–5.
Felner, A., Stern, R., Shimony, S. E., Boyarski, E., Goldenberg, M.,
Sharon, G., Sturtevant, N., Wagner, G., & Surynek, P . (2017).Search-based optimal solvers for the multi-agent pathﬁnding prob-
lem: Summary and challenges. In Symposium on combinatorial
search .
Framinan, J. M., Fernandez-Viagas, V ., & Perez-Gonzalez, P . (2019).
Using real-time information to reschedule jobs in a ﬂowshop withvariable processing times. Computers & Industrial Engineering,
129, 113–125.
Garey, M. R., & Johnson, D. S. (1975). Complexity results for multi-
processor scheduling under resource constraints. SIAM Journal on
Computing, 4 (4), 397–411.
Gerkey, B. P ., & Mataric, M. J. (2004). A formal analysis and taxonomy
of task allocation in multi-robot systems. International Journal of
Robotics Research, 23 (9), 939–954.
Gini, M. L. (2017). Multi-robot allocation of tasks with temporal and
ordering constraints. In AAAI conference on artiﬁcial intelligence
(AAAI) , pp. 4863–4869.
Gombolay, M. C., Wilcox, R., & Shah, J. A. (2018). Fast scheduling
of robot teams performing tasks with temporospatial constraints.IEEE Transactions on Robotics (TRO), 34 (1), 220–239.González-Neira, E., Montoya-Torres, J., & Barrera, D. (2017). Flow-
shop scheduling problem under uncertainties: Review and trends.International Journal of Industrial Engineering Computations,8(4), 399–426.
Hönig, W., Kiesel, S., Tinka, A., Durham, J., & Ayanian, N. (2018).
Conﬂict-based search with optimal task assignment. In Interna-
tional conference on autonomous agents and multiagent systems
(AAMAS) .
Hyland, M., & Mahmassani, H. S. (2018). Dynamic autonomous vehi-
cle ﬂeet operations: Optimization-based strategies to assign A Vsto immediate traveler demand requests. Transportation Research
Part C: Emerging Technologies, 92, 278–297.
Johannsmeier, L., & Haddadin, S. (2016). A hierarchical human–robot
interaction-planning framework for task allocation in collabora-tive industrial assembly processes. IEEE Robotics and Automation
Letters, 2 (1), 41–48.
Kartal, B., Nunes, E., Godoy, J., & Gini, M. L. (2016). Monte Carlo
tree search for multi-robot task allocation. In AAAI conference on
artiﬁcial intelligence (AAAI) , pp. 4222–4223.
Kochenderfer, M. J. (2015). Decision making under uncertainty: Theory
and application . MIT Press.
Kok, J. R., Spaan, M. T., & Vlassis, N. (2003). Multi-robot decision
making using coordination graphs. International Conference on
Advanced Robotics (ICAR), 3, 1124–1129.
Lanctot, M., Zambaldi, V ., Gruslys, A., Lazaridou, A., Tuyls, K., Péro-
lat, J., Silver, D., & Graepel, T. (2017). A uniﬁed game-theoreticapproach to multiagent reinforcement learning. In Advances in
neural information processing systems , pp. 4190–4203.
Lau, H. C., Sim, M., & Teo, K. M. (2003a). V ehicle routing problem
with time windows and a limited number of vehicles. European
Journal of Operational Research, 148 (3), 559–569.
Lau, H. C., Sim, M., & Teo, K. M. (2003b). V ehicle routing problem
with time windows and a limited number of vehicles. European
Journal of Operational Research, 148 (3), 559–569.
Laumond, J. P ., et al. (1998). Robot motion planning and control (V ol.
229). Springer.
Lenstra, J. K., Kan, A. R., & Brucker, P . (1977). Complexity of machine
scheduling problems. In Annals of discrete mathematics ,V o l .1 ,
Elsevier, pp. 343–362.
Lerman, K., Jones, C. V ., Galstyan, A., & Mataric, M. J. (2006). Analysis
of dynamic task allocation in multi-robot systems. International
Journal of Robotics Researh (IJRR), 25 (3), 225–241.
Lin, X., Janak, S. L., & Floudas, C. A. (2004). A new robust opti-
mization approach for scheduling under uncertainty: I—Boundeduncertainty. Computers & Chemical Engineering, 28 (6–7), 1069–
1085.
Littman, M. L. (1994). Markov games as a framework for multi-agent
reinforcement learning. In Machine learning , Elsevier, pp. 157–
163.
Liu, L., & Shell, D. A. (2011). Assessing optimal assignment under
uncertainty: An interval-based algorithm. The International Jour-
nal of Robotics Research, 30 (7), 936–953.
Mataric, M. J., Sukhatme, G. S., & Østergaard, E. H. (2003). Multi-robot
task allocation in uncertain environments. Autonomous Robots,
14(2–3), 255–263.
Munkres, J. (1957). Algorithms for the assignment and transportation
problems. Journal of the Society for Industrial and Applied Math-
ematics, 5 (1), 32–38.
Nunes, E., Manner, M. D., Mitiche, H., & Gini, M. L. (2017). A tax-
onomy for task allocation problems with temporal and orderingconstraints. Robotics and Autonomous Systems, 90, 55–70.
Ooonovan, R., Uzsoy, R., & McKay, K. N. (1999). Predictable schedul-
ing of a single machine with breakdowns and sensitive jobs.International Journal of Production Research, 37 (18), 4217–4233.
Péret, L., & Garcia, F. (2013). Online resolution techniques. In Markov
decision processes in artiﬁcial intelligence , pp. 153–184.
123
Autonomous Robots
Pinedo, M. (2012). Scheduling (V ol. 29). Springer.
Rahmani, D., & Heydari, M. (2014). Robust and stable ﬂow shop
scheduling with unexpected arrivals of new jobs and uncertainprocessing times. Journal of Manufacturing Systems, 33 (1), 84–
92.
Raman, V ., Donzé, A., Sadigh, D., Murray, R. M., & Seshia, S. A.
(2015). Reactive synthesis from signal temporal logic speciﬁca-tions. In International conference on hybrid systems: Computation
and control , pp. 239–248.
Sharon, G., Stern, R., Felner, A., & Sturtevant, N. (2012). Conﬂict-based
search for optimal multi-agent path ﬁnding. In AAAI conference
on artiﬁcial intelligence (AAAI) .
Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An intro-
duction . MIT press.
Szelke, E., & Kerr, R. M. (1994). Knowledge-based reactive scheduling.
Production Planning & Control, 5 (2), 124–145.
Timotheou, S. (2010). Asset-task assignment algorithms in the presence
of execution uncertainty. The Computer Journal, 54 (9), 1514–
1525.
Timotheou, S. (2011). Network ﬂow approaches for an asset-task
assignment problem with execution uncertainty. In Computer and
information sciences , Springer, pp. 33–38.
V odopivec, T., Samothrakis, S., & Ster, B. (2017). On Monte Carlo
tree search and reinforcement learning. Journal of Artiﬁcial Intel-
ligence Research, 60, 881–936.
Wang, Z., & Gombolay, M. (2020). Learning scheduling policies for
multi-robot coordination with graph attention networks. IEEE
Robotics and Automation Letters, 5 (3), 4509–4516.
Yan, Z., Jouandeau, N., & Chérif, A. A. (2012). Multi-robot heuristic
goods transportation. In IEEE international conference on intelli-
gent systems , pp. 409–414.
Zhang, C., Song, W., Cao, Z., Zhang, J., Tan, P . S., & Chi, X. (2020).
Learning to dispatch for job shop scheduling via deep reinforce-ment learning. In Advances in neural information processing
systems (NIPS), V ol. 33.
Publisher’s Note Springer Nature remains neutral with regard to juris-
dictional claims in published maps and institutional afﬁliations.
Shushman Choudhury is a Ph.D.
student in Computer Science atStanford University. He is gen-erally interested in computational
decision-making for intelligent
systems and spaces. In his Ph.D.,he has developed hierarchicalalgorithms for coordinated multi-robot networks. His work has beennominated for Best Multi-RobotPaper at ICRA 2020 and featuredin BBC Digital Planet. He hasalso contributed to many open-sourced repositories for planningand decision-making methods.
Prior to joining Stanford, Shushman obtained an MS in Robotics fromCarnegie Mellon University in 2017 and a B.Tech in Computer Sci-ence and Engineering from IIT Kharagpur in 2015.
Jayesh K. Gupta is Senior
Researcher at Microsoft. Prior tojoining Microsoft and during thework in this paper, he was a Com-puter Science Ph.D. student atStanford University. He was advi-sed by Mykel Kochenderfer. He isinterested in enabling autonomousagency for the real world. To thisend, he studies how agents can
learn better models of the world
for effective control, learn fromeach other, and work well togetherin a team. Jayesh is an avid con-tributor/maintainer of a variety of
open-source software in the JuliaPOMDP ecosystem. Previously,Jayesh obtained his Bachelors in Electrical Engineering from IIT Kan-pur in 2015.
Mykel J. Kochenderfer received
the B.S. and M.S. degrees in com-puter science from Stanford Uni-versity, Stanford, CA, USA, andthe Ph.D. degree from The Uni-versity of Edinburgh, Edinburgh,U.K. He is currently an Asso-ciate Professor of Aeronautics andAstronautics with Stanford Uni-versity. He is also the Director ofthe Stanford Intelligent SystemsLaboratory, conducting research
on advanced algorithms and ana-
lytical methods for the design ofrobust decision-making systems.
Dorsa Sadigh is an assistant pro-
fessor in Computer Science andElectrical Engineering at StanfordUniversity. Her research interestslie in the intersection of robotics,learning, and control theory.Speciﬁcally, she is interested indeveloping algorithms for safe andadaptive human-robot interaction.Dorsa has received her doctoraldegree in Electrical Engineeringand Computer Sciences (EECS)from UC Berkeley in 2017, andhas received her bachelor’s degreein EECS from UC Berkeley in
2012. She is awarded the NSF CAREER award, the AFOSR Y oung
Investigator Program Award, the IEEE TCCPS early career award, the
Google Faculty Award, and the Amazon Faculty Research Award.
123
Autonomous Robots
Jeannette Bohg is an Assistant
Professor of Computer Science atStanford University. She was agroup leader at the AutonomousMotion Department (AMD) of theMPI for Intelligent Systems untilSeptember 2017. Before joiningAMD in January 2012, JeannetteBohg was a PhD student at theDivision of Robotics, Perception
and Learning (RPL) at KTH in
Stockholm. In her thesis, she pro-posed novel methods towardsmulti-modal scene understandingfor robotic grasping. She also
studied at Chalmers in Gothenburg and at the Technical University inDresden where she received her Master in Art and Technology andher Diploma in Computer Science, respectively. Her research focuseson perception and learning for autonomous robotic manipulation andgrasping. She is speciﬁcally interesting in developing methods thatare goal-directed, real-time and multi-modal such that they can pro-vide meaningful feedback for execution and learning. Jeannette Bohghas received several awards, most notably the 2019 IEEE InternationalConference on Robotics and Automation (ICRA) Best Paper Award,the 2019 IEEE Robotics and Automation Society Early Career Awardand the 2017 IEEE Robotics and Automation Letters (RA-L) BestPaper Award.
123

Shushman Choudhury
Jayesh Gupta
Mykel J
Kochenderfer
Dorsa Sadigh
Jeannette Bohg