Tuning the Hyperparameters of Anytime Planning:
A Metareasoning Approach with Deep Reinforcement Learning.

Tuning the Hyperparameters of Anytime Planning:
A Metareasoning Approach with Deep Reinforcement Learning
Abhinav Bhatia,1Justin Svegliato,2Samer B. Nashed,1Shlomo Zilberstein1
1College of Information and Computer Sciences, University of Massachusetts Amherst
2Department of Electrical Engineering and Computer Sciences, University of California Berkeley
abhinavbhati@cs.umass.edu, jsvegliato@berkeley.edu, snashed@cs.umass.edu, shlomo@cs.umass.edu
Abstract
Anytime planning algorithms often have hyperparameters
that can be tuned at runtime to optimize their performance.
While work on metareasoning has focused on when to inter-
rupt an anytime planner and act on the current plan, the scope
of metareasoning can be expanded to tuning the hyperparam-
eters of the anytime planner at runtime. This paper introduces
a general, decision-theoretic metareasoning approach that op-
timizes both the stopping point and hyperparameters of any-
time planning. We begin by proposing a generalization of the
standard meta-level control problem for anytime algorithms.
We then offer a meta-level control technique that monitors
and controls an anytime algorithm using deep reinforcement
learning. Finally, we show that our approach boosts perfor-
mance on a common benchmark domain that uses anytime
weighted A* to solve a range of heuristic search problems
and a mobile robot application that uses RRT* to solve mo-
tion planning problems.
Introduction
Anytime algorithms often have hyperparameters that can
be tuned at runtime to optimize their performance in a
given scenario—a specific problem instance and time con-
straint. Simply put, an anytime algorithm is an algorithm
that gradually improves its current solution at runtime and
can be interrupted at any time for that solution (Zilberstein
1996). This offers a trade-off between solution quality and
computation time that has proven to be useful in real-time
decision-making, such as motion planning (Karaman et al.
2011), heuristic search (Burns, Ruml, and Do 2013), object
detection (Karayev, Fritz, and Darrell 2014), belief space
planning (Spaan and Vlassis 2005), and probabilistic infer-
ence (Ramos and Cozman 2005). Naturally, to manage this
trade-off, current work on metareasoning focuses on deter-
mining when to interrupt an anytime planner and act on the
current plan. However, the scope of metareasoning can ide-
ally be expanded to tuning the hyperparameters of an any-
time planner at runtime to optimize its performance.
There has been substantial work on metareasoning that
determines when to interrupt an anytime algorithm and act
on the current solution. Generally, these methods monitor
Copyright © 2022, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.and control an anytime algorithm by tracking its perfor-
mance and calculating its stopping point at runtime. For ex-
ample, an early method estimates the optimal stopping point
by solving a sequential decision problem with dynamic pro-
gramming (Hansen and Zilberstein 2001) while a more re-
cent method estimates the optimal stopping point by predict-
ing its performance online (Svegliato, Wray, and Zilberstein
2018). These methods, however, cannot tune the hyperpa-
rameters of an anytime algorithm at runtime.
Methods for tuning the hyperparameters of an anytime al-
gorithm at runtime have largely been designed for specific
anytime algorithms and often lack formal analysis and gen-
erality while requiring expertise in the implementation.
In this paper, we introduce a general, decision-theoretic
metareasoning approach for both optimal stopping and hy-
perparameter tuning of anytime planning. Our approach ap-
proximates the problem of monitoring and controlling an
anytime algorithm as a Markov decision process (MDP). Its
states represent the quality and computation time of the cur-
rent solution and any other features needed to summarize the
internal state of the algorithm, the instance of the problem,
or the performance of the underlying system, while its ac-
tions represent either interrupting the algorithm or executing
the algorithm for another time step while tuning its inter-
nal hyperparameters. Our approach solves this MDP using
deep reinforcement learning (RL) to obtain a policy for both
optimal stopping and hyperparameter tuning of the anytime
algorithm: it performs a series of episodes, each of which
applies the anytime algorithm to some generated instance
of a specific problem. Our experiments on distinct problem
paradigms— anytime weighted A* for heuristic search and
RRT* for motion planning—show that deep RL is a natu-
ral approach to metareasoning for anytime algorithms since
the performance of an anytime algorithm is generally hard
to predict, but an abundance of simulations of the algorithm
can be easily performed.
Our main contributions are: (1) a generalization of the
standard meta-level control problem for anytime algorithms,
(2) a meta-level control technique that monitors and controls
an anytime algorithm based on deep RL, and (3) experiments
showing that our approach boosts performance on a common
benchmark domain that uses anytime weighted A* to solve
a range of heuristic search problems and a mobile robot ap-
plication that uses RRT* for motion planning.
Related Work
The literature on automated online hyperparameter tuning
for general algorithms is vast. Commonly used methods in-
clude local search (Adenso-Diaz and Laguna 2006; Hutter,
Hoos, and St ¨utzle 2007; Hutter et al. 2009b), genetic algo-
rithms (Ans ´otegui, Sellmann, and Tierney 2009), racing al-
gorithms (Birattari et al. 2002, 2010), and sequential model-
based optimization (Hutter et al. 2009a, 2010; Hutter, Hoos
et al. 2011).
These methods have largely been designed for general al-
gorithms and generally do not exploit anytime algorithms.
By applying deep RL to anytime algorithms in particular,
our approach avoids the drawbacks often imposed by cur-
rent methods. First, unlike methods that only support nu-
merical hyperparameters and deterministic algorithms, our
approach supports both categorical hyperparameters and
stochastic algorithms. Next, unlike methods that only op-
timize an algorithm’s hyperparameters on a single known
problem instance, our approach optimizes an algorithm’s hy-
perparameters for each problem instance drawn from a prob-
lem distribution . In fact, our approach can tailor the algo-
rithm’s hyperparameters to a specific instance of a problem
at runtime—making adjustments in a non-myopic, decision-
theoretic manner. Finally, unlike methods that always exe-
cute an algorithm until completion, our approach learns to
terminate an algorithm early when it is beneficial.
Only one other approach that we are aware of proposes
using reinforcement learning for automated hyperparameter
tuning (Biedenkapp et al. 2020). Similar to our approach,
it represents the problem as an MDP to be solved using
deep reinforcement learning. However, while it provides a
template for general algorithms, our approach is tailored to
anytime algorithms, optimizing both the stopping point and
hyperparameters, with specific MDP characteristics and re-
ward structure that maximizes the value of the final solution
produced by the anytime algorithm prior to termination.
There has been a large body of work on optimal stop-
ping for anytime algorithms. The earliest approach, namely
fixed allocation , executes the algorithm until a stopping
point determined prior to runtime (Horvitz 1987; Boddy and
Dean 1994). While fixed allocation is effective given neg-
ligible uncertainty in the performance of the anytime al-
gorithm, there is often substantial uncertainty in real-time
planning (Paul et al. 1991). Hence, a more sophisticated ap-
proach, namely monitoring and control , tracks the perfor-
mance of the algorithm and estimates a stopping point at
runtime periodically (Horvitz 1990; Zilberstein and Russell
1995; Hansen and Zilberstein 2001; Lin et al. 2015; Sveg-
liato, Wray, and Zilberstein 2018; Svegliato and Zilberstein
2018; Svegliato, Sharma, and Zilberstein 2020). Our ap-
proach not only determines the stopping point but also tunes
the hyperparameters of an anytime algorithm at runtime.
Finally, there are specialized methods for heuristi-
cally tuning certain hyperparameters of anytime weighted
A* (Hansen and Zhou 2007; Sun, Druzdzel, and Yuan 2007;
Thayer and Ruml 2009; Bhatia, Svegliato, and Zilberstein
2021) and RRT* (Urmson and Simmons 2003; Akgun and
Stilman 2011; Kiesel, Burns, and Ruml 2012), without op-
timizing the stopping point. In this work, we apply our ap-proach to optimize both the stopping point and tune selected
hyperparameters of anytime weighted A* and RRT* at run-
time, experimenting with the vanilla versions of these algo-
rithms for simplicity and generality as our approach focuses
broadly on anytime algorithms instead of heuristic search or
motion planning.
Standard Meta-Level Control Problem
We begin by reviewing the standard meta-level control prob-
lem for anytime algorithms. This requires a function that de-
scribes the utility of a solution computed by an anytime algo-
rithm in terms of its quality and computation time (Horvitz
and Rutledge 1991). We define this function below.
Definition 1. Atime-dependent utility function U: Φ×
Ψ→Rrepresents the utility U(q, t)of a solution of quality
q∈Φat time step t∈Ψ.
A time-dependent utility function can be expressed as the
difference between an intrinsic value function that describes
the utility of a solution given its quality but not its computa-
tion time and a cost of time that describes the utility of a so-
lution given its computation time but not its quality (Horvitz
1988). We present this property as follows.
Definition 2. A time-dependent utility function U: Φ×
Ψ→Ristime-separable if the utility U(q, t)of a solution
of quality q∈Φat time step t∈Ψcan be expressed as the
difference between two functions U(q, t) =UI(q)−UC(t)
where UI: Φ→R+is the intrinsic value function and
UC: Ψ→R+is the cost of time .
The standard meta-level control problem for anytime al-
gorithms is the problem of determining when to interrupt an
anytime algorithm and act on the current solution (Horvitz
1990; Zilberstein 1996). Ideally, the solution to this problem
is the optimal stopping point because it results in the opti-
mum of the time-dependent utility function. However, the
optimal stopping point of an anytime algorithm can be chal-
lenging to determine given substantial uncertainty over the
performance of the anytime algorithm.
Generalized Meta-Level Control Problem
In this section, we propose a generalization of the standard
meta-level control problem for anytime algorithms. Intu-
itively, an anytime algorithm can be viewed as containing an
internal hyperparameter that either interrupts or executes the
algorithm for another time step (i.e., stopping ) along with a
set of internal hyperparameters that adjust the internal op-
eration of the algorithm (i.e., hyperparameter tuning ). This
results in a new meta-level control problem for both opti-
mal stopping and hyperparameter tuning of an anytime algo-
rithm. As we discuss later in the paper, our approach mon-
itors and controls an anytime algorithm by expressing this
new meta-level control problem as a deep RL problem. We
present a description of an anytime algorithm below.
Definition 3. Ananytime algorithm ,Λ, contains an inter-
nal hyperparameter Θ0={STOP,CONTINUE }that either
interrupts or executes the algorithm for another time step ∆
along with a set of internal hyperparameters {Θ1, . . . , ΘℓΘ}
that adjusts the internal operation of the algorithm.
Figure 1: An illustration of anytime weighted A*.
Anytime weighted A* , an anytime variant of the
A* heuristic search algorithm (Hansen, Zilberstein, and
Danilchenko 1997; Likhachev, Gordon, and Thrun 2004;
Aine, Chakrabarti, and Kumar 2007; Hansen and Zhou
2007; Thayer and Ruml 2010) is a common example of an
anytime algorithm. We use anytime weighted A* to illustrate
our approach throughout the paper. The algorithm (1) uses
an inadmissible heuristic to quickly find suboptimal solu-
tions, (2) continues the search after each solution is found,
(3) provides an error bound on each suboptimal solution,
and (4) guarantees an optimal solution at termination. No-
tably, the standard evaluation function f(n) =g(n) +h(n)
used to select the next node for expansion from the open list
is replaced with a weighted evaluation function fw(n) =
g(n) +w·h(n), where the path cost g(n)is the cost of the
path from the start node to a node nand the heuristic h(n)
is the estimated cost from a node nto the goal node. Intu-
itively, by weighting the heuristic h(n)more heavily than
the path cost g(n)given any weight w≥1, the algorithm ex-
pands nodes that appear closer to reaching any solution in-
stead of nodes that lead to the optimal solution, potentially
speeding up the search at the expense of solution quality.
Figure 1 shows typical performance curves of two runs of
anytime weighted A* with different weights that each solve
a given instance of a problem. With deadlines , a weight of
2.0leads to better quality at Contract 1 while a weight of 1.5
results in better quality at Contract 2 . Without deadlines , a
weight of 2.0leads to better quality in the short term but
worse quality in the long term while a weight of 1.5results
in worse quality in the short term but better quality in the
long term. This leads to an important question: is it possible
to tune the hyperparameters of an anytime algorithm at run-
time to optimize its performance with orwithout deadlines?
We answer this question by offering a simple metareasoning
framework for learning optimal stopping and hyperparame-
ter tuning of anytime algorithms with deep RL.
Our metareasoning approach that uses deep RL represents
the generalization of the standard meta-level control prob-
lem for anytime algorithms as an MDP. An MDP is a for-
mal decision-making model for reasoning in fully observ-
able, stochastic environments that can be defined by a tuple
⟨S, A, T, R, s 0⟩, where Sis a finite set of states, Ais a fi-
nite set of actions, T:S×A×S→[0,1]represents the
probability of reaching a state s′∈Safter performing an
action a∈Ain a state s∈S,R:S×A×S→Rrep-resents the expected immediate reward of reaching a state
s′∈Safter performing an action a∈Ain a state s∈S,
ands0∈Sis a start state. A solution to an MDP is a pol-
icyπ:S→Aindicating that an action π(s)∈Ashould
be performed in a state s∈S. A policy πinduces a value
function Vπ:S→Rrepresenting the expected discounted
cumulative reward Vπ(s)∈Rfor each state s∈Sgiven
a discount factor 0≤γ < 1. An optimal policy π∗maxi-
mizes the expected discounted cumulative reward for every
states∈Sby satisfying the Bellman optimality equation
V∗(s) = max a∈AP
s′∈ST(s, a, s′)[R(s, a, s′)+γV∗(s′)].
Here, we express the generalization of the standard meta-
level control problem for anytime algorithms as an MDP
with a specific state and action space. The states have state
factors that reflect the quality and computation time of the
current solution along with state factors that reflect the in-
ternal state of the algorithm, the instance of the problem, or
the performance of the underlying system. The actions have
anaction factor that reflect the internal hyperparameter that
either interrupts or executes the algorithm for another time
step along with action factors that reflect the internal hyper-
parameters that adjust internal operation of the algorithm.
We formalize the generalized meta-level control problem for
anytime algorithms as an MDP in the following way.
Definition 4. Thegeneralized meta-level control problem
for monitoring and controlling an anytime algorithm, Λ,
is represented by an MDP ⟨Φ,Ψ, F, S, A, T, R, s 0⟩given a
time-dependent utility function U: Φ×Ψ→R:
•Φ ={q0, q1, . . . , q NΦ}is a set of qualities.
•Ψ ={t0, t1, . . . , t NΨ}is a set of time steps.
•F=F0×F1×···× FNFis a set of features that summa-
rize the internal state of the algorithm, the instance of the
problem, or the performance of the underlying system.
•S= Φ×Ψ×Fis a set of states of computation: each
states∈Sindicates that the algorithm has a solution of
quality q∈Φat time step t∈Ψwith a feature f∈F.
•A= Θ 0×Θ1× ··· × ΘNΘis a set of actions
of computation: the internal hyperparameter Θ0=
{STOP,CONTINUE }either interrupts or executes the al-
gorithm for another time step while the internal hyper-
parameters Θ1, . . . , ΘNΘadjust its internal operation.
•T:S×A×S→[0,1]is an unknown, possibly nonsta-
tionary transition function that represents the probability
of reaching a state s′= (q′, t′, f′)∈Safter performing
an action a∈Ain a state s= (q, t, f )∈S.
•R:S×A×S→Ris a reward function that represents
the expected immediate reward, R(s, a, s′) =U(q′, t′)−
U(q, t), of reaching a state s′= (q′, t′, f′)∈Safter
performing an action a∈Ain a state s= (q, t, f )∈S.
•s0∈Sis a start state s0= (q0, t0, f0)∈Sthat indi-
cates that the algorithm has a solution of quality q0∈Φ
at time step t0∈Ψwith a feature f0∈F.
Note that the reward function is consistent with the objective
of optimizing the time-dependent utility: executing the any-
time algorithm until a solution of quality q∈Φat time step
t∈Ψwith a feature f∈Fgives a cumulative reward that is
equal to the time-dependent utility U(q, t). This is a form of
reward shaping —equivalent to emitting a reward of U(q, t)
once at the end of an episode in terms of the objective—that
accelerates reinforcement learning by guiding the agent with
a reward at each time step (Ng, Harada, and Russell 1999).
Many metareasoning approaches to anytime algorithms
use only the quality and computation time of the current so-
lution as the state of computation (Zilberstein and Russell
1995). This state of computation, however, will likely not
satisfy the Markov property. It could therefore benefit from
features that summarize the internal state of algorithm, the
instance of the problem, or the performance of the underly-
ing system. As an example, in a domain that uses anytime
weighted A* to solve an instance of a travelling salesman
problem, there could be features for the mean of the g- and
h-values on the open list of anytime weighted A*, the num-
ber of cities in the instance at hand (Hutter et al. 2014), or the
processor usage of the system. Our metareasoning approach
can naturally use a complex representation for the state of
computation to approximate the Markov property by lever-
aging the advantages of using a neural network as a function
approximation with reinforcement learning.
We show that an optimal policy for the meta-level control
problem produces optimal meta-level control of an anytime
algorithm under certain conditions below.
Remark 1. If the change in the current solution of the any-
time algorithm given a state of computation s∈Sand an
action of computation a∈Asatisfies the Markov property,
the optimal policy π∗:S→Aresults in optimal stopping
and hyperparameter tuning.
Proof Sketch. This follows directly from the Markov prop-
erty: a transition to a successor state of computation s′∈S
only depends on the current state of computation s∈Sand
the current action of computation a∈A.
Meta-Level Control with Deep RL
Our meta-level control technique based on deep RL learns
both optimal stopping and hyperparameter tuning for an
anytime algorithm: it performs a series of episodes that each
use the anytime algorithm to solve a generated instance of
a specific problem. Generally, deep RL has been effective
across a variety of applications, such as Atari (Mnih et al.
2015), chess (Silver et al. 2018), and StarCraft (Vinyals et al.
2019). A deep RL agent learns a policy as a neural network
by performing actions and observing rewards in the world,
making it a natural approach to the meta-level control prob-
lem for anytime algorithms for a few reasons. First, by bal-
ancing exploitation and exploration, it can learn how to ad-
just the algorithm’s internal hyperparameters without know-
ing the transition function. Next, by ignoring large unreach-
able regions of the state space, it can reduce the overhead
of learning how to adjust the algorithm’s internal hyperpa-
rameters. Finally, by using a neural network that extracts the
relationship between large input and output spaces, it can
encode the effects of the algorithm’s internal hyperparame-
ters on its internal state in a way that generalizes to internal
states that are unfamiliar or have not been seen yet.
Algorithm 1 shows our meta-level control technique that
uses deep Q-learning (Mnih et al. 2015). Each episode (LineAlgorithm 1: Our meta-level control technique.
Input: Anytime algorithm Λ, action-value network N, step
sizeα1, target action-value network step size α2,
exploration strategy E, experience buffer capacity
ℓ1, number of episodes ℓ2, initialization period ℓ3,
minibatch size ℓ4, and duration ∆
Output: An action-value function Q
1B←EXPERIENCE BUFFER (ℓ1)
2Q←NEURAL NETWORK (N)
3ˆQ←Q
4fori= 1,2, . . . , ℓ 2do
5 P←SAMPLE PROBLEM DISTRIBUTION ()
6 Λ.SETUP(P)
7 t←0
8 st←(Λ.GETΦ(),Λ.GETΨ(),Λ.GETF())
9 at←πQ
E(st)
10 Λ.START(at.Θ1, . . . , a t.ΘℓΘ)
11 SLEEP(∆)
12 while Λ.RUNNING ()do
13 st+1←(Λ.GETΦ(),Λ.GETΨ(),Λ.GETF())
14 rt←R(st, at, st+1)
15 B.APPEND ((st, at, rt, st+1))
16 ifB.SIZE()≥ℓ3then
17 M←B.SAMPLE MINIBATCH (ℓ4)
18 ˆL(r, s′):=r+γmax a′∈AˆQ(s′, a′)
19 L(s, a, r, s′):= (ˆL(r, s′)−Q(s, a))2
20 Q.BACKPROPAGATE (M,L, α1)
21 ˆQ←(1−α2)·ˆQ+α2·Q
22 t←t+ 1
23 at←πQ
E(st)
24 ifat.Θ0=STOPthen
25 Λ.STOP()
26 break
27 Λ.CONTINUE (at.Θ1, . . . , a t.ΘℓΘ)
28 SLEEP(∆)
29 return Q
4) starts by executing the anytime algorithm for a time step
(of a duration ∆) on a generated instance of a specific prob-
lem (Lines 5-11). For each time step as the anytime algo-
rithm is executing (Line 12), there are several steps. First, the
experience buffer is updated with the current state of com-
putation, the current action of computation, the current re-
ward, and the next state of computation (Lines 13-15). Next,
if the size of the experience buffer exceeds the initializa-
tion period, we sample a minibatch (Lines 16-17). With that
minibatch, the temporal-difference error is used to update
the action-value network via backpropagation and then the
target action-value network is updated via a moving average
(Lines 17-21). Finally, the anytime algorithm is either inter-
rupted or executed for another time step while adjusting its
internal hyperparameters by following the policy computed
from the action-value network and the exploration strategy
(Lines 22-28).
Internal State
Problem System
Internal Parameters Anytime W eighted A*
Object-Level ProcessMeta-Level Process
State Action
control monitorFigure 2: An example of a metareasoning architecture.
Anytime Weighted A* Example
We now apply our approach to anytime weighted A*. Recent
work on anytime weighted A* focuses on selecting the best
static weight for a problem (Hansen and Zhou 2007), choos-
ing the best static weight for an instance of a problem (Sun,
Druzdzel, and Yuan 2007), and changing the weight at run-
time heuristically (Thayer and Ruml 2009). There is also
work that shows that anytime weighted A* can be improved
via restarting when a solution is found (Richter, Thayer, and
Ruml 2010) and work that analyzes the failure conditions
of anytime weighted A* with respect to its weight (Wilt and
Ruml 2012). Overall, recent work highlights the difficulty of
adjusting the weight of anytime weighted A* at runtime.
The meta-level control problem for anytime weighted A*,
Λ, is an MDP ⟨Φ,Ψ, F, S, A, T, R, s 0⟩.Φ = [0 ,1]is the set
of qualities. Ψ = [0 , τ]is the set of time steps with a dead-
lineτ.Fis the set of features such that the feature w∈W
is the current weight, the features µg∈Randµh∈Rare
the mean of the g- and h-values on the open list, the fea-
tures σg∈Randσh∈Rare the standard deviation of the
g- and h-values on the open list, the features g∈Rand
h∈Rare the minimum g- and h-values on the open list,
the feature ζ∈Ris the value log(n)for the number of
nodes non the open list, the feature ¯q∈Ris the h-value
of the initial state divided by the minimum f-value on the
open list, the feature h0is the h-value of the initial state, the
feature ρgh∈[−1,1]is the correlation between the g- and
h-values on the open list, the feature κ∈Kis the settings
for the instance of the problem, and the feature χ∈[0,1]
is the processor usage of the system. Ais the set of actions:
Θ0∈ {STOP,CONTINUE }interrupts or executes the algo-
rithm for another time step ∆while Θ1={←,→}adjusts
the weight w∈Wgradually by shifting its pointer to the
set of weights W={1,1.5,2,3,4,5}left or right, which
is sufficient when the state transitions are also gradual. Note
thatS,A,T,R, and s0follow directly from the generalized
meta-level control problem for anytime algorithms.
Similar to recent work on metareasoning (Svegliato et al.
2019, 2022), the metareasoning architecture in Figure 2 has
ameta-level process that monitors and controls an object-
insert() insert() insert()
delete()*Figure 3: A modified implementation of anytime weighted
A* that manages multiple open lists for different weights.
level process that executes anytime weighted A*.
Anytime weighted A* involves a simple modification to
allow its weight to be adjusted at runtime. Instead of insert-
ing/deleting a node into/from a single open list for a static
weight, the algorithm inserts/deletes this node into/from |W|
virtual open lists each ordered by the fw-value of a weight
w∈ W as shown in Figure 3 such that each open list has
a different ordering of the same exact nodes. This leads to
a worst-case time complexity of O(|W|logn)for sequen-
tially inserting/deleting a node across all |W|virtual open
lists of size n. In practice, thanks to the efficient implemen-
tation of anytime weighted A* used throughout our experi-
ments, we confirmed that the overhead of managing multiple
virtual open lists is negligible (less than 1%).
Experiments
We now evaluate our approach on a common benchmark do-
main that uses anytime weighted A* to solve a range of
heuristic search problems and a mobile robot application
that uses RRT* to solve motion planning problems.1
In our experiments, our approach and the standard ap-
proaches each solve 1000 random test instances of a given
problem using the anytime algorithm from both domains.
For each random test instance, we continuously measure the
solution quality produced by the anytime algorithm across
all approaches. A solution quality q= 0 means no solution
was computed while a solution quality q= 1means an opti-
mal solution was computed. Formally, for any instance of a
problem, the quality of a given solution is ideally defined as
the approximation ratio, q=c∗/c, where c∗is the cost of the
optimal solution and cis the cost of the given solution. How-
ever, since computing the optimal solutions for the complex
problems in our experiments is often infeasible, we estimate
the quality of a given solution as the approximation ratio,
q= ˆc∗/c, where ˆc∗is a lower bound on the cost of the opti-
mal solution and cis the cost of the given solution, following
earlier work (Hansen and Zilberstein 2001).
Naturally, any meta-level control problem for anytime al-
gorithms requires a time-dependent utility function. In our
experiments, we consider a common stopping -deadline set-
ting: there is an increasing urgency for a solution to incen-
tivize early termination of the anytime algorithm along with
1A public Julia library offers the implementation of our ap-
proach and the RL environments for anytime weighted A* and
RRT*: ⟨https://github.com/bhatiaabhinav/Metareasoning.jl ⟩.
a mandatory deadline of τseconds that incurs a severe dead-
line penalty Υ. This is common in planning and robotics
where an autonomous system urgently needs a solution but
also needs that solution prior to a mandatory deadline. More
formally, given a solution of quality q∈Φat time step
t∈Ψ, the time-dependent utility function is as follows:
U(q, t) = [t≤τ]·(UI(q)−UC(t))−[t > τ ]·Υ,
where UI(q) = ιqis the intrinsic value function and
UC(t) =eβt−1is the cost of time given the scalar ιandβ
selected in practice based on the value of a solution and the
urgency for a solution (Hansen and Zilberstein 2001). We
want to highlight that our approach generally supports any
well-behaved time-dependent utility function in addition to
the time-dependent utility function considered in this paper.
Our approach is trained on random training instances for
each problem in the domains. These problems only involve
a few hours of training. Algorithm 1 uses typical settings for
deep Q-learning. The action-value network Nis a fully con-
nected neural network with two hidden layers of 64and32
nodes with ReLU activation and a linear output layer of 6
nodes. The step size α1is0.0001 . The target action-value
network step size α2is0.001. The exploration strategy Eis
ϵ-greedy action selection with an exploration probability ϵ
that is annealed from 1to0.1over1000 episodes. The ex-
perience buffer capacity ℓ1is∞. The number of episodes ℓ2
is15000 and30000 for the anytime weighted A* and RRT*
domains. The initialization period ℓ3is1000 . The minibatch
sizeℓ4is128. The duration ∆is1/20th of the deadline. Our
experiments were run on an AMD Ryzen 3900X CPU with
32GB of RAM using a fixed set of random seeds. Note that
the random training instances for each problem differ from
the random test instances to ensure that our approach gener-
alizes to unseen or unfamiliar instances of each problem.
Common Benchmark Domain
We first experiment with a common benchmark domain that
uses anytime weighted A* to solve a range of heuristic
search problems by comparing our approach to the stan-
dard approaches. For our approach, we consider two vari-
ants: D QNτcan adjust the weight but always executes any-
time weighted A* until the deadline τwhile D QN(t) can
both adjust the weight and interrupt anytime weighted A* at
an earlier stopping point t. The standard approaches always
execute anytime weighted A* until the deadline τand use
either a static weight (denoted by a number ) or a dynamic
weight that decreases from the highest weight after each so-
lution (denoted by D EC), given a set of common weights
of1,1.5,2,3,4, and 5(Richter, Thayer, and Ruml 2010).
Overall, as we discuss later, our results demonstrate that our
approach outperforms the standard approaches regardless of
whether we always execute anytime weighted A* until the
deadline (D QNτ) or can interrupt anytime weighted A* at
an earlier stopping point (D QN(t)).
We provide a description of the heuristic search prob-
lems below. Each problem is selected to reflect problems
that require different static weights and problems for which
counterintuitive behavior of anytime weighted A* was re-
ported (Wilt and Ruml 2012). The parameters of each prob-lem are selected to avoid trivializing the problem by either
not having enough time so that no approach finds any solu-
tion or having too much time so that every approach finds the
optimal solution before the deadline. For the time-dependent
utility function, the scalars ιandβare1andln(1.25), the
deadline penalty Υis roughly ∞, and the deadline is τ= 1
second corresponding to roughly 100000 ,100000 ,50000 ,
and500000 node expansions for the S P, ISP, TSP, and G NP
benchmark problems on our system using our implementa-
tion. This means that there is a cost of time UC(t)and a
mandatory deadline τ. Note that we enforce the node expan-
sion limit instead of the deadline for reproducibility.
Sliding Puzzle An S Pinstance has J=j2−1tiles with
each tile ilabeled from 1toJin aj×jgrid. Every tile
must be moved from an initial position to a desired position
given a unitcostc(i) = 1 for moving a tile i. The sum of the
Manhattan distances from the current position of each tile
to its desired position is used as an admissible and consis-
tent heuristic function h. The number of tiles Jis15. The
difficulty of an instance, as measured by the h-value of the
initial configuration of all tiles, is chosen randomly between
35and45. The MDP includes the setting Kthat represents
the difficulty for the instance of the problem.
Inverse Sliding Puzzle An I SPinstance is the same as an
SPinstance except that there is an inverse costc(i) = 1 /ifor
moving a tile i. This means that the sum of the Manhattan
distances from the current position of each tile to its desired
position, weighted by the cost of moving each tile, is used
as an admissible and consistent heuristic function h.
Traveling Salesman Problem A T SPinstance has Jcities
that must be visited along an optimal route given a cost for
each edge between a pair of cities. A percentage of the edges
have an infinite cost to control its sparsity. The total cost
of a minimum spanning tree across the unvisited cities with
an infinite cost for no feasible tour is used as an admissible
and consistent heuristic function h. The number of cities J
is chosen randomly between 25and35. The percentage of
edges with an infinite cost is chosen randomly between 0%
and90%. The cost for each edge between a pair of cities is
chosen randomly ∈(0,1]. The MDP includes the settings
Kthat represent the number of cities and the percentage of
edges with an infinite cost for the problem instance.
Grid Navigation Problem A G NPinstance has a grid of
j×jcells with a percentage of the cells marked as obstacles.
The obstacles of shape 1×iare scattered randomly through-
out the grid. The goal is to find the shortest path from an
initial position of (1,1)to a goal position (j, j)by moving
north, east, south, or west, which incurs unit cost. The Man-
hattan distance from the current position to the goal position
is used as an admissible and consistent heuristic function h.
The grid is set to j= 1000 andi= 10 . The density of
the obstacles is chosen randomly between 5%and10%. The
MDP includes the setting Kfor the density of the obstacles.
Common Benchmark Results Figure 4 demonstrates that
our D QNτapproach outperforms the best standard approach
on each benchmark problem: it exhibits a better mean time-
Figure 4: The box plots of the final time-dependent utilities produced by anytime weighted A* for each approach over all
instances of S P, ISP, TSP, and G NP(from lefttoright ). Note that the crosses represent the mean, the bullets represent the
outliers, and the time-dependent utilities can be negative due to the cost of time UC(t)and the deadline penalty Υ.
Figure 5: An analysis of our approach (from lefttoright ). (a) For S P, a histogram showing the distribution of every instance over
theutility error for each of our approaches. We define utility error as the normalized difference between the final time-dependent
utility of each of our approaches and the final time-dependent utility of the best approach that always executes anytime weighted
A* until the deadline. (b) For S P, a smoothed line chart showing the final solution quality ( left y-axis ), the final time-dependent
utility ( left y-axis ), and the final computation time ( right y-axis ) for each training episode of our D QN(t) approach. (c) For S P, a
bar chart showing the importance of the top 10features used by our D QN(t) approach. We define importance as the normalized
mean absolute weight on a feature in the input layer of the neural network of our D QN(t) approach. (d) On a selected instance
of G NP, a line chart showing how the solution quality ( left y-axis ), the time-dependent utility ( left y-axis ), and the weight of
anytime weighted A* ( right y-axis ) vary with the computation time as anytime weighted A* solves the select instance using our
DQNτapproach ( dotted lines ) and our D QN(t) approach ( solid lines ). We emphasize that our D QNτapproach executes anytime
weighted A* until the deadline while our D QN(t) approach interrupts anytime weighted A* at the earlier stopping point.
dependent utility than D ECfor S Pandw= 4 for I SPand
a comparable time-dependent utility to D ECfor T SPand
w= 5for G NP. Hence, we observe that our D QNτapproach
is better than or comparable to the best standard approaches
without any need to adjust the weight manually for each
benchmark problem. More importantly, once we introduce
stopping on top of adjusting the weight automatically, our
DQN(t) approach further outperforms the best standard ap-
proach on each benchmark problem. This highlights that our
deep RL metareasoning approach for anytime algorithms
can learn both stopping and hyperparameter tuning unlike
existing methods that only optimize for one or the other.
Figure 5 offers an analysis of our approach. Figure 5(a)
shows that our D QN(t) and D QNτapproaches exhibit a util-
ity error of 0for about 900and850instances (out of 1000 ) of
the S Pbenchmark problem. This means it leads to the high-
est time-dependent utility more often than the standard ap-
proaches, which explains our results in Figure 4. Figure 5(b)
shows that, as our D QN(t) approach learns to optimize the
time-dependent utility for the S Pbenchmark problem, it ini-
tially focuses on improving the quality of the solution and
later focuses on reducing the computation time by terminat-
ing early. Hence, at the end of training, the agent learns how
to balance the quality of the solution and the computation
time. Figure 5(c) shows that the decisions of our D QN(t)approach are informed by key features like the quality qof
the current solution, the heuristic value h0of the starting
state, the current weight wof anytime weighted A*, the up-
per bound ¯qon the solution quality, and the computation
timet. Figure 5(d) shows a selected instance of the G NP
benchmark problem as it solved by our D QNτapproach and
our D QN(t) approach: the former terminates with a solution
of slightly higher quality but lower time-dependent utility
while the latter terminates with a solution of slightly lower
quality but higher time-dependent utility. Thus, we observe
that interrupting anytime weighted A* at an earlier stopping
point once its performance diminishes is beneficial.
Mobile Robot Application
We now experiment with the mobile robot application that
uses RRT* to solve motion planning problems by compar-
ing our approach to a standard approach. RRT* is a popular
algorithm that converges in the limit to an optimal motion
plan from a start state to a goal state by rapidly expanding a
tree via sampling the map randomly (Karaman et al. 2011).
Typically, RRT* has two hyperparameters: a growth factor
limiting how much the tree grows for each sample and an
area of focus biasing where each sample is drawn. Recent
work on RRT* focuses on heuristically tuning the growth
factor and area of focus (Urmson and Simmons 2003; Ak-
Figure 6: The performance curves showing how the mean
solution quality varies with the number of samples for all
instances of the motion planning problem.
gun and Stilman 2011; Kiesel, Burns, and Ruml 2012).
In particular, we compare our approach to a standard ap-
proach that executes RRT* with a small and large static
growth factor and an area of focus that spans the entire map.
Each motion planning problem is a map with a high den-
sity of obstacles where the start and goal states are in the
bottom-left and top-right corners. For the time-dependent
utility function, the scalars ιandβare1and0, the dead-
line penalty is roughly ∞, and the deadline τcorresponds
to1000 samples. This means that there is a mandatory dead-
lineτbut no cost of time UC(t). Note that we again enforce
a sample limit instead of the deadline for reproducibility.
The meta-level control problem for RRT* is similar to
anytime weighted A* but with a distinct set of features and
actions of computation. The features include the growth fac-
tor, the position of the area of focus, the percentage of sam-
ples that have expanded the tree, a lower bound on the re-
maining distance to the goal state, an estimate of the free
space of the map, an estimated average size of the obsta-
cle, and a score for each possible area of focus that consid-
ers an estimated probability of improving the current path,
the fraction of the current path within that area of focus, the
fraction of the tree within that area of focus, and the average
curvature of the current path within that area of focus. The
actions of computation include the internal hyperparameter
that increases or decreases the growth factor and the internal
hyperparameter that moves the position of the area of focus.
Mobile Robot Results Figure 6 shows that our approach
outperforms the standard approach to RRT*. At the maxi-
mum number of samples, our approach produces a higher
mean solution quality than the standard approach by adjust-
ing the growth factor and the area of focus. In particular, our
approach achieves a mean solution quality of roughly 0.73
while the standard approach to RRT* achieves a mean solu-
tion quality of 0.62and0.59for the large and small growth
factor. Moreover, over any number over 450 samples, our
approach is comparable to or better than the standard ap-
proach to RRT*. This indicates that it is sampling in a more
efficient way that results in better motion plans. Overall, we
find these results especially encouraging because our deep
RL architecture that was originally configured for anytime
weighted A* did not need to be updated for RRT*.
Figure 7 illustrates qualitatively how our approach adjusts
RRT* on selected instance. In general, our approach guides
the tree from the start state to the goal state of the map by
1
2
3
4Figure 7: The evolution of RRT* over the number of sam-
ples for our approach on a selected instance of the motion
planning problem from Checkpoint 1to4. The black shapes
are the obstacles, the green andredcircles are the start and
goal states, the blue lines are the current tree, the cyan cir-
cles are the samples, the purple line is the current path, and
theorange box is the current area of focus.
focusing on the frontier that has been less explored to com-
pute an initial path quickly. That is, from Checkpoint 1to4,
the area of focus is typically placed at the frontier of the ex-
isting tree of RRT*. Intuitively, our approach shifts the area
of focus to the frontier to compute an initial motion plan as
quickly as possible ( shown ). Thereafter, once our approach
has computed an initial motion plan, it favors the center of
the map near the initial motion plan to improve it as rapidly
as possible ( not shown ). Overall, our approach learns how to
shift the area of focus to key areas of the map, outperforming
the standard approach that samples from the entire map, by
reducing how long it takes to compute an initial motion plan
and how long it takes to improve that initial motion plan.
Conclusion
We present a general, decision-theoretic metareasoning ap-
proach that optimizes both the stopping point and hyperpa-
rameters of anytime planning. It not only boosts the perfor-
mance of an anytime algorithm by tuning its hyperparame-
ters at runtime on a specific instance of a problem, but also
eliminates any need for manual hyperparameter exploration.
Most importantly, we show that our meta-level control tech-
nique learns how to monitor and control anytime algorithms
that reflect distinct problem solving paradigms—heuristic
search and motion planning—without changing any of the
deep RL settings. Future work will explore additional any-
time algorithms commonly used in planning and robotics.
Acknowledgments
This work was supported by the NSF GRFP DGE-1451512
and the NSF grants IIS-1813490 and IIS-1954782.
References
Adenso-Diaz, B.; and Laguna, M. 2006. Fine-tuning of algorithms
using fractional experimental designs and local search. Operations
research , 54(1).
Aine, S.; Chakrabarti, P.; and Kumar, R. 2007. AWA*: A window
constrained anytime heuristic search algorithm. In 20th IJCAI .
Akgun, B.; and Stilman, M. 2011. Sampling heuristics for optimal
motion planning in high dimensions. In IROS .
Ans´otegui, C.; Sellmann, M.; and Tierney, K. 2009. A gender-
based genetic algorithm for the automatic configuration of algo-
rithms. In 15th CP .
Bhatia, A.; Svegliato, J.; and Zilberstein, S. 2021. On the benefits
of randomly adjusting anytime weighted A*. In 12th SOCS .
Biedenkapp, A.; Bozkurt, H. F.; Eimer, T.; Hutter, F.; and Lindauer,
M. 2020. Dynamic algorithm configuration: Foundation of a new
meta-algorithmic framework. In 24th ECAI .
Birattari, M.; St ¨utzle, T.; Paquete, L.; and Varrentrapp, K. 2002. A
racing algorithm for configuring metaheuristics. In 4th GECCO .
Birattari, M.; Yuan, Z.; Balaprakash, P.; and St ¨utzle, T. 2010. F-
Race and iterated F-Race: An overview. Experimental Methods for
the Analysis of Optimization Algorithms .
Boddy, M.; and Dean, T. L. 1994. Deliberation scheduling for prob-
lem solving in time-constrained environments. AI, 67(2).
Burns, E.; Ruml, W.; and Do, M. B. 2013. Heuristic search when
time matters. JAIR , 47.
Hansen, E. A.; and Zhou, R. 2007. Anytime heuristic search. JAIR ,
28.
Hansen, E. A.; and Zilberstein, S. 2001. Monitoring and control of
anytime algorithms: A dynamic programming approach. AI, 126.
Hansen, E. A.; Zilberstein, S.; and Danilchenko, V . A. 1997. Any-
time Heuristic Search: First Results. Technical Report 97-50, Com-
puter Science Department, University of Massachusetts Amherst.
Horvitz, E. 1988. Reasoning under varying and uncertain resource
constraints. In 7th AAAI .
Horvitz, E.; and Rutledge, G. 1991. Time-dependent utility and
action under uncertainty. In 7th UAI .
Horvitz, E. J. 1987. Reasoning about beliefs and actions under
computational resource constraints. In 3rd UAI .
Horvitz, E. J. 1990. Computation and action under bounded re-
sources . Ph.D. thesis, Stanford University, CA.
Hutter, F.; Hoos, H. H.; Leyton-Brown, K.; and Murphy, K. 2010.
Time-bounded sequential parameter optimization. In 4th LION .
Hutter, F.; Hoos, H. H.; Leyton-Brown, K.; and Murphy, K. P.
2009a. An experimental investigation of model-based parameter
optimisation: SPO and beyond. In 11th GECCO .
Hutter, F.; Hoos, H. H.; Leyton-Brown, K.; and St ¨utzle, T.
2009b. ParamILS: An automatic algorithm configuration frame-
work. JAIR , 36.
Hutter, F.; Hoos, H. H.; and St ¨utzle, T. 2007. Automatic algorithm
configuration based on local search. In 22nd AAAI .
Hutter, F.; Hoos, H. H.; et al. 2011. Sequential model-based opti-
mization for general algorithm configuration. In 5th LION .
Hutter, F.; Xu, L.; Hoos, H. H.; and Leyton-Brown, K. 2014. Al-
gorithm runtime prediction: Methods & evaluation. AI, 206.Karaman, S.; Walter, M. R.; Perez, A.; Frazzoli, E.; and Teller, S.
2011. Anytime motion planning using the RRT*. In ICRA .
Karayev, S.; Fritz, M.; and Darrell, T. 2014. Anytime recognition
of objects and scenes. In 27th CVPR .
Kiesel, S.; Burns, E.; and Ruml, W. 2012. Abstraction-guided sam-
pling for motion planning. In 5th SOCS .
Likhachev, M.; Gordon, G.; and Thrun, S. 2004. ARA*: Anytime
A* with provable bounds on sub-optimality. In 17th NeurIPS .
Lin, C. H.; Kolobov, A.; Kamar, E.; and Horvitz, E. 2015. Metar-
easoning for planning under uncertainty. In 24th IJCAI .
Mnih, V .; Kavukcuoglu, K.; Silver, D.; et al. 2015. Human-level
control through deep reinforcement learning. Nature , 518(7540).
Ng, A. Y .; Harada, D.; and Russell, S. J. 1999. Policy invariance
under reward transformations: Theory and application to reward
shaping. In 16th ICML .
Paul, C. J.; Acharya, A.; Black, B.; and Strosnider, J. K. 1991. Re-
ducing problem-solving variance to improve predictability. Com-
munications of the ACM , 34(8).
Ramos, F. T.; and Cozman, F. G. 2005. Anytime anyspace proba-
bilistic inference. IJAR , 38(1).
Richter, S.; Thayer, J. T.; and Ruml, W. 2010. The joy of forgetting:
Faster anytime search via restarting. In 20th ICAPS .
Silver, D.; Hubert, T.; Schrittwieser, J.; Antonoglou, I.; Lai, M.;
Guez, A.; Lanctot, M.; Sifre, L.; Kumaran, D.; Graepel, T.; et al.
2018. A general reinforcement learning algorithm that masters
chess, shogi, and Go through self-play. Science , 362(6419).
Spaan, M. T.; and Vlassis, N. 2005. Perseus: Randomized point-
based value iteration for POMDPs. JAIR , 24.
Sun, X.; Druzdzel, M. J.; and Yuan, C. 2007. Dynamic weighting
A* search-based MAP algorithm for Bayesian networks. In 20th
IJCAI .
Svegliato, J.; Basich, C.; Saisubramanian, S.; and Zilberstein, S.
2022. Metareasoning for safe decision making in autonomous sys-
tems. In ICRA .
Svegliato, J.; Sharma, P.; and Zilberstein, S. 2020. A model-free
approach to meta-level control of anytime algorithms. In ICRA .
Svegliato, J.; Wray, K. H.; Witwicki, S. J.; Biswas, J.; and Zilber-
stein, S. 2019. Belief space metareasoning for exception recovery.
InIROS .
Svegliato, J.; Wray, K. H.; and Zilberstein, S. 2018. Meta-level
control of anytime algorithms with online performance prediction.
In27th IJCAI .
Svegliato, J.; and Zilberstein, S. 2018. Adaptive metareasoning for
bounded rational agents. In 1st Workshop on AEGAP .
Thayer, J.; and Ruml, W. 2009. Using distance estimates in heuris-
tic search. In 19th ICAPS .
Thayer, J.; and Ruml, W. 2010. Anytime heuristic search: Frame-
works and algorithms. In 3rd SOCS .
Urmson, C.; and Simmons, R. 2003. Approaches for heuristically
biasing RRT growth. In IROS .
Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.;
Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds, T.;
Georgiev, P.; et al. 2019. Grandmaster level in StarCraft II using
multi-agent reinforcement learning. Nature , 575(7782).
Wilt, C.; and Ruml, W. 2012. When does weighted A* fail? In 5th
SOCS .
Zilberstein, S. 1996. Using anytime algorithms in intelligent sys-
tems. AI Magazine , 17(3).
Zilberstein, S.; and Russell, S. J. 1995. Approximate reasoning
using anytime algorithms. In Natarajan, S., ed., Imprecise and Ap-
proximate Computation . Springer.

Abhinav Bhatia
Justin Svegliato
Samer B
Nashed
Shlomo Zilberstein


Selecting the Partial State Abstractions of MDPs:
A Metareasoning Approach with Deep Reinforcement Learning.

Selecting the Partial State Abstractions of MDPs:
A Metareasoning Approach with Deep Reinforcement Learning
Samer B. Nashed1∗, Justin Svegliato2∗, Abhinav Bhatia1, Stuart Russell2, Shlomo Zilberstein1
Abstract — Markov decision processes (MDPs) are a common
general-purpose model used in robotics for representing se-
quential decision-making problems. Given the complexity of
robotics applications, a popular approach for approximately
solving MDPs relies on state aggregation to reduce the size of
the state space but at the expense of policy fidelity—offering
a trade-off between policy quality and computation time.
Naturally, this poses a challenging metareasoning problem:
how can an autonomous system dynamically select different
state abstractions that optimize this trade-off as it operates
online? In this paper, we formalize this metareasoning problem
with a notion of time-dependent utility and solve it using deep
reinforcement learning. To do this, we develop several general,
cheap heuristics that summarize the reward structure and
transition topology of the MDP at hand to serve as effective
features. Empirically, we demonstrate that our metareasoning
approach outperforms several baseline approaches and a strong
heuristic approach on a standard benchmark domain.
I. I NTRODUCTION
MDPs are a common general-purpose model used in
robotics for representing sequential decision-making prob-
lems [23]. However, the complexity of solving MDPs scales
poorly with the number of features reasoned about in the
environment, limiting their applicability. To address this
limitation, a range of approximate solvers for MDPs have
been proposed that seek to trade a small reduction in policy
quality for a large reduction in computation time.
A particularly effective approximate solver for MDPs,
recently proposed by Nashed et al. [17], solves a sequence
of partially abstract MDPs in order to solve an MDP. In
a partially abstract MDP, some states are considered at
maximum fidelity while other states are considered at lower
fidelity using an abstract representation. This can greatly
reduce the size of the state space while still resulting in a near
optimal policy by using a detailed representation for states
where it is most necessary. Still, for a partially abstract MDP
to be effective, it requires a suitable abstraction function that
maps a state in an MDP to an abstract state in an abstract
MDP. Since there has been substantial work on generating
abstraction functions for planners, ranging from symbolic
planners [6], [27], [13] to stochastic planners [1], [7], [26],
this paper assumes that a suitable abstraction function already
exists via either learning or careful expert design.
Given a specific abstraction function, a partially abstract
MDP uses an expansion strategy to determine which states
This work was supported by NSF grants IIS-1813490 and IIS-1954782.
∗Both authors contributed equally.
1University of Massachusetts, Amherst, MA, USA. Emails: {snashed,
abhinavbhati, shlomo }@cs.umass.edu
2University of California, Berkeley, CA, USA. Emails: {jsvegliato,
russell }@berkeley.edu
Fig. 1. Two partially abstract MDPs that were constructed using different
expansion strategies: a cheap expansion strategy that often results in lower
policy quality ( top) and an expensive expansion strategy that often results in
higher policy quality ( bottom ). Small circles are ground states, large circles
are abstract states, and arrows are transitions between states. The red ground
state is the current state, the green ground states have high reward, and the
green abstract states contain a ground state that has high reward.
of the MDP to reason about at maximum fidelity and
which states of the MDP to reason about at lower fidelity.
Concretely, this means that the expansion strategy selects
the abstract states to expand in the partially abstract MDP.
Ideally, any approach to selecting an expansion strategy
(illustrated in Fig. 1) should optimize a formal notion of
time-dependent utility by managing the trade-off between
policy quality and computation time given the domain of
operation and the available computational resources. Most
importantly, such an approach should generalize to any MDP
and require little knowledge in the details of that MDP.
In this paper, we therefore (1) offer a metareasoning
approach to selecting different expansion strategies that
optimizes a formal notion of time-dependent utility and (2)
express it as deep reinforcement learning problem. Moreover,
we (3) propose several general, cheap heuristics that summa-
rize the reward structure and transition topology of the MDP
at hand to generate effective features for deep reinforcement
learning. Empirically, we demonstrate that our metareasoning
approach outperforms several baseline approaches and a
strong heuristic approach on a standard benchmark domain.
II. R ELATED WORK
There are many approaches to approximately solving
MDPs, including performing dynamic programming, using
partial policies, and employing state abstractions. See [17]
for a thorough discussion of how these approaches relate
to partially abstract MDPs. Similarly, reinforcement learning
with time-dependent utility, introduced by Horvitz [10], has
been used in a variety of metareasoning problems [24], [22],
[4]. Although our approach uses reinforcement learning and
time-dependent utility similar to this work, we focus on a
different problem—that of learning how to select different
expansion strategies dynamically online during operation.
Online planning and learning over different abstractions
in general is a common problem across many areas of
artificial intelligence and encompasses several related sub-
problems. These include handling the non-Markovian nature
of state and action abstractions [2], learning context-specific
independences present in certain tasks [5], and learning
temporal abstractions in the form of progressively more
abstract skill controllers [14]. However, here, we restrict our
attention to state abstractions in the form of state aggregation
where multiple states in a larger (ground) problem form a
single state in a smaller (abstract) problem.
Online selection of state abstractions has been studied in
the context of both reinforcement learning and planning.
In reinforcement learning, abstractions are generally used if
the state space is large and training data is sparse, which
leads to poor experiential coverage. Methods include learning
the best state abstraction from a set of state abstractions
via hypothesis testing [12] and dynamically selecting state
abstractions of increasing granularities based on confidence
intervals of Q-values [25]. In planning, similar techniques
have been to applied to sample-based tree search algorithms.
For example, the PARSS algorithm is an algorithm that
adjusts state abstractions during tree search by starting with
coarse state abstractions and refining them given the variance
of the Q-values over actions at a specific abstract state [11].
An extensive body of research has investigated reasoning
over state abstractions during planning. Early work proposed
a hierarchy of state abstractions, represented as factored
semi-MDPs, that may have multiple intermediate state ab-
stractions that can be swapped in and out depending on
the environment [21]. Later work proposed algorithms for
dynamically eliminating state factors in states that were
estimated to not impact the policy by comparing two partially
abstract policies made with different state abstractions [3].
Finally, there have been specific applications, such as multi-
agent planning, where specialized partition schemes have
been introduced and adapted to an online setting [16].
This paper proposes a metareasoning framework that takes
advantage of powerful deep reinforcement learning methods
to learn a policy for selecting different expansion strategies.
Following work on SSPs, we use general, cheap heuristic
features that avoid relying on the specifics of an MDP. Most
importantly, we define this as a metareasoning problem that
optimizes a formal notion of time-dependent utility.III. B ACKGROUND
In this section, we review the formal definitions of a
ground MDP, an abstract MDP, and a partially abstract MDP.
a) Ground MDPs: A ground MDP is a tuple M=
⟨S, A, T, R ⟩. The space of states is S. The space of actions
isA. The transition function T:S×A×S→[0,1]
represents the probability of reaching a state s′∈Safter
performing an action a∈Ain a state s∈S. The reward
function R:S×A→Rrepresents the immediate reward
of performing an action a∈Ain a state s∈S. A
solution is a policy π:S→Aindicating that an action
π(s)∈Ashould be performed in a state s∈S. A policy
πinduces a value function Vπ:S→Rrepresenting
the expected discounted cumulative reward Vπ(s)∈Rfor
each state s∈Sgiven a discount factor 0≤γ < 1.
An optimal policy π∗maximizes the expected discounted
cumulative reward for each state s∈Sgiven the equation
V∗(s) = max a∈A
R(s, a) +γP
s′∈ST(s, a, s′)V∗(s′)
.
b) Abstract MDPs: Specifying an abstract MDP ¯Mof
a ground MDP Mrequires two functions [15]. First, an
abstraction function ϕ:S→¯Smaps a ground state s∈S
to an abstract state ¯s∈¯S. Second, an inverse abstraction
function ϕ−1:¯S→ P(S)maps an abstract state ¯s∈¯Sto
a set of ground states S⊆ P(S), where P(S)is the power
set of S. The condition ϕ(s) = ¯s⇔s∈ϕ−1(¯s)must hold
for each ground state s∈Sand abstract state ¯s∈¯S.
An abstract MDP is a tuple ¯M=⟨¯S, A, ¯T,¯R⟩[15].
The space of abstract states is ¯S={ϕ(s)|s∈
S}such that a set of ground states Sis abstracted by
an abstraction function ϕ. The space of ground actions
isA. The abstract transition function is ¯T(¯s, a,¯s′) =P
s∈ϕ−1(¯s)ψ(s)P
s′∈ϕ−1(¯s′)T(s, a, s′). The abstract reward
function is ¯R(¯s, a) =P
s∈ϕ−1(¯s)ψ(s)R(s, a). Note that a
weighting function ψ:S→[0,1]represents the probability
of being in a ground state s∈Sin an abstract state ϕ(s)∈¯S.
c) Partially Abstract MDPs: A partially abstract MDP
˜Mcombines a ground MDP Mand an abstract MDP ¯M
as a tuple ˜M=⟨˜S, A, ˜T,˜R⟩[17]. The space of partially
abstract states is ˜S=α∪βwith a set of ground states
α={ϕ−1(¯s)|¯s∈Γ}and a set of abstract states β={¯S\Γ}
such that a set of expanded abstract states Γ⊆¯Sis expanded
by an inverse abstraction function ϕ−1. The space of ground
actions is A. The partially abstract transition function ˜T:
˜S×A×˜S→[0,1]is composed of the ground/abstract
transition functions Tand¯T:
˜T(˜s, a,˜s′) =

T(˜s, a,˜s′) if˜s∈α,˜s′∈αP
s′∈ϕ−1(˜s′)T(˜s, a, s′) if˜s∈α,˜s′∈βP
s∈ϕ−1(˜s)ψ(s)T(s, a,˜s′)if˜s∈β,˜s′∈α
¯T(˜s, a,˜s′) if˜s∈β,˜s′∈β
The partially abstract reward function ˜R:˜S×A→Ris
composed of the ground/abstract reward functions Rand¯R:
˜R(˜s, a) =(
R(˜s, a)if˜s∈α
¯R(˜s, a)if˜s∈β
IV. S ELECTING PARTIAL STATE ABSTRACTIONS
The problem of selecting an expansion strategy to de-
termine the abstract states to expand in a partially abstract
MDP involves managing the trade-off between policy quality
and computation time. We frame this as a metareasoning
problem, the main advantage being that it expresses this
trade-off in terms of time-dependent utility, providing deep
reinforcement learning with an appropriate objective. Meth-
ods for similar metareasoning problems typically use heuris-
tics based on statistical measures to manage this trade-off.
Here, we introduce the first approach that selects expansion
strategies for partially abstract MDPs decision-theoretically.
A. Metareasoning for Partial State Abstractions
We begin by introducing the metareasoning problem for
partial state abstractions. This problem requires a time-
dependent utility that represents the utility of a policy in
terms of its quality and computation time. Intuitively, a policy
of a specific quality computed in a second has higher utility
than a policy of the same quality computed in an hour. A
time-dependent utility is therefore expressed as the difference
between an intrinsic value that reflects the utility of a policy
given its quality (but not computation time) and a time cost
that reflects the utility of a policy given its computation time
(but not quality) [10]. We define this function below.
Definition 1. Given a policy of quality q∈Φand compu-
tation time t∈Ψ, atime-dependent utility U: Φ×Ψ→R
can be expressed as the difference between two functions
U(q, t) =UI(q)−UC(t)where UI: Φ→R+is the intrinsic
value andUC: Ψ→R+is the time cost .
Given this time-dependent utility, the one-step metareason-
ing problem for partial state abstractions is the problem of
selecting the abstract states to expand in a given partially
abstract MDP. Naturally, a solution to this problem must
optimize time-dependent utility: we must select the abstract
states to expand in the partially abstract MDP that balances
the quality and computation time of its resulting policy.
Formally, given a set of abstract states Γi∈ P(¯S)to expand
in a partially abstract MDP ˜Miand its resulting policy πiof
policy quality q(πi)and computation time t(πi), this one-
step metareasoning problem is as follows.
arg max
Γi∈P(¯S)U(q(πi), t(πi))
This can be challenging to solve given substantial uncertainty
over the policy πiresulting from a partially abstract MDP
˜Mithat expands the abstract states Γi∈ P(¯S).
In real-time settings, an autonomous system often lazily
plans and acts online. Hence, during operation, we assume
that the autonomous system is either (1) executing an old
local policy πwhen it encounters a visited current state sor
(2) solving for a new local policy π′when it encounters an
unvisited current state s′. We can therefore view the union
of each local policy πias a joint global policy πΥas in
our recent work [17] that grows in quality and computation
time with each local policy πi. Intuitively, this presents asequential metareasoning problem for selecting the abstract
states to expand in a sequence of partially abstract MDPs
where the resulting local policies πiof each partially abstract
MDP ˜Mitogether compose a joint global policy πΥthat
must optimize time-dependent utility. Formally, given the
abstract states Υ = [Γ 1, . . . , Γh]expanded in a sequence
of partially abstract MDPs [˜M1, . . . , ˜Mh]over the unvisited
states{s1, . . . s h} ∈Shand the joint global policy πΥof
quality q(πΥ)and computation time t(πΥ), this sequential
metareasoning problem is as follows.
arg max
ΥU(q(πΥ), t(πΥ))
In practice, it is often beneficial to approximate this
sequential metareasoning problem as a sequence of inde-
pendent one-step metareasoning problems as follows.
arg max
Γ1∈P(¯S)U(q(π1), t(π1)) +···+ arg max
Γh∈P(¯S)U(q(πh), t(πh))
B. Reinforcement Learning for Partial State Abstractions
We now cast the sequential metareasoning problem for
partial state abstractions as an MDP. Each time an unvisited
state si∈Sis encountered, the MDP must select the
abstract states Γito expand in the partially abstract MDP ˜Mi.
Intuitively, the states include the quality and computation
time of the current joint global policy along with the reward
structure and transition topology of the ground MDP and
abstract MDP while the actions include expansion strategies
that select the abstract states to expand in the partially
abstract MDP. We define this metareasoning problem below.
Definition 2. The sequential metareasoning problem for
partial state abstractions is a tuple ⟨Φ,Ψ, F,ˆS,ˆA,ˆT,ˆR⟩
given a ground MDP Mand an abstract MDP ¯M:
•Φ ={q0, q1, . . . , q NΦ}is a set of qualities.
•Ψ ={t0, t1, . . . , t NΨ}is a set of computation times.
•F=F0×F1× ··· × FNFis a set of features that
summarize the reward structure and transition topology
of the ground MDP Mand abstract MDP ¯M.
•ˆS= Φ×Ψ×Fis a set of states of computation: each
state s∈ˆSreflects the current joint global policy πΥ
of quality q(πΥ)∈Φand computation time t(πΥ)∈Ψ.
•ˆAis a set of actions of computation: the set of expansion
strategies that each select different abstract states Γito
expand in a partially abstract MDP ˜Mi.
•ˆT:ˆS×ˆA×ˆS→[0,1]is an unknown transition function
that represents the probability of reaching state s′=
(q′, t′, f′)∈ˆSafter performing action a∈ˆAin state
s= (q, t, f )∈ˆS.
•ˆR:ˆS×ˆA×ˆS→Ris a reward function that represents
the expected immediate reward, ˆR(s, a, s′) =U(q′, t′)−
U(q, t), of reaching state s′= (q′, t′, f′)∈ˆSafter
performing action a∈ˆAin state s= (q, t, f )∈ˆS.
Note that the reward function is consistent with the
objective of optimizing the time-dependent utility: execut-
ing a sequence of expansion strategies until a joint global
policy πΥof quality q(πΥ)∈Φand computation time
t(πΥ)∈Ψemits a cumulative reward equal to the time-
dependent utility U(q(πΥ), t(πΥ)). This is a form of reward
shaping —equivalent to emitting a reward of U(q, t)once at
the end of an episode in terms of the objective—that guides
reinforcement learning with a reward at each time step [18].
We use deep reinforcement learning to learn an optimal
metareasoning policy by performing a series of simulations
that each use an expansion strategy to select the abstract
states to expand in a sequence of partially abstract MDPs.
Here, an agent learns a policy as a neural network by
performing actions and observing rewards in the environ-
ment, making it a good fit for metareasoning for three rea-
sons. First, by balancing exploitation and exploration, it can
learn how to select an expansion strategy given the reward
structure and transition topology of the ground MDP and
abstract MDP. Next, by ignoring large unreachable regions
of the state space, it can reduce the overhead of learning
which expansion strategy to select. Finally, by using a neural
network that extracts the relationship between large input
and output spaces, it can encode the effects of an expansion
strategy on the resulting policy of a partially abstract MDP
in a way that generalizes to novel states of computation.
C. Calculating Time-Dependent Utility
Typically, in metareasoning, a solution quality qis defined
as the approximation ratio, q=c∗
c, where c∗is the cost of
the optimal solution and cis the cost of the given solution.
However, since computing the cost of an optimal solution to
a complex problem is often infeasible, a solution quality can
be estimated as the approximation ratio, q=¯c∗
c, where ¯c∗is
a lower bound on the cost of the optimal solution and cis
the cost of the given solution. Generally, a solution quality
q= 0 means no solution was computed while a solution
quality q= 1 means an optimal solution was computed.
We need a specific definition of solution quality in the
context of MDPs. Here, the quality q(π)of a policy πis
defined as the approximation ratio,
q(π) =Vπ
V∗=P
s∈Sd(s)Vπ(s)P
s∈Sd(s)V∗(s),
where Vπis the value function of the policy πandV∗is the
value function of the optimal policy π∗, given a probability
d(s)of starting in a state s∈S. However, since computing
the value of an optimal policy of a complex MDP is often
infeasible, the optimal value function V∗must be replaced
with an upper bound on the value function ¯V∗.
Given the quality q(πΥ)and computation time t(πΥ)of
the current joint global policy πΥ, we can define the time-
dependent utility U(q(πΥ), t(πΥ))using an intrinsic value
UI(q(πΥ))and a time cost UC(t(πΥ)). First, given a tunable
parameter α, we model the intrinsic value as UI(q(πΥ)) =
αq(πΥ). Second, given a tunable parameter β, we model the
time cost as UC(t(πΥ)) =P
i∈h[eβt(πi)−1]such that πiis
the local policy solved for the unvisited states {s1, . . . s h} ∈
Sh. The rates αandβare typically given in the problem
depending on the value and urgency for a policy [8].
Given this time-dependent utility, it is possible to ex-
press the reward function of the metareasoning problem.Formally, given the current state of computation s=
(q(πΥ), t(πΥ),·)∈ˆSand the successor state of computation
s′= (q(π′
Υ), t(π′
Υ),·)∈ˆSthat reflect the current joint global
policy πΥand successor joint global policy π′
Υalong with
an expansion strategy a∈ˆAused to solve for a new local
policy πthat improves the joint global policy πΥ, we can
express the reward function in the following way.
ˆR(s, a, s′) =U(q(π′
Υ), t(π′
Υ))−U(q(πΥ), t(πΥ))
=α[q(π′
Υ)−q(πΥ)]−eβt(π)
V. R EPRESENTING THE STATE OF COMPUTATION
In this section, we introduce 6 features that compose the
state of computation in the sequential metareasoning problem
for partial state abstractions. These features can easily be
computed for a ground MDP Mand abstract MDP ¯Mand
reflect their reward structure ortransition topology .
A. Reward Structure
We define 3 features below describing the availability of
immediate reward around the current ground/abstract state.
1)Reward Frequency :The feature f1is the number of
positive reward ground states reachable within hactions of
the current ground state normalized by the total number of
reachable ground states.
2)Reward Proximity :The feature f2is the minimum
number of actions required to reach the nearest positive
reward ground state from the current ground state normalized
by the diameter diam( M) of the ground MDP M.
3)Reward Information :A main weakness of state ab-
stractions is that they induce artificial information boundaries
within the state space. For example, when a set of ground
states is compressed into an abstract state, the abstract MDP
loses information about any ground state that has successor
ground states in other abstract states. This is because succes-
sor ground states may be aggregated with other ground states
that are not reachable in a single action, which is detrimental
when a ground state with high reward successor ground states
can no longer be distinguished from a ground state without
high reward successor ground states. Therefore, the feature
f3is1/(1 +|diam(¯s)−δ|), where diam( ¯s) is the diameter
of the graph of the ground states in the current abstract state
¯s∈¯Sandδis the distance to the nearest high reward ground
state such that this value approaches 1or0as these ground
states move toward or away from this boundary.
B. Transition Topology
We define 3 features below describing the local transition
topology surrounding the current ground/abstract state.
1)Transition Entropy :The feature f4is the entropy of
the abstract successor state distribution at the current abstract
state assuming that actions are selected randomly. This is a
rough measure of the probability that actions performed at
the current abstract state will transition to different abstract
states that may be worth reasoning over more closely. A
higher entropy at the current abstract state indicates a higher
probability of transitioning to different abstract states.
Fig. 2. An example of (k, h)-reachability where k= 3 andh≤4. The
green region is allstates reachable from a given ground state swithin k
actions. The yellow /redregions are the states from which the set of important
ground states SGis still reachable/unreachable within hactions.
2)State Visitation :The feature f5is the expected dis-
counted number of times that the current abstract state will be
visited given an abstract start state distribution and an optimal
abstract policy. We compute this for all abstract states ¯s′∈¯S
by performing dynamic programming with the equation
λ(¯s′) =¯d(¯s′) +γX
¯s∈¯S¯T(¯s,¯π∗(¯s),¯s′)λ(¯s),
where ¯π∗(¯s)is the optimal abstract policy and ¯d(¯s′)is the
abstract start state distribution.
3)Important Ground State Reachability :The feature f6
is a novel measure of reachability from a given ground state
to a set of nearby important ground states.
Definition 3. A set of important ground states SGis(k,h)-
reachable from a given ground state sif, after performing
any arbitrary sequence of actions a1, . . . , a k, there is at least
one important ground state sg∈SGthat is still reachable
within hactions given a probability ϵ >0.
Similar to recent work on SSPs [20], [19], this measure
establishes an envelope (illustrated in Fig. 2) of ground states
in which the probability of reaching a set of important ground
states SGfrom a given ground state sis always greater than
zero. In general, while certain MDPs may have transition
topologies that permit calculating (k, h)-reachability exactly,
it usually must be estimated. To do this, we provide a
constant time estimation procedure in Algorithm 1. Here,
the accuracy of the estimate improves with the number of
samples parameterized by nandm. We choose kto be
proportional to the diameter of the current abstract state as
this is the maximum number of actions that can performed
by the local policy solved for the current abstract state.
VI. E XPERIMENTS
We now evaluate the proposed approach (D QN) against a
set of baseline approaches on a standard benchmark domain.
a) Hypothesis: Any approach to the metareasoning
problem for partial state abstractions should try to optimize
time-dependent utility by selecting the abstract states to
expand in a sequence of partially abstract MDPs. Ideally,
the approach should identify two cases. First, there are
cases in which cheap and expensive expansion strategiesAlgorithm 1: ESTIMATE (k, h)-REACHABILITY
1:Input: An MDP M, a ground state s, a set of important ground
states SG, and the parameters k,h,n, and m
2:Output: The probability that the set of important ground states SG
are(k, h)-reachable from the ground state s
3:Sk← ∅
4:fori∈ {1, . . . , n }do
5: s′←s
6: forj∈ {1, . . . , k }do
7: s′←SIMULATE RANDOM ACTION (M, s′)
8: Sk←Sk∪ {s′}
9:σ←0,ρ← ∅
10:forsk∈Skdo
11: fori∈ {1, . . . , m }do
12: s′←sk
13: forj∈ {1, . . . , h }do
14: s′←SIMULATE RANDOM ACTION (M, s′)
15: ifs′∈SGor∃p∈ρsuch that s′∈pthen
16: σ←σ+ 1
17: ρ←ρ∪PATH(sk, s′)
18: break
19:return σ/n
result in roughly equal policy quality , reducing computation
time at negligible sacrifice to policy quality. Second, there
are cases in which expensive expansion strategies result in
much higher policy quality than a cheap expansion strategy,
boosting policy quality at marginal amortized computation
time. Our hypothesis is that the proposed approach will
learn to exploit these two cases and hence optimize time-
dependent utility beyond the baseline approaches.
b) Experimental Setup: All approaches were evaluated
on100random simulations. For each simulation, we record
three metrics: the values for the policy quality ,computation
time, and time-dependent utility of the final policy. The
proposed approach was trained on1000 random simulations
using deep Q-learning with standard settings. The neural
network has two hidden layers of 64and32nodes with ReLU
activation and a linear output layer of 3nodes. The step
size is 0.0001 . The exploration strategy is ϵ-greedy action
selection with an exploration probability ϵthat is annealed
from 1to0.1over 1000 episodes. The experience buffer
capacity is ∞. The number of steps is 20000 . The buffer ini-
tialization period is 200. The target network update interval
is1000 . The minibatch size is 64. All simulations for training
and evaluation were generated using different randomization
seeds to measure generalizability to unfamiliar simulations.
c) Standard Benchmark Domain: We consider the
Earth observation domain proposed in early work on ground
MDPs [9] and recently modified in recent work on partially
abstract MDPs [17]. In this domain, a satellite orbiting Earth
indefinitely must take photos of points of interest Pwith
weather levels Wthat change stochastically. The satellite
starts at longitude x∈Xwith its camera focused at latitude
y∈Y. Given the rates ∆Yand∆X, the satellite can
then either do N OOPERATION , shift its camera N ORTH to
latitude (y+ ∆ Y)∈Y, shift its camera S OUTH to latitude
(y−∆Y)∈Y, or take an I MAGE of a point of interest at
latitude y∈Yand longitude x∈X. Concurrent to each
action, the satellite orbits from east to west described by
Naive Greedy Proactive Hybrid DQN
Approach0.00.51.0Policy Quality
Computation Time [seconds]
Time-Dependent UtilityFig. 3. Left: The policy quality of the final policy relative to the optimal policy over all evaluation simulations for each approach. Center: The frequency
of computation times of the final policy over all evaluation simulations for each approach. Right: The distribution of time-dependent utilities of the final
policy over all evaluation simulations for each approach. Putting these figures together, we observe that the proposed approach optimizes time-dependent
utility more effectively than the baseline approaches by learning how to manage the trade-off between policy quality and computation time.
Fig. 4. An example policy for the proposed approach that selects expansion
strategies. There are eight abstract states that each contain 2×4ground states
where hatch marks denote a point of interest. Each band within an abstract
state represents a specific expansion strategy: blue for N AIVE ,orange for
GREEDY , and green for P ROACTIVE such that darker shading denotes
higher probability. This policy shows how the proposed approach exploits
reward structure and transition topology of the earth observation MDP to
dynamically select the expansion strategy optimizing time-dependent utility.
longitude ((x+ ∆ X) mod |X|)∈Xwhere the modulo
operator creates periodic boundary conditions to represent
continuous orbits around earth. Most importantly, given the
IMAGE action, the satellite earns a reward proportional to
image quality such that image quality is a function of the
weather w∈W. The formal definitions of the ground,
abstract, and partially abstract MDPs are in recent work [17].
d) Baseline Approaches: We consider pure and hybrid
approaches that expand the current abstract state anda set of
informative abstract states . The N AIVE approach expands no
informative abstract states. The G REEDY approach expands
informative abstract states that contain a point of interest
within 1abstract state of the current abstract state. The
PROACTIVE approach expands informative abstract states
that are reachable from the current abstract state to any
abstract state that contains a point of interest within 2abstract
states of the current abstract state. The H YBRID approach
uses either the N AIVE , GREEDY , or P ROACTIVE approach
depending on the (k, h)-reachability of the current ground
state and the occupancy frequencies of the abstract MDP.
e) Experimental Results: Fig. 3 shows that the pro-
posed approach optimizes time-dependent utility beyond the
baseline approaches. First, N AIVE , GREEDY , and P ROAC -
TIVE exhibit poor time-dependent utility ( 29.6,36.6,32.2).
This is because they lead to either high policy quality in too
much computation time or low policy quality. Next, H YBRID
exhibits better time-dependent utility ( 40.7) by heuristically
reasoning over expansion strategies via careful expert design.
Finally, D QNexhibits the best time-dependent utility ( 44.7)
by performing explicit optimization using deep reinforce-
ment learning. Overall, the proposed approach decision-
theoretically selects expansion strategies (like in Fig. 4)
based on whether investing computation time would result
in a worthwhile improvement in policy quality.VII. C ONCLUSION
This paper introduces the metareasoning problem of se-
lecting different expansion strategies and solves it using deep
reinforcement learning with several general, cheap heuristics
that reflect the MDP at hand. Empirically, we show that
our metareasoning approach outperforms several baseline
approaches and a strong heuristic approach on a standard
benchmark domain. In future work, we will explore the
generalizability of this work to MDPs of varied topologies.
REFERENCES
[1] D. Abel, D. Arumugam, L. Lehnert, and M. Littman. State abstractions for
lifelong reinforcement learning. In ICML , 2018.
[2] A. Bai, S. Srivastava, and S. J. Russell. Markovian state and action abstractions
for MDPs via hierarchical MCTS. In IJCAI , 2016.
[3] J. Baum, A. E. Nicholson, and T. I. Dix. Proximity-based non-uniform
abstractions for approximate planning. JAIR , 43, 2012.
[4] A. Bhatia, J. Svegliato, S. B. Nashed, and S. Zilberstein. Tuning the hyperparam-
eters of anytime planning: A metareasoning approach with deep reinforcement
learning. In ICAPS , 2022.
[5] R. Chitnis, T. Silver, B. Kim, et al. CAMPS: Learning context-specific
abstractions for efficient planning. arXiv:2007.13202 , 2020.
[6] N. S. Flann. Learning appropriate abstractions for planning in formation
problems. In 6th ICML , 1989.
[7] X. Fu, G. Yang, P. Agrawal, and T. Jaakkola. Learning task informed abstractions.
In38th ICML , 2021.
[8] E. A. Hansen and S. Zilberstein. Monitoring and control of anytime algorithms.
AIJ, 126(1-2):139–157, 2001.
[9] A. Hertle, C. Dornhege, T. Keller, R. Mattm ¨uller, et al. An experimental
comparison of classical, fond and probabilistic planning. In KI, 2014.
[10] E. Horvitz and G. Rutledge. Time-dependent utility and action under uncertainty.
In7th UAI , 1991.
[11] J. Hostetler, A. Fern, and T. Dietterich. Sample-based tree search with fixed and
adaptive state abstractions. JAIR , 60, 2017.
[12] N. Jiang, A. Kulesza, and S. Singh. Abstraction selection in model-based
reinforcement learning. In ICML , 2015.
[13] C. A. Knoblock. Generating abstractions for planning. AIJ, 68(2), 1994.
[14] G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto. Robot learning from
demonstration by constructing skill trees. IJR, 31(3), 2012.
[15] L. Li, T. J. Walsh, and M. L. Littman. Towards a unified theory of state abstraction
for MDPs. In ISAIM , 2006.
[16] A. Ma, M. Ouimet, and J. Cort ´es. Dynamic domain reduction for multi-agent
planning. In MRS , 2017.
[17] S. B. Nashed, J. Svegliato, M. Brucato, C. Basich, R. Grupen, and S. Zilberstein.
Solving Markov decision processes with partial state abstractions. In ICRA , 2021.
[18] A. Y . Ng, D. Harada, and S. J. Russell. Policy invariance under reward
transformations. In ICML , 1999.
[19] L. Pineda and S. Zilberstein. Soft labeling in stochastic shortest path problems.
In18th AAMAS , 2019.
[20] L. E. Pineda, K. H. Wray, and S. Zilberstein. Fast SSP solvers using short-sighted
labeling. In 31st AAAI , 2017.
[21] K. Steinkraus and L. P. Kaelbling. Combining dynamic abstractions in large
MDPs. Technical report, MIT, 2004.
[22] J. Svegliato, P. Sharma, and S. Zilberstein. A model-free approach to meta-level
control of anytime algorithms. In ICRA , 2020.
[23] J. Svegliato, K. H. Wray, S. J. Witwicki, J. Biswas, and S. Zilberstein. Belief
space metareasoning for exception recovery. In IROS , 2019.
[24] J. Svegliato, K. H. Wray, and S. Zilberstein. Meta-level control of anytime
algorithms with online performance prediction. In 27th IJCAI , 2018.
[25] M. Tamassia, F. Zambetta, W. L. Raffe, F. Mueller, and X. Li. Dynamic choice
of state abstraction in Q-learning. In ECAI , 2016.
[26] M. Tomar, A. Zhang, R. Calandra, M. E. Taylor, and J. Pineau. Model-invariant
state abstractions for model-based RL. arXiv:2102.09850 , 2021.
[27] A. Unruh and P. S. Rosenbloom. Abstraction in problem solving and learning.
In11th IJCAI , 1989.

Samer B
Nashed
Justin Svegliato
Abhinav Bhatia
Stuart Russell
Shlomo Zilberstein