Competence-Aware Systems.

Artificial Intelligence 00 (2022) 1–34Artificial Intelligence
Competence-Aware Systems 1
Connor Basich∗a, Justin Svegliatob, Kyle H. Wrayc, Stefan Witwickic, Joydeep Biswasd, 2
Shlomo Zilbersteina
3
aUniversity of Massachusetts Amherst, Amherst, MA, USA {cbasich, shlomo}@cs.umass.edu 4
bUniversity of California, Berkeley, Berkeley, CA, USA, jsvegliato@berkeley.edu 5
cAlliance Innovation Lab Silicon Valley, Santa Clara, CA, USA, {kyle.wray, stefan.witwicki}@nissan-usa.com 6
dThe University of Texas at Austin, Austin, TX, USA joydeepb@cs.texas.edu 7
∗Corresponding Author 8
Abstract 9
Building autonomous systems for deployment in the open world has been a longstanding objective in both artificial intelligence
and robotics. The open world, however, presents challenges that question some of the assumptions often made in contemporary AI
models. Autonomous systems that operate in the open world face complex, non-stationary environments wherein enumerating all
situations the system may face over the course of its deployment is intractable. Nevertheless, these systems are expected to operate
safely and reliably for extended durations. Consequently, AI systems often rely on some degree of human assistance to mitigate
risks while completing their tasks, and are hence better treated as semi-autonomous systems . In order to reduce unnecessary reliance
on humans and optimize autonomy, we propose a novel introspective planning model— competence-aware systems (CAS)—that
enables a semi-autonomous system to reason about its own competence and allowed level of autonomy by leveraging human
feedback or assistance. A CAS learns to adjust its level of autonomy based on experience and interactions with a human authority
so as to reduce improper reliance on the human and optimize the degree of autonomy it employs in any given circumstance. To
handle situations in which the initial CAS model has insu fficient state information to properly discriminate feedback received from
humans, we introduce a methodology called iterative state space refinement that gradually increases the granularity of the state
space online. The approach exploits information that exists in the standard CAS model and requires no additional input from the
human. The result is an agent that can more confidently predict the correct feedback from the human authority in each level of
autonomy, enabling it learn its competence in a larger portion of the state space.
Keywords: probabilistic planning, human-agent systems, competence-aware systems, risk-aware autonomy, adjustable 10
autonomy, decision making under uncertainty 11
1. Introduction 12
Autonomous systems are increasingly deployed in the open world , involving highly complex and unstructured 13
domains. Examples of these systems include space exploration rovers [36, 63], autonomous underwater vehicles [20, 14
54, 85], service robots [10, 43, 59], and autonomous vehicles [15, 16, 28]. Because it is infeasible to completely 15
model the open world, these systems must rely on approximate models of their domains that may not be su fficient for 16
handling every situation [78, 89], introducing potentially risky behavior when the system attempts to act autonomously 17
where it is not competent to do so. Nevertheless, these systems are expected to maintain safe and reliable operation 18
over the course of potentially long-term deployments. To accomplish that, they often rely on various forms of human 19
supervision, assistance, and intervention. In that sense, many of the sophisticated AI systems under development 20
today are at best semi-autonomous in that they operate autonomously only under certain conditions, and otherwise 21
require human intervention in order to complete their assigned tasks [25, 104]. 22
Basich et al. /Artificial Intelligence 00 (2022) 1–34 2
Reliance on human assistance has been explored extensively to address the limited competence of autonomous 23
systems [33, 37, 46, 62, 66, 77, 93]. Often, this has been explored in the context of varying levels of autonomy , a 24
paradigm for modeling gradations in autonomous behavior within a human-agent team [64, 83], where each level 25
of autonomy corresponds to some set of constraints, limitations, or requirements on autonomous operation. For 26
example, on the two extremes would be full autonomous operation, and full human control (no autonomy). This 27
paradigm has already taken hold in several industrial applications where safety and reliability are critical, including 28
driving automation [76], robotic medical devices [6, 34, 101], and autonomous legal reasoning [31, 32]. 29
Human assistance may be available in di fferent forms or modalities, corresponding to di fferent degrees of com- 30
petence of a semi-autonomous system. Di fferent forms of human assistance compensate for the limitations imposed 31
in each level of autonomy and consequently mitigate the potential for risky behavior, while still ensuring that the 32
system’s task is completed. For example, Veloso et al. [92, 93] designed the CoBot system that can aid humans in an 33
office environment as an assistive robot in a variety of pick-up and delivery tasks. However, as the CoBot has no arms 34
to grasp objects, it cannot perform its tasks entirely autonomously, and must instead seek assistance from humans to 35
compensate for its limitation, for example by placing or removing objects in its basket. Ficuciello et al. [34] proposed 36
a level of autonomy framework for a surgical assistive medical robot with four levels of autonomy, where the lowest 37
two involve purely assistive actions to aid the human who is the primary executor, and the highest two involve fully 38
autonomous execution by the robot with assistance from the human in the form of surgical strategy selection. 39
In this work, we are primarily concerned with the risk associated with a system that operates at a level of autonomy 40
that is inappropriate for a task given its capabilities; for instance, an o ffice robot that autonomously handles fragile 41
items it is not competent to handle safely (i.e., without a high risk of breaking). Hence, we aim to develop systems 42
that are aware of their own competence , which we define to be the optimal level of autonomy to employ in any given 43
situation conditioned on the availability of suitable human assistance. A system that is aware of its own competence 44
when generating plans can therefore mitigate the potential for risky behavior by optimizing the degree of human 45
assistance that it requests, leveraging the human where the system’s competence is low, and acting autonomously 46
where it is high. 47
To further mitigate risks, humans may impose constraints on autonomous operation based on the perceived com- 48
petence of the system, for instance, by allowing them to intervene in time to prevent risky behavior or by disallowing 49
autonomous behavior entirely. In fact, the perceived risks may be outside the scope of what the autonomous system 50
can detect or reason about, hence enabling us to mitigate a broader range of risks. For example, a robot’s sensors may 51
be unable to perceive black ice on a sidewalk, or a nearby obstacle in dense fog, leading to risky behavior if left to 52
operate without supervision in these conditions. 53
Determining the exact competence of an autonomous system at design time can be very di fficult, particularly 54
when the environment is not fully specified or is simply too complex to fully anticipate. For example, a self-driving 55
car may initially be authorized to drive autonomously without supervision only on highways and during the daytime 56
with clear weather. Hence, an initial level of autonomy may be determined a priori through testing and evaluation, 57
but adjustments must be made when the system is deployed. Even when developers aim to err on the side of caution, 58
initializing the level of autonomy to be below the system’s true competence, it is possible to unintentionally infer 59
from initial testing that the system is more competent than it really is [69, 89]. Therefore, developing mechanisms to 60
explicitly represent, reason about, and adjust the level of autonomy is critical for the success of autonomous systems 61
deployed in the open world. 62
We propose a planning model called competence-aware system (CAS) for operating at multiple levels of autonomy 63
where each level is associated with di fferent forms of human assistance that compensate for the constrained abilities 64
of the system. Motivated by ideas from collaborative control [35], the structure of a CAS is illustrated in Figure 1. 65
The model associates with each type of human assistance a set of feedback signals that the system can receive from 66
the human, the likelihood of which can be learned over time. This model enables the system to operate more reliably 67
in the open world, reduce improper reliance on the human and ultimately optimize the autonomous behavior of the 68
system [5]. To address situations where the initial domain model has insu fficient information to correctly model 69
human feedback, we introduce an iterative approach to refine the system’s state space in order to better discriminate 70
human feedback, producing a more nuanced partitioning of the state-action space with di fferent levels of competence, 71
and allowing the system to better learn and act at its true competence [4]. 72
One of the main characteristics of CAS is that the system must recognize the limits on its autonomy, but it is 73
not required to know the reasons for these restrictions. This could be seen as a limitation, but we argue that it is an 74
Basich et al. /Artificial Intelligence 00 (2022) 1–34 3
Figure 1: An overview of how competence modeling impacts planning and execution. Here, the system’s current state is provided
as input to the system’s policy which traditionally would only output an action, but in our case also outputs a level of autonomy
determined by the competence model in which to perform the action. The level of autonomy dictates the type and degree of human
assistance used in the execution of the intended action. The human assistance can also provide additional feedback to the system,
which can be used to update and refine the competence model online.
advantage because it allows us to build autonomous systems that respect constraints on autonomy derived from human 75
knowledge that is beyond the scope of the system’s reasoning abilities. While we allow situations in which the system 76
does not have complete knowledge of the risks that justify the limitations on autonomy, the system may acquire that 77
knowledge over time. 78
Our contributions are three fold: (1) a mathematically rigorous formalization of competence for automated deci- 79
sion making; (2) a planning framework for a competence-aware system that integrates a model of competence with 80
a planning model to enable the system to reduce unnecessary reliance on humans and optimize its autonomous be- 81
havior; and (3) a method called iterative state space refinement that enables a competence-aware system to refine the 82
granularity of its state representation online. We provide a theoretical analysis of our model and algorithm, a concrete 83
example of a CAS and considerations in its design and implementation, empirical evaluations of our contributions in 84
simulation, and the lessons learned from a preliminary testing of the approach on an autonomous vehicle prototype. 85
2. Related Work 86
Researchers in automated planning [38] and reinforcement learning [87] have produced a vast literature devoted 87
to models, languages and algorithms that enable agents to reason about their environment and choose actions intelli- 88
gently. In this work, we specifically focus on advancing proactive reasoning under uncertainty about when and how 89
to obtain human assistance in order to improve goal achievement or safety. We discuss below three areas of research 90
that are particularly relevant to competence aware systems. 91
2.1. Systems with Variable Levels of Autonomy 92
Recognizing the value of human knowledge in planning has led to several research e fforts on human-agent collab- 93
oration in automated planning and control. Mixed-initiative planning /control [14, 18, 33, 37] is a paradigm based on 94
mixed-initiative interaction [1, 48] wherein multiple di fferent agents, generally a human an an autonomous system, 95
can take the initiative to act at di fferent stages to best utilize their respective abilities. Recent work has investigated 96
applying mixed-initiative control in the context of variable autonomy [23] in which the level of autonomy (LoA) can 97
change dynamically online. Chiou et al. [22] introduced the expert-guided mixed-initiative control switcher , which 98
dynamically adjusts the level of autonomy based on a comparison of the expected performance of a task expert and the 99
observed performance of the current system. Petousakis et al. [66] extended this approach by explicitly modeling the 100
cognitive availability of the human based on real-time vision of the human to better inform the LoA switching decision 101
between the autonomous agent and the human. Our work di ffers from this prior work in several key aspects. First, we 102
assume that an automated planner determines the level of autonomy for the human-agent team, thereby designating the 103
workload to both the human and the autonomous agent rather than allowing for each to initiate control on their own. 104
Second, we are focused on the problem of learning the true competence of the human-agent system online through 105
the acquisition of feedback from the human in response to actions taken by the agent at di fferent levels of autonomy. 106
Basich et al. /Artificial Intelligence 00 (2022) 1–34 4
Finally, much of the previous work is either tied to, or focused on, systems with only two levels of autonomy—no 107
autonomy and full autonomy—whereas we emphasize a general model for arbitrary levels of autonomy. 108
Rigter et al. [72] considered a similar setting in which control of a system is selected from a set of autonomous 109
controllers and a human operator. To reduce the reliance on the human over time, they propose to learn one of the 110
controllers online from demonstrations gained from the human operator. While similarly motivated, we consider a 111
slightly di fferent problem setting. First, we consider one agent operating in di fferent levels of autonomy, each of which 112
may involve some degree of human assistance, rather than all-or-nothing involvement of the human, and allow for the 113
level to change at every time step, rather than being fixed throughout an episode. The idea of learning a controller 114
from human demonstrations is similar to how we propose to learn a model of the human’s transition function when 115
they are in control, but in our case we use it only to predict their behavior, not to learn or alter autonomous control. 116
Symbiotic autonomy is similar in that the aim is to enable the completion of complex tasks by distributing tasks 117
and information across multiple agents. However, the term has been used both to represent human-agent systems 118
where the two agents act asynchronously to perform tasks for each other , that is both the human and agent may seek 119
assistance from the other to complete their task [75, 92, 93], as well as systems in which there is a smart environment in 120
addition to the autonomous agent and human that provides auxiliary information to the autonomous agent to facilitate 121
it [19, 25, 77]. Generally, our work di ffers in that we do not consider the environment and we emphasize the use of 122
human assistance to better facilitate the completion of the autonomous agent’s task, rather than asynchronously acting 123
in order to help the other agent with their task. 124
Adjustable autonomy [13, 29, 62, 80, 81, 91, 103] is a closely related paradigm for human-agent teams that is 125
characterized by the ability to dynamically change between di fferent levels, or modes, of autonomy, each of which 126
corresponds to some set of constraints or allowances that a ffect the actions the human-agent team can successfully 127
perform. It is worth noting that these approaches are largely complementary, and there has been work specifically 128
designed to combine multiple of these approaches [13, 60]. Our work falls generally in the category of adjustable 129
autonomy, but adds two important capabilities to such systems, on top of the fundamental notion of competence. 130
First, we explicitly model multiple forms of human feedback and use this feedback to enable a semi-autonomous 131
system to learn its competence over time. Second, in the CAS model the system learns a predictive model of the 132
human’s feedback allowing the system to converge to the optimal level of autonomy over time. 133
2.2. Learning from Human Feedback 134
Our approach is highly related to the general area of learning from human feedback. In reinforcement learning, 135
some work has investigated the e ffect of additional information provided by a guiding human. Specifically, Chernova 136
and Veloso [21] consider the inclusion of a guidance period after a robot’s action which can restrict the set of actions 137
that the robot can take in the next step to improve the e fficiency of the learning process. Moreira et al. [61] apply this 138
method in the context of deep reinforcement learning to expedite the learning process of a deployed system in a new 139
environment. Similarly, Rosenstein and Barto [74] propose a generalization to the actor-critic reinforcement learning 140
framework [3] that includes a supervisor who can provide additional feedback to the system in the form of auxiliary 141
guiding rewards, action selection guidance, or even direct control of the system. These di ffer from our work in that 142
we assume that the agent has access to a well-defined and fully-specified model of its domain, including the reward 143
(or cost) function from which to compute an optimal policy, and hence we are not concerned with learning a better 144
world model online (rather, we are only concerned with learning the system’s competence model online). 145
On the other hand, Knox et al. [50, 51] proposed a framework for training a robot solely from human feedback 146
(sometimes called interactive shaping or interactive reinforcement learning) in which the human supervising the robot 147
provides real-valued rewards for the actions that were just taken by the robot in a way that is assumed to account for 148
the long-term impacts of the action. However, in our work we are not training the agent to act by learning a reward 149
function, but rather providing the agent labeled data from which it can compute a distribution that is integrated into 150
an explicit transition function. Additionally, we do not consider the use of real-valued feedback from the human, but 151
rather discrete information tokens. More similar to our learning setting, Gri ffith et al. [41] proposed an approach in 152
which the agent learns two policies in parallel, one derived from reward signals from the environment, and one derived 153
from “right /wrong” labels from the human in order to infer what the human believes is the optimal policy, and then 154
combines the two policies into one that is used for action exploitation. The key di fference from Gri ffith et al. [41] 155
is that we seek to learn a predictive model of the human’s feedback rather than what the human believes the correct 156
policy to be, and then use this predictive model to analytically determine the optimal policy given the domain model. 157
Basich et al. /Artificial Intelligence 00 (2022) 1–34 5
Finally, Ramakrishnan et al. [70] examined a problem similar to what we consider in Section 5, wherein an 158
autonomous agent trained in simulation may have “blind spots” when deployed in real-world environments driven by 159
missing or ignoring features that are important in the real-world. Similar to how our method exploits human feedback 160
to identify new features that the human is using in generating their feedback, their method applies imitation learning 161
to demonstrations collected from the human to identify features used by the human but not by the agent. Our work 162
differs primarily in the type of information that the human provides to the system as well as how the missing features 163
are used. We integrate them into the existing model to improve the accuracy of the predicted human feedback which 164
consequently improves the quality of the overall policies generated by the system. On the other hand, [70] use the 165
learned information to learn blind spot models in the real world to perform safe transfer-of-control to a human when 166
encountering a blind spot to avoid potentially dangerous situations. 167
2.3. Competence Modeling 168
The term competence has been used widely in the context of intelligent systems. The classification literature, in 169
particular, has often defined the term as some measure of performance based on standard metrics for classification 170
systems on their input space [53], including accuracy estimation [98], potential function estimates [71], Bayes-based 171
confidence measures [47], relative performance to random guessing or otherwise randomized classifiers [95], and 172
probabilistic models [56, 96, 97]. More recently, Platanios et al. [67] defined the competence of a curriculum learner 173
to be the proportion of training data that the learner is allowed to use at any given time based on the di fficulty of 174
training samples, and Rabiee et al. [68] proposed competence as a distribution over failure classes that is learned 175
via introspective perception in the context of robotic path-planning. Common across these examples is an evaluative 176
approach to defining competence; that is, competence is a measure of the performance of a system or algorithm. Most 177
closely related to the formalization of competence presented in this paper was suggested by Smyth and McKenna 178
[84] who defined the competence of a case-based reasoning (CBR) system as the set of problems that the system 179
can solve successfully. The authors provide a rigorous model and analysis of competence for CBR systems, but the 180
work is highly specific to CBR systems on non-probabilistic domains, and consequently does not apply to stochastic 181
decision-making processes considered in this work. Rather, our aim is to enable a system to handle allproblems by 182
utilizing the appropriate degree of human assistance to ensure safe operation. 183
Instead, we borrow insights from the definitions of competence posed in the context of human workers. While 184
many definitions have been proposed over the last several decades [30, 79, 86, 90], they are largely atomistic and 185
lacking a well-defined mathematical representation. Gilbert [39] defined it as a function of the ratio of valuable 186
accomplishments to costly behavior, which while mathematically precise, leaves unaddressed both the relative perfor- 187
mative capabilities of di fferent agents with respect to a given task’s satisfactory completion, an essential component 188
of competence [42], as well as competence as an indication of authoritative permissibility. However, this definition 189
together with the definition of competency as a performance capability implying performance at a stated level [90] 190
inspires our definition formalized in Section 3.4. Intuitively, we propose that the competence of a system, much like 191
that of a human, is the optimal level of autonomy to use conditioned on available resources. For example, we might 192
say that a competent worker is one that knows when to perform tasks autonomously, when to ask for help and what 193
type of help to ask for, or when to reach for additional sources of aid and information (e.g., via Internet search) to 194
determine how to solve their task safely and reliably. Note that even human workers, when starting a new job for 195
example, may not initially know their exact competence and instead must learn “on the fly” where and when they 196
should solicit di fferent forms of aid or assistance. 197
3. Competence-Aware Systems 198
We start with a description of a general competence-aware system that can operate in and plan for multiple levels 199
of autonomy . Each level of autonomy is defined by a unique set of constraints on autonomous operation and consists 200
of different forms of human feedback that can be provided to the autonomous agent. To enable the agent to reason 201
about its own competence, it must have access to three di fferent models: a domain model , anautonomy model , and 202
afeedback model . Throughout this section, we use the problem setting in Example 1 as a running example to better 203
illustrate the concepts and terminology that we introduce throughout the paper. 204
Basich et al. /Artificial Intelligence 00 (2022) 1–34 6
Figure 2: Illustration of Example 1.205
Example 1. An autonomous vehicle (AV) with a human driver (shown in 206
blue in Figure 2) encounters an obstruction (e.g., a parked truck) block- 207
ing its lane on a one-lane road (red). In order to overtake the obstruction, 208
the AV would need to drive around the obstruction necessarily driving 209
through the oncoming tra ffic’s lane. In the oncoming lane, there may or 210
may not be a vehicle (yellow), but while stopped behind the obstruction, 211
the AV cannot detect it. The AV may Stop to let oncoming tra ffic go past 212
or see if the obstruction resolves itself (e.g., starts moving again), Edge 213
into the oncoming lane to gain better visibility without risking crashing, 214
orGoand begin passing the obstruction through the oncoming lane. 215
3.1. Domain Model 216
Thedomain model describes the environment in which the agent operates and the dynamics of the agent’s actions 217
within that environment. We model this as a stochastic shortest path (SSP) problem, a commonly used form of Markov 218
decision process (MDP) for reasoning in fully-observable, stochastic environments where the objective is to find the 219
least-cost path from a start state to a goal state [9]. For the purposes of this paper, we consider goal-oriented cost- 220
minimizing problems as they align more naturally with the problem domains that are considered in our experiments. 221
On the other hand, extending the theory to mixed-observable and partially-observable MDPs introduces additional 222
sources of uncertainty, particularly with respect to human interaction, that are non-trivial to handle in our model. A 223
discussion of these challenges can be found later in Section 7.3. 224
Definition 1. Adomain model ,D, is an SSP represented by the tuple ⟨S,A,T,C,s0,G⟩where: 225
•S is a finite set of states, 226
•A is a finite set of actions, 227
•T:S×A→∆|S|is a transition function where T (s,a)describes the probability distribution over successor 228
states when taking an action a ∈A in state s∈S , 229
•C:S×A→R+is a cost function where C (s,a)describes the cost of taking action a ∈A in state s∈S , 230
•s0∈S is the initial state, and 231
•G⊂S is the finite set of goal states. 232
A solution to an SSP is a mapping π:S→A, called a policy , that indicates that action π(s) is taken by the agent 233
in state s. A policyπinduces the state–value function Vπ:S→R 234
Vπ(s)=C(s,π(s))+X
s′∈ST(s,π(s),s′)Vπ(s′) (1)
which represents the expected cumulative cost of reaching any sg∈Gfrom state s∈Sfollowing the policy π. Any 235
policy that minimizes this function is referred to as an optimal policy and denoted π∗; formally: 236
π∗:=argmin
π∈ΠVπ(2)
However, the existence of an optimal solution to the SSP is guaranteed only under the condition that there exists 237
aproper policy , i.e. a policy under which a goal state is reachable from all states with probability 1, and that all 238
improper policies generate infinite cost when starting from at least one state; under this assumption, the optimal value 239
function is also unique. 240
Basich et al. /Artificial Intelligence 00 (2022) 1–34 7
Levels of Autonomy Human Involvement
l0No Autonomy Human driver fully in control of vehicle.
l1Verified Autonomy Autonomous agent in control of vehicle conditioned on explicit approval
from human for maneuver prior to execution.
l2Supervised Autonomy Autonomous agent in control of vehicle conditioned on a human driver
supervising the system ready and capable of taking control.
l3Unsupervised Autonomy Autonomous agent in unconditional control of vehicle, possibly with (but
not requiring) a human who can take over control.
Table 1: Levels of autonomy with L={l0,l1,l2,l3}where l0→l1→l2→l3.
3.2. Autonomy Model 241
Theautonomy model describes the levels of autonomy that the agent can operate in, restrictions on the situations 242
under which each level is allowed, the utilities of each level, and a set of system sub-competencies. 243
Definition 2. Anautonomy model ,A, is represented by the tuple ⟨L,κ,µ⟩where: 244
•Lis the finite, partially ordered set of levels of autonomy where each level l ∈L corresponds to some set of 245
constraints on the system’s autonomy, 246
•κ:S×L× A→P (L)is the autonomy profile where κ(s,l,a)returns the subset of levels of autonomy L ⊆L 247
allowed when performing action a ∈A in state s∈S given that the agent just acted in level l ∈L, and 248
•µ:S×L× A×L→ R+is the cost of autonomy where µ(s,l,a,l′)describes the cost of taking action a ∈A in 249
level l′∈Lin state s∈S given that the agent just acted in level l ∈L. 250
While most interpretations of levels of autonomy, as discussed in Section 1, are presented as ordered sets of 251
increasing autonomy, in general this need not be the case. In fact, in some cases di fferent levels of autonomy may be 252
directly compared. Hence, we choose to model ours more generally as a partially ordered set1where li≤ljif and only 253
if, given any task ( s0,G),Vli(s0)≤Vlj(s0) where Vliis the value function induced by the optimal policy when the level 254
of autonomy is fixed at li. Note that we consider two levels, liandlj, to be adjacent ifli<lj∧∄lk∈L| li<lk<lj. The 255
constraints corresponding to each level of autonomy can be technical in nature, i.e., internally imposed constraints 256
such as requiring human supervision in poor weather conditions that may be known a priori to cause errors, as 257
well as externally imposed constraints such as ethical or legal requirements. Each constraint is associated with a 258
corresponding form of human assistance or involvement. Intuitively, the higher the level of autonomy, the lower the 259
cost of human involvement, although this is not a requirement of the model. An example of a set of levels of autonomy 260
can be seen in Table 1. 261
Additionally, κcan be defined to not only reflect hard constraints such as ethical, legal, or technical constraints [40, 262
55, 57, 88] that are fixed throughout the system’s deployment, but also tentative constraints that can be updated over 263
time. Tentative constraints allow for a period of learning or adjustment early in the deployment of the system as 264
the human familiarizes themselves with the system, or the system learns to act appropriately in its environment. An 265
example of di fferent constraints on autonomy can be seen in Table 2. 266
The cost of autonomy, µ, is the cost associated with the act of operating in a given level of autonomy and is distinct 267
from the base domain cost of the action’s execution. For example, in a level of autonomy that requires tele-operation 268
from an o ff-site human to provide verification to a waiting autonomous vehicle, there may be an additional cost of 269
operating in that level corresponding to the amount of time waiting to reach an available tele-operator and receive 270
feedback. In a system with a finite energy supply that can perform sensing and perception at di fferent levels of fidelity 271
(corresponding to di fferent levels of autonomy), each level may utilize a di fferent amount of energy. 272
1Lcould be structured as a polytree or an arbitrary directed acyclic graph, however, for the sake of clarity we do not consider such levels of
autonomy in this paper.
Basich et al. /Artificial Intelligence 00 (2022) 1–34 8
Constraints on Autonomy
Ethical The A V may not be allowed to initiate a transfer of control to a human that is drowsy or otherwise deemed
unfit to operate the vehicle safely.
Legal The A V may not be allowed to operate autonomously inside of a school zone.
Technical The A V may be disallowed from operating autonomously in snowy weather due to the interference of per-
ception and object detection systems.
Tentative The A V may be initialized to drive in l1when it has no visibility, but may learn to perform the action Edge in
l3as it introduces an allowable level of risk by the human in the car.
Table 2: Examples of di fferent types of constraints on autonomy.
3.3. Feedback Model 273
The feedback model describes the agent’s knowledge about and predictions of its interactions with the human, 274
including the types of feedback it can receive from the human, how likely each possible type of feedback is at any 275
given time, and the expected cost to the human for assisting the agent. 276
Definition 3. Afeedback model ,F, is represented by the tuple ⟨Σ,λ,ρ,τH⟩, where: 277
•Σis the finite set of feedback signals that the agent can receive from the human, 278
•λ:S×L× A×L→ ∆|Σ|is the feedback profile where λ(s,l,a,l′)represents the probability distribution over 279
feedback signals that the agent will receive when performing action a ∈A in level l′∈L in state s∈S given 280
that the agent just operated in level l ∈L, 281
•ρ:S×L× A×L→ R+is the human cost function where ρ(s,l,a,l′)represents the cost to the human when the 282
agent performs action a ∈A in level l′∈Lin state s∈S given that the agent just operated in level l ∈L, and 283
•τH:S×A→∆|S|is the human state transition function where τH(s,a)represents the probability distribution 284
over successors states s′∈S when the human takes control of the system when the agent attempts to perform 285
action a∈A in state s∈S . 286
Although there are many forms of human feedback that have been studied, we limit our focus specifically to 287
feedback signals which are represented as discrete tokens of feedback that the human can provide to the autonomous 288
agent, either implicitly (e.g. facial gestures or body posture), or explicitly (e.g., verbal responses or physical control), 289
as opposed to real-valued reward signals [50, 51] or full demonstrations [24, 70, 72]. The primary reason is to keep 290
the feedback signals semantically simple in the sense that they are represented compactly by the system while still 291
being easily and unambiguously associated with the human’s intentions. This reduces the overhead associated with 292
the human-agent interactions. Each feedback signal may be associated with a distinct level, or subset of levels, 293
of autonomy and a corresponding form of human involvement. An example of this can be seen in Table 3. Future 294
directions of research may investigate extending these feedback signals to address such questions as how to learn from 295
feedback when there is a degree of severity associated with it, how to handle proactive feedback which is intended by 296
the human to be for inferred future states or trajectories, or feedback in the form of direct action commands. 297
The human cost function, ρ, is the cost to the human when operating in a given level and hence is separate from the 298
costs incurred directly by the autonomous agent. This cost may often be related to the human’s opportunity cost for 299
being unable to engage in other activities while assisting the autonomous agent. However, it may additionally capture 300
other costs to the human, such as additional stress or work added to them in addition to the time they spend assisting 301
(assisting two di fferent actions which take the same time may require di fferent levels of exertion from the human, 302
for example supervising an autonomous action making a left turn, or manually making the left turn). In practice, the 303
human’s cost function may be non-Markovian; for instance becoming fatigued after repeatedly performing manual 304
control, or becoming frustrated after extended periods of oscillating between di fferent levels of autonomy, constantly 305
shifting the demand on the human. While this can be coarsely approximated by conditioning the cost on the previous 306
Basich et al. /Artificial Intelligence 00 (2022) 1–34 9
Feedback Signal Interaction Levels of Autonomy
∅No feedback N/A {l0,l2,l3}
⊕Approval Verbal or Tactile Response {l1}
⊖Disapproval Verbal or Tactile Response {l1}
⊘Override Arrested Control {l2,l3}
Table 3: Each feedback signal is provided via a fixed and known interaction; for instance, the feedback signal approval may be
provided either by a verbal “Yes” from the human, or via a tactile response such as pressing a button on a touchscreen, similarly
fordisapproval .Override may be recognized by any form of arrested control by the human during autonomous operation, for
instance braking, accelerating, or steering while the A V is in control. Each signal is only recognized when the A V is operating at
the corresponding level of autonomy.
level of autonomy (as done here), one can improve this by maintaining a model of the human’s state, similar to what 307
is done by Costen et al. [26]. 308
IfλandτHare known exactly a priori then the system’s true competence (Definition 10) can be immediately 309
computed exactly under any κ, and the problem reduces to a straightforward planning problem. Furthermore, in some 310
problem instances where the feedback model is known exactly there may be no need to even constrain the policy 311
space at all (i.e. κ(s,a)=Lfor every ( s,a)∈S×A). This is the case when the feedback mechanisms are su fficient to 312
prevent the agent from taking actions that would violate hard constraint; for example, if the human authority always 313
overrides an action at a level that would violate an ethical, legal, or technical constraint. This introduces a trade-o ffin 314
distributing the burden of e ffort between the designers of the system and the operator of the system to ensure safe and 315
reliable operation in all cases. 316
However, in this work we are primarily concerned with systems where λandτH, and by consequence the system’s 317
true competence, are unknown a priori . In this case, they must be estimated by functions ˆλand ˆτH, which are based 318
on observed data collected online through interactions with the human at various levels of autonomy that can generate 319
feedback signals. These feedback signals can be analogously treated as labels in a labeled data set where the data is 320
the state, action, and level that generated the feedback signal. In Section 5, we address situations where the human’s 321
model of the world does not align with that of the autonomous agent, leading to feedback that is poorly discriminated 322
by the agent, which reduces its ability to learn from the signals it receives from the human. 323
Note that, in many real-world problems, the process of acquiring feedback signals may not be instantaneous, and 324
in some cases could require a complex process of fully or partially transferring control to and from a human over 325
an indefinite amount of time, where each element of the transfer process, such as the communication interface, is 326
important. The problem of transfer of control in semi-autonomous systems has been separately studied [81, 99]; 327
however, for the sake of clarity, we do not model this process explicitly in this work as we focus on the orthogonal 328
problem of modeling levels of autonomy and competence. 329
3.4. Competence-Aware Systems 330
Acompetence-aware system (CAS) represents a planning problem that accounts for the di fferent levels of auton- 331
omy available to the agent and factors in the agent’s expectations regarding the likelihood and cost of human feedback 332
(e.g., assistance, queries, intervention, etc.). The objective of a solution to a CAS planning problem is to create a plan 333
that best balances the cost of reaching the goal with the cost of human assistance to achieve the most cost-e ffective 334
strategy given the constraints of the problem. Hence, the CAS uses the autonomy model to proactively generate plans 335
that operate across multiple levels of autonomy by leveraging the feedback model to predict the likelihood of di fferent 336
feedback signals in order to optimize the level of autonomy and minimize the reliance on humans. To this end, we 337
represent a CAS as a multi-objective planning problem. 338
Example 2. A competence-aware system with four levels of autonomy—verified, supervised, unsupervised, and no 339
autonomy—and four type of feedback signals—approval, disapproval, override, and no feedback. The policy, π, 340
constrained by the autonomy profile κ, produces an action a at a level l to be performed in state s. The level l 341
determines the execution process of the action a, as depicted in the lower section of the figure. Certain levels may 342
Basich et al. /Artificial Intelligence 00 (2022) 1–34 10
l0 No Autonomy
l1 Verified Autonomy
l2 Supervised Autonomy
l3 Unsupervised Autonomy
⊕ Approval
⊖ Disapproval
⊘ Override
∅ No Feedback
κ Autonomy Profile
λ Feedback Profile
τH Human Transition Function
GE Gated Exploration
−−→AH Agent-to-Human Transfer of Control
−−→HA Human-to-Agent Transfer of Control
Figure 3: Illustration of Example 2
prompt the human for feedback, with a possibility of complete transfer of control from the autonomous agent to the 343
human. After the action is executed and data is collected, internal model parameters, λandτH, are updated. Finally, 344
the agent may perform gated exploration (Definition 8) to update the autonomy profile κ, although in practice this 345
would be performed on a less frequent basis. 346
Definition 4. Acompetence-aware system Sis represented by the tuple ⟨S,A,T,C,s0,G⟩, where: 347
•S=S×Lis a set of factored states, each comprised of a domain state s ∈S and a level of autonomy l ∈L. 348
•A=A×Lis a set of factored actions, each comprised of a domain action a ∈A and a level of autonomy l ∈L. 349
•T:S×A→∆|S|is a transition function where T(s,a)represents the distribution over successor states when 350
taking action a∈A in state s∈S . 351
•C=h
Cµ ρiTis a vector of cost functions. 352
•s0∈S is the initial state where s0=⟨s0,l⟩for some l∈L. 353
•G⊂S is the set of goal states. 354
A CAS state s∈Srepresents the CAS’s current domain state sand the level of autonomy, l, that the CAS 355
performed its last action in. The purpose of including the previous level of autonomy in the state representation is to 356
capture the fact that human feedback can vary depending on the level of autonomy that the agent was just operating 357
in (for instance, a human may be less likely to override the system if they were previously engaged in supervising 358
the system); additionally, we may want to discourage the system from oscillating between levels of autonomy by 359
imposing a small cost every time the system changes levels. Note that, one can set G=ˆS×L for some ˆS⊆Sto 360
indicate that the level of autonomy does not impact the goal condition or state, for instance setting G=G×L. 361
A CAS action a∈Arepresents a domain action ato be performed at a given level of autonomy lwhich may alter 362
both the mechanics of how the action is executed, the form and degree of involvement by the human authority in the 363
execution of the action, and the types of feedback that the agent can receive from the human authority. 364
Tis a transition function that represents the probability distribution over both how the state will change and which 365
feedback signal, if any, the agent will receive from the human when performing an action conditioned on the level 366
the action is being performed in, the current state, and the previous level that the agent had operated in (i.e. the 367
timestep prior to the current one). For example, the likelihood of a human override may decrease if the system had 368
Basich et al. /Artificial Intelligence 00 (2022) 1–34 11
already been acting under supervision than if they had been acting without supervision, as the human may have a 369
better understanding of what the system is doing. 370
Example 3. GivenLandΣ, we can specify the state transition function of this CAS. Given s=(s,l),s′=(s′,l′), 371
anda=(a,l′), we define T as follows: 372
T(s,a,s′)=τH(s,a,s′), if l=l0,
λ(⊕|s,a)T(s,(a,l2),s′)+λ(⊖|s,a)[s=s′],if l=l1,
λ(∅|s,a)T(s,a,s′)+λ(⊘|s,a)τH(s,a,s′),if l∈{l2,l3},(3)
where [·]denotes Iverson brackets. Intuitively, Equation 3 states that when the agent operates in l 0, it follows the 373
transition dynamics of the human who takes control. When operating in l 1, the probability it arrives in state s′is 374
the probability it is approved to take the action times the probability of the state change following T under level l 2, 375
plus the probability that it is disapproved and the state is the same. In levels l 2and l 3, the probability it arrives in 376
state s′is the probability it succeeds following T without any human intervention plus the probability that the human 377
overrides it and takes it to that state. In general, we expect the probability of an override to be lower (or even 0) in l 3378
as supervision is not required. 379
Figure 4: Illustration of a policy space,
Π, constrained by three di fferent auton-
omy profiles, κ1,κ2, andκ3.A solution to a given CAS is a policy πthat maps states and levels s∈S 380
to actions and levels a∈A. Multi-objective decision making has been well- 381
studied [73], and for our purposes we assume a scalarized approach [73] with 382
a scalarization function fparameterized by a weight vector w. A common 383
approach is simply based on a linear combination of the cost functions in C, 384
e.g., C=wh
Cµ ρiT. With some modifications, the problem could be 385
extended to handle both lexicographic models [100] and constrained mod- 386
els [2]. However, the properties that we derive for the scalarized model may 387
not necessarily hold for arbitrary multi-objective models, and would need 388
to be re-examined in those contexts. Additionally, we restrict the CAS to 389
only consider policies that are allowed under the autonomy profile κin the 390
following way. 391
Definition 5. Leta=⟨a,l⟩. Given s=⟨s,l′⟩ ∈S , we say that (s,a)is 392
allowed if l∈κ(s,a), and a policy πisallowed if for every s∈S ,(s,π(s))is 393
allowed. 394
We denote the set of allowable policies given κasΠκand require that the policy returned by solving the CAS, 395
π∗, is always taken from argminπ∈ΠκVπ(s0). An illustration of how di fferent autonomy profiles can constrain the full 396
policy space, Π, can be seen in Figure 4. 397
In general, a competence-aware system planning model is not guaranteed to be a valid stochastic shortest path 398
problem (see Proposition 1) due to the possible e ffects thatκandλcan have on the existence of a proper policy, 399
although in some cases they may only induce dead-ends away from the initial state for which there is existing work 400
on how to handle [52]. However, one can ensure that there is a proper policy with the inclusion of a level of autonomy 401
with a property similar to level l0in Table 1 which allows for (at potentially high cost) the deterministic completion 402
of any action or task, guaranteeing the existence of a proper policy. Note that we do not need to worry about the 403
possibility of ρorµinducing zero-cost cycles as they are non-negative cost functions, and the domain model is, by 404
assumption, a valid SSP. 405
4. Properties of a Competence-Aware System 406
In this section, we will discuss the central properties of a CAS that will allow us to prove several key results of 407
competence-aware systems. Henceforth, we will assume that there exists a singular human authority that the semi- 408
autonomous system in a CAS interacts with, and we will use the notation Hto refer to them. 409
Basich et al. /Artificial Intelligence 00 (2022) 1–34 12
Definition 6. Thehuman authority ,His represented by the tuple ⟨FH,λH,κH⟩where: 410
•FHis the set of features used by Hwhen providing feedback, 411
•λH:S×A→∆|Σ|is a stationary distribution of feedback signals that Hfollows, and 412
•κH:S×A→P (L)is the fixed mapping from state-action pairs to sets of autonomy levels that Hwill allow 413
the autonomous agent to operate in with nonzero probability. 414
Intuitively, κHrepresents the human authority’s belief of the agent’s competence; by definition any level not 415
contained in the image of κHwill never be allowed by H. 416
First, we begin with a simple proof that a CAS model is, in general, not guaranteed to be a valid stochastic shortest 417
path problem due to the lack of a proper policy. 418
Proposition 1. There exists a competence-aware system Sthat does not admit a proper policy. 419
Proof. LetSbe a CAS with exactly one level of autonomy, l, where the level of autonomy works as follows: when 420
the agent attempts to execute action a, they must first query the human to obtain a binary yesornofeedback signal. 421
If the signal is yesthen the agent may attempt to execute the action according to its model. If the signal is nothen 422
the agent may not attempt to execute the action in its current state. Let ( s0,l)∈Sdenote the initial state and assume 423
(s0,l)<G, where Sis the state space of SandGis the set of goals. Let λH(yes|(s0,l),(a,l))=0.0 for every action 424
a∈A(where Ais the action set). As the agent will never be able to transition out of its state which is not a goal state 425
by assumption, it is clear that there exists no proper policy. 426
Second, a fundamental component of the CAS model is the ability to adjust its autonomy profile over time using 427
what it has learned in order to optimize its autonomy by reducing unnecessary reliance on human assistance. However, 428
before operating in a new level of autonomy, the system may have no knowledge of how the human will interact with 429
it in that level, i.e., the feedback profile in that new level may be initialized by default to some baseline distribution. 430
As a result it is necessary that the system explore levels of autonomy that it predicts are more cost e ffective than its 431
current allowed levels, so that it may learn whether or not it is competent to act in those levels. 432
Allowing the system to alter its own autonomy profile, however, can lead to severe consequences in the real world 433
if not done carefully, mitigating the risk-awareness we aim to endow via the competence modeling. Therefore, we 434
propose two notions to ensure a measure of safety and risk-sensitivity in a competence-aware system. The first is 435
level-safety which is a notion of the safety of the level of autonomy that the system is using and is conditioned on 436
both the agent and the human; intuitively, a CAS is level-safe if it cannot act in levels that the human authority 437
would not allow. Second is gated exploration which is a simply extension to standard exploration methods used in 438
reinforcement learning in which the system must obtain permission from a human before exploring a new (disallowed) 439
level of autonomy, ensuring that level-safety is never violated. 440
Example 4. An autonomous vehicle is initialized to only use levels {l0,l1,l2}when executing the overtaking maneuver, 441
but learns that there is a very low likelihood of an override by the human authority during the day with clear visibility 442
and sparse tra ffic. Hence, it expects based on estimated costs that its competence is in fact l 3which is initially 443
disallowed to ensure safety at initial deployment. It therefore queries the human to approve it to update its autonomy 444
profileκby adding level l 3under the stated conditions. 445
Definition 7. A CASSislevel-safe underκifκ(s,a)⊆κH(s,a)for every (s,a)∈S×A. 446
Definition 8. We define the gated-exploration strategy for (s,a)∈S×A as follows: let adj (l,l′)=1if l=l′or l and 447
l′are adjacent inLand 0 otherwise, and let adj (κ(s,a),l′)=1if l′∈κ(s,a)or adj (l,l′)==1for some l∈κ(s,a). 448
Let P l(L)be a distribution over Lsuch that P l(l′)=0if ad j (l,l′)==0, and let l∗∼Pl(L). If l∗∈κ(s,a)do nothing, 449
otherwise, query the human authority Hto allow for the level exploration. If the query returns a positive response, 450
setκ(s,a)←κ(s,a)∪{l∗}, and otherwise do nothing. 451
Proposition 2. LetSbe a CAS with initial autonomy profile κ0. IfSis level-safe under κ0and follows the gated- 452
exploration strategy, then Swill be level-safe under κtfor any t≥0. 453
Basich et al. /Artificial Intelligence 00 (2022) 1–34 13
Proof. This is straightforward to observe by applications of the definitions. If Sis level-safe under κ0, then for all 454
(s,a)∈S×A,κ0(s,a)⊆κH(s,a) by definition. If there exists t>0 for which κt(s,a),κ0(s,a) for some ( s,a)∈S×A, 455
then there is some l∗∈κt(s,a)\κ0(s,a). By the definition of gated exploration and κH, it must be that l∗∈κH(s,a), 456
and henceκt(s,a)⊆κH(s,a). As ( s,a) is arbitrary, this holds for all ( s,a)∈S×A, and henceSis level-safe. 457
Next, we introduce a notion of feedback consistency which is a property of how consistent the human authority is 458
in providing the same feedback given the same query by the acting agent. 459
Definition 9. Let FH={FH
1,...,FH
n}be the set of features used by the human authority, H, and let SH=FH
1×···× 460
FH
n×L. The ground truth feedback function is a deterministic mapping f :SH×A→Σ.Hisperfectly consistent 461
ifλH(f(s,a)|s,a)=1∀s∈S,a∈A. IfλH(f(s,a)|s,a)≥ϵforϵ∈(0,1)∀s∈S,a∈A, thenHisϵ-consistent . 462
Unless otherwise stated, we assume that the human authority is ϵ–consistent henceforth. We now define three 463
central properties of a CAS. 464
Definition 10. LetλHbe the stationary distribution of feedback signals that the human authority follows. The com- 465
petence of CASS, denotedχS, is a mapping from S×A to the optimal (least-cost) level of autonomy given perfect 466
knowledge of λH. Formally: 467
χS(s,a)=argmin
l∈Lq∗(s,(a,l);λH) (4)
where q∗(s,(a,l);λH)is the cumulative expected cost under the optimal policy π∗when taking action a=(a,l)in state 468
s conditioned on the human authority’s feedback distribution, λH. 469
Fundamentally, the system’s competence for executing action ain state s,χS(s,a), is the most beneficial (e.g. cost 470
effective) level of autonomy were it to know the true human feedback distribution. When Lis an ordered set, we 471
expect this to generally be the highest level of autonomy allowed by the human; however, this need not be the case. In 472
principle, the highest allowed level of autonomy could require more frequent human interventions, e.g. due to lower 473
levels of trust by the human in the system [44], that may render it less e fficient overall relative to a lower level of 474
autonomy. 475
It is important to note that this definition of competence relies on λH, and hence is a definition of competence on 476
the overall human-agent system , and is explicitly not just a measure of the underlying agent’s technical capabilities 477
(i.e.D). A corollary of this fact is that the CAS is only as competent as the human authority believes it to be; a human 478
authority that has a poor understanding of the system’s abilities could lead to the system having a lower competence 479
than a human authority that knows perfectly the limitations and capabilities of the system. One reason for modeling 480
competence in this manner is to avoid relying on arbitrary thresholding based on evaluative metrics to determine when 481
a system is competent or not. 482
We say that a CAS Sisλ-stationary if, in expectation, any new feedback drawn from the true distribution λHwill 483
not affectλenough to change the optimal level of autonomy for any s∈Sanda∈A. We show below that, under 484
standard assumptions, Swill converge to λ-stationarity. 485
Definition 11. LetSbe a CAS and let U (λ)be the q-value of (s,a)under the optimal policy given λwhereSexecuted 486
the action a in level l in state s. We define the expected value of sample information (EVSI) on σ∈Σfor(s,a)to be: 487
X
σ∈Σmax
l∈LZ
ΛU(l,λ)λ(σ|s,a,l)p(λ)dλ−max
l∈LZ
ΛU(l,λ)p(λ)dλ. (5)
Definition 12. LetSbe a CAS.Sisλ-stationary if for every state s=(s,l)∈S , and every action a ∈A, the expected 488
value of sample information on σ∈Σfor(s,a)(Eq. 5) is less than ϵfor anyϵgreater than 0. 489
Proposition 3. Letλs,a
tbe the random variable representing λ(s,a)after having received t feedback signals for (s,a) 490
where each signal is sampled from the true distribution λH(s,a). Then, as t→∞ , the sequence{λs,a
t}converges in 491
distribution to λs,a
H=E[λH(s,a)]. 492
Proof. As each signal is drawn from λH(s,a) i.i.d, then by a straightforward application of the law of large numbers 493
the sequence will converge in probability to λs,a
H, which directly implies the claim. 494
Basich et al. /Artificial Intelligence 00 (2022) 1–34 14
Theorem 1. LetSbe a CAS, and let λs,a
tbe the random variable representing λ(s,a)after having received t feedback 495
signals for (s,a)where each signal is sampled from the true distribution λH(s,a). As t→∞ , if no (s,a)is starved,S 496
will converge to λ-stationarity. 497
Proof. Lets∈Sanda∈A. As sandaare arbitrary and we assume that no ( s,a) is starved, it is su fficient to show
convergence to stationarity for ( s,a) ast→∞ . By Proposition 3, {λs,a
t}will converge to λs,a
Hin distribution given our
assumptions. Because {λs,a
t}converges in distribution, lim t→∞Pr(|λs,a
t−λs,a
H|>ϵ)=0∀ϵ >0. Therefore, in the limit
the probability that λ=λs,a
Hafter tsteps, pt(λ), defines a Dirac delta function with point mass centered at λH. Hence
we get that, lim t→∞EVSI (Eq. 5)
=
lim
t→∞X
σ∈Σmax
l∈LZ
ΛU(λ,l)λ(σ|s,∅,a,l)pt(λ)dλ
−
lim
t→∞max
l∈LZ
ΛU(λ,l)pt(λ)dλ
=X
σ∈σmax
l∈LU(λH,l)λH(σ|s,∅,a,l)
−
max
l∈LU(λH,l)
=X
σ∈Σmax
l∈LU(λH,l)(1−λH(σ|s,∅,a,l))
=max
l∈LU(λH,l)
1−X
σ∈ΣλH(σ|s,∅,a,l)
=max
l∈LU(λH,l)(1−1)
=0.
498
Second, we say that a CAS Sislevel-optimal in some state if, under its current optimal policy, the action it takes 499
in that state is performed at its competence for that state-action pair. 500
Definition 13. LetSbe a CAS.Sislevel-optimal in state sif 501
π∗(s)=(a,χS(s,a)) (6)
If this holds for all states we say that Sislevel-optimal . Similarly,Sisγ-level-optimal if this holds in γ|S|states for 502
γ∈(0,1). 503
The primary goal of a competence-aware system is to reach level-optimality while maintaining level-safety . As 504
we have already shown that a CAS will maintain level-safety under the gated-exploration strategy (given an initial, 505
level-safe autonomy profile), we therefore want to show that under certain conditions, a competence-aware system S 506
will be guaranteed to reach level-optimality. In other words, that the system is guaranteed to reach a point where it 507
operates at its competence in all situations. 508
To prove that a competence-aware system will reach level-optimality, we rely on the notion of gated exploration as 509
detailed in Definition 8. However, we also require the following exploitation approach: ifShas reached λ-stationarity 510
then it no longer explores under the exploration strategy and instead exploits its knowledge by deterministically 511
selecting the optimal level of autonomy at that point, i.e. for any given ( s,a)∈S×A, the system will use a level 512
l∈argminl∈κ(s,a)q(s,(a,l);ˆλ). However, as the theory only proves convergence to λ-stationarity (that is, an expected 513
value of sample information of 0 over all σ∈Σfor every ( s,a)∈S×A) in the limit, we instead simply require 514
that for any fixed z∈R+, sufficiently small, the system will switch to exploitation once the expected value of sample 515
information falls below zeverywhere which will happen in finite time. We will refer to this below as exploitation 516
under stationarity . 517
Definition 14. LetSbe a CAS, and let κtrepresent the autonomy profile κat time t. Given s∈S and a∈A, we say 518
that l∈L is reachable from κtfor(s,a)if there exists at least one path from κt(s,a)to l∈L, where all levels along 519
the path are in κH(s,a). 520
In the following text, let κtrefer to the autonomy profile, κ, after the tthfeedback signal has been received. 521
Basich et al. /Artificial Intelligence 00 (2022) 1–34 15
Human 1 Human 2
ˆFσ1σ2σ3σ1σ2σ3
f1 0.171 -0.146 -0.055 0.222 0.255 -0.410
f2 0.293 -0.158 -0.209 -0.037 -0.109 0.111
f3-0.399 0.267 0.220 -0.212 -0.197 0.361
f4 0.375 -0.335 -0.103 0.384 -0.170 -0.313
f5-0.379 0.257 0.205 -0.372 0.311 0.208
f6 0.064 0.043 -0.141 0.045 -0.183 0.069
f7-0.030 0.118 -0.104 0.044 -0.019 -0.036
f8 0.179 -0.110 -0.112 0.044 -0.019 -0.036
f9 0.085 -0.093 -0.002 -0.062 0.027 0.051
f10 0.108 -0.151 0.038 -0.237 0.104 0.193
f11 0.175 -0.059 -0.168 0.325 0.295 -0.549
Table 5: The correlation matrices of each override signal with each feature.
Theorem 2. LetSbe a CAS that follows the gated exploration strategy and performs exploitation under stationarity, 522
whereχS(s,a)is reachable from κ0for all (s,a)∈S×A. Then if no (s,a)is starved, as t→∞ ,Swill converge to 523
level-optimality. 524
Proof. Fixs∈Sand threshold z≪1∈R+. We need to show that in the limit, π∗(s)=(a,χS(s,a)). By Proposition 1, 525
Swill converge to λ-stationarity for ( s,a) for all a∈A. Hence there is a finite point tat which the expected value of 526
information on Σfalls below zfor (s,a) for every a∈AandSwillexploit under stationarity fors. That is, at such 527
time,π∗(s)=(a,argminl∈κt(s,a)(q∗(s,(a,l)). By Proposition 3, this value is exactly the definition of χS(s,a) provided 528
thatχS(s,a)∈κt(s,a). By assumption, χS(s,a) is reachable from κ0(s,a)⊆κH(s,a), so given that under the gated 529
exploration strategy, there is a nonzero probability of reaching χS(s,a), and as sis arbitrary, we are done. 530
5. Improving Competence Online 531
As discussed in Section 1, many problems in the open world are too complex to fully specify a priori all features 532
that will be relevant over the course of the system’s deployment, even with expert knowledge of the domain. This is 533
particularly prevalent with features that may not directly impact the technical functionality of the autonomous agent 534
(e.g. its domain model) but rather are factors that influence the human’s feedback which may encompass additional 535
features that a ffect other elements such as comfort or social behavior [8, 58]. Preliminary analysis of override data 536
collected on a real autonomous vehicle prototype from two di fferent safety drivers corroborates this claim. Here, 537
the A V could either be in supervised autonomy , or could defer full control to the human; overrides corresponded to 538
braking or accelerating registered by the human driver while the A V was operating in supervised autonomy. 539
The results of this analysis can be seen in Table 5 where we provide the correlation matrix for each type of 540
override with every feature used by the CAS model implemented on the A V for each human safety driver. These 541
results demonstrate two important facts. First, the di fference in correlation matrices between Human 1 and Human 542
2 illustrate that feedback, and the features which determine that feedback, can vary significantly between humans, 543
meaning there is no “one-size-fits-all” feedback model. Second, the lack of any feature having a correlation coe fficient 544
greater that±0.4 indicates that it is challenging, even with expert input, to capture all of the causal features used by 545
all humans a priori . If the CAS model does not represent certain features in its model that are used by the human in 546
deciding their feedback signals (either explicitly or implicitly), the human’s feedback may appear inconsistent or even 547
random, leading to low competence and a potentially high degree of improper reliance on the human stemming from 548
an underspecified model. Consequently, for these systems to be most e ffective in the real world it is important that 549
they are equipped with a means of updating their model online to better align with the human’s model so that they can 550
better predict the correct feedback likelihoods. 551
To address this shortcoming, we propose a method for providing a CAS the ability to improve its competence over 552
time by increasing the granularity of its state representation through online model updates. The approach works by 553
Basich et al. /Artificial Intelligence 00 (2022) 1–34 16
identifying states that are deemed indiscriminate under the system’s current feedback profile, i.e. unable to predict 554
human feedback with high confidence, and attempts to find the feature, or set of features, that is available to the system 555
but currently unused that best discriminates human feedback, leading to a more nuanced drawing of the boundaries 556
between regions of the state space with di fferent levels of competence. An example of this process can be viewed 557
in Figure 6. By exploiting the existing information available in a standard CAS model (namely, the existing human 558
feedback) to identify where features may be missing and should be added, our approach adds no additional work to the 559
human at all. Additionally, when the missing features impact only the human’s feedback profile (and not the system’s 560
technical capabilities), or when using a CAS with levels of autonomy that involve forms of human assistance that main- 561
tain safe operation (like that which is described in the running example) we only need to modify the state space directly, 562
and not the transition or cost functions, enabling the entire process to be performed online and fully autonomously. 563
Figure 5: Illustration of Example 5564
Example 5. Recall the scenario in our running example, where the AV 565
(blue) must overtake an obstacle blocking its lane (red) by driving into 566
the oncoming tra ffic’s lane (yellow). Now, consider the existence of a 567
trailing vehicle (or vehicles) waiting behind the AV (green); the existence 568
of trailing vehicles may not be included in the state representation of the 569
domain model as they do not a ffect the decision making of the AV from a 570
technical perspective (that is, they do not influence the success or failure 571
probabilities of each action, do not influence the safety of the actions, 572
and short of rear-ending the AV do not directly alter the AV’s state), and 573
serve only to increase the state space of the planner. However, it may 574
be the case that the human in the AV is actually more likely to override 575
safe behavior, such as waiting if there is an oncoming vehicle, and take 576
manual control of the vehicle due to the social pressure exerted by the 577
trailing vehicle’s existence. 578
5.1. Indiscriminate States 579
LetSbe a competence-aware system. In practice, when a robotic system is deployed into the open world, both 580
the exact environment the system will operate in, and the human authority it will interact with, may not be known 581
a priori . Naively including all possible features available to the system from perception or external sources in its 582
planning model may make planning intractable without benefit in the case where many of the features do not add 583
useful information for decision making and serve only to increase the number of states. Hence, we assume that Shas 584
available to it a complete feature space that can be partitioned into an active feature space that is used bySand an 585
inactive feature space that is not yet used by Sin its planning model. However, as Sreceives additional feedback over 586
time,Swill learn to exploit some of the inactive features, adding them to its state representation to more e ffectively 587
align its features with those used by the human authority. 588
Definition 15. Given the complete feature space F={F1,F2,...,Fn}available toS, the active feature space is 589
denoted as ˆF⊆F, and the inactive feature space as˘F=F\ˆF. 590
We say that a state s∈Sisindiscriminate if, intuitively, the active feature space is missing features needed 591
to properly discriminate the feedback received from the human for the state s. The condition states more precisely 592
that for at least one action there must be no feedback signal that, under the system’s current feedback profile, can 593
be predicted with high probability. The intuition is that, under the assumption of ϵ-consistency and a ground truth 594
feedback, situations where the agent cannot predict feedback with high probability indicate that a feature may be 595
missing from its state representation causing the probability mass to be normalized over the remaining features in its 596
active feature space. We formalize this below. 597
Definition 16. Let the human authority Hbeϵ-consistent for ϵ >1
|Σ|. A state s∈S isindiscriminate if there exists at 598
least one action, a∈A, where for every feedback signal σ∈Σ, we have the following: 599
λ(σ|s,a)≤1−δ δ∈(1−ϵ,1−1
|Σ|) (7)
Basich et al. /Artificial Intelligence 00 (2022) 1–34 17
Figure 6: An illustration of iterative state space refinement .S(ˆFi) represents the state space given the active feature set ˆFi. The
middle row depicts a “zoomed in” view of a small part of the state space. We can see that originally, with active feature set ˆF1, there
are only two states in the subspace: s1ands2. The top row depicts the key information found by our algorithm: first, it identifies
thats2is an indiscriminate state givenλ, and finds the discriminator D1(represented by the red line) which then partitions s2into
two states: s21ands22. The process repeats once more, finding that s1is also an indiscriminate state, and finding discriminator D2
which partitions s1into four states: s11,s12,s13, and s14.
Here,δis referred to as the discrimination slack , and determines the predictive confidence needed for a state to be 600
declared indiscriminate; the lower the slack is set, the higher the confidence needed. The discrimination slack serves 601
to provide a formal trade-o ffmechanism between increasing the complexity of the underlying planning model, and the 602
completeness of the competence-aware model. The determination of how to set δmay be done via expert knowledge, 603
offline evaluations, or could even be tuned online in a dynamic fashion. To avoid considering states that have a very 604
small amount of data (and hence may be deemed “indiscriminate” due to chance), we consider only states for which 605
the system has collected a su fficient amount of data (which may be determined simply via a fixed threshold, or based 606
on some statistical analysis). 607
Given the notion of an indiscriminate state, we can now define the central concept of this approach. A discrimi- 608
nator is, intuitively, any subset of the inactive feature space that could help the agent to better discriminate feedback 609
fromHfor an indiscriminate state. For example, consider the autonomous vehicle agent in Running Example 5 that 610
initially does not consider the existing of a trailing vehicle in its active feature set. Suppose that the human always 611
overrides the vehicle and takes manual control when there is a trailing vehicle if the A V waits for too long before 612
proceeding around the obstruction to maintain safe operation. Without this additional feature in its model, the agent 613
may perceive having received “noisy”, or even seemingly random, feedback from the human authority, leading to a 614
feedback profile with low predictive capabilities and a poor competence model, resulting in the A V conservatively 615
transferring control to the human when performing an overtake in situations where it was actually competent to act 616
autonomously. By providing the agent with the ability to add these features to its active feature space, the agent’s new 617
feedback profile will be able to predict the correct feedback signal in more situations with higher probability. 618
5.2. Iterative State Space Refinement 619
Definition 17. Adiscriminator is any subset of ˘F which, if added to ˆF, will improve the performance of λby at least 620
α, for someα∈(0,1). 621
The larger that αis set, the stricter the requirement is on including a new feature. Determining αcan be as 622
simple as setting it to be a fixed threshold, or can be via more sophisticated means such as based on the value of 623
information or other information-theoretic metrics. The methodology for selecting discriminators is well explored 624
in the feature selection literature and not the focus of this contribution; standard approaches include mRMR [65], 625
JMI [17], and correlation-based methods [82]. We define a discriminator as a subset because there may be causal 626
features which if added individually do not help to discriminate the human’s feedback, but when added together do 627
(i.e. they are only meaningful in the context of each other). The size of feature subsets to consider when selecting 628
potential discriminators is therefore an important parameter of the approach, but we note that, if desired, one could 629
also take an iterative approach, running the algorithm with increasing size until a discriminator is found. 630
Basich et al. /Artificial Intelligence 00 (2022) 1–34 18
Algorithm 1: Single–Step State Space Refinement
Input: A CASS, datasetD, slackδ, and threshold M
Result: An updated CASS
1S∗←{}
2fors∈S.GetStates ()do
3 fora∈S.GetActions ()do
4 ifmaxσ∈Σλ(σ|s,a)≤1−δand
5 maxσ∈ΣPr[Obs(D(s,a))|σis ground truth] <pϵ
6 S∗←S∗∪{s}
7 end
8 end
9end
10ifS∗=∅
11 returnS
12end
13s∗∼S∗
14Dtrain,Dval←Split(D)
15D←FindDiscriminators (Dtrain,˘F,s)
16ford∈Ddo
17λd←train (ˆF1×···× ˆF|ˆF|×d,Dtrain)
18end
19d∗=argmaxd∈DEvaluate (λd,Dval)
20ifValidate (d∗,S)isTrue
21 ˆF←ˆF∪d∗
22S′←Update (S)
23end
24returnS′Algorithm 1 presents the pseudocode of 631
our approach for improving the competence 632
of a CAS via iterative partitioning of the 633
state space by adding new features to the 634
state representation over time. The algorithm 635
first identifies the current set of indiscriminate 636
states (Lines 1–9). To avoid labeling sparsely 637
sampled state-action pairs as indiscriminate 638
through chance, we limit the process to only 639
consider certain state-action pairs. In particu- 640
lar, only those where the probability of having 641
observed all labeled instances of that element 642
in the existing dataset D, referred to in Algo- 643
rithm 1 as Obs(D(s,a)), is at least some thresh- 644
oldpϵconditioned on the assumption that there 645
exists a true correct feedback signal returned 646
with probability at least ϵby the human for 647
every state-action pair (Line 5). Next, the al- 648
gorithm samples an indiscriminate state from 649
the set (Line 13) and identifies the most likely 650
discriminators for that state using any stan- 651
dard feature selection technique (in our case, 652
we used mRMR [65] with the FCQ methodol- 653
ogy [102]) (Line 15). For each potential dis- 654
criminator, a new feedback profile is trained 655
using a portion of the full dataset with the dis- 656
criminator temporarily added to the active fea- 657
ture set (Lines 16–18). The discriminator that 658
leads to the best performing feedback profile, 659
in our case the highest Matthews correlation 660
coefficient, is selected for validation (Line 19). 661
If validation is successful, the discriminator is 662
added to the active feature set and the system is updated (Lines 20–23). 663
In the design and usage of Algorithm 1, we make two key assumptions. First, we assume that the initial transition 664
function provided in the domain model is sufficiently correct for any scenario where the agent is allowed, under κH, 665
to act autonomously. We aim to improve the robustness of deployed systems where accounting for every scenario a 666
priori is infeasible, but where the scenarios that are considered a priori are well-designed. 667
Second, we assume that the human authority has a su fficient understanding of the agent’s capabilities to both 668
prevent the execution of an action that the agent cannot perform successfully and also provide consistent feedback. 669
We make this assumption for two reasons. First, there are di fferent ways to improve the human authority’s under- 670
standing of the system’s capabilities so that it has the appropriate trust [45], or reliance, on the system. These include 671
pre-deployment training, standardized feedback criteria, and expert knowledge of the system. Second, recognizing 672
potential failures and handling fault recovery are separate areas of active research [7, 27, 94] that are orthogonal to 673
what we examine here. 674
Critically, under these assumptions, we do not need to update the domain model’s transition or reward functions 675
directly at any point . It su ffices for the agent to be able to discriminate between actions that it has the competence to 676
perform autonomously and actions that require human involvement because, under the first assumption, Tis correct 677
when the agent is allowed to execute an action autonomously. Consequently, the only elements of the CAS transition 678
function, T, that are marginally dependent on features added to the state representation are λandτH. AsλandτH679
are learned online from observed feedback, we can directly compute the respective new distributions over ˆF′from 680
the current dataset which in turn updates the transition function as λandτHare both parameters of T. We suggest 681
that when one or both of these assumptions do not hold it is possible to use our approach as a means of identifying 682
Basich et al. /Artificial Intelligence 00 (2022) 1–34 19
the missing features and subsequently improving the system’s competence by directly updating the transition and cost 683
functions (e.g. via software updates). 684
A natural question is whether in the process of adding a discriminator to make some indiscriminate states discrim- 685
inate, we will, as an unintended by-product, make some discriminate state indiscriminate. 686
Remark 1. Adding a discriminator will never cause a discriminate state to become indiscriminate. 687
While possibly not obvious a priori, this remark is trivially true. Observe that any given discriminate state will 688
either be a ffected by the discriminator or it will not. If it is not a ffected, the feedback profile for the state will not 689
change. If the state is a ffected, then the initial state in question by definition no longer exists. More importantly, we 690
want to ensure that every state is eventually properly discriminated given a su fficient set of features. 691
The following proposition states that if every feature that the human uses to determine their feedback is available 692
to the robot, then there must be a point in time at which the robot has fully discriminated all states, and no state will 693
become indiscriminate past that point. 694
Proposition 4. Let I tbe the number of indiscriminate states at time t, and let λs,a
tbe the random variable representing 695
λ(s,a)after having received t feedback signals for (s,a)where each signal is sampled from the true distribution 696
λH(s,a). If FH⊆F,Hisϵ-consistent, δ>0and no (s,a)∈S×A is starved, then there exists some t∗>0for which 697
It′=0for all t′>t∗. 698
Proof. First, observe that as FH⊆F, if there is a point at which FH⊆ˆF, then because the sequence {λs,a
t}converges 699
in distribution by Proposition 3, lim t→∞Pr(|λs,a
t−λs,a
H|>γ)=0∀γ>0,(s,a)∈A×A. Hence, there exists some t∗>0 700
for which Pr(|λs,a
t−λs,a
H|>δ)=0 at which point it is clear that no state will be indiscriminate under δ. Consequently, 701
for the claim to not hold, it must be the case that for every t>0,FH\(FH∩ˆF),∅. Pick such a t, sufficiently large, 702
for which there is an indiscriminate state s∈S. There is some subset, G⊆FH\(FH∩ˆF), which is a discriminator 703
ofs. As this holds for all t>0 and s∈S, we either reach a satisficing t∗where FH\(FH∩ˆF),∅, and hence are 704
done, or where FH⊆ˆFwhich contradicts our assumption. 705
6. Empirical Evaluations 706
To test the competence-aware system, we implemented the CAS model in two simulated autonomous vehicle 707
domains at di fferent levels of decision-making abstraction. The first domain is a high-level navigation problem in 708
which an autonomous vehicle must plan (and execute) the optimal route to take between two locations conditioned 709
on its knowledge about di fferent intersections and streets and its own competence in performing di fferent maneuvers 710
at the various locations. The second takes a more fine-grained look at one of the maneuvers that can be performed 711
in the first domain, namely passing an obstacle that is blocking its lane, and is modeled after the domain depicted in 712
Example 1. 713
We evaluated our iterative state space refinement approach (Algorithm 1) on both of these domains as well, where 714
the key di fference is that the CAS model is missing features in its initial active feature space that do not impact its 715
transition model (that is, what it is technically capable of doing), but impact the human’s feedback signal likelihoods 716
regardless. We test our approach for multiple di fferent simulated humans, each of whom uses di fferent auxiliary 717
features in determining their feedback. We describe an overview of the domains below, and include additional exper- 718
imental details in Appendix A. 719
6.1. Autonomous Vehicle Navigation 720
6.1.1. Domain Description 721
In this domain, an autonomous vehicle operates in a known map represented by a directed graph G=(V,E) where 722
each vertex v∈Vrepresents an intersection and each edge e∈Erepresents a road; the graph used can be seen in 723
Figure 7 and is modeled after locations in the area of Amherst, Massachusetts. The autonomous vehicle is tasked with 724
navigating the map safely from a start vertex to a goal vertex. 725
Each vertex (intersection) state is represented by an ID for the vertex, a boolean indicator of the presence of 726
pedestrians, a boolean indicator of the presence of an occlusion limiting or blocking visibility, the number of other 727
Basich et al. /Artificial Intelligence 00 (2022) 1–34 20
Figure 7: A depiction of the map used for our simulated navigation domain with actual locations from OpenStreetMap (left) and
the abstracted representation of the navigation graph (right).
vehicles at the intersection (0-4), and the vehicle’s heading. Each edge (road) state is represented by a start vertex 728
ID, a destination vertex ID, the number of drivable lanes on the current road segment, the direction of travel, and a 729
boolean indicator of the presence of an obstruction blocking the agent’s lane. Additionally, each edge is associated 730
with a known length and speed of travel. Model parameters dictating the probabilities of each state variable (e.g. the 731
probability of a pedestrian being at a given intersection upon reaching it) are assumed to be known o ffline and given 732
as part of the model input. 733
In vertex states, the agent can either Go Straight ,Turn Right ,Turn Left ,U-Turn , each of which has a 734
cost of 10.0, or Wait , which has a cost of 1.0. All maneuvers succeed deterministically. In edge states, the agent 735
can either Continue orOvertake an obstruction, each with unit cost. Overtake is assumed to succeed with 736
probabilities [0 .2,0.5,0.8] depending on the number of lanes. Continue fails deterministically in the presence of 737
an obstruction, and if there is no obstruction transitions the agent to the end-vertex of the edge with probability 738
p∝speed (e)/length(e) or otherwise to the same edge with some probability of an obstruction occurring. We model 739
the expected duration as part of the transition function, rather than the cost function, to allow for the development of 740
an obstruction in the A V’s lane while traversing an edge segment which may be very long in real life. 741
We consider the following levels of autonomy, L={l0,l1,l2,l3}where l3does not require any involvement from 742
the human at all (i.e. we assume the probability of an override is 0), l2allows the agent to execute an action under 743
supervision, during which the human may override the action if they deem it unsafe, l1which requires explicit approval 744
from the human for an action prior to its execution during which, if approval is received, the agent may attempt to 745
execute the action under supervision, and if the action is disapproved by the human the agent must select a di fferent 746
action to perform, and l0which requires full transfer of control to the human to complete the action. 747
The autonomy profile, κ, is initialized toLin edge states without an obstruction and otherwise to {l0,l1,l2}. The 748
feedback profile, λ, is initialized to be uniformly random over the possible feedback signals. There is an associated 749
cost of 10.0 to the human for operating in l0, as the human is required to manually control the vehicle, a cost of 2.0 for 750
operating in l1, a cost of 1.0 in level l2, and no additional cost to the human when operating in l3. The system incurs a 751
cost of 3.0 when receiving a negative response in l1and a cost of 10.0 when receiving an override in l2as we assume 752
that the human completes the intended action. 753
6.1.2. Results 754
To validate the CAS model in the A V navigation domain, we randomly selected a start node and goal node each 755
episode to ensure that the system had the ability to visit the entirety of the graph. We repeated this for four di fferent 756
human authorities where we varied their consistency: 0 .8, 0.9, 1.0 (i.e. perfectly consistent), and, in the final case, 757
a human who starts with a very low consistency (0 .6) to reflect their poor understanding of the capabilities of the 758
system, but increases their consistency by a small amount (0 .1) each episode to reflect their improved understanding 759
Basich et al. /Artificial Intelligence 00 (2022) 1–34 21
of the capabilities of the system over time as they interact with it. Figures 8, 9 and 10 report the results from the 760
experiment conducted in the autonomous vehicle navigation domain. 761
Figure 8: Empirical results from simulations of a fixed route
(12→7) showing the expected cost (top) to goal of a CAS
and the average cost (bottom) over 100 trials with a CAS (blue)
and without a CAS (red) as a function of the number of signals
received.Figure 8 depicts the results on a fixed route (node 762
12 to node 7 in Figure 7). The top graph shows the ex- 763
pected cost of the route and the bottom graph shows 764
the actual mean cost (averaged over 100 simulations) 765
of the CAS (blue) compared against an agent just us- 766
ing the domain model agnostic to its competence, with 767
a human overriding as necessary (i.e. e ffectively al- 768
ways operating in level l2) (red). These results demon- 769
strate that by learning an accurate competence model 770
and incorporating that into the planning model, a CAS 771
can e fficiently ( <40 feedback signals) improve both 772
its average performance and expected performance, 773
significantly outperforming a system that is agnostic 774
to its competence and the dynamics of human interac- 775
tion. These experiments were taken from the human 776
with consistency ϵ=0.9 but we note that very similar 777
results were obtained in all cases. 778
Figure 9 depicts in the top two rows the conver- 779
gence of the level-optimality of the competence-aware 780
system as a function of the number of feedback signals 781
received, and in the bottom row the number of signals 782
received over the course of 100 episodes (where each 783
episode is a random route) for a system with a CAS 784
(blue) and a system without a CAS (red). Each graph 785
corresponds to a human authority with a di fferent con- 786
sistency,ϵ, as detailed above. In all cases, the level 787
optimality reaches 100% over all reachable states in 788
the domain. Interestingly, in Figure 9d, the results are 789
more comparable to a human with a fixed consistency 790
of 0.9 or 1.0 in the level-optimality convergence rate 791
than they are to a human with a fixed consistency of 792
0.8 which requires roughly twice as many feedback 793
signals to converge to level-optimality. This demon- 794
strates that even a CAS with a human who starts with 795
an initially poor understanding of the system’s capabilities, and consequently low consistency, can e fficiently reach 796
level-optimality if the human’s understanding and consistency improves at a consistent rate. The figures in the bottom 797
row illustrate that without a CAS the number of feedback signals provided by the human grows linearly, demonstrat- 798
ing the significant disparity in burden placed upon the human in a system without a CAS model compared to a system 799
with a CAS model. We only depict the results for 0 .8 and 1.0 for the sake of space, but the results look very similar for 800
allϵ-consistencies considered. Overall these results demonstrate the primary goal of the CAS model which is that it 801
enables a system to e fficiently reach level-optimality, optimizing the trade-o ffbetween autonomous performance and 802
human assistance, thereby reducing the net burden placed on the human over the course of the system’s operation. 803
Figure 10 depicts the change in routes taken between the first episode and the 100th episode for the CAS model 804
for four fixed routes. Here, purple denotes parts of the route taken that are the same, red denotes parts of the route that 805
are taken in the first episode but not the 100th, and blue denotes parts of the route that are taken in the 100th episode 806
but not the first. This figure illustrates the macro policy changes made as the CAS learns its competence—namely 807
altering its route to avoid states or trajectories of low competence which would require excessive human assistance— 808
in addition to the micro changes of selecting which level of autonomy to use in any given situation. In general, we find 809
that the A V’s behavior changes to avoid areas densely populated with pedestrians, occlusions, and single lane roads, 810
such as downtown Amherst (nodes 8-11) and University of Massachusetts Amherst (nodes 6-8). 811
Basich et al. /Artificial Intelligence 00 (2022) 1–34 22
(a)ϵ=0.8
 (b)ϵ=0.9
(c)ϵ=1.0
 (d)ϵ=0.6:1.0
(e)ϵ=0.8
 (f)ϵ=1.0
Figure 9: Empirical results from the autonomous vehicle navigation domain with varying levels of human consistency showing
the level-optimality as a function of the number of feedback signals received (9a – 9d) and the number of feedback signals received
over the first 100 routes executed (9e - 9f). In Figure 9d, the human consistency increases after each route is executed, mimicking
a human whose consistency improves the more it interacts with the system.
Basich et al. /Artificial Intelligence 00 (2022) 1–34 23
Figure 10: Comparison of routes taken before and after the CAS learns its competence. Purple indicates shared route, red indicates
route taken by starting model alone, blue indicates route taken by ending model alone. Green and yellow circles denote start and
end nodes respectively.
6.2. Autonomous Vehicle Obstacle Passing 812
6.2.1. Domain Description 813
Figure 11: Illustration of the A V
obstacle passing domain.In this domain, modeled after the problem depicted in Example 1, an au- 814
tonomous vehicle must overtake an obstacle that is blocking its lane on a one-lane 815
road. Importantly, this maneuver required that the A V drive into the oncoming 816
traffic’s lane in order to overtake the obstacle, a potentially dangerous maneuver. 817
Each state is represented by the vehicle’s position (0-4), the position of an on- 818
coming vehicle (0-3, or unknown), and whether the oncoming vehicle has given 819
priority to the A V to attempt its overtake. Model parameters dictating the behav- 820
ior of oncoming vehicles is assumed to be known o ffline and given as part of the 821
model input. 822
The autonomous vehicle can perform the following actions: Wait ,Edge , and 823
Go.Edge provides visibility of oncoming tra ffic to the A V if unknown and oth- 824
erwise advances the A V’s position with probability 0.5. Godeterministically ad- 825
vances the A V’s position, which results in a crash if the A V and an oncoming 826
vehicle share the same position. Stop holds the A V’s position, during which time 827
the oncoming vehicles position may change (or become empty), or the oncoming 828
vehicle may give priority to the A V . If the A V has priority it is assumed that the on- 829
coming tra ffic will stay stopped until the A V has finished its overtake. All actions 830
have unit cost, and crashing incurs a very high cost. 831
Basich et al. /Artificial Intelligence 00 (2022) 1–34 24
(a) Autonomous Vehicle Obstacle Passing Domain Level-
Optimality
 (b)Autonomous Vehicle Obstacle Passing Average Cost
Figure 12: Empirical results from the autonomous vehicle obstacle passing domain depicting the level-optimality (left) over all
reachable states (red) and the full state space (blue), and the average cost (right) over 1000 simulations, as a function of the number
of feedback signals received.
We consider the following levels of autonomy, L={l0,l1,l2}where l2does not involve the human at all, l1allows 832
the agent to execute an action under supervision, during which the human may override the action if they deem it 833
unsafe, and l0which requires full transfer of control to the human to complete the action. Note that we do not include 834
the level l1from the prior domain (referred to earlier as “verified autonomy” in Table 1) due to the second-to-second 835
nature of decision making in this safety-critical domain, where prompting the human for explicit approval before 836
every action may be impractical or even dangerous. 837
The autonomy profile, κ, is initialized to{l0,l1}in all cases; i.e., in such a safety critical domain it is expected that, 838
initially, the human is always aware and ready to override the system. As above, the feedback profile λis initialized 839
to be uniformly random. The human incurs a cost of 10.0 when the CAS operates in l0but is assumed to complete 840
the maneuver successfully (i.e., the human does not give back control part way through passing the obstacle), a cost 841
of 1.0 when supervising in l2, and no cost in l3. The system receives a penalty of 10.0 when being overridden by the 842
human. 843
6.2.2. Results 844
In the A V obstacle passing domain, the problem—i.e., the initial state and goal state—stayed fixed each episode. 845
Figures 12a and 12b report the results from the experiment conducted in the autonomous vehicle obstacle passing 846
domain. Figure 12a shows the level-optimality of the CAS over all states in the domain and all reachable states 847
(each episode) plotted against the number of feedback signals received from the human, in this case consisting only 848
of overrides. The figure illustrates that the CAS is able to converge to level-optimality on all reachable states in the 849
domain with slightly more than 100 feedback signals. The slower convergence rate is due to a stricter requirement 850
on gated exploration due to the more safety-critical nature of the domain (see Appendix A for details). 100% Level- 851
optimality is not reached on the whole state space due to the absence of a portion of the state space ever being visited 852
(or even reachable), preventing the human authority from providing any feedback for actions taken in those states. 853
Figure 12b reports the expected cost of overtaking the obstacle and illustrates that the expected cost decreases as 854
the level-optimality increases, corroborating the results from the previous domain. This also demonstrates that, in 855
certain domains, performance may be improved to near optimal performance without even needing to converge to full 856
level-optimality across the entire state space due to variations in state reachability trends. 857
6.3. Iterative State Space Refinement 858
To validate the iterative state space refinement method, we implemented Algorithm 1 and compared the perfor- 859
mance of a CAS with Algorithm 1 and a CAS without it on both of the domains defined above (autonomous vehicle 860
navigation and autonomous vehicle obstacle passing). In both experiments we considered di fferent human users of 861
the autonomous vehicle system, each of whose feedback was conditioned not just on the features already used by the 862
CAS model that directly impacted the CAS’s technical performance (i.e., the existence of a pedestrian, an occlusion, 863
Basich et al. /Artificial Intelligence 00 (2022) 1–34 25
etc.) but additionally on auxiliary features which are tracked by the autonomous vehicle but not included in its a priori 864
planning model, as the features in question are di fferent for each person, and do not (directly) impact the transition 865
and cost dynamics of the system. 866
In the A V navigation domain, the inactive feature set included the following features: whether the A V has a trailing 867
vehicle, a vehicle to its left, or a vehicle to its right, whether the A V has been “waiting” to move, whether it is daytime 868
or nighttime, and whether it is sunny, rainy, or snowy. In the A V obstacle passing domain, we consider the same 869
inactive features except whether there is a vehicle to the A V’s left or right, as the problem is for single lane roads. 870
In the A V navigation domain, we consider two “people” implemented as software agents: the first person is 871
cautious with low trust in letting the A V operate in challenging environmental conditions (even though they do not 872
impact the A V in simulation), for instance taking over control when the system attempts an overtake on a road segment 873
when it is either snowing or rainy and night time. The intuition here is that the weather conditions impacts the human’s 874
ability to fully assess the situation and hence the veracity of the A V’s actions, prompting them to take control of the 875
vehicle themselves. We refer to them as “Cautious”. The second person is motivated by more social factors, and is 876
more likely to take control of the vehicle when there is a trailing vehicle the A V is blocking, and or when the A V has 877
been stopped for too long (either on a road segment behind an obstruction, or at an intersection). We refer to them as 878
“Conscientious”. 879
In the A V obstacle passing domain, we consider three “people” implemented as software agents (see Appendix 880
A for more details): the first is motivated by the same features as the first person above; we again refer to them as 881
“Cautious”. The second person is motivated by whether there is a trailing vehicle that they are blocking, prompting 882
them to take control if the A V waits to long to attempt its overtake; we also refer to them as “Conscientious”. The 883
third person is in a rush and takes over control if the A V is waiting too long or doesn’t go when it has priority; we 884
refer to them as “Rushed”. Each simulated person is perfectly consistent up to some fixed noise ϵ, within which they 885
return uniformly random feedback. 886
We note that in both domains, some inactive features are never used by any of the humans simulated, and hence 887
we aim to show that our approach does not simply “pick all features" in the inactive feature space. Additionally, one 888
important distinction between the two domains is that the additional inactive features may change at each new state 889
in the A V navigation domain, but are fixed in the A V obstacle passing domain at the beginning of each episode due to 890
the short time horizon of the problem. Details of the simulated humans can be found in Appendix A. 891
6.3.1. Results 892
Figure 13 shows the results of our experiment, comparing the performance of a CAS with and without the iterative 893
state space refinement (ISSR) approach (Algorithm 1) implemented, on the A V navigation domain with random routes 894
each episode. Figure 14 shows the results for the A V obstacle passing domain. In Figure 13, we can see that the CAS 895
with the ISSR implemented converges to higher level-optimality on all state in the domain, and 100% level-optimality 896
on all states visited each episode, leading to far fewer feedback signals from the human, for both human authorities. 897
Additionally, in both cases, the only features added to the active feature space where the features in the inactive feature 898
space that were actually used by the humans in determining their feedback. 899
Figure 14 shows the results for the A V obstacle passing domain. Note that we include results on all reachable 900
states here because the additional features stay fixed through each episode, whereas in the A V Navigation domain, 901
they can change throughout an episode and the transition dynamics are (by design) not modeled by the agent. 902
There are several key takeaways from these graphs. First, if we consider the level-optimality over all states in 903
the domain, it is higher for the ISSR-CAS in the cases of all three human authorities, than for the CAS without 904
ISSR active, indicating that our approach is enabling the CAS to generalize its competence model to a larger portion 905
of the (unvisited) state space. We remark that by adding features in order to refine the state space, the number of 906
states increases multiplicatively with each feature added, meaning that not only is the ISSR-CAS level-optimal in a 907
larger portion of the state space, that directly translates to being level-optimal in a larger number of unique situations. 908
More important are the results depicting the level-optimality over all visited states each episode; here, we see that 909
this reaches 100%, or near 100%, for all 3 human authorities with fewer than 50 feedback signals. However, we 910
observe an interesting phenomenon for the CAS without ISSR active; namely, we see several clusters of green at the 911
far right (at which point no additional feedback signals were received). This phenomenon is due to the fact that the 912
CAS learns to operate in l0, that is, full human control, in a large portion of the statespace because it cannot properly 913
discriminate the feedback received from the human conditioned on features in the inactive feature space, which is 914
Basich et al. /Artificial Intelligence 00 (2022) 1–34 26
(a)Person 1 (Cautious)
(b)Person 2 (Conscientious)
Figure 13: Iterative state space refinement results for two human authorities in the autonomous vehicle navigation domain, showing
the level optimality after each episode as a function of the number of feedback signals with (left) and without (right) Algorithm 1
implemented. Colors indicate the level-optimality over states visited during each episode (green) and the full state space (blue).
correct for certain settings of these features (which, to reiterate, are set and fixed at the start of each episode), but not 915
for others. However, because the state space is not refined enough to consider these decision boundaries, the CAS 916
learns to operate at the incorrect level of autonomy (relative to the full feature space) in certain conditions. 917
These results demonstrate that the ISSR method is e ffective at enabling a competence-aware system to improve its 918
competence online when missing from its active feature space features used by its human authority. 919
7. Discussion and Future Work 920
7.1. Autonomy Profile Initialization 921
Because we restrict the system to choose policies from Πκ, if the autonomy profile κis altered, so too is the space of 922
allowed policies. Hence, there is a trade-o ffwhen setting the initial constraints on the allowed autonomy of the system, 923
i.e.,κ. One can take a conservative approach and constrain the system significantly, for instance setting |κ(s,a)|=1 924
so that a single level is deterministically selected for every ( s,a)∈S×A, reducing the problem complexity to solving 925
the underlying domain model. However, doing so risks a globally sub-optimal policy with respect to Land may, 926
depending on the initial κ, make reaching the globally optimal policy impossible. On the other extreme, one can take 927
a risky approach and not constrain the system at all a priori, leaving the decision of choosing the level of autonomy 928
completely up to the system when solving its model. This approach, while necessarily containing the optimal policy 929
(subject to the agent’s model) is naturally slower due to the larger policy space and inherently less safe as the agent 930
Basich et al. /Artificial Intelligence 00 (2022) 1–34 27
(a)Person 1 (Cautious)
(b)Person 2 (Conscientious)
(c)Person 3 (Rushed)
Figure 14: Iterative state space refinement results for three human authorities in the autonomous vehicle obstacle passing domain,
showing the level optimality after each episode as a function of the number of feedback signals with (left) and without (right)
Algorithm 1 implemented. Colors indicate the level-optimality over states visited during each episode (green), all reachable states
each episode (red), and the full state space (blue).
Basich et al. /Artificial Intelligence 00 (2022) 1–34 28
can take actions in undesirable levels. Figure 4 illustrates di fferent partitionings of the policy space under di fferent 931
autonomy profiles. 932
We propose that in practice, the desired initialization is somewhere in the middle; κshould be less constraining 933
in situations where the expected cost of failure is relatively low, and more constraining in situations where it is high. 934
While the model makes no such requirements, in many practical settings such information may be at least partially 935
known a priori for a specific domain. For instance, in an autonomous vehicle, κshould be more constraining initially 936
in situations involving pedestrians, poor visibility, or chaotic environments such as large intersections with multiple 937
vehicles; however, initial testing may indicate that driving along a highway is low-risk and may not require a highly 938
constraining κ. 939
7.2. Model Assumptions 940
We now discuss the practical considerations of the two main assumptions made in Section 3.4: (1) the human au- 941
thority,H, provides consistent feedback and (2) the human authority’s feedback comes from a stationary, Markovian 942
distribution. 943
Implicit in Assumption (1) is that humans respond appropriately to each situation, possibly with some noise 944
representing the likelihood of human error. However, because of the limited scope of the system’s domain model, it 945
could be that perfectly consistent feedback from H’s perspective is perceived to be random by the system, particularly 946
when it is not aware of the domain features that explain the human feedback. As an example, consider a robot that can 947
open ‘push’ doors and cannot open ‘pull’ doors, but does not model this discriminating feature. If the robot cannot 948
discriminate between these types of doors, consistent and correct human feedback (approving autonomously opening 949
‘push’ doors only) may be perceived by the robot to be arbitrary or random. Although in practice one may wish 950
to avoid such situations, we emphasize that the system will still converge to its competence for the state features it 951
uses—possibly a low competence—when the feedback distribution appears to be random. 952
Assumption (2)—the human feedback distribution λHis stationary and Markovian from the start—implies that 953
the human has good knowledge of the system from the start. That may not be realistic in certain domains. It is more 954
likely that the feedback signals may vary based upon the observed performance of the system over time. However, 955
as the human authority observes the system’s performance, it is reasonable to assume that their feedback distribution 956
will eventually reach a stationary point as long as the system’s underlying capabilities stay fixed. Therefore, even if 957
there are erroneous feedback signals provided early in this process, in the limit the system will still converge to its 958
competence. Two possible means of expediting this is to introduce a training phase at the beginning of the system’s 959
deployment to allow the human to observe the system’s performance and develop accurate expectations regarding the 960
system’s capabilities, and to introduce standardized feedback criteria that is made known to the human a priori . 961
7.3. Partially Observable Models 962
As stated in Section 3, the CAS is designed to handle fully-observable sequential decision-making models like 963
SSPs and, more generally, MDPs, but is not immediately compatible with partially observable models (or mixed- 964
observability models) despite partial observability and other limitations on state observability being a natural con- 965
tributor to limitations on system competence. The two main barriers in directly applying the CAS to models like a 966
POMDP are (1) the challenge of appropriately associating feedback signals with domain states for learning purposes 967
when the system only has access to a belief state at any given time, and (2) the challenge in defining the competence 968
of a belief-state, where the system implicitly does not know its true state. Future work will consider ways in which 969
we can extend both the representation of feedback signals and the definition of competence, and consequently the 970
CAS model, to such domains in a well-defined manner, for instance by changing the definition of competence from a 971
function on states to a function on observations. 972
8. Conclusion 973
We introduce a new framework for representing, learning, and reasoning with self-competence models in semi- 974
autonomous systems. Competence in our approach represents the level of autonomy that the system can handle 975
reliably based on human feedback. More precisely, we define competence as the optimal level of autonomy in any 976
Basich et al. /Artificial Intelligence 00 (2022) 1–34 29
given situation, consistent with perfect human feedback. We present a novel decision-making framework, competence- 977
aware systems , that enables a semi-autonomous system to learn its own competence over time through interactions 978
with a human authority. The result is a system that can handle risky scenarios by relying on the human authority to 979
compensate for limitations or constraints on its autonomous abilities, while simultaneously optimizing its autonomous 980
operation to reduce unnecessary reliance on humans. 981
We illustrate the operation of a competence-aware system with a running example and prove several theoretical 982
properties of the CAS model. In particular, we prove that under standard convergence assumptions the model will 983
converge to level-optimality , guaranteeing that the system consistently operates at its competence. We test the e fficacy 984
of our model empirically on two simulated autonomous vehicle domains, at di fferent levels of reasoning abstraction, 985
and demonstrate that the competence-aware system can e fficiently reach high level-optimality, optimizing the trade- 986
offbetween its own autonomous operation and human assistance, and leading to less burden on the human and a more 987
cost-e ffective overall plan. 988
Preliminary internal testing on an autonomous vehicle prototype suggests that designing a perfectly specified CAS 989
model for real-world, highly-unstructured domains is a non-trivial task. Even with expert domain knowledge, an initial 990
model may be missing features used by the human in determining their feedback for the CAS. To avoid solving this 991
naively with the inclusion of all possible system features in the CAS’s domain model (many of which would serve no 992
functional purpose but would cause the state space to explode and render planning intractable), we devise the iterative 993
state space refinement approach. Described in Algorithm 1, the approach provides a competence-aware system the 994
means to gradually refine its state representation online, enabling it to better identify the boundaries between state- 995
action pairs with di fference competences. This ability is particularly relevant in the context of systems deployed in 996
the real world where human feedback may be conditioned on features that are unspecified or unknown a priori . Such 997
features may not impact the original stated objectives of the system, but could influence unstated human preferences, 998
trust, safety, and social conscientiousness. We prove that, when possible, this approach is guaranteed to reach a 999
point where all states are discriminated, and demonstrate empirically that a CAS with this approach implemented far 1000
outperforms a CAS without it when the CAS cannot properly learn from human feedback due to missing state features. 1001
In particular, the modified CAS requires both fewer total feedback signals from the human, placing less burden on 1002
the human, and is more sample e fficient with the feedback it receives in learning its competence, leading to a higher 1003
level-optimality for the CAS. 1004
The primary direction of future work lies in extending competence-aware systems to models with limited state ob- 1005
servability, such as MOMDPs and POMDPs. This includes devising a method of associating human feedback acquired 1006
in belief-states with underlying states in the domain, when the system does not know which state is responsible for 1007
the feedback, and generalizing competence to belief-states in a well-defined way that still captures the risk-sensitive 1008
semantics of the current approach. We are also interested in extending our model of human feedback to account for 1009
temporal uncertainty about the feedback signals, and to handle both proactive and retroactive feedback that is not 1010
necessarily associated with the action being currently executed. 1011
Acknowledgements 1012
This work was supported in part by the National Science Foundation grants IIS-1724101 and IIS-1954782 and in 1013
part by the Alliance Innovation Lab Silicon Valley. 1014
References 1015
[1] Allen, J.E., Guinn, C.I., Horvtz, E., 1999. Mixed-initiative interaction. IEEE Intelligent Systems and their Applications 14, 14–23. doi: 10. 1016
1109/5254.796083 . 1017
[2] Altman, E., 1999. Constrained Markov decision processes: stochastic modeling. Routledge. 1018
[3] Barto, A.G., Sutton, R.S., Anderson, C.W., 1983. Neuronlike adaptive elements that can solve di fficult learning control problems. IEEE 1019
Transactions on Systems, Man, and Cybernetics SMC-13, 834–846. 1020
[4] Basich, C., Svegliato, J., Beach, A., Wray, K.H., Witwicki, S., Zilberstein, S., 2021. Improving competence via iterative state space 1021
refinement, in: IEEE /RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE. pp. 1865–1871. 1022
[5] Basich, C., Svegliato, J., Wray, K.H., Witwicki, S., Biswas, J., Zilberstein, S., 2020. Learning to optimize autonomy in competence-aware 1023
systems, in: International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pp. 123–131. 1024
[6] Beal, J., Rogers, M., 2020. Levels of autonomy in synthetic biology engineering. Molecular Systems Biology 16, e10019. doi: 10.15252/ 1025
msb.202010019 . 1026
Basich et al. /Artificial Intelligence 00 (2022) 1–34 30
[7] Beck, A.B., Schwartz, A.D., Fugl, A.R., Naumann, M., Kahl, B., 2015. Skill-based exception handling and error recovery for collaborative 1027
industrial robots., in: IROS FinE-R Workshop, pp. 5–10. 1028
[8] Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., Tanaka, F., 2018. Social robots for education: A review. Science Robotics 3, 1029
eaat5954. 1030
[9] Bertsekas, D.P., Tsitsiklis, J.N., 1991. An analysis of stochastic shortest path problems. Mathematics of Operations Research 16, 580–595. 1031
[10] Biswas, J., Veloso, M., 2016. The 1,000-km challenge: Insights and quantitative and qualitative results. IEEE Intelligent Systems 31, 86–96. 1032
doi:10.1109/MIS.2016.53 . 1033
[11] Blaom, A.D., Kiraly, F., Lienart, T., Simillides, Y ., Arenas, D., V ollmer, S.J., 2020. MLJ: A julia package for composable machine learning. 1034
Journal of Open Source Software 5, 2704. doi: 10.21105/joss.02704 . 1035
[12] Bonet, B., Ge ffner, H., 2003. Labeled rtdp: Improving the convergence of real-time dynamic programming., in: International Conference 1036
on Automated Planning and Scheduling (ICAPS), pp. 12–21. 1037
[13] Bradshaw, J.M., Jung, H., Kulkarni, S., Johnson, M., Feltovich, P., Allen, J., Bunch, L., Chambers, N., Galescu, L., Je ffers, R., et al., 2005. 1038
Kaa: Policy-based explorations of a richer model for adjustable autonomy, in: International Joint Conference on Autonomous Agents and 1039
Multiagent Systems (AAMAS), pp. 214–221. 1040
[14] Bresina, J., Jónsson, A., Morris, P., Rajan, K., 2005. Mixed-initiative activity planning for Mars rovers, in: International Joint Conference 1041
on Artificial Intelligence (IJCAI), pp. 1709–1710. 1042
[15] Broggi, A., Bertozzi, M., Fascioli, A., Bianco, C.G.L., Piazzi, A., 1999. The ARGO autonomous vehicle’s vision and control systems. 1043
International Journal of Intelligent Control and Systems 3, 409–441. 1044
[16] Broggi, A., Cerri, P., Felisa, M., Laghi, M.C., Mazzei, L., Porta, P.P., 2012. The VisLab intercontinental autonomous challenge: An extensive 1045
test for a platoon of intelligent vehicles. International Journal of Vehicle Autonomous Systems 10, 147–164. doi: 10.1504/IJVAS.2012. 1046
051250 . 1047
[17] Brown, G., Pocock, A., Zhao, M.J., Luján, M., 2012. Conditional likelihood maximisation: A unifying framework for information theoretic 1048
feature selection. The Journal of Machine Learning research 13, 27–66. 1049
[18] Bruemmer, D.J., Few, D.A., Boring, R.L., Marble, J.L., Walton, M.C., Nielsen, C.W., 2005. Shared understanding for collaborative control. 1050
IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans 35, 494–504. 1051
[19] Capobianco, R., Gemignani, G., Iocchi, L., Nardi, D., Riccio, F., Vanzo, A., 2016. Contexts for symbiotic autonomy: Semantic mapping, 1052
task teaching and social robotics, in: AAAI Workshop on Symbiotic Cognitive Systems. 1053
[20] Cashmore, M., Fox, M., Larkworthy, T., Long, D., Magazzeni, D., 2014. AUV mission control via temporal planning, in: IEEE International 1054
Conference on Robotics and Automation (ICRA), pp. 6535–6541. 1055
[21] Chernova, S., Veloso, M., 2009. Interactive policy learning through confidence-based autonomy. Journal of Artificial Intelligence Research 1056
34, 1–25. doi: 10.1613/jair.2584 . 1057
[22] Chiou, M., Hawes, N., Stolkin, R., 2021. Mixed-initiative variable autonomy for remotely operated mobile robots. ACM Transactions on 1058
Human-Robot Interaction (THRI) 10, 1–34. 1059
[23] Chiou, M., Hawes, N., Stolkin, R., Shapiro, K.L., Kerlin, J.R., Clouter, A., 2015. Towards the principled study of variable autonomy in 1060
mobile robots, in: IEEE International Conference on Systems, Man, and Cybernetics (ICSMC), IEEE. pp. 1053–1059. 1061
[24] Clouse, J.A., 1996. On integrating apprentice learning and reinforcement learning. Ph.D. thesis. University of Massachusetts. 1062
[25] Coradeschi, S., Sa ffiotti, A., 2006. Symbiotic robotic systems: Humans, robots, and smart environments. IEEE Intelligent Systems 21, 1063
82–84. doi: 10.1109/MIS.2006.59 . 1064
[26] Costen, C., Rigter, M., Lacerda, B., Hawes, N., 2022. Shared autonomy systems with stochastic operator models, in: International Joint 1065
Conferences on Artificial Intelligence Organization (IJCAI), pp. 4614–4620. 1066
[27] Das, D., Banerjee, S., Chernova, S., 2021. Explainable ai for robot failures: Generating explanations that improve user assistance in fault 1067
recovery, in: ACM /IEEE International Conference on Human-Robot Interaction (HRI), pp. 351–360. 1068
[28] Dickmanns, E.D., 2007. Dynamic vision for perception and control of motion. Springer Science & Business Media. 1069
[29] Dorais, G., Bonasso, R.P., Kortenkamp, D., Pell, B., Schreckenghost, D., 1999. Adjustable autonomy for human-centered autonomous 1070
systems, in: IJCAI Workshop on Adjustable Autonomy Systems, pp. 16–35. 1071
[30] Dubois, D.D., 1998. The competency casebook: Twelve studies in competency-based performance improvement. Human Resource Devel- 1072
opment. 1073
[31] Eliot, L., 2020a. Legal judgment prediction (ljp) amid the advent of autonomous ai legal reasoning. arXiv preprint arXiv:2009.14620 . 1074
[32] Eliot, L., 2020b. An ontological AI-and-law framework for the autonomous levels of AI legal reasoning. arXiv preprint arXiv:2008.07328 . 1075
[33] Ferguson, G., Allen, J.F., Miller, B.W., et al., 1996. TRAINS-95: Towards a mixed-initiative planning assistant, in: AAAI International 1076
Conference on Artificial Intelligence Planning Systems (AIPS), pp. 70–77. 1077
[34] Ficuciello, F., Tamburrini, G., Arezzo, A., Villani, L., Siciliano, B., 2019. Autonomy in surgical robots and its meaningful human control. 1078
Journal of Behavioral Robotics 10, 30–43. doi: 10.1515/pjbr-2019-0002 . 1079
[35] Fong, T., Thorpe, C., Baur, C., 2003. Multi-robot remote driving with collaborative control. IEEE Transactions on Industrial Electronics 1080
50, 699–704. 1081
[36] Gao, Y ., Chien, S., 2017. Review on space robotics: Toward top-level science through space exploration. Science Robotics 2. doi: 10. 1082
1126/scirobotics.aan5074 . 1083
[37] Ghalamzan, E.A.M., Abi-Farraj, F., Giordano, P.R., Stolkin, R., 2017. Human-in-the-loop optimisation: mixed initiative grasping for 1084
optimally facilitating post-grasp manipulative actions, in: IEEE /RSJ International Conference on Intelligent Robots and Systems (IROS), 1085
IEEE. pp. 3386–3393. 1086
[38] Ghallab, M., Nau, D., Traverso, P., 2016. Automated planning and acting. Cambridge University Press. 1087
[39] Gilbert, T.F., 1996. Human Competence: Engineering Worthy Performance. 1088
[40] Greenblatt, N.A., 2016. Self-driving cars and the law. IEEE Spectrum 53, 46–51. 1089
[41] Gri ffith, S., Subramanian, K., Scholz, J., Isbell, C.L., Thomaz, A.L., 2013. Policy shaping: Integrating human feedback with reinforcement 1090
learning. Advances in Neural Information Processing Systems 26. 1091
Basich et al. /Artificial Intelligence 00 (2022) 1–34 31
[42] Hager, P., Gonczi, A., 1996. What is competence? Medical Teacher 18, 15–18. doi: 10.3109/01421599609040255 . 1092
[43] Hawes, N., Burbridge, C., Jovan, F., Kunze, L., Lacerda, B., Mudrova, L., Young, J., Wyatt, J., Hebesberger, D., Kortner, T., et al., 1093
2017. The STRANDS project: Long-term autonomy in everyday environments. IEEE Robotics & Automation Magazine 24, 146–156. 1094
doi:10.1109/MRA.2016.2636359 . 1095
[44] Hewitt, C., Politis, I., Amanatidis, T., Sarkar, A., 2019. Assessing public perception of self-driving cars: The autonomous vehicle acceptance 1096
model, in: International Conference on Intelligent User Interfaces, pp. 518–527. 1097
[45] Ho ff, K.A., Bashir, M., 2015. Trust in automation: Integrating empirical evidence on factors that influence trust. Human Factors 57, 1098
407–434. doi: 10.1177/0018720814547570 . 1099
[46] Holtz, J., Guha, A., Biswas, J., 2018. Interactive robot transition repair with smt, in: International Joint Conference on Artificial Intelligence 1100
(IJCAI), pp. 4905–4911. URL: https: //joydeepb.com /Publications /ijcai2018_srtr.pdf, doi: 10.24963/ijcai.2018/681 . 1101
[47] Huenupán, F., Yoma, N.B., Molina, C., Garretón, C., 2008. Confidence based multiple classifier fusion in speaker verification. Pattern 1102
Recognition Letters 29, 957–966. doi: 10.1016/j.patrec.2008.01.015 . 1103
[48] Jiang, S., Arkin, R.C., 2015. Mixed-initiative human-robot interaction: definition, taxonomy, and survey, in: IEEE International Conference 1104
on Systems, Man, and Cybernetics (ICSMC), IEEE. pp. 954–961. 1105
[49] Kaelbling, L.P., Littman, M.L., Moore, A.W., 1996. Reinforcement learning: A survey. Journal of Artificial Intelligence Research (JAIR) 4, 1106
237–285. 1107
[50] Knox, W.B., Breazeal, C., Stone, P., 2012. Learning from feedback on actions past and intended, in: ACM /IEEE International Conference 1108
on Human-Robot Interaction, Late-Breaking Reports Session (HRI), Citeseer. 1109
[51] Knox, W.B., Stone, P., Breazeal, C., 2013. Training a robot via human feedback: A case study, in: International Conference on Social 1110
Robotics (ICSR), Springer. pp. 460–470. 1111
[52] Kolobov, A., Mausam, Weld, D.S., 2012. A theory of goal-oriented MDPs with dead ends, in: Conference on Uncertainty in Artificial 1112
Intelligence (UAI), pp. 438–447. 1113
[53] Kuncheva, L.I., 2004. Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons. 1114
[54] Kunz, C., Murphy, C., Singh, H., Pontbriand, C., Sohn, R.A., Singh, S., Sato, T., Roman, C., Nakamura, K., Jakuba, M., et al., 2009. Toward 1115
extraplanetary under-ice exploration: Robotic steps in the Arctic. Journal of Field Robotics 26, 411–429. doi: 10.1002/rob.20288 . 1116
[55] Lin, P., 2016. Why ethics matters for autonomous cars, in: Autonomous driving. Springer, pp. 69–85. 1117
[56] Lysiak, R., Kurzynski, M., Woloszynski, T., 2014. Optimal selection of ensemble classifiers using measures of competence and diversity of 1118
base classifiers. Neurocomputing 126, 29–35. doi: 10.1016/j.neucom.2013.01.052 . 1119
[57] Maurer, M., Gerdes, J.C., Lenz, B., Winner, H., 2016. Autonomous driving: technical, legal and social aspects. Springer. 1120
[58] McQuillin, E., Churamani, N., Gunes, H., 2022. Learning socially appropriate robo-waiter behaviours through real-time user feedback, in: 1121
ACM /IEEE International Conference on Human-Robot Interaction (HRI), pp. 541–550. 1122
[59] Meeussen, W., Marder-Eppstein, E., Watts, K., Gerkey, B.P., 2011. Long term autonomy in o ffice environments, in: Robotics: Science and 1123
Systems (RSS) Alone Workshop. 1124
[60] Mo ffitt, V .Z., Franke, J.L., Lomas, M., 2006. Mixed-initiative adjustable autonomy in multi-vehicle operations. Association for Unmanned 1125
Vehicle Systems International . 1126
[61] Moreira, I., Rivas, J., Cruz, F., Dazeley, R., Ayala, A., Fernandes, B., 2020. Deep reinforcement learning with interactive feedback in a 1127
human–robot environment. Applied Sciences 10, 5574. 1128
[62] Mostafa, S.A., Ahmad, M.S., Mustapha, A., 2019. Adjustable autonomy: A systematic literature review. Artificial Intelligence Review 51, 1129
149–186. doi: 10.1007/s10462-017-9560-8 . 1130
[63] Mustard, J.F., Beaty, D.W., Bass, D.S., 2013. Mars 2020 science rover: Science goals and mission concept, in: AAS Division for Planetary 1131
Sciences Annual Meeting, pp. 211–217. 1132
[64] Parasuraman, R., Sheridan, T.B., Wickens, C.D., 2000. A model for types and levels of human interaction with automation. IEEE Transac- 1133
tions on Systems, Man, and Cybernetics-Part A: Systems and Humans 30, 286–297. 1134
[65] Peng, H., Long, F., Ding, C., 2005. Feature selection based on mutual information criteria of max-dependency, max-relevance, and min- 1135
redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 1226–1238. doi: 10.1109/TPAMI.2005.159 . 1136
[66] Petousakis, G., Chiou, M., Nikolaou, G., Stolkin, R., 2020. Human operator cognitive availability aware mixed-initiative control, in: IEEE 1137
International Conference on Human-Machine Systems (ICHMS), IEEE. pp. 1–4. 1138
[67] Platanios, E.A., Stretcu, O., Neubig, G., Poczos, B., Mitchell, T.M., 2019. Competence-based curriculum learning for neural machine 1139
translation. arXiv preprint arXiv:1903.09848 . 1140
[68] Rabiee, S., Basich, C., Wray, K.H., Zilberstein, S., Biswas, J., 2022. Competence-aware path planning via introspective perception. IEEE 1141
Robotics and Automation Letters doi: 10.1109/LRA.2022.3145517 . 1142
[69] Rabiee, S., Biswas, J., 2019. IVOA: Introspective vision for obstacle avoidance, in: IEEE /RSJ International Conference on Intelligent 1143
Robots and Systems (IROS), pp. 1230–1235. 1144
[70] Ramakrishnan, R., Kamar, E., Nushi, B., Dey, D., Shah, J., Horvitz, E., 2019. Overcoming blind spots in the real world: Leveraging 1145
complementary abilities for joint execution, in: AAAI conference on Artificial Intelligence (AAAI), AAAI Press. pp. 6137–6145. doi: 10. 1146
1609/aaai.v33i01.33016137 . 1147
[71] Rastrigin, L.A., Erenstein, R.H., 1981. Method of collective recognition. Energoizdat 595, 37. 1148
[72] Rigter, M., Lacerda, B., Hawes, N., 2020. A framework for learning from demonstration with minimal human e ffort. IEEE Robotics and 1149
Automation Letters 5, 2023–2030. doi: 10.1109/LRA.2020.2970619 . 1150
[73] Roijers, D.M., Vamplew, P., Whiteson, S., Dazeley, R., 2013. A survey of multi-objective sequential decision-making. Journal of Artificial 1151
Intelligence Research 48, 67–113. 1152
[74] Rosenstein, M.T., Barto, A.G., 2004. Supervised actor-critic reinforcement learning, in: Si, J., Barto, A.G., Powell, W.B., Wunsch, D. 1153
(Eds.), Handbook of Learning and Approximate Dynamic Programming. IEEE Press, pp. 359–380. 1154
[75] Rosenthal, S., Biswas, J., Veloso, M.M., 2010. An e ffective personal mobile robot agent through symbiotic human-robot interaction., in: 1155
International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS), pp. 915–922. 1156
Basich et al. /Artificial Intelligence 00 (2022) 1–34 32
[76] SAE On-Road Automated Vehicle Standards Committee, 2014. Taxonomy and definitions for terms related to on-road motor vehicle 1157
automated driving systems. SAE Standards Journal 3016, 1–16. 1158
[77] Sa ffiotti, A., Broxvall, M., Gritti, M., LeBlanc, K., Lundh, R., Rashid, J., Seo, B., Cho, Y .J., 2008. The PEIS-ecology project: Vision and 1159
results, in: IEEE /RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2329–2335. 1160
[78] Saisubramanian, S., Zilbertsein, S., 2019. Adaptive outcome selection for planning with reduced models, in: IEEE /RSJ International 1161
Conference on Intelligent Robots and Systems (IROS), pp. 1655–1660. 1162
[79] Sampson, D., Fytros, D., 2008. Competence models in technology-enhanced competence-based learning, in: Handbook on Information 1163
Technologies for Education and Training. Springer, pp. 155–177. 1164
[80] Scerri, P., Pynadath, D.V ., Tambe, M., 2001. Adjustable autonomy in real-world multi-agent environments, in: International Conference on 1165
Autonomous Agents AGENTS, pp. 300–307. 1166
[81] Scerri, P., Pynadath, D.V ., Tambe, M., 2002. Towards adjustable autonomy for the real world. Journal of Artificial Intelligence Research 17, 1167
171–228. doi: 10.1613/jair.1037 . 1168
[82] Senliol, B., Gulgezen, G., Yu, L., Cataltepe, Z., 2008. Fast correlation based filter with a di fferent search strategy, in: IEEE International 1169
Symposium on Circuits and Systems (ISCIS), pp. 1–4. 1170
[83] Sheridan, T.B., 1992. Telerobotics, Automation, and Human supervisory control. Cambridge, MA: MIT Press. 1171
[84] Smyth, B., McKenna, E., 2001. Competence models and the maintenance problem. Computational Intelligence 17, 235–249. doi: 10.1111/ 1172
0824-7935.00142 . 1173
[85] Sousa, A., Madureira, L., Coelho, J., Pinto, J., Pereira, J., Sousa, J.B., Dias, P., 2012. LAUV: The man-portable autonomous underwater 1174
vehicle. International Federation of Automatic Control 45, 268–274. doi: 10.3182/20120410-3-PT-4028.00045 . 1175
[86] Sternberg, R.J., Kolligian Jr., J., 1990. Competence considered. Yale University Press. 1176
[87] Sutton, R.S., Barto, A.G., 2018. Reinforcement learning: An introduction. MIT press. 1177
[88] Svegliato, J., Nashed, S.B., Zilberstein, S., 2021. Ethically compliant sequential decision making, in: AAAI Conference on Artificial 1178
Intelligence (AAAI), pp. 11657–11665. 1179
[89] Svegliato, J., Wray, K., Witwicki, S., Biswas, J., Zilberstein, S., 2019. Belief space metareasoning for exception recovery, in: IEEE /RSJ 1180
International Conference on Intelligent Robotics and Systems (IROS), pp. 1224–1229. 1181
[90] Unit, F.E., 1984. Towards a competence-based system: An FEU view. 1182
[91] Vecht, B., 2009. Adjustable autonomy: Controlling influences on decision making. Utrecht University. 1183
[92] Veloso, M., Biswas, J., Coltin, B., Rosenthal, S., Kollar, T., Mericli, C., Samadi, M., Brandao, S., Ventura, R., 2012. Cobots: Collaborative 1184
robots servicing multi-floor buildings, in: IEEE /RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5446–5447. 1185
[93] Veloso, M.M., Biswas, J., Coltin, B., Rosenthal, S., 2015. CoBots: Robust symbiotic autonomous mobile service robots, in: International 1186
Joint Conference on Artificial Intelligence (IJCAI), pp. 4423–4429. 1187
[94] Visinsky, M.L., Cavallaro, J.R., Walker, I.D., 1994. Robotic fault detection and fault tolerance: A survey. Reliability Engineering & System 1188
Safety 46, 139–158. 1189
[95] Woloszynski, T., Kurzynski, M., 2009. On a new measure of classifier competence applied to the design of multiclassifier systems, in: 1190
International Conference on Image Analysis and Processing (ICIAP), pp. 995–1004. 1191
[96] Woloszynski, T., Kurzynski, M., 2011. A probabilistic model of classifier competence for dynamic ensemble selection. Pattern Recognition 1192
44, 2656–2668. doi: 10.1016/j.patcog.2011.03.020 . 1193
[97] Woloszynski, T., Kurzynski, M., Podsiadlo, P., Stachowiak, G.W., 2012. A measure of competence based on random classification for 1194
dynamic ensemble selection. Information Fusion 13, 207–213. doi: 10.1016/j.inffus.2011.03.007 . 1195
[98] Woods, K., Kegelmeyer, W.P., Bowyer, K., 1997. Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on 1196
Pattern Analysis and Machine Intelligence 19, 405–410. doi: 10.1109/34.588027 . 1197
[99] Wray, K.H., Pineda, L., Zilberstein, S., 2016. Hierarchical approach to transfer of control in semi-autonomous systems, in: International 1198
Joint Conference on Artificial Intelligence (IJCAI), pp. 517–523. 1199
[100] Wray, K.H., Zilberstein, S., Mouaddib, A.I., 2015. Multi-objective MDPs with conditional lexicographic reward preferences, in: AAAI 1200
conference on Artificial Intelligence (AAAI), pp. 3418–3424. 1201
[101] Yang, G.Z., Cambias, J., Cleary, K., Daimler, E., Drake, J., Dupont, P.E., Hata, N., Kazanzides, P., Martel, S., Patel, R.V ., et al., 2017. 1202
Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Science Robotics 2, 8638. doi: 10.1126/ 1203
scirobotics.aam8638 . 1204
[102] Zhao, Z., Anand, R., Wang, M., 2019. Maximum relevance and minimum redundancy feature selection methods for a marketing machine 1205
learning platform, in: IEEE International Conference on Data Science and Advanced Analytics (DSAA), IEEE. pp. 442–452. 1206
[103] Zieba, S., Polet, P., Vanderhaegen, F., Debernard, S., 2010. Principles of adjustable autonomy: A framework for resilient human-machine 1207
cooperation. Cognition, Technology & Work 12, 193–203. doi: 10.1007/s10111-009-0134-7 . 1208
[104] Zilberstein, S., 2015. Building strong semi-autonomous systems, in: AAAI Conference on Artificial Intelligence, pp. 4088–4092. 1209
Basich et al. /Artificial Intelligence 00 (2022) 1–34 33
Appendix A. 1210
In this section we describe additional details of our experimentation. In all of our experiments, our models were 1211
solved using LRTDP [12], and the feedback profiles were implemented as random forests using the Julia package 1212
DecisionTree.jl in the Julia MLJ framework [11] with default parameters. In our implementation of Algorithm 1, our 1213
validation step simply required a Matthews correlation coe fficient that was (1) positive (i.e. better than random) on 1214
the validation data set and (2) better than the Matthews correlation coe fficient of the current feedback profile on the 1215
same validation data set (with the discriminator masked out) by at least 0.2. 1216
Gated Exploration 1217
In all experiments, we used the gated exploration strategy as defined in Definition 8. While a variety of di fferent 1218
distributions could be used for the exploration strategy, we use an extension of the standard Boltzmann softmax 1219
distribution [49] over q-values in the adjacency set of l∈L: 1220
P(l′)=adj(κ(s,a),l′)exp(−q(s,(a,l′);ˆλ))
P
l′′∈Ladj(κ(s,a),l′′) exp(−q(s,(a,l′′);ˆλ))(A.1)
where q(s,(a,l);ˆλ)=C(s,(a,l))+P
s′∈ST(s,(a,l),s′)V(s′;ˆλ) is the expected cumulative reward when taking action 1221
(a,l)∈Ain state s∈Sconditioned on the current feedback profile ˆλ. 1222
To improve exploration e fficiency, we introduce a potential-based mechanism in our experiments in which, for 1223
each s∈Sanda∈A, we maintain a potential for each level l∈L,γs,a,l, which is updated at each level-exploration 1224
step, defined as 1225
γt+1
s,a,l←0 l′is chosen
min
γt
s,a,l+P(l),1
otherwise(A.2)
whereγt
lis the potential at time tandP(l) is defined in Equation A.1. For readability purposes, define γt(s,a,l) := 1226
γt
s,a,l; given this potential function we can slightly alter Equation A.1 to be 1227
ˆP(l′)=adj(l,l′)exp(γt(s,a,l′))P
l′′∈Ladj(l,l′′) exp(γt(s,a,l′′))(A.3)
which defines a new distribution from which to sample new levels of autonomy to explore. 1228
In our experiments, a potential matrix was initialized for the CAS model and updated each time the autonomy 1229
profile was updated via gated exploration. Gated exploration was implemented by sampling from the above distribu- 1230
tion to update the autonomy profile for each ( s,a) input by including the sampled level if not in κ(s,a) already, and 1231
otherwise doing nothing. The “gated” element was simulated in all experiments by observing the likelihood of an 1232
override, and adding the highest level (the only level disallowed initially) if sampled if the likelihood is below 0.15 1233
for the A V navigation domain or below 0.05 for the A V obstacle passing domain. 1234
Simulated feedback 1235
All human feedback in our experiments is fully simulated; the feedback of each simulated agent is determined 1236
by set rules based on the state and action up to their consistency ϵ. In the other 1−ϵpart of the time we return a 1237
random feedback signal drawn uniformly from the possible feedback signals for the given level of autonomy. Below, 1238
we describe the rules behind the simulated feedback in our experiments. The first two cases refer to feedback rules 1239
present across all simulated humans for the base domain. The rest of the cases refer to feedback rules present for spe- 1240
cific simulated humans. Note that all feedback rules mentioned directly correspond to competences of no autonomy 1241
when the human would override or disapprove an action, and unsupervised autonomy otherwise; there is no situation 1242
in our domain where the optimal action to perform is in verified or supervised autonomy given a perfect model of the 1243
human’s feedback. 1244
1245
Autonomous Vehicle Navigation The human overrides or disapproves overtaking an obstruction in an edge state 1246
when there is only a single lane, preferring to it themselves. When making a right turn at an intersection, which 1247
Basich et al. /Artificial Intelligence 00 (2022) 1–34 34
is considered a generally safe maneuver, the human overrides the maneuver if there is on occlusion, a pedesrtrian, 1248
and at least one other vehicle, indicating the presence of numerous other actors in a potentially chaotic environment. 1249
When going straight, making a left turn, or making a U-turn at an intersection, which are considered more challenging 1250
maneuvers as all potential cross-tra ffic must be considered, the human will override if there is an occlusion limiting 1251
visibility and a pedestrian or more than one vehicle, or if there is a pedestrian and more than two vehicles even without 1252
an occlusion limiting visibility. In all other cases, the human approves or does not override the system’s behavior. 1253
1254
Autonomous Vehicle Obstacle Passing The human overrides the action Stop if the A V is fully in the oncoming 1255
lane or if they can see that there is no oncoming vehicle. The human overrides the action Edge if the A V has visibility 1256
of oncoming tra ffic as the A V should either commit to the overtake (if safe to do so) or stop and wait until the overtake 1257
is safe. Finally, the human overrides the action Goif there is no visibility of oncoming tra ffic, or if there isoncoming 1258
traffic and the A V does not have priority to go. 1259
1260
A V Navigation – Cautious The human overrides or disapproves the vehicle from acting at all if the weather is 1261
snowy and it is night time, preferring to drive the whole way in these conditions. The human overrides or disapproves 1262
the overtake of a vehicle if it is snowy, or if it is rainy, nighttime, and a two-lane road. At intersections, the human 1263
also prefers to take control if it is rainy and nighttime. 1264
1265
A V Navigation – Conscientious The human overrides or disapproves the vehicle’s maneuver if there is a trailing 1266
vehicle when overtaking an obstruction, or if there is a trailing vehicle when the A V is at an intersection and either 1267
takes the Wait action or otherwise if there is at least one additional vehicle at the intersection, to hurry the A V through 1268
the intersection. 1269
1270
A V Obstacle Passing – Cautious The human overrides the vehicle if it is either snowy or rainy and nighttime, as 1271
the human does not trust the A V to handle the potentially dangerous maneuver in these conditions where the human 1272
feels less sure of what the A V can detect. 1273
1274
A V Obstacle Passing – Conscientious The human overrides the vehicle if there is a trailing vehicle and the vehi- 1275
cle takes the action Stop , or is stuck waiting with a trailing vehicle and takes the action Edge , as they feel socially 1276
pressured to execute the overtake expediently by the presence of the trailing vehicle. 1277
1278
A V Obstacle Passing – Rushed The human overrides the vehicle if it is stuck waiting, takes the action Stop , or 1279
if the vehicle has priority but does not take the action Go. 1280