Unifying Principles and Metrics for Safe and Assistive AI.

Unifying Principles and Metrics for Safe and Assistive AI
Siddharth Srivastava
School of Computing, Informatics, and Decision Systems Engineering
Arizona State University, Tempe, AZ 85281 USA
siddharths@asu.edu
Abstract
The prevalence and success of AI applications have been tem-
pered by concerns about the controllability of AI systems and
AI’s impact on the future of work. These concerns reﬂect two
aspects of a central question: how would humans work with
AI systems? While research on AI safety focuses on design-
ing AI systems that allow humans to safely instruct and con-
trol AI systems, research on AI and the future of work focuses
on the impact of AI on humans who may be unable to do so.
This Blue Sky Ideas paper proposes a unifying set of declar-
ative principles that enable a more uniform evaluation of ar-
bitrary AI systems along multiple dimensions of the extent
to which they are suitable for use by speciﬁc classes of hu-
man operators. It leverages recent AI research and the unique
strengths of the ﬁeld to develop human-centric principles for
AI systems that address the concerns noted above.
Introduction
Recent years have witnessed immense progress in research
on safe and assistive AI systems as well as on the poten-
tial impact of AI on the future of work. These directions of
research address two sides of a common, fundamental con-
cern: how would humans work with AI systems? While re-
search on AI safety focuses on designing AI systems that
allow humans to safely instruct and control them (e.g., (Rus-
sell, Dewey, and Tegmark 2015; Zilberstein 2015; Hadﬁeld-
Menell et al. 2016; Russell 2017; Hadﬁeld-Menell et al.
2017)), research on AI and the future of work focuses on
the impact of AI on members of the workforce who may be
unable to do so (Arntz, Gregory, and Zierahn 2016; Manyika
et al. 2017; Nedelkoska and Quintini 2018).
This paper presents the view that in addition to the pro-
ductive streams of research outlined above, we need uni-
fying metrics and declarative objectives that would allow
a more uniform evaluation of AI systems on the extent to
which an AI system is suitable for working with speciﬁc
classes of human operators . It also presents a common prin-
ciple for human-centered AI systems that allows the devel-
opment of such metrics. Consequently, rather than proposing
a speciﬁc new design for AI systems, the focus of this paper
Copyright c
2021, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.is on elucidating the declarative principles and types of met-
rics that would lead to concerted progress on the problem.
The advantage of this declarative approach to framing the
problem is that it enables an assessment of progress indepen-
dent of the internal design being used in an AI system, and it
will help draw out the strengths and weaknesses of different
design approaches. Without such a speciﬁcation, design dif-
ferences can make solution paradigms difﬁcult to compare.
E.g., one might develop a complex system architecture that
builds user proﬁles and provides appropriate assistance. This
system would have very different input requirements and de-
sign and performance parameters than a formulation that ad-
dresses the same problem by computing assistance policies
using planning under partial observability while incorporat-
ing the value of information to learn more about a user’s
poorly articulated objectives and constraints. Better declar-
ative objectives and metrics for assistive AI systems would
also help ensure that, regardless of the methods being used,
progress amounts to advancement towards safe and assistive
AI systems. More pragmatically, such metrics will not only
help end-users assess the utility of a given AI system but
they will also help AI researchers and developers identify
more readily the dimensions along which further research
will be beneﬁcial for applications of their interest.
The next section presents a succession of intuitive princi-
ples for safe and assistive AI systems, and shows that eval-
uating the compatibility of a system with such principles (in
particular P2) helps clarify the required types of metrics.
The paper concludes by drawing the attention of our com-
munity towards research on the operationalization of such
metrics along with promising research directions on devel-
oping systems that do well on them.
Unifying Principles for Safe and Assistive
AI Systems
We focus on taskable AI systems that carry out user-assigned
high-level tasks using arbitrary mechanisms for reason-
ing and planning over multiple time steps. E.g., household
robots that can be given objectives such as setting the ta-
ble or doing laundry, co-manufacturing robots that can assist
workers in creating complex assemblies with heavy compo-
nents, digital assistants that can plan a vacation given the
user’s preferences, etc. Such systems serve as sound inte-
grative platforms and model end-to-end applications where
the AI system is responsible for assistance in the execution
of long-horizon tasks.
AI systems are frequently evaluated in terms of perfor-
mance measures such as the computational complexity of
computing the required behavior, training data requirements,
and the quality of the computed behavior in terms of execu-
tion time, resources used, risk of unsafe outcomes etc. We
can consider systems that optimize such performance met-
rics as Level 0 of assistive AI systems.
Level I of assistive AI systems Recent AI research has
also focused on assistive properties of AI systems. We begin
with a rather common-sensical principle deﬁning Level I of
such safe and assistive AI systems:
P1:An AI system must make it easy for its operators touse
itsafely .
The italicized terms denote dimensions along which com-
patibility of AI systems with principle P1can be evaluated;
while a lot of current AI research utilizes one or more of
these dimensions for evaluation, a closer analysis reveals
some new insights.
In the context of this paper we consider using an AI sys-
temto be synonymous with instructing it to change its be-
havior as desired. Different interfaces may be used for this,
including programming, text, speech, gestures etc. We con-
sider the operators of an AI system as those persons who use
it in the sense described above. For instance, if a self-driving
car gives all its passengers the right to give it instructions,
then all of them are its operators; if it gives instruction-rights
to only a qualiﬁed set of users, perhaps adults who pass an
assisted-driving exam, then the class of operators is deﬁned
by that qualiﬁcation exam. Safety refers to the overall safety
of the AI system’s interactions with its environment, which
may be physical or online. These dimensions of compatibil-
ity with P1serve as natural dimensions for evaluating Level
I assistive AI systems:
E1. How inclusive is the set of operators ? Systems whose
operators require PhD-level expertise in AI may be less
desirable for broader deployments.
E2. How easy is it for the system’s operators to change its
behavior?
E3. Which set of tasks can the AI system be used for?
E4. What form of safety guarantees does the system pro-
vide? Systems that are unable to provide an upper bound
on expected risks are clearly less desirable than those
that stipulate conditions under which upper bounds can
be provided.
P1serves to highlight the interplay between these dimen-
sions of compliance and evaluation. Safety guarantees are
often inversely related with the size of the operator set. A
system may provide a high level of safety, but only under
the requirement that its operators take extensive training pro-
grams. At one end of the spectrum, automated robot vacuum
cleaners require almost no prior skills and perform limited,well-deﬁned tasks. Safety issues are still present—a robot
vacuum cleaner may pull on electrical cables that have been
left on the ﬂoor; auto-completion software may send emails
to unintended recipients. However, the lower expected dam-
age from using such applications has made them broadly
accepted in society. AI-powered industrial robots are at the
other end of the spectrum: these devices require specialized
training as well as operating environments in order to ensure
safety (see for instance, ISO/TS 15066:2016 on collabora-
tive robots). Typically, these systems operate in regions that
humans cannot access when the robot is active unless the hu-
man is hand-guiding the robot within a safe operating enve-
lope that limits the speed and range of operation. Their func-
tionality is closely controlled and monitored. Only skilled
engineers change their functionality while day-to-day oper-
ators monitor execution of predictable behavior and control
run-stops from designated safe areas. Similarly, the safety
of current airborne drone operations is ensured by requiring
specially trained drone operators (Marshall 2020).
Thus, principle P1holds for AI systems today with vary-
ing degrees of compliance along E1-E4. The examples
above illustrate how practical implementations often rely
upon implicitly deﬁned operator classes to provide accept-
able levels of safety. Such approaches rely upon limiting
the users and the scope of a system to achieve an accept-
able level of compatibility with P1: it is easy for such sys-
tems’ users to operate it safely because the set of users is re-
quired to be sufﬁciently skilled, and its functionality is suf-
ﬁciently limited for that group to be able to safely use the
device. However, a broader emphasis on the need to specify
safety guarantees with respect to different classes of opera-
tors would help mitigate some of the risks associated with
broadly deployed AI systems.
In contrast to classical automated systems, AI systems
feature a more nuanced interplay between the class of tasks
that a system can be used for ( E3) and the other dimensions
above. Traditionally deployed systems (even automated sys-
tems) have a very well-deﬁned boundary of use cases. This
allows for an easier classiﬁcation of safety. Taskable AI sys-
tems on the other hand, are expected to change their be-
havior and functionality as they learn and adapt to new
environments or new user-given objectives. For such sys-
tems, we need better methods for deriving the range of in-
structions that different operator-groups are allowed to pro-
vide. Scenarios like the self-driving car that needs to decide
whether to obey a child’s instruction allude to this require-
ment. Methods for assessing user and AI competency can
also allow the AI system to expand its range of functionality
by drawing upon the expertise of its operator (Basich et al.
2020) while ensuring an acceptable level of safety.
A major limitation of P1and its associated metrics is that
it does not evaluate the amount of training required for an
individual to qualify as an operator for the system. This cre-
ates a blind-spot in evaluation of the ease-of-use or safety of
an AI system: since user-training occurs outside the require-
ments outlined by P1, an unsafe AI system (or one that is
deployed in an unsafe manner) would simply claim that its
so-called operator was insufﬁciently trained!
Furthermore, if P1were a sufﬁcient categorization of safe
and assistive AI systems, we would have no need for ex-
plainable AI as compliance with P1does not require the sys-
tem to be easy to understand.
An implicit emphasis on assessing AI systems only along
some aspects of P1may also explain the increasing preva-
lence of concerns about the workers who may be left behind
in the future workplace. From this perspective it is unsur-
prising that these concerns have gained renewed interest at a
time when AI applications have reached a level of maturity
where they are being used by non-AI-experts in situations
that have some inherent safety risks. However, the “assis-
tive” nature of such AI systems is undermined by the need
for highly skilled individuals who could safely debug, un-
derstand and modify the behavior of such systems.
Level II of assistive AI systems In order to address the
limitations of P1, we consider the following as a guiding
principle for safe and assistive AI:
P2:An AI system must make it easy for its operators to
learn how to useitsafely .
P2changes the notion of operators from those who are
qualiﬁed to use a given AI system to those who are quali-
ﬁed to start learning how to use it. In addition to the metrics
associated with P1,P2introduces a new dimension:
E5. How easy is it to learn how to use the AI system? What
are the expected prerequisites and costs of training for its
operators? Can training be provided on-the-job?
This dimension could also be viewed as evaluating the re-
sources required to train operators for P1systems. Most AI
systems in use today would perform poorly on this new di-
mension, and consequently, on compatibility with P2as a
whole. Explainable AI (e.g., (Ribeiro, Singh, and Guestrin
2016; Hayes and Shah 2017; Chakraborti et al. 2017; Hoff-
man et al. 2018; Gunning and Aha 2019; Weld and Bansal
2019; Anderson et al. 2019; Eiﬂer et al. 2020)) plays a key
role along this dimension because systems that are easy to
understand or that can explain themselves naturally make it
easier for people to learn how to use them.
P2leverages the unique strengths of AI as a ﬁeld of re-
search. AI research already addresses the problem of esti-
mating users’ skills; research on intelligent tutoring systems
and AI for education addresses the problem of identifying
skill gaps. This can be used to determine the minimal dif-
ferential training to be provided to an operator. P2places
the onus of training on the deployed AI system and opens
up a new direction of interdisciplinary research connecting
existing research directions in AI with research in human-
systems engineering and in industrial engineering for the
development of productive training modalities and the op-
erationalization of metrics for E5. It also allows AI systems
to formally characterize different scopes of functionality for
different classes of operators, e.g., operators that use man-
ufacturing robots for pre-determined tasks, those that give
the robots new instructions, or those that are ready to learn
about giving the robot new instructions.
P2is not required for every AI system— P1would be suf-
ﬁcient for systems that place minimal requirements on op-
erator qualiﬁcations (e.g., robot vacuum cleaners) and fornon-adaptive AI systems that require a small set of oper-
ators. On the other hand, P2serves as a better declarative
foundation for evaluating taskable AI systems that are meant
to assist large numbers of non-AI-experts on a wide range of
tasks. Increasing concerns about job roles that would feature
a high-degree of interaction with AI systems (and the work-
ers that are likely to be left behind) allude to the pressing
need for including E5, a dimension for evaluation under P2
(and not P1) as a part of an AI system’s evaluation.
AI systems that are not beneﬁcial (either in terms of AI
safety or in terms of the future of work) fare poorly on P2.
E.g., systems that can thwart their users’ objectives by wire-
heading and those that may derive incorrect objective func-
tions from user instructions make it difﬁcult for an operator
to learn how to provide instructions that are speciﬁc enough
to be safe, and fare poorly along E5. Similarly, systems that
require extensive training investment to be used effectively
and safely fail along E5. In this way P2serves as a unifying
principle encompassing research on AI safety as well as on
AI for a beneﬁcial future of work.
Promising Directions of Research
P2serves as a declarative principle for guiding research on
assistive AI systems as well as for developing metrics for
evaluating AI systems and their deployments. Converting
this principle into tangible metrics calls for interdisciplinary
research including AI and other ﬁelds associated with hu-
man factors. The increasing prevalence of research thrusts
on safe and assistive AI systems (Fern et al. 2014; Russell,
Dewey, and Tegmark 2015; Amodei et al. 2016; Gunning
and Aha 2019) makes this a particularly opportune phase
for formalizing the metrics and the interfaces required for
evaluating AI systems for compatibility with P2along di-
mensions E1-E5.
Recent research on AI safety and explainable AI devel-
ops methods improving the ease of use and safety of AI
systems along P2(see, for instance, the ICML 2020 Work-
shop on Explainable AI). Placing AI systems that compute
user-skill aligned explanations (Sreedharan, Srivastava, and
Kambhampati 2018; Sreedharan et al. 2019) in a loop with
AI systems for identifying user-skills and skill-gaps can help
develop AI systems that gradually present users with new
functionality and explain it, thereby training their users on-
the-ﬂy and as needed. Such systems would be better tuned
towards P2, and towards addressing the underlying prob-
lems of AI safety and the future of work.
Critically, ensuring progress towards safe and assistive AI
systems requires that AI systems with arbitrary internal de-
signs support assessment along the metrics developed for
E1-E5 . This raises a new set of research questions: Can
we develop non-intrusive AI-interface requirements for sup-
porting such evaluations in the face of changing operating
environments and objectives? The need for such interfaces
is even more pressing for systems that learn and those that
undergo system updates after deployment. What is the min-
imal external interface that an AI system must support so as
to allow its independent evaluation? How would changing
the nature of such interfaces change the complexity of con-
ducting such an evaluation? One would expect that AI sys-
tems that offer more transparency would be easier to evalu-
ate. Could we use the inherent reasoning capabilities of AI
systems to develop interface requirements that would allow
more adept systems to make such evaluations easier? E.g.,
rather than testing a manufacturing robot to discover its re-
sponse to every possible situation, could we ask higher-level
queries such as “under which situations would you be able to
create the proposed assembly?” Clearly, the ease of assess-
ment of an AI system would depend on the class of queries
that it can answer.
Recent work suggests that a minimal query-response in-
terface for AI systems that connects the system with a sim-
ulator and observes its responses to high-level instructions
can be quite powerful. Such an interface has a few distinct
advantages. Current AI systems are already tested with sim-
ulators and they are inherently required to be able to take
user instructions, so these interface requirements can be
considered to be natural. They also allow the autonomous
synthesis of query-policies : running the query-policy on a
black-box taskable AI system can help construct an inter-
pretable model of the limits and capabilities of that sys-
tem (Verma, Marpally, and Srivastava 2021). Such models
can be used to support the evaluations discussed above.
Extensions of such interface requirements to arbitrary AI
systems would help ensure that our AI systems are amenable
to independent evaluation. Such a paradigm would allow
users to assess their AI systems while freeing AI researchers
and developers to utilize arbitrary internal implementations.
Systems with interfaces that support more efﬁcient and accu-
rate independent assessment would be rewarded with greater
public adoption of their products.
Progress on these threads would help prevent undesirable
situations such as insufﬁcient support for independent evalu-
ation of powerful AI systems, and the negative consequences
of deployment of an insufﬁciently evaluated system.
Acknowledgements
I would like to thank members of the ASU Future of Work
(ASUFoW) project, members of the Center for Human-
Compatible AI (CHAI) and members of the Autonomous
Agents and Intelligent Robots (AAIR) Lab research group
at ASU for the many fruitful discussions leading to the pre-
sented ideas, as well as the anonymous reviewers for their
helpful comments on the paper. This work was supported
in part by the NSF under grants OIA 1936997, and IIS
1942856.
References
Amodei, D.; Olah, C.; Steinhardt, J.; Christiano, P.; Schul-
man, J.; and Man ´e, D. 2016. Concrete problems in AI safety.
arXiv preprint arXiv:1606.06565 .
Anderson, A.; Dodge, J.; Sadarangani, A.; Juozapaitis, Z.;
Newman, E.; Irvine, J.; Chattopadhyay, S.; Fern, A.; and
Burnett, M. 2019. Explaining reinforcement learning to
mere mortals: an empirical study. In Proc. IJCAI .
Arntz, M.; Gregory, T.; and Zierahn, U. 2016. The Risk of
Automation for Jobs in OECD Countries. Organisation forEconomic Cooperation and Development (OECD) Social,
Employment and Migration Working Papers (189).
Basich, C.; Svegliato, J.; Wray, K. H.; Witwicki, S.; Biswas,
J.; and Zilberstein, S. 2020. Learning to Optimize Auton-
omy in Competence-Aware Systems. In Proc. AAMAS .
Chakraborti, T.; Sreedharan, S.; Zhang, Y .; and Kambham-
pati, S. 2017. Plan Explanations as Model Reconciliation:
Moving Beyond Explanation as Soliloquy. In Proc. IJCAI .
Eiﬂer, R.; Cashmore, M.; Hoffmann, J.; Magazzeni, D.; and
Steinmetz, M. 2020. A New Approach to Plan-Space Expla-
nation: Analyzing Plan-Property Dependencies in Oversub-
scription Planning. In Proc. AAAI .
Fern, A.; Natarajan, S.; Judah, K.; and Tadepalli, P. 2014. A
decision-theoretic model of assistance. Journal of Artiﬁcial
Intelligence Research 50: 71–104.
Gunning, D.; and Aha, D. W. 2019. DARPA’s explainable
artiﬁcial intelligence program. AI Magazine 40(2): 44–58.
Hadﬁeld-Menell, D.; Dragan, A.; Abbeel, P.; and Russell, S.
2017. The Off-Switch Game. In Proc. IJCAI .
Hadﬁeld-Menell, D.; Russell, S. J.; Abbeel, P.; and Dragan,
A. 2016. Cooperative inverse reinforcement learning. In
Proc. NeurIPS .
Hayes, B.; and Shah, J. A. 2017. Improving robot controller
transparency through autonomous policy explanation. In
Proc. HRI .
Hoffman, R. R.; Mueller, S. T.; Klein, G.; and Litman, J.
2018. Metrics for explainable AI: Challenges and prospects.
arXiv preprint arXiv:1812.04608 .
Manyika, J.; Chui, M.; Miremadi, M.; Bughin, J.; George,
K.; Willmott, P.; and Dewhurst, M. 2017. A future that
works: Automation, employment, and productivity. Tech-
nical report, McKinsey Global Institute.
Marshall, A. 2020. No, Amazon Won’t Deliver You a Bur-
rito by Drone Anytime Soon. Wired (September 5, 2020).
Nedelkoska, L.; and Quintini, G. 2018. Automation, skills
use and training. Organisation for Economic Cooperation
and Development (OECD) Social, Employment and Migra-
tion Working Papers (202).
Ribeiro, M. T.; Singh, S.; and Guestrin, C. 2016. Why
should I trust you?: Explaining the predictions of any clas-
siﬁer. In Proceedings of the 22nd ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data Min-
ing. ACM.
Russell, S. 2017. Provably beneﬁcial artiﬁcial intelligence.
Exponential Life, The Next Step .
Russell, S.; Dewey, D.; and Tegmark, M. 2015. Research
priorities for robust and beneﬁcial artiﬁcial intelligence. AI
Magazine 36(4): 105–114.
Sreedharan, S.; Srivastava, S.; and Kambhampati, S. 2018.
Hierarchical Expertise-Level Modeling for User Speciﬁc
Contrastive Explanations. In Proc. IJCAI .
Sreedharan, S.; Srivastava, S.; Smith, D. E.; and Kambham-
pati, S. 2019. Why Couldn’t You do that, HAL? Explaining
Unsolvability of Classical Planning Problems. In Proc. IJ-
CAI.
Verma, P.; Marpally, S. R.; and Srivastava, S. 2021. Asking
the Right Questions: Learning Interpretable Action Models
Through Query Answering. In Proc. AAAI .
Weld, D. S.; and Bansal, G. 2019. The challenge of crafting
intelligible intelligence. Communications of the ACM 62(6):
70–79.
Zilberstein, S. 2015. Building strong semi-autonomous sys-
tems. In Proc. AAAI .


X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback.

Published as a conference paper at ICLR 2021
X2T: T RAINING AN X-TO-TEXT TYPING INTERFACE
WITH ONLINE LEARNING FROM USERFEEDBACK
Jensen Gao, Siddharth Reddy, Glen Berseth
University of California, Berkeley
sgr@berkeley.eduNicholas Hardy, Nikhilesh Natraj
University of California, San Francisco
Karunesh Ganguly
University of California, San Francisco
karunesh.ganguly@ucsf.eduAnca D. Dragan, Sergey Levine
University of California, Berkeley
ABSTRACT
We aim to help users communicate their intent to machines using ﬂexible, adaptive
interfaces that translate arbitrary user input into desired actions. In this work, we
focus on assistive typing applications in which a user cannot operate a keyboard,
but can instead supply other inputs, such as webcam images that capture eye gaze
or neural activity measured by a brain implant. Standard methods train a model
on a ﬁxed dataset of user inputs, then deploy a static interface that does not learn
from its mistakes; in part, because extracting an error signal from user behavior
can be challenging. We investigate a simple idea that would enable such interfaces
to improve over time, with minimal additional effort from the user: online learn-
ing from user feedback on the accuracy of the interface’s actions. In the typing
domain, we leverage backspaces as feedback that the interface did not perform the
desired action. We propose an algorithm called x-to-text (X2T) that trains a pre-
dictive model of this feedback signal, and uses this model to ﬁne-tune any existing,
default interface for translating user input into actions that select words or charac-
ters. We evaluate X2T through a small-scale online user study with 12 participants
who type sentences by gazing at their desired words, a large-scale observational
study on handwriting samples from 60 users, and a pilot study with one partic-
ipant using an electrocorticography-based brain-computer interface. The results
show that X2T learns to outperform a non-adaptive default interface, stimulates
user co-adaptation to the interface, personalizes the interface to individual users,
and can leverage ofﬂine data collected from the default interface to improve its
initial performance and accelerate online learning.
1 I NTRODUCTION
Recent advances in user interfaces have enabled people with sensorimotor impairments to more
effectively communicate their intent to machines. For example, Ward et al. (2000) enable users to
type characters using an eye gaze tracker instead of a keyboard, and Willett et al. (2020) enable
a paralyzed human patient to type using a brain implant that records neural activity. The main
challenge in building such interfaces is translating high-dimensional, continuous user input into
desired actions. Standard methods typically calibrate the interface on predeﬁned training tasks for
which expert demonstrations are available, then deploy the trained interface. Unfortunately, this
does not enable the interface to improve with use or adapt to distributional shift in the user inputs.
In this paper, we focus on the problem of assistive typing: helping a user select words or characters
without access to a keyboard, using eye gaze inputs (Ward et al., 2000); handwriting inputs (see
Figure 7 in the appendix), which can be easier to provide than direct keystrokes (Willett et al., 2020);
or inputs from an electrocorticography-based brain implant (Leuthardt et al., 2004; Silversmith et al.,
2020). To enable any existing, default interface to continually adapt to the user, we train a model
using online learning from user feedback. The key insight is that the user provides feedback on
the interface’s actions via backspaces , which indicate that the interface did not perform the desired
1
Published as a conference paper at ICLR 2021
User sees list of words and system’s current prediction of desired wordECoGbrain-computer interface records user’s neural activity
Reward modelPredicted wordBackspace or continue
Default interfaceOur interface
Reward
User input
Train reward modelInput-action-reward log dataBinary feedback (e.g., from button press)
Anonymized userAnonymized user
Figure 1: We formulate assistive typing as a human-in-the-loop decision-making problem, in which the inter-
face observes user inputs (e.g., neural activity measured by a brain implant) and performs actions (e.g., word
selections) on behalf of the user. We treat a backspace as feedback from the user that the interface performed
the wrong action. By training a model online to predict backspaces, we continually improve the interface.
action in response to a given input. By learning from this naturally-occurring feedback signal instead
of an explicit label, we do not require any additional effort from the user to improve the interface.
Furthermore, because our method is applied on top of the user’s default interface, our approach is
complementary to other work that develops state-of-the-art, domain-speciﬁc methods for problems
like gaze tracking and handwriting recognition. Figure 1 describes our algorithm: we initialize our
model using ofﬂine data generated by the default interface, deploy our interface as an augmentation
to the default interface, collect online feedback, and update our model.
We formulate assistive typing as an online decision-making problem, in which the interface receives
observations of user inputs, performs actions that select words or characters, and receives a reward
signal that is automatically constructed from the user’s backspaces. To improve the default inter-
face’s actions, we ﬁt a neural network reward model that predicts the reward signal given the user’s
input and the interface’s action. Upon observing a user input, our interface uses the trained reward
model to update the prior policy given by the default interface to a posterior policy conditioned on
optimality, then samples an action from this posterior (see Figure 1). We call this method x-to-text
(X2T), where xrefers to the arbitrary type of user input; e.g., eye gaze or brain activity.
Our primary contribution is the X2T algorithm for continual learning of a communication interface
from user feedback. We primarily evaluate X2T through an online user study with 12 participants
who use a webcam-based gaze tracking system to select words from a display. To run ablation
experiments that would be impractical in the online study, we also conduct an observational study
with 60 users who use a tablet and stylus to draw pictures of individual characters. The results show
that X2T quickly learns to map input images of eye gaze or character drawings to discrete word
or character selections. By learning from online feedback, X2T improves upon a default interface
that is only trained once using supervised learning and, as a result, suffers from distribution shift
(e.g., caused by changes in the user’s head position and lighting over time in the gaze experiment).
X2T automatically overcomes calibration problems with the gaze tracker by adapting to the mis-
calibrations over time, without the need for explicit re-calibration. Furthermore, X2T leverages
ofﬂine data generated by the default interface to accelerate online learning, stimulates co-adaptation
from the user in the online study, and personalizes the interface to the handwriting style of each user
in the observational study.
2
Published as a conference paper at ICLR 2021
2 L EARNING TO INFER INTENT FROM USERINPUT
In our problem setting, the user cannot directly perform actions; e.g., due to a sensorimotor impair-
ment. Instead, the user relies on an assistive typing interface, where the user’s intended action is
inferred from available inputs such as webcam images of eye gaze or handwritten character draw-
ings. As such, we formulate assistive typing as a contextual bandit problem (Langford & Zhang,
2008; Yue & Joachims, 2009; Li et al., 2010; Lan & Baraniuk, 2016; Gordon et al., 2019). At each
timestep, the user provides the interface with a context x2X, whereXis the set of possible user
inputs (e.g., webcam images). The interface then performs an action u2U, whereUis the set of
possible actions (e.g., word selections). We assume the true reward function is unknown, since the
user cannot directly specify their desired task (e.g., writing an email or ﬁlling out a form). Instead
of eliciting a reward function or explicit reward signal from the user, we automatically construct a
reward signal from the user’s backspaces . The key idea is to treat backspaces as feedback on the
accuracy of the interface’s actions.
Our approach to this problem is outlined in Figure 1. We aim to minimize expected regret, which,
in our setting, is characterized by the total number of backspaces throughout the lifetime of the
interface. While a number of contextual bandit algorithms with lower regret bounds have been
proposed in prior work (Lattimore & Szepesv ´ari, 2020), we use a simple strategy that works well in
our experiments: train a neural network reward model to predict the reward given the user’s input and
the interface’s action, and select actions with probability proportional to their predicted optimality.
Our approach is similar to prior work on deep contextual multi-armed bandits (Collier & Llorens,
2018) and NeuralUCB (Zhou et al., 2019), except that instead of using Thompson sampling or UCB
to balance exploration and exploitation, we use a simple, stochastic policy.
2.1 M ODELING USERBEHAVIOR AND FEEDBACK
Unlike in the standard multi-armed bandit framework, we do not get to observe an extrinsic reward
signal that captures the underlying task that the user aims to perform. To address this issue, we infer
rewards from naturally-occurring user behavior. In particular, in the assistive typing setting, we take
advantage of the fact that we can observe when the user backspaces ; i.e., when they delete the most
recent word or character typed by the interface. To infer rewards from backspaces, we make two
assumptions about user behavior: (1) the user can perform a backspace action independently of our
interface (e.g., by pressing a button); (2) the user tends to backspace incorrect actions; and (3) the
user does not tend to backspace correct actions. Hence, we assign a positive reward to actions that
were not backspaced, and assign zero reward to backspaced actions. Formally, let r2f0;1gdenote
this reward signal, where r= 0indicates an incorrect action and r= 1indicates a correct action.
2.2 T RAINING THE REWARD MODEL TO PREDICT FEEDBACK
In order to perform actions that minimize expected regret – i.e., the total number of backspaces over
time – we need to learn a model that predicts whether or not the user will backspace a given action
in a given context. To do so, we learn a reward model p(rjx;u), wherepis a neural network and
are the weights. Since the reward r2f0;1gcan only take on one of two values, pis a binary
classiﬁer. We train this binary classiﬁer on a dataset Dof input-action-reward triples (x;u;r). In
particular, we ﬁt the model pby optimizing the maximum-likelihood objective; i.e., the binary
cross-entropy loss (see Equation 2 in the appendix).
Since X2T learns from human-in-the-loop feedback, the amount of training data is limited by how
frequently the user operates the interface. To reduce the amount of online interaction data needed
to train the reward model, we use ofﬂine pretraining. We assume that the user already has access
to some default interface for typing. We also assume access to an ofﬂine dataset of input-action
pairs generated by the user and this default interface. We assign zero rewards to the backspaced
actions and positive rewards to the non-backspaced actions in this ofﬂine dataset, and initially train
our reward model to predict these rewards given the user’s inputs and the default interface’s actions.
Thus, when X2T is initially deployed, the reward model has already been trained on the ofﬂine data,
and requires less online interaction data to reach peak accuracy.
3
Published as a conference paper at ICLR 2021
Algorithm 1 X-to-Text (X2T)
Require ;init .default interface, pretrained reward model parameters
while truedo
xpuser(x) .user gives input
u(ujx)/p(r= 1jx;u)(ujx) .interface performs action
r 0if user backspaces else 1 .infer reward from user feedback
D D[f (x;u;r)g .store online input-action-reward data
 +rP
(x;u;r)Dlog (p(rjx;u)) .update reward model w/SGD
2.3 U SING THE REWARD MODEL TO SELECT ACTIONS
Even with ofﬂine pretraining, the initial reward model may not be accurate enough for practical
use. To further improve the initial performance of our interface at the onset of online training,
we combine our reward model p(rjx;u)with the default interface (ujx). We assume that
is a stochastic policy and that we can evaluate it on speciﬁc inputs, but do not require access to
its implementation. We set our policy (ujx) =p(ujx;r= 1) to be the probability of an action
conditional on optimality, following the control-as-inference framework (Levine, 2018). Applying
Bayes’ theorem, we get p(ujx;r= 1)/p(r= 1jx;u)p(ujx). The ﬁrst term is given by our reward
modelp, and the second term is given by the default interface. Combining these, we get the policy
(ujx)/p(r= 1jx;u)(ujx): (1)
This decomposition of the policy improves the initial performance of our interface at the onset of
online training, and guides exploration for training the reward model. It also provides a framework
for incorporating a language model into our interface, as described in Section 4.3.
Our x-to-text (X2T) method is summarized in Algorithm 1. In the beginning, we assume the user has
already been operating the default interface for some time. In doing so, they generate an ‘ofﬂine’
dataset that we use to train the initial reward model parameters init. When the user starts using X2T,
our interface already improves upon the default interface by combining the default interface with
the initial reward model via Equation 1. As the user continues operating our interface, the resulting
online data is used to maintain or improve the accuracy of the reward model. At each timestep, the
user provides the interface with input xpuser(x). Although standard contextual bandit methods
assume that the inputs xare i.i.d., we ﬁnd that X2T performs well even when the inputs are correlated
due to user adaptation (see Section 4.2) or the constraints of natural language (see Section 4.4). The
interface then uses the policy in Equation 1 to select an action u. We then update the reward model
p, by taking one step of stochastic gradient descent to optimize the maximum-likelihood objective
in Equation 2. Appendix A.1 discusses the implementation details.
3 R ELATED WORK
Prior methods for training interfaces with supervised learning typically collect a dataset of input-
action pairs, then ﬁt a model that predicts actions given inputs. These methods tend to either assume
access to ground-truth action labels from the user (Anumanchipalli et al., 2019; Wang et al., 2016;
Karamcheti et al., 2020), or assume the user always intends to take the optimal action (Gilja et al.,
2012; Dangi et al., 2013; 2014; Merel et al., 2015). Unfortunately, the user may not always be
able to provide ground-truth action labels. Furthermore, in order for the system to compute optimal
actions, the user must be instructed to perform speciﬁc calibration tasks for which the optimal policy
is already known. These calibration tasks may not be representative of the tasks that the user intends
to perform. This can lead to a distribution mismatch between the inputs that the model is trained
on during the calibration phase, and the inputs that the model is evaluated on at test time when the
user performs their desired tasks. Standard methods address this problem by periodically repeating
the calibration process (Willett et al., 2020; Prorokovi ´c et al., 2020), which can be time-consuming,
disruptive, and requires assumptions about when and how frequently to re-calibrate. X2T overcomes
these issues by continually learning from user feedback on tasks that naturally arise as the user types,
rather than imposing separate training and test phases.
Extensive prior work on text entry systems (MacKenzie & Tanaka-Ishii, 2010) enables users to type
using eye gaze (Ward et al., 2000), Braille (Oliveira et al., 2011), gestures (Jones et al., 2010), and
4
Published as a conference paper at ICLR 2021
palm keyboards for wearable displays (Wang et al., 2015). X2T differs in that it enables the user to
type using arbitrary inputs like eye gaze or handwriting, rather than a ﬁxed type of input that restricts
the ﬂexibility of the system and must be anticipated in advance by the system designer.
X2T trains a typing interface through reinforcement learning (RL) with human-in-the-loop feedback
instead of an explicit reward function. COACH (MacGlashan et al., 2017; Arumugam et al., 2019),
TAMER (Knox & Stone, 2009; Warnell et al., 2017), and preference learning (Sadigh et al., 2017;
Christiano et al., 2017) also train an RL agent without access to extrinsic rewards, but require ex-
plicit user feedback on the agent’s behavior. X2T differs in that it learns from naturally-occurring
feedback, which requires no additional effort from the user to train the agent. Other prior work trains
RL agents from implicit signals, such as electroencephalography (Xu et al., 2020), peripheral pulse
measurements (McDuff & Kapoor, 2019), facial expressions (Jaques et al., 2017; Cui et al., 2020),
and clicks in web search (Radlinski & Joachims, 2006). X2T differs in that it trains an interface that
always conditions on the user’s input when selecting an action, rather than an autonomous agent
that ignores user input after the training phase. Furthermore, X2T focuses on the assistive typing
domain, where, to the best of our knowledge, backspaces have not yet been used to train an interface
through RL. Related work on assistive robotic teleoperation interfaces proposes human-in-the-loop
RL methods that minimally modify user actions (Pilarski et al., 2011; Broad et al., 2017; Reddy
et al., 2018; Schaff & Walter, 2020; Du et al., 2020; Jeon et al., 2020). X2T differs in that it learns to
operate on arbitrary types of user inputs, instead of assuming the user provides suboptimal actions.
4 E XPERIMENTAL EVALUATION
We seek to answer the following questions: Q1(Sec. 4.1): Does X2T improve with use and learn to
outperform a non-adaptive interface? Q2(Sec. 4.2): Does the user adapt to the interface while the
interface adapts to the user? Q3(Sec. 4.3): Does X2T personalize the interface to different input
styles? Q4(Sec. 4.4): Do ofﬂine pretraining and an informative prior policy accelerate online learn-
ing? Q5(Sec. 4.4): Does online learning improve the interface beyond the initialization provided
by ofﬂine pretraining? Q6(Sec. 4.5): Can X2T improve the accuracy of a brain-computer interface
for typing? To answer Q1-2 , we run a user study with 12 participants who use webcam images of
their eyes to type via gaze (see Figure 8). To answer Q3-5 , we conduct an observational study with
prerecorded images of handwritten characters drawn by 60 users with a tablet and stylus (see Figure
7). To answer Q6, we conduct a pilot study with one participant using an electrocorticography-based
brain-computer interface. In our experiments, we use default interfaces that are not necessarily the
state of the art in gaze tracking or handwriting recognition, but are instead chosen to test the hy-
potheses in Q1-6 . Appendix A.1 describes the experiment design in detail.
4.1 A DAPTING THE INTERFACE TO THE USER
In this experiment, we aim to test X2T’s ability to improve over time, relative to a non-adaptive,
default interface. To that end, we formulate a gaze-based word selection task in which we display
a list of words to the user (see Figure 8), ask them to look at their desired word, record an image
from their webcam that captures their eye gaze, and predict their desired word. To measure objective
performance, we ask the user to type speciﬁc goal sentences. To simplify the experiment, we restrict
the action spaceU=f1;2;3;:::;8gto the eight buttons on the screen, and always assign the next
word in the sentence to one of those eight buttons (see Figure 8).
We evaluate (1) a default interface that uses iTracker (Krafka et al., 2016) to estimate the user’s
2D gaze position on the screen and select the nearest button, and (2) X2T. We calibrate the default
interface once at the beginning of each experimental condition for each user, by asking the user to
look at each of the eight buttons one by one, recording 20 eye image samples for each button in 2
cycles, and training a 2D gaze position estimator using the iTracker method. After calibration, the
default interface stays ﬁxed, and does not adapt to the user within the session; in part, because we do
not know in advance which word the user intends to select, so we cannot automatically harvest the
necessary paired data of gaze images and targets to re-calibrate the interface. Instead of periodically
interrupting the user to gather new paired data and re-calibrate, X2T continually adapts to the user
throughout the session, following Algorithm 1. Appendix A.1 describes the implementation of the
default interface and X2T in further detail. We measure the performance of each method using the
ground-truth accuracy of action selections. To access ground-truth actions for calculating accuracy,
5
Published as a conference paper at ICLR 2021
Default interface is inaccurate, but X2T is accurateDefault interface is accurate(a)(b)(c)
Figure 2: An online user study with 12 participants in the gaze tracking domain that addresses Q1: does X2T
improve with use and learn to outperform a non-adaptive interface? (a)X2T predicts the user’s intended action
more accurately than the default interface, and the gap between the two methods grows over time. We smooth
the curves using a moving average with a window size of 20 interactions, and measure standard error across
the 12 users. (b)X2T improves the performance of 10 out of the 12 users, and the improvement from X2T is
smaller when the default interface already performs well. Each orange circle represents one of the 12 users.
The dashed gray line shows default-equivalent performance, and the dotted orange lines show the difference
between X2T and default performance. Per-user accuracy is averaged across 250 interactions. (c)As shown in
the screenshot in Figure 8, the user is shown a display of eight words arranged in a circle. Here, we plot the
default interface’s 2D gaze position estimates given user inputs intended to select the bottom-most button. The
widely-scattered 2D estimates show that action prediction is particularly hard for this button, perhaps because
the user’s eyes tend to be more obscured when they are looking down. By training a reward model on user
feedback, X2T helps the interface recover from incorrect gaze estimates.
we instruct the user to try to select a speciﬁed word from the list displayed on their screen. Note
that this instruction is not an essential component of X2T, and is used purely to evaluate objective
performance in this experiment.
The results in panel (a) of Figure 2 show that at the onset of online learning, both X2T and the
default interface predict the user’s intended action with the same accuracy, but quickly diverge. The
default interface’s performance degrades over time, potentially due to distribution shift in the user’s
inputs caused by changes in head position, background lighting, and other visual conditions. In
contrast, X2T maintains the interface’s strong initial performance throughout the experiment, by
continually updating the reward model to reﬂect changes in the user’s inputs. Panel (b) shows that
X2T signiﬁcantly improves the performance of 10 out of 12 participants, and that there are dimin-
ishing returns to X2T when the default interface already performs well. We ran a one-way repeated
measures ANOV A on the action prediction accuracy dependent measure from the default and X2T
conditions, with the presence of X2T as a factor, and found that f(1;11) = 17:23;p< 0:01. The
subjective evaluations in Table 2 in the appendix corroborate these results: users reported feeling
that X2T selected the words they wanted and improved over time more than the default interface.
Panel (c) qualitatively illustrates how X2T helps the interface recover from incorrect 2D gaze posi-
tion estimates: each green ‘x’ shows that even when the default interface estimates a gaze position
far from the intended button, which would normally cause an incorrect action prediction, the reward
model can adjust the overall action prediction back to the correct button via Equation 1. We also
ﬁnd that X2T performs well even when the user’s feedback is slightly noisy: the user backspaces
mistakes, and does not backspace correct actions, in 98:6%of their interactions.
4.2 U SERADAPTATION TO THE INTERFACE
In the previous experiment, we tested X2T’s ability to adapt the interface to the user. Prior
work on human-machine co-adaptation (Taylor et al., 2002; Carmena, 2013; Shenoy & Car-
mena, 2014; Nikolaidis et al., 2017) and the evolution of communication protocols between hu-
mans (Hawkins et al., 2020) suggests that an adaptive interface may not only learn to assist a
static user, but even stimulate the user to adapt their input style to the interface. In this ex-
periment, we investigate whether the user adapts to the gaze-based interface described in Sec-
tion 4.1. To do so, we perform a counterfactual experiment: instead of training and evalu-
ating X2T on inputs xgenerated by the user while they were typing with X2T, we train and
6
Published as a conference paper at ICLR 2021
evaluate X2T on inputs xgenerated by the user while they were typing with the default inter-
face. During the counterfactual experiment, instead of asking the user for new inputs, we sim-
ply replay the old inputs that were intended for the default interface, and automate backspaces.
Figure 3: A counterfactual experiment with the on-
line user study data that addresses Q2: does the user
adapt to the interface while the interface adapts to the
user? Training and evaluating X2T on user inputs orig-
inally intended for the default interface leads to worse
performance than doing so on user inputs intended for
X2T (orange vs. teal). Evaluating the default inter-
face on inputs intended for X2T leads to worse perfor-
mance than evaluating X2T on the same inputs (orange
vs. gray). These results suggest that the user adapts
their input style speciﬁcally to X2T, and that this user
co-adaptation improves performance. We smooth the
curves using a moving average with a window size of
20 interactions, and measure standard error across the
12 users.This enables us to test if the user adapted their
inputs to X2T, or if the user provided the same
distribution of inputs to both the default inter-
face and X2T.
The results in Figure 3 suggest that the user
does indeed adapt their input style to the in-
terface, and that this user adaptation improves
performance. Comparing X2T’s actual perfor-
mance (orange curve) to X2T’s counterfactual
performance on replayed user inputs that the
user originally intended for the default inter-
face (teal curve), we see that X2T is able to
perform much better with inputs intended for
X2T compared to inputs intended for the de-
fault interface. From this result alone, one
might infer that the user provided X2T with
inputs that were more generically predictable
than the inputs they provided to the default in-
terface. However, by comparing the default in-
terface’s performance on replayed user inputs
that the user originally intended for X2T (gray
curve) to X2T’s performance on the same in-
puts (orange curve), we see that X2T performs
better than the default interface on the same in-
puts. This suggests that the user’s inputs to
X2T are not merely easier to predict, but in
fact adapted speciﬁcally to X2T. In other words,
X2T stimulates user co-adaptation to the inter-
face.
4.3 P ERSONALIZING THE INTERFACE FOR DIFFERENT USERS
In this experiment, we demonstrate X2T’s ability to operate on a different type of user input: draw-
ings of characters (see Figure 7 in the appendix); which, for some users, can be easier to provide
than direct keystrokes (Willett et al., 2020). We also investigate to what extent X2T learns a person-
alized interface that is uniquely adapted to the individual user. To that end, we analyze handwriting
samples from 60 users, collected through a tablet and stylus, from the UJI Pen Characters Database
(Llorens et al., 2008). Each sample consists of a sequence of 2D positions that traces the trajectory
of the user’s pen tip as they draw a known character. We conduct an observational study with this
data by sampling goal sentences, replaying a randomly-selected handwriting sample of the user’s
desired next character from the goal sentence, and treating each drawing as the user input x; akin
to the replay experiment in Section 4.2. This observational study is equivalent to an online study,
except that it does not permit user co-adaptation to the interface since it replays logged user inputs,
and assumes that the feedback signal is not noisy when automating backspaces. To test X2T’s ro-
bustness to noise and distributional shift in the user’s input, we perturb the replayed pen tip positions
by adding Brownian noise (see Figure 7 in the appendix).
We evaluate (1) a default interface trained to classify handwritten EMNIST characters (Cohen
et al., 2017), and (2) X2T. We intentionally train the default interface on EMNIST images in-
stead of UJI Pen Characters drawings, to model real-world applications with a distribution mis-
match between the training data and test data, as discussed in Section 3. To address the chal-
lenge of selecting from 27 possible character actions, we use a language model (Dai et al., 2019)
to predict the prior likelihood of an action pLM(utju0:t1)given the preceding characters u0:t1
in the user’s text ﬁeld. We use this language model in both the default interface and X2T. In
particular, we set (ujx)/p(ujx)pLM(uju0:t1), wherepis the EMNIST image classiﬁer.
7
Published as a conference paper at ICLR 2021
Evaluation
Training User 1 User 2 User 3 User 4
User 1 .995 .033 .025 .017
User 2 .018 .463 .009 .061
User 3 .000 .002 .975 .039
User 4 .018 .002 .025 .993
Table 1: An observational study with 60 users in
the handwriting recognition domain that addresses Q3:
does X2T personalize the interface to different input
styles? We measure action prediction accuracy across
1000 interactions, and randomly sample users 1-4 from
the pool of 60 users. The interface trained on user iis
substantially more accurate when evaluated on inputs
from user ithan on inputs from user j, suggesting that
the learned interface is personalized to each individual
user.As in Section 4.1, the default interface stays
ﬁxed throughout the experiment and does not
adapt to the user, since re-training the default
interface would require interrupting the user to
collect paired data.
The results in Figure 4 show that X2T signiﬁ-
cantly outperforms the default interface (orange
vs. gray). We ran a one-way repeated measures
ANOV A on the action prediction accuracy de-
pendent measure from the default and X2T con-
ditions, with the presence of X2T as a factor,
and found that f(1;59) = 37:46;p< 0:0001 .
Furthermore, Table 1 shows that X2T learns an
interface that is particularly suited to the user
whose data it was trained on: when an interface trained on user i’s data is evaluated on data from
userj6=iinstead of user i, performance degrades substantially.
4.4 A BLATION EXPERIMENTS
Figure 4: An observational study with 60 users in the
handwriting recognition domain that addresses Q4-5 :
do ofﬂine pretraining and an informative prior policy
accelerate online learning, and does online learning im-
prove the interface beyond the initialization provided
by ofﬂine pretraining? In this ablation experiment, we
remove each of the three components – ofﬂine pretrain-
ing, an informative prior policy, and online learning –
one by one, and ﬁnd that each component is critical for
maintaining high action prediction accuracy at different
stages of the experiment.In this experiment, we aim to test the impor-
tance of three components of X2T: ofﬂine pre-
training, an informative prior policy, and online
learning. As discussed in Sections 2.2 and 2.3,
to improve the initial performance of X2T and
accelerate online learning, we pretrain on of-
ﬂine data collected using the default interface,
and incorporate prior knowledge from the de-
fault interface into the prior policy in Equa-
tion 1. Using the handwriting recognition task
from Section 4.3, we conduct ablation experi-
ments in which we drop out each of the three
components, one by one, and measure any re-
sulting changes in performance. In the ﬁrst con-
dition, we test the effect of not performing of-
ﬂine pretraining, by initializing X2T with ran-
dom weights init. In the second, we test the
effect of not incorporating prior knowledge into
our policy, by setting the prior policy to be a
uniform distribution over actions. In the third,
we test the effect of not learning online and in-
stead relying solely on ofﬂine pretraining, by
freezing the reward model parameters after of-
ﬂine pretraining and not storing online data.
The results in Figure 4 show that ofﬂine pre-
training is helpful at the onset of online learn-
ing (orange vs. teal), but does not have a sub-
stantial effect on performance after some online
data has been collected. This is unsurprising,
since leveraging ofﬂine data is only necessary when insufﬁcient online data has been collected. Us-
ing the default interface as a prior policy in Equation 1 is critical for X2T’s performance throughout
the experiment (orange vs. red). This is also unsurprising, given that the default interface contains
useful prior knowledge about how to interpret user inputs. Online learning is not particularly helpful
at the onset of the experiment (orange vs. purple), but has a substantial effect on performance over
time. This result shows that learning from on-policy data is critical for X2T. In other words, X2T
learns best when it learns from its own mistakes, rather than the mistakes of the default interface.
4.5 P ILOT STUDY WITH A BRAIN -COMPUTER INTERFACE
8
Published as a conference paper at ICLR 2021
Figure 5: A pilot study with one participant that ad-
dresses Q6: can X2T improve the accuracy of a brain-
computer interface (BCI) for typing? We ﬁnd that X2T
adapts to recent input-action-reward data and outper-
forms a default interface that is trained via supervised
learning on older, paired input-action data.In this experiment, we demonstrate X2T’s abil-
ity to improve the typing accuracy of a brain-
computer interface (BCI) that uses a 128-
channel chronic electrocorticography (ECoG)
implant to measure neural activity (Leuthardt
et al., 2004; Silversmith et al., 2020). Each
user input has 128 features, which are obtained
by pre-processing the raw ECoG signals (Ap-
pendix A.1.3 discusses the details). The results
in Figure 5 show that, after ofﬂine pretraining
on a dataset of 2913 input-action-reward tu-
ples, X2T achieves 60.7% action prediction ac-
curacy during a new session of 300 steps. We
train a default interface on paired data of inputs
and ground-truth actions collected before the
ofﬂine data. To evaluate the default interface,
we conduct a counterfactual experiment (sim-
ilar to those in previous sections) on the 300
steps of data from the X2T evaluation session.
The default interface only achieves 46.3% ac-
curacy. Interestingly, the default interface never predicts the top-most button action, leading to a
0% recall rate for that action. This is most likely due to changes in the feature processing pipeline
or other kinds of distributional shift in the user inputs over time. In contrast, X2T achieves 70.2%
recall on the top-most button action, suggesting that it can overcome these challenges.
5 D ISCUSSION
In our online user study on gaze-based word selection, we show that X2T learns to outperform a
non-adaptive interface in under 30 minutes (including ofﬂine data collection), and that the user si-
multaneously adapts their input style to the interface. The observational study on handwritten char-
acter recognition shows that X2T can also personalize the interface to individual users.Additionally,
our pilot study with a brain-computer interface user shows that X2T improves the recall rate of one
particular action from 0% to 70.2%. These experiments broadly illustrate how online learning from
user feedback can be used to train a human-machine interface.
One limitation of this work is that we assume backspaces can be generated independently of X2T.
This assumption restricts X2T to settings where the user can communicate a binary signal; e.g.,
through a button press or a sip-and-puff interface. One direction for future work is to relax this
assumption by training a binary classiﬁer to generate feedback signals from, e.g., facial expressions
or brain activity. X2T is also limited in that the beneﬁt from using X2T to ﬁne-tune the default
interface may decrease as the default interface is improved, as suggested by the diminishing returns
in Figure 2. One direction for future empirical work is to test X2T with state-of-the-art default in-
terfaces. In spite of these limitations, methods like X2T that learn from their mistakes and stimulate
user co-adaptation provide a general mechanism for improving user interfaces; not only for assistive
typing, but also for other domains, such as brain-computer interfaces for prosthetic limb control and
augmented reality devices for visually-impaired users.
6 A CKNOWLEDGMENTS
Thanks to Sarah Seko, Reza Abiri, Yasmin Graham, and members of the Neural Engineering and
Plasticity lab at UC San Francisco for helping to conduct the brain-computer interface experiments
in Section 4.5. Thanks to members of the InterACT and RAIL labs at UC Berkeley for feedback on
this project. This work was supported in part by an NVIDIA Graduate Fellowship, AFOSR FA9550-
17-1-0308, NSF NRI 1734633, NIH New Innovator Award [1 DP2 HD087955], and UCSF Weill
Institute for Neurosciences.
9
Published as a conference paper at ICLR 2021
REFERENCES
Gopala K Anumanchipalli, Josh Chartier, and Edward F Chang. Speech synthesis from neural
decoding of spoken sentences. Nature , 2019.
Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement learning
from policy-dependent human feedback. arXiv preprint arXiv:1902.04257 , 2019.
Alexander Broad, Todd David Murphey, and Brenna Dee Argall. Learning models for shared control
of human-machine systems with unknown dynamics. In Robotics: Science and Systems , 2017.
Jose M Carmena. Advances in neuroprosthetic learning and control. PLoS Biology , 2013.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony
Robinson. One billion word benchmark for measuring progress in statistical language modeling.
arXiv preprint arXiv:1312.3005 , 2013.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep re-
inforcement learning from human preferences. In Neural Information Processing Systems , 2017.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist
to handwritten letters. In International Joint Conference on Neural Networks , 2017.
Mark Collier and Hector Urdiales Llorens. Deep contextual multi-armed bandits. arXiv preprint
arXiv:1807.09809 , 2018.
Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Niekum, and W Bradley Knox.
The empathic framework for task learning from implicit human feedback. arXiv preprint
arXiv:2009.13649 , 2020.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi-
nov. Transformer-xl: Attentive language models beyond a ﬁxed-length context. arXiv preprint
arXiv:1901.02860 , 2019.
Siddharth Dangi, Amy L Orsborn, Helene G Moorman, and Jose M Carmena. Design and analysis
of closed-loop decoder adaptation algorithms for brain-machine interfaces. Neural computation ,
2013.
Siddharth Dangi, Suraj Gowda, Helene G Moorman, Amy L Orsborn, Kelvin So, Maryam Shanechi,
and Jose M Carmena. Continuous closed-loop decoder adaptation with a recursive maximum
likelihood algorithm allows for rapid performance acquisition in brain-machine interfaces. Neural
computation , 2014.
Yuqing Du, Stas Tiomkin, Emre Kiciman, Daniel Polani, Pieter Abbeel, and Anca Dragan. Ave:
Assistance via empowerment. arXiv preprint arXiv:2006.14796 , 2020.
Vikash Gilja, Paul Nuyujukian, Cindy A Chestek, John P Cunningham, M Yu Byron, Joline M
Fan, Mark M Churchland, Matthew T Kaufman, Jonathan C Kao, Stephen I Ryu, et al. A high-
performance neural prosthesis enabled by control algorithm design. Nature neuroscience , 2012.
Ethan K Gordon, Xiang Meng, Matt Barnes, Tapomayukh Bhattacharjee, and Siddhartha S Srini-
vasa. Adaptive robot-assisted feeding: An online learning framework for acquiring previously-
unseen food items. arXiv preprint arXiv:1908.07088 , 2019.
Robert D Hawkins, Michael C Frank, and Noah D Goodman. Characterizing the dynamics of
learning in repeated reference games. Cognitive Science , 2020.
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jos ´e Miguel Hern ´andez-Lobato, Richard E
Turner, and Douglas Eck. Sequence tutor: Conservative ﬁne-tuning of sequence generation mod-
els with kl-control. In International Conference on Machine Learning , 2017.
Hong Jun Jeon, Dylan P Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions.
arXiv preprint arXiv:2005.03210 , 2020.
10
Published as a conference paper at ICLR 2021
Eleanor Jones, Jason Alexander, Andreas Andreou, Pourang Irani, and Sriram Subramanian. Ges-
text: accelerometer-based gestural text-entry systems. In SIGCHI Conference on Human Factors
in Computing Systems , 2010.
Siddharth Karamcheti, Dorsa Sadigh, and Percy Liang. Learning adaptive language interfaces
through decomposition. arXiv preprint arXiv:2010.05190 , 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 , 2014.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer
framework. In International Conference on Knowledge Capture , 2009.
Kyle Krafka, Aditya Khosla, Petr Kellnhofer, Harini Kannan, Suchendra Bhandarkar, Wojciech
Matusik, and Antonio Torralba. Eye tracking for everyone. In IEEE Conference on Computer
Vision and Pattern Recognition , 2016.
Andrew S Lan and Richard G Baraniuk. A contextual bandits framework for personalized learning
action selection. In Educational Data Mining , 2016.
John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side
information. In Neural Information Processing Systems , 2008.
Tor Lattimore and Csaba Szepesv ´ari.Bandit algorithms . Cambridge University Press, 2020.
Eric C Leuthardt, Gerwin Schalk, Jonathan R Wolpaw, Jeffrey G Ojemann, and Daniel W Moran.
A brain–computer interface using electrocorticographic signals in humans. Journal of Neural
Engineering , 2004.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review.
arXiv preprint arXiv:1805.00909 , 2018.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to
personalized news article recommendation. In International Conference on World Wide Web ,
2010.
David Llorens, Federico Prat, Andr ´es Marzal, Juan Miguel Vilar, Mar ´ıa Jos ´e Castro, Juan-
Carlos Amengual, Sergio Barrachina, Antonio Castellanos, Salvador Espana Boquera, Jon Ander
G´omez, et al. The ujipenchars database: a pen-based database of isolated handwritten characters.
InInternational Conference on Language Resources and Evaluation , 2008.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, David Roberts, Matthew E Taylor, and
Michael L Littman. Interactive learning from policy-dependent human feedback. arXiv preprint
arXiv:1701.06049 , 2017.
I Scott MacKenzie and Kumiko Tanaka-Ishii. Text entry systems: Mobility, accessibility, universal-
ity. 2010.
Joseph G Makin, David A Moses, and Edward F Chang. Machine translation of cortical activity to
text with an encoder–decoder framework. Technical report, 2020.
Daniel McDuff and Ashish Kapoor. Visceral machines: Reinforcement learning with intrinsic
physiological rewards. In International Conference on Learning Representations , 2019. URL
https://openreview.net/forum?id=SyNvti09KQ .
Josh Merel, David Carlson, Liam Paninski, and John P Cunningham. Neuroprosthetic decoder
training as imitation learning. arXiv preprint arXiv:1511.04156 , 2015.
Stefanos Nikolaidis, Yu Xiang Zhu, David Hsu, and Siddhartha Srinivasa. Human-robot mutual
adaptation in shared autonomy. In ACM/IEEE International Conference on Human-Robot Inter-
action , 2017.
Jo˜ao Oliveira, Tiago Guerreiro, Hugo Nicolau, Joaquim Jorge, and Daniel Gonc ¸alves. Brailletype:
unleashing braille over touch screen mobile phones. In IFIP Conference on Human-Computer
Interaction , 2011.
11
Published as a conference paper at ICLR 2021
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and
Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic
reinforcement learning. In IEEE International Conference on Rehabilitation Robotics , 2011.
Krsto Prorokovi ´c, Michael Wand, and J ¨urgen Schmidhuber. Meta-learning for recalibration of emg-
based upper limb prostheses. 2020.
Filip Radlinski and Thorsten Joachims. Evaluating the robustness of learning from implicit feed-
back. arXiv preprint cs/0605036 , 2006.
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Shared autonomy via deep reinforcement
learning. arXiv preprint arXiv:1802.01744 , 2018.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn-
ing of reward functions. In Robotics: Science and Systems , 2017.
Charles Schaff and Matthew R Walter. Residual policy learning for shared autonomy. arXiv preprint
arXiv:2004.05097 , 2020.
Krishna V Shenoy and Jose M Carmena. Combining decoder design and neural adaptation in brain-
machine interfaces. Neuron , 2014.
Daniel B Silversmith, Reza Abiri, Nicholas F Hardy, Nikhilesh Natraj, Adelyn Tu-Chan, Edward F
Chang, and Karunesh Ganguly. Plug-and-play control of a brain–computer interface through
neural map stabilization. Nature Biotechnology , 2020.
Dawn M Taylor, Stephen I Helms Tillery, and Andrew B Schwartz. Direct cortical control of 3d
neuroprosthetic devices. Science , 2002.
Cheng-Yao Wang, Wei-Chen Chu, Po-Tsung Chiu, Min-Chieh Hsiu, Yih-Harn Chiang, and Mike Y
Chen. Palmtype: Using palms as keyboards for smart glasses. In International Conference on
Human-Computer Interaction with Mobile Devices and Services , 2015.
Sida I Wang, Percy Liang, and Christopher D Manning. Learning language games through interac-
tion. arXiv preprint arXiv:1606.02447 , 2016.
David J Ward, Alan F Blackwell, and David JC MacKay. Dasher—a data entry interface using
continuous gestures and language models. In ACM Symposium on User Interface Software and
Technology , 2000.
Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive
agent shaping in high-dimensional state spaces. arXiv preprint arXiv:1709.10163 , 2017.
Francis R Willett, Donald T Avansino, Leigh R Hochberg, Jaimie M Henderson, and Krishna V
Shenoy. High-performance brain-to-text communication via imagined handwriting. bioRxiv ,
2020.
Alan Wrench. The mocha-timit articulatory database, 1999.
Duo Xu, Mohit Agarwal, Faramarz Fekri, and Raghupathy Sivakumar. Accelerating reinforcement
learning agent with eeg-based implicit human feedback. arXiv preprint arXiv:2006.16498 , 2020.
Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a
dueling bandits problem. In International Conference on Machine Learning , 2009.
Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with upper conﬁdence
bound-based exploration. arXiv preprint arXiv:1911.04462 , 2019.
12
Published as a conference paper at ICLR 2021
A A PPENDIX
A.1 I MPLEMENTATION DETAILS
Stochastic gradient descent. We use Adam (Kingma & Ba, 2014) to optimize the binary cross-
entropy loss described in Section 2.2,
`() =X
(x;u;r)2Drlog (p(r= 1jx;u)) + (1r) log (1p(r= 1jx;u)): (2)
We set gradient norm clipping to 10 in all experiments.
Ofﬂine pretraining. To pretrain X2T on ofﬂine data, we ﬁrst train the reward model pto con-
vergence on the ofﬂine data, and store the learned network weights init. Once a minimum of four
online input-action-reward triples have been collected, after every interaction, we update the reward
modelpby taking one gradient step. We use the values initto initialize the network weights (e.g.,
instead of a random initialization).
A.1.1 O NLINE USERSTUDY : TYPING WITH EYEGAZE
Eye image features. Instead of operating directly on 224x224 images of the user’s eyes, we use the
activations of the last fully-connected layer in the iTracker model (Krafka et al., 2016) as the input x
to our reward model p(rjx;u). When training the reward model p, we freeze the iTracker network
weights used to generate this 128-dimensional input x.
Reward model architecture and learning. We set the learning rate for Adam to 103, and batch
size to 128. We do not perform online updates to the reward model parameters whilejDj<4.
In our experiments, for every input x, there is exactly one desired action uthat will result in
a positive reward r= 1, while all other actions u6=uresult in a zero reward r= 0. In
other words, p(r= 1jx;u) =p(u=ujx). Hence, we structure the reward model for X2T as
p(r= 1jx;u) =f(ujx), wherefis an action classiﬁer. Furthermore, since we expect the re-
ward model to learn to implicitly estimate the user’s gaze position in order to predict actions,
we directly incorporate this inductive bias into the model: we structure the action classiﬁer as
f(ujx)/exp (kg(x)pos(u)k2), whereg(x)outputs a 2D position of the user’s estimated
gaze, and pos(u)is the known 2D position of the button for action u. Note that even though g(x)
outputs a 2D position, the parameters are still trained on the reward prediction objective in Equa-
tion 2 (e.g., instead of a 2D gaze position prediction objective). We represent gusing a feedforward
neural network with one hidden layer containing 64 units, a dropout layer with a dropout rate of 0.3
between the hidden layer and the output layer, and ReLU activations. At each timestep t, we record
10 eye imagesfxi
tg10
i=1at a sampling rate of 10 Hz, and average our predictions over these 10 inputs.
Speciﬁcally, for X2T, we set g(xt) =1
10P10
i=1g(xi
t). For the default interface, we average the 2D
gaze position estimates across the 10 samples before predicting the action whose button position is
nearest to the average gaze position estimate. We initialize initwith the ofﬂine pretraining scheme
described in Section 2.2, using 250 input-action-reward triples collected with the default interface.
Experiment design. We recruited 11 male and 1 female participants, with an average age of 21.
Each participant was provided with the rules of the task and a short practice period of 20 interactions
to familiarize themselves with the system. Each interaction – which consisted of providing an input,
observing the interface’s action, and deciding whether or not to backspace – took an average of 4
seconds. Each participant completed three phases of experiments: A, B, and C. In phase A, they
operate the default interface for 250 steps, generating an ofﬂine dataset of input-action-reward triples
that we use to initialize X2T. In phase B, they operate X2T for 250 steps. In phase C, they operate
the default interface for 250 steps. To avoid the confounding effects of user learning or fatigue over
time, we counterbalance the order of phase B and C: six randomly-selected participants completed
phase B before C, and the other six participants completed phase C before B. Phase A is used
solely to generate ofﬂine data to initialize X2T in phase B. One participant’s room lighting changed
substantially during phase B. Since their performance during phase A was substantially better than
during phase B, we use their phase A data (instead of phase B data) to measure the default interface’s
performance on this one participant. We sample goal sentences from the MOCHA-TIMIT database
(Wrench, 1999), following prior work on speech interfaces (Makin et al., 2020). We set the same
goal sentences for each user in each condition.
13
Published as a conference paper at ICLR 2021
Deterministic policy. In our experiments, we ﬁnd that sampling actions u(ujx)from the
stochastic policy does not substantially improve exploration, and can in fact degrade perfor-
mance by not always choosing the optimal action. This is most likely due to the use of the default
interface in Equation 1, which already provides an effective exploration mechanism. Hence,
instead of randomly sampling actions, we deterministically select the highest-likelihood action:
u arg maxu(ujx).
A.1.2 O BSERVATIONAL STUDY : TYPING BY DRAWING CHARACTERS
Perturbing user inputs. We perturb the character drawings in the UJI Pen Characters Database
by decomposing each drawing into a sequence of pen tip velocity vectors, adding Brownian noise
to each velocity, then integrating over the perturbed velocities to yield a complete, perturbed draw-
ing (see Figure 7 in the appendix). We compute the Brownian noise by sampling an independent
Gaussian noise vector with zero mean and variance of 2104at each timestep, and summing over
these noise vectors from time 0 to time tto compute the Brownian noise vector for time t. The
same Brownian noise vector is applied to all the velocity segments in a given drawing. Each user
inputxis a 28x28 image of the user’s complete, perturbed drawing of a given character. There are
27 possible characters in the action space U: 26 lower-case letters, and space (which we represent
using the digit 7).
Reward model architecture and learning. We set the learning rate for Adam to 5104, batch
size to 128, pretrain on the ofﬂine data for 20 epochs, and sample actions from the stochastic policy
described in Equation 1 (instead of the deterministic policy described in Appendix A.1.1). We also
limit the size of the replay buffer Din Algorithm 1 to the latest 500 input-action-reward triples. We
do not perform online updates to the reward model parameters whilejDj<100. We represent the
reward model pas a neural network with the following architecture: 28x28 input layer, 32x5x5
convolutional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.5, 64x5x5 convolu-
tional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.3, and a fully-connected output
layer. We structure the reward model as an action classiﬁer, as in Section A.1.1. We initialize init
with the ofﬂine pretraining scheme described in Section 2.2, using 1000 input-action-reward triples
collected with the default interface.
Experiment design. The UJI Pen Characters Database (Llorens et al., 2008) contains handwriting
samples from 60 users. For each user, it includes two repetitions of lowercase letters, uppercase
letters, and 10 digits, for a total of 1364 samples per user. In our observational study, we randomly
sample target sentences, and replay user inputs that attempt to type the characters in those target
sentences. In the replays, we automatically backspace incorrect actions; a realistic modeling choice,
given that in the online user study in Section 4.1, the user did indeed backspace mistakes, and did
not backspace correct actions, in 98:6%of their interactions. As in the online user study, we run
each user’s data through two conditions: default and X2T. Since this is an observational study, we
do not need to counterbalance the order of the two conditions. For the personalization experiment in
Table 1, we assign a randomly-selected, constant value to the random seed of the Brownian noise for
each user in each condition. This ensures that comparisons between entry (i;i)and entries (i;j6=i)
in the 4x4 table are not confounded by differences in the input noise, and are only inﬂuenced by
systematic differences in the users’ individual handwriting styles. As in the online user study, we
sample goal sentences from the MOCHA-TIMIT database (Wrench, 1999), and set the same goal
sentences for each user in each condition.
Language model. We use the Transformer-XL language model (Dai et al., 2019) – speciﬁcally,
the word-level version that is pretrained on the One Billion Word dataset (Chelba et al., 2013) – to
compute the prior likelihood pLM(utju0:t1). To compute character-level likelihoods, we marginal-
ize over the 60000 words in the language model’s vocabulary that occur the most frequently in the
One Billion Word training corpus, not including words with punctuation and converting upper-case
to lower-case characters. We feed the previously-typed characters in the current sentence (i.e., not
including the previous sentences) as context to the language model.
14
Published as a conference paper at ICLR 2021
A.1.3 P ILOT STUDY WITH A BRAIN -COMPUTER INTERFACE
Due to constraints on the duration and number of sessions we were able to conduct with the partici-
pant, we made two simpliﬁcations to the interface: we simpliﬁed the display to use 4 buttons instead
of 8 buttons, and we automated backspaces.
Reward model architecture and learning. We set the learning rate for Adam to 104, batch size to
128, and maximum buffer size jDjto 1000. Unlike the previous experiments, we did not use dropout.
We used an L2 regularization constant of 0.001 for X2T, and 0.01 for the default interface. For both
the default interface and reward model, we used a feedforward network architecture with 3 layers
of 256 hidden units each. We deterministically selected actions with the maximum conditional
likelihood under the policy, instead of sampling from the stochastic policy in Equation 1. When
pretraining the reward model on ofﬂine data, we initialized the reward model network weights with
the default interface network weights. We set the prior policy in Equation 1 to be a uniform
prior, instead of using the default interface. To accommodate noise in the user inputs, we require 4
consecutive bins of input to be mapped to the same action by the policy before executing that action;
after 10 bins with no string of 4 consecutive, equal actions, the majority action is executed. In the
counterfactual experiment with the default interface, if we reach the end of the bins for a given step
without a string of 4 consecutive, equal actions, the majority action is executed. After an action is
executed and reward collected, we add an input-action-reward tuple for the input at each bin to the
bufferD.
Experiment design. We conducted three experimental sessions: the ﬁrst on 2/26/21, in which we
collected 997 input-action-reward samples; the second on 3/5/21, which yielded 1916 samples; and
the third on 3/13/21, which yielded 1984 samples. The default interface was trained on 3686 samples
(left button intended: 959, down: 916, right: 897, up: 914) recorded prior to 2/26/21. There were
3 seconds of no input before each word selection. The interface was paused intermittently during
each session to accommodate user fatigue.
Feature processing. The ECoG signals were binned at a frequency of 2 Hz. Each user input has
128 dimensions, which consist of high-band frequencies of raw ECoG signal recorded during each
binning period. The remaining implementation details are described in Silversmith et al. (2020).
15
Published as a conference paper at ICLR 2021
p-value Default Interface X2T
The system selected the words I wanted < :05 4.50 5.42
The system improved over time < :05 3.17 4.75
I improved at using the system over time > :05 4.08 4.42
The system did not select the words I wanted < :05 4.08 2.83
The system got worse over time > :05 3.25 2.42
I got worse at using the system over time > :05 3.25 2.67
I backspaced when there was a mistake > :05 5.42 5.83
I ignored mistakes (did not backspace them) > :05 2.33 2.17
Table 2: Subjective evaluations from the 12 participants in the online user study. Means reported below for
responses on a 7-point Likert scale, where 1 = Strongly Disagree, 4 = Neither Disagree nor Agree, and 7 =
Strongly Agree. p-values from a one-way repeated measures ANOV A with the presence of X2T as a factor
inﬂuencing responses.
Figure 6: To measure X2T’s sensitivity to noise in the reward signal, we conduct a counterfactual experiment
with the eye gaze user study data, similar to Section 4.2. In this experiment, we ﬂip the reward rto1r
with probability p, which we call the reward mislabeling rate. We then train our reward model on these noisy
rewards. X2T outperforms the default interface for a wide range of mislabeling rates, and only performs
worse than the default interface when the rewards are completely random (i.e., the mislabeling rate is 50%).
In practice, we ﬁnd that users’ backspaces tend to follow the assumptions in Section 2.1, which leads to a
relatively low empirical mislabeling rate of 1.4%.
16
Published as a conference paper at ICLR 2021
Figure 7: Handwriting samples from the UJI Pen Characters Database that have been perturbed by adding
Brownian noise to pen tip velocities. The left-most column shows the true input, and each successive column
shows the perturbed input after 100-timestep intervals. We used the character ’7’ in place of the space character.
User sees list of words Webcam records image of user’s eye gaze
Reward modelPredicted wordBackspace or continue
Default interfaceOur interface
Reward
User input
Train reward modelInput-action-reward log dataBinary feedback (e.g., from button press)
Figure 8: An illustration of the eye gaze experiments in Section 4.1
17

Jensen Gao
Siddharth Reddy
Glen Berseth
Nicholas Hardy
Nikhilesh Natraj
Karunesh Ganguly
Anca Dragan
Sergey Levine