Principles of Explanatory Debugging
to Personalize Interactive Machine Learning
Todd Kulesza1Margaret Burnett1Weng-Keen Wong1Simone Stumpf2
1Oregon State University
School of EECS
Corvallis, Oregon2City University London
Centre for HCI Design
London, United Kingdom
{kuleszto, burnett, wong}@eecs.oregonstate.edu, Simone.Stumpf.1@city.ac.uk
ABSTRACT
How can end users efﬁciently inﬂuence the predictions that
machine learning systems make on their behalf? This paper
presents Explanatory Debugging, an approach in which the
system explains to users how it made each of its predictions,
and the user then explains any necessary corrections back to
the learning system. We present the principles underlying this
approach and a prototype instantiating it. An empirical evalua-
tion shows that Explanatory Debugging increased participants’
understanding of the learning system by 52% and allowed
participants to correct its mistakes up to twice as efﬁciently as
participants using a traditional learning system.
Author Keywords
Interactive machine learning; end user programming.
H.5.2. [ Information interfaces and presentation (e.g. HCI) ]
User interfaces
INTRODUCTION
In many machine learning applications, the user has a concept
in mind—songs he wants to hear, junk mail she doesn’t want
to see, anomalous transactions he wants to detect—that the
machine learning system is trying to identify. An increasing
body of research has begun to explore what users can do when
their machine learning systems misidentify their concepts [e.g.,
1,2,18,26,38,45]. Much of this work shares a common chal-
lenge: how can end users effectively and efﬁciently personalize
the predictions or recommendations these learning systems
make on their behalf?
For example, engineer Alice has email folders for her project
group at work, an ofﬁce sports pool, and individual folders
for colleagues she frequently collaborates with; she wants her
email program to automatically categorize new email into the
appropriate folder. This requires ﬁne-grained control—perhaps
she wants email from her colleague Bob to go to the “sports”
folder if it mentions something about hockey, the “project”
folder if it references something related to their current project,
or the “Bob” folder if neither of those conditions are met.
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for proﬁt or commercial advantage and that copies bear this notice and the full citation
on the ﬁrst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior speciﬁc permission and/or a
fee. Request permission from permissions@acm.org.
IUI 2015, April 26–May 1, 2015, Atlanta, GA, USA.
Copyright 2015 ACM 978-1-4503-3306-1/15/03. . . $15.00.
http://dx.doi.org/10.1145/2678025.2701399However, Alice does not have enough email from Bob to
accurately train a learning system to recognize these three
concepts, and by the time she acquires it, the hockey season
may be over or her group’s project may change.
Our proposed solution is Explanatory Debugging, which is
an explanation-centric approach to help end users effectively
and efﬁciently personalize machine learning systems. We say
“debugging” because personalizing a learning system is an
end-user debugging problem: as discussed in [14], the user is
trying to identify and correct faults in the system’s reasoning
that are responsible for predictions that failed to meet the
user’s expectations.
In Explanatory Debugging, the system explains the reasons for
its predictions to its end user, who in turn explains corrections
back to the system. We hypothesize that this cycle of expla-
nations will help users build useful mental models—internal
representations that allow people to predict how a system will
behave [ 17]—and thus allow them to personalize their learning
system better and faster than traditional black-box systems.
In the earlier example, such an approach could show Alice all
the words the system uses to identify email about her group’s
project, the sports pool, or any other concept. She could then
add additional words she knows to be relevant, or remove
words that she knows are irrelevant. She would not need to
worry that her corrections might confuse the system (e.g.,
telling it that a message from Bob about hockey belongs in the
sports folder and hoping it will identify that the topic is the
reason, not the sender), nor would she need to spend weeks or
months acquiring new training instances.
In this paper we present the Explanatory Debugging approach
and instantiate it in EluciDebug, the ﬁrst interactive system
designed to help end users build useful mental models of a
machine learning system while simultaneously allowing them
to explain corrections back to the system. We evaluate Ex-
planatory Debugging against a traditional black-box learning
system to determine whether Explanatory Debugging helped
end users build better mental models, and if so, whether these
mental models helped users more efﬁciently personalize their
learning system’s behavior.
RELATED WORK
The study of how end users interactively control statistical
machine learning systems is gaining increased attention. Fails
et al. ﬁrst popularized the phrase interactive machine learning
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
126
Topic Modeling / Robotics
in a paper describing how an iterative train-feedback-correct
cycle allowed users to quickly correct the mistakes made by an
image segmentation system [ 11]. Since then, researchers have
explored using this cycle of quick interactions to train instance-
based classiﬁers [ 5,13], enable better model selection by end
users [ 2,12,40], elicit labels for the most important instances
(e.g., active learning) [ 7,37], and to improve reinforcement
learning for automated agents [ 21]. These use cases largely
treat the machine learning system as a “black box”—users can
try to personalize the system by providing it with different
inputs (e.g., labeled instances), but are unable to see why
different inputs may cause the system’s outputs to change.
An alternative to the “black box” model is a “white box” model
[16], in which the machine learning system is made more trans-
parent to the user. Increased transparency has been associated
with many beneﬁts, including increased user acceptance of (or
satisfaction with) recommendations and predictions [ 4,8,42],
improved training of intelligent agents [ 41], and increased trust
in the system’s predictions [ 10]. In this paper, however, we
primarily care about transparency because it has been shown to
help users understand how a learning system operates [ 25,30].
Recent work has suggested that as end users better understand
how a machine learning system operates, they are better able
to personalize it [ 12,23]. This aligns with work from other
domains, which has found that explanation-based debugging
approaches can help end users correct a program’s faults
[22,36]. Further, research has shown that in the absence
of transparency, users experience difﬁculty building accurate
mental models of learning systems [ 23,43]. People will still
build mental models of the system, but such ﬂawed mental
models often lead to misinformed user behavior [19, 32].
Some researchers have sought to expand the types of inputs
interactive machine learning systems can process, and this can
also increase system transparency. For example, user feedback
may take the form of model constraints [ 18], critiques to
the model’s search space [ 45], or adjustments to the model’s
weights of evidence [ 26]. In addition to supporting novel user
feedback mechanisms, these approaches increase transparency
by enabling users to view such facets as the existing model
constraints [ 18], the model’s feature set [ 45], or the weights
of evidence responsible for each prediction [26].
All of these works have increased our understanding of how
human factors impact machine learning, but none have evalu-
ated the ability of an interactive machine learning system to (1)
help users build useful mental models of how it operates such
that (2) they can efﬁciently personalize the system. Kulesza
et al. found a link between mental models and personalization
ability, but relied on human instructors to help participants
build useful mental models [ 23]. Other work has studied the
impact of machine-generated explanations on users’ mental
models, but did not explore whether this helped participants’
personalize their learning systems [ 8,25,30]. In this paper we
rely on machine-generated explanations to help users build
accurate mental models and then evaluate their resulting ability
to personalize a learning system.EXPLANATORY DEBUGGING
In 2010, Kulesza et al. proposed the idea of Explanatory
Debugging—a two-way exchange of explanations between
an end user and a machine learning system [ 24]—but did not
provide principles underlying the approach. In this section we
build upon that proposal by describing principles to ground
and support replication of Explanatory Debugging.
Explainability
Our ﬁrst principle for Explanatory Debugging is Explainabil-
ity: accurately explain the learning system’s reasons for each
prediction to the end user. This principle builds upon research
showing that end users who received explanations about a
learning system’s reasoning were able to better personalize
classiﬁers and recommenders [ 4,18,24]. Without explana-
tions, however, users struggle to build accurate mental models
of such systems [ 23,30,43]. This suggests in situ explanations
are a necessary condition to help end users learn how a ma-
chine learning system operates. To help users build useful
mental models of the learning system, these explanations
should observe the following principles:
Principle 1.1: Be Iterative
Research has suggested that users may best personalize a learn-
ing system if they build their mental models while interacting
with it [ 23]. Thus, explanations should support an iterative, in
situlearning process. For example, explanations could take the
form of concise, easily consumable “bites” of information—if
a user is interested in learning more about the system, he or
she can attend to many of these explanations to iteratively
build a higher-ﬁdelity mental model.
Principle 1.2: Be Sound
Explanations should not be simpliﬁed by explaining the model
as if it were less complex than it actually is. In [ 25] this is
referred to as soundness: “the extent to which each component
of an explanation’s content is truthful in describing the under-
lying system.” In [ 23] the authors found a linear correlation
between the quality of a user’s mental model and their ability
to control the learning system as desired, suggesting that
the better someone understands the underlying system, the
better they will be able to control it. Further, [ 25] details the
impact of explanation ﬁdelity on mental model development,
ﬁnding that users did not trust—and thus, were less likely
to attend to—the least sound explanations. Because reducing
soundness reduces both the potential utility of the explanation
and the likelihood that users will invest attention toward it,
Explanatory Debugging entails designing explanations that
are as sound as practically possible.
One method for evaluating explanation soundness is to com-
pare the explanation with the learning system’s mathematical
model. How accurately are each of the model’s terms ex-
plained? If those terms are derived from more complex terms,
is the user able to “drill down” to understand those additional
terms? The more these explanations reﬂect the underlying
model, the more sound the explanation is.
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
127
Topic Modeling / Robotics
Principle 1.3: Be Complete
Kulesza et al. describe completeness as “the extent to which
allof the underlying system is described by the explanation”
[25], so a complete explanation does not omit important infor-
mation about the model. In that study, end users built the best
mental models when they had access to the most complete
explanations, which informed them of all the information
the learning system had at its disposal and how it used that
information [ 25]. Also pertinent is work showing that users
often struggle to understand how different parts of the system
interact with each other [ 26]. Complete explanations that
reveal how different parts of the system are interconnected
may help users overcome this barrier.
One method for evaluating completeness is via Lim and Dey’s
intelligibility types [29], with more complete explanations
including more of these intelligibility types.
Principle 1.4: But Don’t Overwhelm
Balanced against the soundness and completeness principles
is the need to remain comprehensible and to engage user
attention. Findings from [ 25] suggest that one way to engage
user attention is to frame explanations concretely, such as
referencing the predicted item and any evidence the learning
system employed in its prediction. In some circumstances,
selecting a more comprehensible machine learning model may
also be appropriate. For example, a neural network can be
explained as if it were a decision tree [ 9], but this reduces
soundness because a different model is explained. Similarly,
a model with 10,000 features can be explained as if it only
used the 10 most discriminative features for each prediction,
but this reduces completeness by omitting information that
the model uses. Alternative approaches that embody the Ex-
planatory Debugging principles include selecting a machine
learning model that can be explained with little abstraction
[e.g., 27, 38, 39] or using feature selection techniques [47] in
high-dimensionality domains to prevent users from struggling
to identify which features to adjust (as happened in [26]).
Correctability
Our second top-level principle for Explanatory Debugging is
Correctability: allow users to explain corrections back to the
learning system. To enable an iterative cycle of explanations
between the system and the user, in Explanatory Debugging
the machine-to-user explanation should also serve as the user-
to-machine explanation. Research suggests that to elicit cor-
rections from users, this feedback mechanism should embody
the following principles:
Principle 2.1: Be Actionable
Both theory [ 3] and prior empirical ﬁndings [ 6,23,25] suggest
end users will ignore explanations when the beneﬁt of attend-
ing to them is unclear. By making the explanation actionable,
we hope to lower the perceived cost of attending to it by
obviating the need to transfer knowledge from one part of
the user interface (the explanation) to another (the feedback
mechanism). Actionable explanations also fulﬁll three aspects
of Minimalist Instruction [ 44]: (1) people are learning while
performing real work; (2) the explanatory material is tightly
coupled to the system’s current state; and (3) people canleverage their existing knowledge by adjusting the explanation
to match their own mental reasoning.
Principle 2.2: Be Reversible
A risk in enabling users to provide feedback to a machine
learning system is that they may actually make its predictions
worse [e.g., 24,38]. Being able to easily reverse a harmful
action can help mitigate that risk. It may also encourage self-
directed tinkering, which can facilitate learning [ 35]. When
combined with Principle 2.4, reversibility also fulﬁlls a fourth
aspect of Minimalist Instruction [ 44]: help people identify and
recover from errors.
Principle 2.3: Always Honor User Feedback
As Yang and Newman found when studying users of learn-
ing thermostats [ 46], a system that appears to disregard user
feedback deters users from continuing to provide feedback.
However, methods for honoring user feedback are not always
straightforward. Handling user feedback over time (e.g., what
if new instance-based feedback contradicts old instance-based
feedback?) and balancing different types of feedback (e.g.,
instance-based feedback versus feature-based feedback) re-
quires careful consideration of how the user’s feedback will
be integrated into the learning system.
Principle 2.4: Incremental Changes Matter
In [25], participants claimed they would attend to explanations
only if doing so would enable them to more successfully
control the learning system’s predictions. Thus, continued
user interaction may depend upon users being able to see
incremental changes to the learning system’s reasoning after
each interaction (i.e., overcoming the gulf of evaluation that
exists between a user’s mental model and a system’s actual
state [ 33]). This principle is also related to Principle 1.1 (Be
iterative) because our thesis is that users will develop better
mental models iteratively, requiring many interactions with
the learning system. These interactions may not always result
in large, obvious changes, so being able to communicate
the small, incremental changes a user’s feedback has upon
a learning system may be critical to our approach’s feasibility.
ELUCIDEBUG: EXPLANATORY DEBUGGING IN ACTION
We instantiated Explanatory Debugging in a text classiﬁca-
tion prototype we call EluciDebug (Figure 1). We chose text
classiﬁcation because (1) many real-world systems require it
(e.g., spam ﬁltering, news recommendation, serving relevant
ads, search result ranking, etc.) and (2) it can be evaluated
with documents about common topics (e.g., popular sports),
allowing a large population of participants for our evaluation.
We designed EluciDebug to look like an email program with
multiple folders, each representing a particular topic. The
prototype’s machine learning component attempts to automati-
cally classify new messages into the appropriate folder.
While the principles of Explanatory Debugging primarily deal
with the user interface, two principles place constraints on the
machine learning model: (1) it must be able to honor user
feedback in real-time, and (2) it must be explainable with
enough soundness and completeness to allow users to build
useful mental models of how it operates without overwhelming
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
128
Topic Modeling / Robotics
Figure 1. The EluciDebug prototype. (A) List of folders. (B) List of messages in the selected folder. (C) The selected message.
(D) Explanation of the selected message’s predicted folder. (E) Overview of which messages contain the selected word. (F) Complete
list of words the learning system uses to make predictions.
them. Our EluciDebug prototype uses a multinomial naive
Bayes model (MNB) [ 20] with feature selection [ 47] to meet
these constraints. Evaluating the suitability of—or changes
necessary to—other models remains an open question.
The Multinomial Naive Bayes Classiﬁer: A Brief Review
Before describing how we integrated MNB with Explanatory
Debugging, we ﬁrst summarize how MNB operates. An MNB
classiﬁer computes the probability that a given input (e.g., the
document being classiﬁed) has of belonging to each output
(e.g., the possible labels). The output with the highest prob-
ability “wins” and becomes the predicted label for the input.
For example, if MNB calculates that a document has a 70%
probability of being junk mail and a 30% probability of not
being junk mail, the document will be labeled as junk mail.
The equations for computing probability, as deﬁned in [ 20],
are shown below. We use cto represent an individual class
in the collection of potential output classes C,dito represent
an individual document to classify, and assume that the only
features the classiﬁer uses are individual words in the set of
known documents:
Pr(cjdi)=Pr(c)Pr(dijc)
Pr(di)(1)
The term Pr(c)represents the probability that any given docu-
ment belongs to class cand can be estimated by dividing the
number of documents in cby the total number of documents
in the training set. The term Pr(dijc)represents the probabilityof document digiven class cand can be estimated as:
Pr(dijc)=Õ
nPr(wnjc)fni (2)
The term fniis the number of instances of word nin document
diand the term Pr(wnjc)is the probability of word ngiven
class c, estimated with the equation
Pr(wnjc)=pnc+Fnc
N
å
x=1pxc+N
å
x=1Fxc(3)
where Fncis the number of instances of word nin all of the
training documents for class c,Nis the number of unique
words in the training documents for all classes, and pncis
a smoothing term (usually 1) to prevent the equation from
yielding 0 if no documents from class ccontain word wn.
The Explanatory Debugging Principles in EluciDebug
Being Iterative
To help end users iteratively build better mental models, Eluci-
Debug employs two strategies: (1) each explanation focuses
on an individual aspect of the learning system, and (2) layered
explanations are available if the user wants more information.
For example, the Why explanation (introduced later) primarily
tells users about the system’s reasons for a speciﬁc prediction.
Over time, however, the user may learn about the system’s
general operation by observing commonalities in the reasons
for these speciﬁc predictions. Further, this Why explanation
is layered. Part of the explanation shows users which words
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
129
Topic Modeling / Robotics
contributed to the learning system’s prediction—if the user
wants to know more about why the system associated a partic-
ular word with the given topic, hovering over the word yields
a tooltip that explains how the system determines which topic
a word is associated with.
Being Sound
Soundness means that everything an explanation says is true.
Our explanations aim to be sound by accurately disclosing all
of the features the classiﬁer used to make its prediction, as
well as how each feature contributed to the prediction. Eluci-
Debug’s Why explanation is responsible for communicating
much of this information to users (Figure 2).
Soundly explaining the MNB classiﬁer requires explaining
thePr(c)andPr(dijc)terms from Equation 1, as these are the
only terms that impact the model’s predictions 1. Because the
Pr(dijc)term expands into Equation 2, we also need to explain
how word probabilities for each class factor into the prediction.
We can further increase soundness by explaining how these
probabilities are calculated (i.e., by explaining Equation 3).
In EluciDebug we explain the Pr(dijc)term by using a word
cloud to visualize the difference between each feature’s proba-
bility for the two output classiﬁcations (Equation 3) as the
following ratio:Pr(wnjc1)fni
Pr(wnjc2)fni, where c1is the class with the
larger probability for feature wn. We use the result to compute
the feature’s font size, while its font color is based on the
class with the larger probability. For example, in Figure 2
the word stanley is larger than tiger because its ratio of word
probability is correspondingly larger, and it is blue because its
probability of occurring in the hockey class is larger than its
probability of occurring in the baseball class. Hovering over a
word in this cloud shows a tooltip that explains the word’s size
was determined by a combination of (1) how many times it
appeared in each class, and (2) any adjustments the user made
to the word’s importance (Equation 3).
The second component of a sound explanation of the MNB
classiﬁer is the Pr(c)term from Equation 1. We explain this
term via a bar graph visualization of the number of items in
each class (Figure 2, middle).
The top-to-bottom design of this entire Why explanation, along
with the text that describes Part 1 andPart 2, is intended to
teach the user that both word presence (the Pr(dijc)term) and
folder size (the Pr(c)term) play a role in each of the classiﬁer’s
predictions. The result, shown at the bottom of Figure 2,
explains how these two parts are combined to determine the
classiﬁer’s certainty in its prediction.
Being Complete
Completeness means telling the whole truth. For the MNB
classiﬁer, this means not only explaining each of the terms
from Equation 1, but also all of the information the classiﬁer
could use, where it came from, and how likely it is that each
prediction is correct. To help ensure completeness, we turn to
1ThePr(di)term in the denominator of Equation 1 is only used to normalize
the result to fall within the range 0–1; it does not impact the model’s
prediction.
Figure 2. The Why explanation tells users how features and
folder size were used to predict each message’s topic. This ﬁgure
is a close-up of Figure 1 part D.
Lim and Dey’s schema of intelligibility types. The results of
[25] suggest that a complete explanation should include Lim
and Dey’s why, inputs, model, and certainty types. Lim’s work
suggests the usefulness of the what if type in scenarios where
the user is attempting to change the behavior of a classiﬁer
[28], so we included this intelligibility type as well.
We thus designed our EluciDebug explanations to detail all
of the information the classiﬁer could potentially use when
making predictions. The Why explanation shown in Figure 2
tells users that both feature presence and folder size played
a role in each prediction (the numerator of Equation 1). The
Important words explanations (Figure 3) goes even further,
telling the user allof the features the classiﬁer knows about
and may use in its predictions. Because it tells the user about
the sources of information available to the classiﬁer, this is
also an instantiation of Lim and Dey’s inputs intelligibility
type. To make it clear to users that these features can occur
in all parts of the document—message body, subject line, and
sender—EluciDebug highlights features in the context of each
message (Figure 1, part C).
In [25] the authors found that Lim and Dey’s model intelligi-
bility type was associated with better mental models, but this
intelligibility type was rarely attended to by most participants.
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
130
Topic Modeling / Robotics
To solve this dilemma, we incorporated the model content into
ourWhy explanation—it explains all of the evidence the classi-
ﬁers used, but it also explains where that evidence came from
(e.g., words or folder size) and how it was combined to arrive
at a ﬁnal prediction. This approach has the added advantage of
making the potentially abstract model intelligibility type very
concrete; it is now tailored to each speciﬁc prediction.
A further aspect of completeness is evident in the Feature
overview explanation (Figure 1, part E). This explanation
shows users how many messages contain a given feature and
is intended to help users identify when a feature they think
the computer should pay attention to may not be very useful.
The explanation updates in real-time as the user types in a
potential feature; users do not need to add the feature to view
its potential impact on classiﬁcations, making this an instance
of the what if intelligibility type.
We also include the certainty intelligibility type, instantiated
via the Prediction conﬁdence column (Figure 1, part B), which
reveals the classiﬁer’s conﬁdence in each of its predictions.
Not Overwhelming
To avoid overwhelming users, EluciDebug limits the initial set
of features available to the classiﬁer using information gain
[47]. Because Principle 1.3 states that explanations should
be as complete as possible, users should be able to see all
of the classiﬁer’s features. Given this constraint, we decided
50 would be the upper limit on feature set size. Ofﬂine tests,
however, revealed that MNB’s F 1score did not improve while
the feature set size increased from 10 to 50, so we decided our
classiﬁer would automatically select only the 10 features with
the highest information gain (until the user speciﬁes otherwise
by adding or removing features).
Being Actionable
The Important words explanation (Figure 3) is the primary
actionable explanation in EluciDebug. Users can add words
to—and remove words from—this explanation, which in turn
will add those words to (or remove them from) the machine
learning model’s feature set. Users are also able to adjust the
importance of each word in the explanation by dragging the
word’s bar higher (to make it more important) or lower (to
make it less important), which then alters the corresponding
feature’s weight in the learning model.
TheWhy explanation (Figure 2) is a likely candidate for action-
ability, but we have not yet made it actionable in EluciDebug.
As this explanation only includes features from the selected
message, it cannot replace the Important words explanation
because doing so would interfere with our explanations’ com-
pleteness. It could, however, complement the Important words
explanation by allowing users to directly adjust the importance
of features responsible for the given prediction. For example,
users could drag out from the center of a word to increase
its importance, or drag in toward the center of the word to
decrease its importance. Whether such additional actionability
would help users, however, remains an open question.Being Reversible
EluciDebug includes an undo button for reversing changes to
theImportant words explanation. There is no limit on how
many actions can be un-done because we want users to interact
with the system without fear they will harm its predictions;
regardless of how much they adjust its reasoning, they can
always return to any prior state.
Honoring User Feedback
EluciDebug allows users to provide two types of feedback:
traditional instanced-based feedback, where the user applies a
label 2to an entire item, and feature-based feedback, where the
user tells the classiﬁer an item should be labeled in a certain
manner because of speciﬁc features it contains or feature
values it matches. EluciDebug honors instance-based feedback
in a straightforward manner: once the user labels an item, the
classiﬁer will use it as part of its training set, with no distinc-
tion between older versus more recent feedback. Honoring
feature-based feedback, however, is more complicated.
The smoothing term pncfrom Equation 3 acts as a Bayesian
prior, effectively adding some number of virtual occurrences
(again, traditionally 1) to the number of actual occurrences
of each word in the training data, and we can leverage it to
integrate feature-based feedback. By allowing the user to set
the value of pnc, we are letting the user increase the number
of virtual occurrences of word nin class c. The result is a
classiﬁer that considers word nto be stronger evidence in
favor of class cthan it had before the user’s feedback.
Using the smoothing term as a feature-based feedback mech-
anism, however, has a drawback: Fncmay increase as the
training set size increases, causing the value of pncto become a
smaller component of Equation 3. Thus, a user’s feature-based
feedback could account for less and less of the classiﬁer’s
reasoning as their instance-based feedback increased.
To prevent a user’s feature-based feedback from degrading in
importance over time, we developed a visual feedback mech-
anism (Figure 3) that allows users to specify how important
their feedback should be relative to the model’s internal word
probabilities (the Fterms in Equation 3). The black lines on
the blue and green bars in Figure 3 show the model-computed
probabilities for each feature, which serve as a starting point
for feature-based feedback. Users can tell the system that the
probability of seeing word wnin class cshould be increased
by clicking and dragging its bar higher, which will translate to
an increased value for pnc. If the user later provides additional
instance-based feedback (thus causing Fncto change), pncwill
be automatically recalculated such that the ratios ofN
å
x=1pxcto
N
å
x=1FxcandpnctoFncremain constant.
Revealing Incremental Changes
There are many components of EluciDebug that may change
after each user action, so to avoid confusing users with several
2Alabel is a potential output of the learning system, such as junk mail or
normal mail.
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
131
Topic Modeling / Robotics
Figure 3. The Important words explanation tells users all of the
features the classiﬁer is aware of, and also lets users add, remove,
and adjust these features. Each topic is color-coded (here, blue
forhockey and green for baseball ) with the difference in bar
heights reﬂecting the difference in the word’s probability with
respect to each topic (e.g., the word canadian is roughly twice
as likely to appear in a document about hockey as one about
baseball, while the word player is about equally likely to appear
in either topic). This ﬁgure is an excerpt from Figure 1 part F.
different explanation paradigms, we designed a method that
consistently reveals changes in terms of increases andde-
creases. Increases of any numeric value are identiﬁed by green
“up” arrows, while decreasing numeric values are identiﬁed by
red “down” arrows. Examples of each are shown in Figure 1,
part B. Hovering over either of these arrow icons yields a
tooltip detailing what just changed and how much it changed
by, e.g., “Conﬁdence increased by 9%”. These indicators
reveal changes in the number of messages correctly classiﬁed
in each folder, the total number of messages the machine
learning model currently classiﬁed into each folder, and the
conﬁdence of each prediction.
In addition to numeric change indicators, we also needed
an ordinal change indicator to highlight when a message’s
prediction ﬂipped from one topic to the other. We used a grey
background for these recently-changed predictions (see part B
of Figure 1) and included a tooltip explaining that the user’s
last action resulted in the message’s predicted topic changing.
EVALUATION
We evaluated Explanatory Debugging, as instantiated in Eluci-
Debug, to investigate the following research questions:
RQ1: Does Explanatory Debugging help users personalize a
classiﬁer more efﬁciently than instance labeling?
RQ2: Does Explanatory Debugging help users personalize a
classiﬁer more accurately than instance labeling?
RQ3: Does Explanatory Debugging help users build better
mental models than a traditional black-box approach?
Experiment Design
We used a between-subject, single-factor experimental setup
to evaluate Explanatory Debugging. The factor we varied was
experiment condition: one condition (control ) used a variation
of EluciDebug with all of its explanation and feature-based
feedback capabilities removed (Figure 4), while the second
condition (treatment ) used the EluciDebug prototype described
earlier. In both conditions EluciDebug was setup as a binary
classiﬁer that attempted to predict whether each message
belonged to one of two topics.
Figure 4. Control participants used this variant of EluciDebug,
which lacks explanations and feature-based feedback.
To provide messages for EluciDebug, we selected two con-
cepts from the 20 Newsgroups dataset 3:hockey andbaseball
(the rec.sport.hockey and rec.sport.baseball newgroups, respec-
tively). We used two subgroups of a larger group of related
concepts (rec.sport) to ensure overlap in the terminology of
each concept (e.g., “player” and “team” may be equally rep-
resentative of baseball orhockey). These shared terms help
make the classiﬁcation task more challenging.
To simulate a situation where personalization would be re-
quired because sufﬁcient training data does not yet exist, we
severely limited the size of the machine learning training
set for this experiment. At the start of the experiment this
training set consisted of 5 messages in the Hockey folder
and 5 messages in the Baseball folder, with 1,185 unlabeled
messages in the Unknown folder. The small training set (10
messages) allowed us to evaluate a situation with limited
training data, and the large amount of potential training data
(1,185 messages) allowed us to contrast Explanatory Debug-
ging against black-box instance labeling in a situation where
instance labeling could be expected to succeed.
Participants and Procedure
We recruited 77 participants (27 females, 50 males) from the
local community and university. To ensure that participants
would have little or no prior experience with software debug-
ging or machine learning, we did not accept participants who
had more programming experience than an introductory-level
course. A total of 37 participants experienced the control con-
dition and 40 participants took part in the treatment condition.
A researcher introduced participants to the prototype via a
brief hands-on tutorial that explained how to use it, but did
not discuss how it made predictions. Participants then had
three minutes to explore the prototype on their own. To avoid
learning effects, the tutorial and practice session involved
messages about topics (outer space andmedicine) different
from the main experiment task.
The main experiment task followed the practice session. Par-
ticipants were asked to “make the computer’s predictions as
accurate as possible” and given 30 minutes to work. The soft-
ware logged all participant interactions and logged evaluations
of its internal classiﬁer at 30-second intervals.
3http://qwone.com/~jason/20Newsgroups/
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
132
Topic Modeling / Robotics
After the main task concluded, we assessed participants’ men-
tal models via a questionnaire. This test instrument evalu-
ated how well participants understood the two components
that contribute to the MNB classiﬁer’s predictions: feature
presence and class ratios. Because feature presence can be
easy to detect given certain words (e.g., the word “hockey”
is obviously related to the concept of hockey), we evaluated
participants’ understanding of feature presence using both
“obvious” and “subtle” features. We deﬁne “subtle” features
as words that are not normally associated with a topic, but
appear in the classiﬁer’s training set and thus will impact
classiﬁcation. Participants were given three short messages
about two topics (swimming andtennis) and told that “these are
the only messages the software has learned from”. Participants
in the treatment condition were also given an Important words
explanation similar to the one they saw in the prototype. A
second sheet displayed 12 messages, each only one or two
sentences long, and asked participants which topic the clas-
siﬁer would assign to each message, and why. The messages
were constructed such that only one component—either an
obvious feature, a subtle feature, or class ratios—was entirely
responsible for the classiﬁcation.
To understand participants’ reactions to EluciDebug, the post-
task questionnaire also asked participants about various fea-
tures of the prototype and their perceived task load during the
experiment (via the NASA-TLX questionnaire [15]).
Data analysis
We used non-parametric methods for all statistical analyses. As
suggested in [ 31], we used Mann–Whitney U-tests for interval
and ordinal data, and Spearman’s rfor correlations.
To analyze participants’ mental models, a researcher graded
participant responses to the post-task mental model question-
naires. Because some participants may have randomly guessed
which topic the classiﬁer would predict for each message,
we ignored all predicted topics and only graded the reasons
participants’ stated for the classiﬁer’s prediction. Participants
earned two points for correct reasons and one point for partially
correct reasons. The researcher graded participant responses
without knowing which condition the participant was in (i.e.,
blindly). Each participants’ points were summed to yield a
mental model score with a maximum possible value of 24.
We supplemented our evaluation of classiﬁer accuracy with an
additional ofﬂine experiment using a separate feature selection
method. Recall that EluciDebug limits its classiﬁer to the 10
features with the highest information gain. Text classiﬁers,
however, often include most—if not all—of the words in the
training set as features. Thus, we analyzed participants’ classi-
ﬁers using both HighIG features and Comprehensive features.
For control participants (who could not provide feature-based
feedback), HighIG was recomputed after each message was
labeled and kept the 10 highest information gain features. For
treatment participants, HighIG was never recomputed; instead,
participants needed to modify it manually by adding, removing,
or adjusting features. The Comprehensive feature set included
all words from the set of labeled messages at the end of theexperiment. The classiﬁers participants interacted with used
theHighIG features; the Comprehensive features were only
used for ofﬂine analysis.
RESULTS
Explaining Corrections to EluciDebug
EluciDebug includes several methods for users to explain
corrections to the classiﬁer, and treatment participants made
frequent use of all of them. On average, they added 34.5 new
features, removed 8.2 features (including 7.4 of EluciDebug’s
10 initial features), and made 18.3 feature adjustments (e.g.,
increasing or decreasing a feature’s importance to a topic).
Control participants—who could not provide feature-based
feedback—instead relied on instance-based feedback to adjust
EluciDebug’s predictions, labeling an average of 182 messages
vs. the treatment average of 47 ( W=1395 ,p< :001). Control
participants also examined more messages, averaging 296
message views vs. 151 views ( W=1306 ,p< :001). Treatment
participants thus provided less feedback overall (and needed
to explore less of the dataset while providing it), instead
leveraging EluciDebug’s abilities to target their feedback at
features rather than instances.
This feature-based feedback proved efﬁcient at improving
participants’ classiﬁers. We examined the change in F 1for
each participant’s classiﬁer during the experiment and divided
this by the number of actions the participant made that could
inﬂuence the classiﬁer’s predictions (instances labeled and
features added, removed, or adjusted). The results, shown in
Figure 5 (left), were that treatment participants performed
fewer actions, but each of their actions resulted in larger
classiﬁer improvements than those of control participants.
Treatment participants’ feedback was twice as efﬁcient as
control participants’ using HighIG features (0.16% vs. 0.34%
F1improvement per action, W=207,p< :001), and remained
superior when using the Comprehensive feature set (0.65% vs.
0.97%, W=367, p< :001).
We thus have evidence that when users can only provide a
limited amount of feedback to a learning system (such as
when labeling instances is expensive, insufﬁcient instances
are available for labeling, or the user’s time is constrained),
Explanatory Debugging can result in better classiﬁers than
a traditional black-box instance labeling approach. Indeed,
Figure 5 (right) shows that by the end of the 30-minute ex-
periment, treatment participants had created classiﬁers that
were roughly 10% more accurate than control participants,
averaging F 1scores of 0.85 vs. 0.77 ( W=237, p< :001).
However, our analysis with the Comprehensive feature set
suggests that when the user can label many instances, instance
labeling with a large feature set may be preferable to Ex-
planatory Debugging—at least to initially train a classiﬁer.
The combination of a large training set and many features
allowed control participants’ classiﬁers to edge out those of
treatment participants by about 8% (Figure 5, right). Even
though treatment participants’ feedback was up to twice as
efﬁcient, control participants provided almost four times as
many labeled instances, allowing them to train classiﬁers with
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
133
Topic Modeling / Robotics
0.0%0.5%1.0%
HighIG Comp.F1 change
0.00.51.0
HighIG Comp.F1 scoreFigure 5. (Left) Average classiﬁer F 1improvement per user
action for control (dark blue) and treatment (light orange);
treatment participants controlled their classiﬁers up to twice as
efﬁciently as control participants. (Right) Average classiﬁer F 1
scores per condition. Control participants needed four times as
much data and the Comprehensive feature set to create better
classiﬁers than treatment participants.
an average F 1of 0.94, while treatment participants averaged
0.87 ( W=1290, p< :001).
To verify it was the amount of instance-based feedback that
allowed control participants to outperform treatment partic-
ipants when Comprehensive features were considered, we
analyzed the accuracy of their classiﬁers after the same number
of actions had been performed. Figure 6 shows the F 1scores
after nfeedback actions using the HighIG (solid line) and Com-
prehensive (dotted line) feature sets. Given the same number
of actions, control participants never outperformed treatment
participants. This suggests that when treatment participants
did provide instance-based feedback (which was the only type
of feedback used for the Comprehensive analysis), it was often
more useful than control participants’ feedback.
We also analyzed participant reactions to the two prototype
variations. Treatment participants liked their variant more than
control participants, rating its helpfulness as 4.8 vs. 4.3 on a
6-point scale ( W=474,p=:006). Further, we found no evi-
dence that treatment participants felt Explanatory Debugging
involved more work than black-box instance labeling. We used
the NASA-TLX survey to measure participants’ perceived task
load while attempting to improve their classiﬁer, but found no
evidence of a difference between conditions.
These classiﬁer measures reveal three ﬁndings. First, in situ-
ations where large amounts of training data is unavailable or
expensive to obtain, Explanatory Debugging (as instantiated in
EluciDebug) allows users to successfully train a classiﬁer by
telling it about features instead of instances. Second, the mix
of feature- and instance-based feedback provided by treatment
participants was more efﬁcient than the purely instance-based
feedback provided by control participants, suggesting that
when an end user has a speciﬁc goal in mind (such as our
Alice example from earlier), Explanatory Debugging can help
the user quickly realize their goal.
Third, control participants’ success with using large amounts
of instance-based feedback suggests that in domains where
labeling instances is quick and practical, some combination
of feature- and instance-based feedback may be best. In fact,
such systems may need to emphasize the potential usefulness
of labeling instances. In our experiment, the mere presence of
feature-based feedback tools appears to have biased partici-
pants against instance-based feedback: 3 treatment participants
did not provide any at all, while the smallest number of labeled
0.50.60.70.80.91.0
0102030405060708090100F1 score
# of feedback actionsFigure 6. Treatment participants (light orange) created equiv-
alent or better classiﬁers than control participants (dark blue)
using the same amount of feedback. This held for both the
HighIG (solid) and Comprehensive (dotted) feature sets.
instances from a control participant was 56—more than even
the treatment average of 47 labeled instances.
EluciDebug’s Explanations to End Users
Before users can correct a machine learning system’s explana-
tion of its reasoning, they must ﬁrst understand the explanation.
This understanding is reﬂected in their mental model.
Treatment participants built signiﬁcantly better mental models
than participants in the control condition. As shown in Table 1,
treatment participants scored 52% higher on the mental model
assessment than control participants ( W=259,p< :001).
Much of this difference stems from treatment participants iden-
tifying allof the keywords the classiﬁer used, while control
participants often identiﬁed only the “obvious” keywords. In
fact, treatment participants averaged a score of 14.1 out of 16
(88%) during the keyword assessment, suggesting they ﬁrmly
understood how the classiﬁer involved keywords—regardless
of whether the words had any semantic association with their
topics—in its reasoning.
Table 1 also suggests treatment participants better understood
that the classiﬁer used class ratios as part of its prediction
strategy than participants in the control condition ( W=619:5,
p=:099), but the evidence is weak. Further, a majority of
participants in both conditions failed to answer anyclass ratio
question correctly, suggesting that this explanation either failed
to convey relevant information about how class ratios were
used by the classiﬁer, or failed to attract participants’ attention.
Many control participants wanted the same information avail-
able to the treatment group. As one participant stated:
C1: “More information on how emails are sorted would
help the user target emails to categorize, which would
increase accuracy. ”
Another control participant (C15) described the software as
“annoying”, but that working with it “would have been easier
if we knew how it made predictions”, while still another (C4)
said it was annoying to work with because he “didn’t know
what it was basing its predictions off of”. A fourth participant
(C11) even asked for a similar feedback mechanism as was
available to treatment participants, saying the software was
“time-consuming” because there was “no way to highlight key
words/terms”. A ﬁfth control participant summed it up:
C30: “That was a long 30 minutes. ”
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
134
Topic Modeling / Robotics
Model
componentMax
scoreControl
mean (SD)Treatment
mean (SD)p-value
Obvious features 8 6.7 (2.7) 7.3 (1.8) .345
Subtle features 8 2.8 (2.6 ) 6.8 (1.9) <.001
Class ratios 8 0.6 (1.5) 1.8 (3.0) .099
Total score 24 10.4 (5.3) 15.8 (4.6) <.001
Table 1. Treatment participants ﬁnished the experiment with sig-
niﬁcantly higher mental model scores than control participants.
Participants in the treatment condition, conversely, appreciated
EluciDebug’s explanations and feedback mechanisms:
T24: “Not difﬁcult to understand/operate, doesn’t take a
lot of effort. ”
T40: “It was really fast to get a high degree of accuracy. ”
T37: “I felt in control of all the settings. ”
T6: “It was so simple my parents could use it. ”
Overall, our principled Explanatory Debugging approach suc-
cessfully helped participants develop accurate mental models
of the classiﬁer they used, and participants beneﬁted from this
additional knowledge. Spearman’s rconﬁrms a link between
participants’ mental model scores and their classiﬁer’s F 1
scores (r [75]=:282, p=:013)
DISCUSSION
RQ1 and RQ2: Efﬁcient and Accurate Personalization
Our results suggest that Explanatory Debugging can be an
efﬁcient method for users to personalize a machine learning
system, but that it may not always result in the most accurate
classiﬁers. For example, we found that feature-based feedback
was up to twice as effective as instance-based feedback, but
instance labeling could still yield more accurate classiﬁers
given enough labels and features (in our experiment, four times
as many labels were needed). In situations where labeling
instances is considerably easier or faster than providing feature-
based feedback, users may be better served by labeling a large
number of instances than a small number of features.
However, when users need ﬁne-grained control over a clas-
siﬁer, Explanatory Debugging has two advantages beyond
efﬁciency. First, it does not require much training data to exist,
and can thus help bootstrap a learning system. Even a trained
system that suddenly needs to support a new output type may
beneﬁt from such bootstrapping (this also separates our work
from knowledge-based artiﬁcial intelligence, which is not
retrained post-deployment). Second, the quick improvements—
during even the ﬁrst 10 user actions—treatment participants
made to their classiﬁers suggest that users will remain engaged
with Explanatory Debugging. This matters because research
has shown that if an end-user debugging technique is not
perceived as useful after a small number of interactions, users
are unlikely to continue using it [ 34]. Seeing an immediate
improvement after providing feedback suggests that users
will continue to view and correct the Explanatory Debugging
explanations, while the lack of such an improvement may
discourage users of black-box instance labeling systems from
continuing to provide feedback.RQ3: Mental Models
Explanatory Debugging helped participants build useful men-
tal models, and did so without a perceived increase in task
load. We found no evidence that treatment participants found
the extra information or feedback mechanisms more difﬁcult
to understand or use; instead, their responses suggest they
appreciated having such information available.
Indeed, the fact that many control participants’ requested
explanations similar to those the treatment participants saw
suggests the need for machine learning systems to be able
to explain their reasoning in accordance with Explanatory
Debugging’s Explainability principle. Even if this information
is hidden by default, users should be able to view such explana-
tions on demand. Further, because machine learning systems
are meant to classify items as their user would, the user must
have some method to correct the system’s mistakes. Thus, we
hypothesize that including Explanatory Debugging-style ex-
planations without also supporting our Correctibility principle
will frustrate users—they would see what needs correcting,
but be powerless to ﬁx it.
CONCLUSION
Overall, Explanatory Debugging’s cycle of explanations—
from the learning system to the user, and from the user back
to the system—resulted in smarter users and smarter learning
systems. Participants using Explanatory Debugging under-
stood how the learning system operated about 50% better
than control participants, and this improvement correlated
with the F 1scores of participants’ classiﬁers. Each piece of
feedback provided by Explanatory Debugging participants
was worth roughly two pieces of feedback provided by con-
trol participants; even when we expanded our analysis to
include a comprehensive feature set, treatment participants
still maintained a 50% edge over the control group. Further,
participants liked Explanatory Debugging, rating this variant
of EluciDebug higher than the control group and responding
enthusiastically to the system’s explanations.
This work is but one step toward helping users control machine
learning systems. How Explanatory Debugging may be applied
to more complex problems, such as multiclass classiﬁers or
deep learning algorithms, remains an open question. Future
work could also investigate how Explanatory Debugging im-
pacts user interactions over longer time periods and whether
end users are able to transfer knowledge of one learning system
to more quickly personalize similar systems.
Our results show that when end users want to personalize a
machine learning system, Explanatory Debugging is a more
controllable and satisfying approach than black-box instance
labeling. The approach’s focus on users—in which the system
explains its reasoning and the user explains back corrections
as needed—enables ordinary end users to get the most beneﬁt
from learning systems on which they depend.
ACKNOWLEDGMENTS
We thank Irwin Kwan, Charles Hill, Leah Hanen, Renuka
Bhatt, and our study participants for their help in conducting
this research.
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
135
Topic Modeling / Robotics
REFERENCES
1.Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T.
Power to the people: The role of humans in interactive
machine learning. AI Magazine (in press).
2.Amershi, S., Fogarty, J., Kapoor, A., and Tan, D.
Examining multiple potential models in end-user
interactive concept learning. In Proceedings of the ACM
Conference on Human Factors in Computing Systems
(2010), 1357–1360.
3. Blackwell, A. F. First steps in programming: A rationale
for attention investment models. In Proceedings of the
IEEE 2002 Symposia on Human Centric Computing
Languages and Environments, IEEE (2002), 2–10.
4.Bostandjiev, S., O’Donovan, J., and Höllerer, T.
TasteWeights: A visual interactive hybrid recommender
system. In Proceedings of the 6th ACM Conference on
Recommender Systems (2012), 35–42.
5.Bryan, N. J., Mysore, G. J., and Wang, G. ISSE: An
interactive source separation editor. In Proceedings of
the ACM Conference on Human Factors in Computing
Systems (2014), 257–266.
6.Bunt, A., Lount, M., and Lauzon, C. Are explanations
always important? A study of deployed, low-cost
intelligent interactive systems. In Proceedings of the
2012 ACM International Conference on Intelligent User
Interfaces (2012), 169–178.
7.Cakmak, M., Chao, C., and Thomaz, A. L. Designing
interactions for robot active learners. IEEE Transactions
on Autonomous Mental Development 2, 2 (2010),
108–118.
8.Cramer, H., Evers, V ., Ramlal, S., van Someren, M.,
Rutledge, L., Stash, N., Aroyo, L., and Wielinga, B. The
effects of transparency on trust in and acceptance of a
content-based art recommender. User Modeling and
User-Adapted Interaction 18, 5 (2008), 455–496.
9.Craven, M. W., and Shavlik, J. W. Using neural networks
for data mining. Future generation computer systems 13
(1997), 211–229.
10.Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce,
L. G., and Beck, H. P. The role of trust in automation
reliance. International Journal of Human-Computer
Studies 58, 6 (2003), 697–718.
11.Fails, J. A., Olsen, D. R., and Jr. Interactive machine
learning. In Proceedings of the 8th International
Conference on Intelligent User Interfaces (2003), 39–45.
12.Fiebrink, R., Cook, P. R., and Trueman, D. Human
model evaluation in interactive supervised learning. In
Proceedings of the ACM Conference on Human Factors
in Computing Systems (2011), 147–156.
13.Fogarty, J., Tan, D., Kapoor, A., and Winder, S. CueFlik:
Interactive concept learning in image search. In
Proceedings of the ACM Conference on Human Factors
in Computing Systems (2008), 29–38.
14.Groce, A., Kulesza, T., Zhang, C., Shamasunder, S.,
Burnett, M. M., Wong, W.-K., Stumpf, S., Das, S.,
Shinsel, A., Bice, F., and McIntosh, K. You are the onlypossible oracle: Effective test selection for end users of
interactive machine learning systems. IEEE Transactions
on Software Engineering 40, 3 (2014), 307–323.
15.Hart, S. G., and Staveland, L. E. Development of
NASA-TLX (Task Load Index): Results of empirical and
theoretical research. Advances in Psychology 52 (1988),
139–183.
16.Herlocker, J. L., Konstan, J. A., and Riedl, J. Explaining
collaborative ﬁltering recommendations. In Proceedings
of the 2000 ACM Conference on Computer Supported
Cooperative Work (2000), 241–250.
17.Johnson-Laird, P. N. Mental models: Towards a cognitive
science of language, inference, and consciousness.
Harvard University Press, 1983.
18.Kapoor, A., Lee, B., Tan, D., and Horvitz, E. Interactive
optimization for steering machine classiﬁcation. In
Proceedings of the ACM Conference on Human Factors
in Computing Systems (2010), 1343–1352.
19.Kempton, W. Two theories of home heat control.
Cognitive Science 10 (1986), 75–90.
20.Kibriya, A. M., Frank, E., Pfahringer, B., and Holmes, G.
Multinomial naive Bayes for text categorization revisited.
InAI 2004: Advances in Artiﬁcial Intelligence. Springer
Berlin Heidelberg, Berlin, Heidelberg, 2004, 488–499.
21.Knox, W. B., and Stone, P. Reinforcement learning
from human reward: Discounting in episodic tasks. In
Proceedings of the 21st IEEE International Symposium
on Robot and Human Interactive Communication (2012),
878–885.
22.Ko, A. J., and Myers, B. A. Debugging reinvented. the
13th international conference (2008), 301–310.
23.Kulesza, T., Stumpf, S., Burnett, M. M., and Kwan, I.
Tell me more? The effects of mental model soundness
on personalizing an intelligent agent. In Proceedings of
the ACM Conference on Human Factors in Computing
Systems (2012), 1–10.
24.Kulesza, T., Stumpf, S., Burnett, M. M., Wong, W.-K.,
Riche, Y ., Moore, T., Oberst, I., Shinsel, A., and
McIntosh, K. Explanatory debugging: Supporting
end-user debugging of machine-learned programs. In
Proceedings of the 2010 IEEE Symposium on Visual
Languages and Human-Centric Computing (2010),
41–48.
25.Kulesza, T., Stumpf, S., Burnett, M. M., and Yang, S. Too
much, too little, or just right? Ways explanations impact
end users’ mental models. Proceedings of the 2013 IEEE
Symposium on Visual Languages and Human-Centric
Computing (2013), 3–10.
26.Kulesza, T., Stumpf, S., Wong, W.-K., Burnett, M. M.,
Perona, S., Ko, A. J., and Oberst, I. Why-oriented
end-user debugging of naive Bayes text classiﬁcation.
ACM Transactions on Interactive Intelligent Systems 1, 1
(2011).
27.Lacave, C., and Díez, F. J. A review of explanation
methods for Bayesian networks. The Knowledge
Engineering Review 17, 2 (2002), 107–127.
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
136
Topic Modeling / Robotics
28.Lim, B. Y . Improving understanding and trust with
intelligibility in context-aware applications. PhD thesis,
Carnegie Mellon University, 2012.
29.Lim, B. Y ., and Dey, A. K. Assessing demand
for intelligibility in context-aware applications. In
Proceedings of the 11th International Conference on
Ubiquitous Computing (2009), 195–204.
30.Lim, B. Y ., Dey, A. K., and Avrahami, D. Why and
why not explanations improve the intelligibility of
context-aware intelligent systems. In Proceedings of
the ACM Conference on Human Factors in Computing
Systems (2009), 2119–2128.
31.McCrum-Gardner, E. Which is the correct statistical test
to use? British Journal of Oral and Maxillofacial Surgery
46(2008), 38–41.
32.Norman, D. A. Some observations on mental models.
InHuman-Computer Interaction, R. M. Baecker and
W. A. S. Buxton, Eds. San Francisco, CA, USA, 1987,
241–244.
33.Norman, D. A. The Design of Everyday Things. Revised
and Expanded Edition. Basic Books, 2002.
34.Prabhakararao, S., Cook, C., Ruthruff, J., Creswick, E.,
Main, M., Durham, M., and Burnett, M. M. Strategies and
behaviors of end-user programmers with interactive fault
localization. In Proceedings of the IEEE Symposium on
Human Centric Computing Languages and Environments
(2003), 15–22.
35.Rowe, M. B. Teaching science as continuous inquiry.
McGraw-Hill, 1973.
36.Rugaber, S., Goel, A. K., and Martie, L. GAIA: A
CAD environment for model-Based Adaptation of
game-playing Software Agents. Procedia Computer
Science 16 (2013), 29–38.
37.Settles, B. Active learning literature survey. Tech. Rep.
1648, University of Wisconsin-Madison, 2010.
38.Stumpf, S., Rajaram, V ., Li, L., Wong, W.-K., Burnett,
M. M., Dietterich, T., Sullivan, E., and Herlocker,
J. Interacting meaningfully with machine learningsystems: Three experiments. International Journal of
Human-Computer Studies 67, 8 (2009), 639–662.
39.Szafron, D., Greiner, R., Lu, P., and Wishart, D.
Explaining naïve Bayes classiﬁcations. Tech. Rep.
TR03-09, University of Alberta, 2003.
40.Talbot, J., Lee, B., Kapoor, A., and Tan, D. S.
EnsembleMatrix: Interactive visualization to support
machine learning with multiple classiﬁers. In Proceedings
of the ACM Conference on Human Factors in Computing
Systems (2009), 1283–1292.
41.Thomaz, A. L., and Breazeal, C. Transparency and
socially guided machine learning. In Proceedings of
the 5th International Conference on Development and
Learning (2006).
42.Tintarev, N., and Masthoff, J. Evaluating the effectiveness
of explanations for recommender systems. User Modeling
and User-Adapted Interaction 22, 4-5 (2012), 399–439.
43.Tullio, J., Dey, A. K., Chalecki, J., and Fogarty, J. How
it works: A ﬁeld study of non-technical users interacting
with an intelligent system. In Proceedings of the ACM
Conference on Human Factors in Computing Systems
(2007), 31–40.
44.van der Meij, H., and Carroll, J. M. Principles and
heuristics for designing minimalist instruction. In
Minimalism beyond the Nurnberg funnel, J. M. Carroll,
Ed. MIT Press, Cambridge, MA, 1998, 19–53.
45.Vig, J., Sen, S., and Riedl, J. Navigating the tag genome.
InProceedings of the 16th International Conference on
Intelligent User Interfaces (2011), 93–102.
46.Yang, R., and Newman, M. W. Learning from a learning
thermostat: Lessons for intelligent systems for the home.
InProceedings of the 2013 ACM International Joint
Conference on Pervasive and Ubiquitous Computing
(2013), 93–102.
47.Yang, Y ., and Pedersen, J. O. A comparative study on
feature selection in text categorization. In Proceedings
of the Twentieth International Conference on Machine
Learning (1997), 412–420.
IUI 2015 • Interactive Machine Learning / Decision Making /
March 29–April 1, 2015, Atlanta, GA, USA
137
Topic Modeling / Robotics

Todd Kulesza
Margaret Burnett
Weng-Keen Wong
Simone Stumpf


Transparency and Socially Guided Machine Learning
Andrea L. Thomaz, Cynthia Breazeal
MIT Media Laboratory
20 Ames St. E15-468
alockerd@media.mit.edu
Abstract—In this paper we advocate a paradigm of socially
guided machine learning, designing agents that take better
advantage of the situated aspects of learning. We augmented
a standard Reinforcement Learning agent with the social
mechanisms of attention direction and gaze. Experiments with
an interactive computer game, deployed over the World Wide
Web to over 75 players, show the positive impact of these social
aspects. Allowing the human to direct the agent’s attention
creates a more efﬁcient exploration strategy. Additionally, gaze
behavior lets the learning agent improve its own learning envi-
ronment, using transparency to steer the human’s instruction.
I. INTRODUCTION
Robots that operate in the human environment will need
to be able to learn new skills and tasks ‘on the job’ from
people. It is important to recognize that while the average
consumer is not familiar with machine learning techniques,
they are intimately familiar with various forms of social
learning (e.g., tutelage, imitation). From a Human-Robot
Interaction perspective, this raises two important and related
questions. First, how do people want to teach robots? Second,
how do we design robots that learn effectively from natural
human interaction and instruction? We advocate a paradigm
ofsocially guided machine learning , exploring the ways in
which machine learning can be designed to more fully take
advantage of natural human interaction and tutelage.
We draw our inspiration from Situated Learning Theory, a
ﬁeld of study that looks at the social world of children and
how it contributes to their development. A key concept is
scaffolding , where a teacher provides support such that a
learner can achieve something they would not be able to
accomplish independently [14], [11].
In a situated learning interaction, the teaching and learning
processes are intimately coupled. A goodinstructor maintains
an good mental model of the learner’s state (e.g., what
is understood so far, what remains confusing or unknown)
in order to provide appropriate scaffolding to support the
learner’scurrentneeds.Inparticular,attentiondirectionisone
of the essential mechanisms that contribute to structuring the
learning process [22]. Other scaffolding acts include provid-
ing feedback, structuring successive experiences, regulating
the complexity of information, and otherwise guiding the
learner’s exploration. In general, this is a complex process
where the teacher dynamically adjusts their support based on
the learner’s demonstrated skill level and success.The learner, in turn, helps the instructor by making their
learning process transparent to the teacher through commu-
nicative acts (such as facial expressions, gestures, gaze, or
vocalizations that reveal understanding, confusion, attention),
and by demonstrating their current knowledge and mastery of
the task [13], [1]. Through this reciprocal and tightly coupled
interaction, the learner and instructor cooperate to simplify
thetaskfortheother—makingeachamoreeffectivepartner.
This paper investigates the ways in which the social as-
pects of learning and the dynamics of the teacher-learner
interaction can beneﬁcially impact the performance of a
machine learning agent. We use a computer game platform
as our experimental testbed, Sophie’s Kitchen , where players
interactively train a virtual robot character to perform a task.
We report two interrelated results that show how social
mechanisms can be added to a standard learning process
to improve both sides of the teaching-learning partnership.
Speciﬁcally, we report that
•The ability for the teacher to direct the agent’s atten-
tion has signiﬁcant positive effects on several learning
performance metrics.
•The ability of the agent to use gaze as a transparency
behavior results in measurably better human instruction.
These empirical results illustrate that the ability to use and
leverage social skills is far more than a good interface tech-
nique.Itcanpositivelyimpactthedynamicsoftheunderlying
learning mechanisms to show signiﬁcant improvements in a
real-time interactive learning session.
II. FRAMEWORK
The situated learning process described above stands in
dramatic contrast to typical machine learning scenarios that
have traditionally ignored “teachability issues” such as how
to make the teaching-learning process interactive or intuitive
for a human partner. We advocate a new perspective that
reframes the machine learning problem as an interaction
between the human and the machine. Fig. 1 sketches this
distinction. Typically, the machine learning community has
focused on Fig. 1(a): a human provides input to the learning
mechanism, which performs its task and provides an output
model or classiﬁcation function.
Our social learning perspective models the complete human-
machine system, Fig. 1(b). Adding transparency and aug-
menting the human input to include guidance, introduce key
aspects of social learning, highlighting the reciprocal nature
of the teaching-learning partnership. We need a principled
theory of the content and dynamics of this tightly coupled
process in order to design systems that can learn efﬁciently
and effectively from ordinary users. This approach challenges
the research community to consider many new questions.
A. How can the input channels available to the human
improve the performance of the teaching-learning system?
We can change the input portion of the typical machine
learning process in many ways. Traditionally in supervised
learning, training happens off-line where many examples
are input in batch. However, a situated teaching interaction
may provide beneﬁts over this disembodied, out-of-context
method. For instance, a situated learner can take advantage
of the natural social cues that the human partner will use
(e.g.,referencing,attentiondirection)whichmaysigniﬁcantly
reduce the size of the input space for the machine.
It is important to understand the many ways that natural
human social cues can frame the input for a standard machine
learning process. This paper explicitly examines the effect of
allowing the human to guide the attention of a learner and
to provide feedback during its exploration process.
B. How can the output channels of the learning agent
improve performance of the teaching-learning system?
The output channels of the learner should also take advantage
of the situated nature of learning. In a tightly coupled
interaction, a “black box” learning process does nothing
to improve the quality and relevance of the instruction.
However, transparency of the internal state of the machine
could greatly improve the learning experience. By revealing
what is known and what is unclear, the machine can guide the
teaching process. To be most effective, the machine should
use cues that will be intuitive for the human partner [5],
[2]. For instance, facial expression, eye gaze, and behavior
choices are a signiﬁcant part of this output channel.
Understanding how both expression and behavior can com-
municate appropriate levels of the internal state of the learn-
ing process is an important issue at hand. We explicitly
examines a robot’s use of gaze as a transparency behavior.
C. How can the temporal dynamics of input and output
improve performance of the teaching-learning system?
Finally we must recognize that these input and output chan-
nels interact over time. The dynamics of the interaction
can change the nature of the input from the human. An
MachineLearningML TheoryTraditional Supervised LearningInputOutput(a)
MachineLearningML TheorySocially Guided ML TheoryInput/GuidanceOutputTransparency (b)
Fig. 1. 1(a) is a standard view of a supervised learning process, analyze
input and output a model or classiﬁer, etc. Our approach has the viewpoint
of 1(b), including the human teacher, emphasizing that teaching-learning is a
two-way process. We add transparency, where the machine learner provides
feedback to the human teacher during the learning process; and we augment
the human input with guidance. We aim to enhance the performance of the
tightly coupled partnership of a machine learner with a human teacher.
incremental, on-line learning system creates a very different
experience for the human than a system that must receive a
full set of training examples before its performance can be
evaluated. The ability to gauge the system’s relative level
of knowledge in various situations may help the trainer
pick “better” examples for the system. Further, a transparent
learner may help a non-expert human teacher answer a
difﬁcult question: when is learning ﬁnished?
This paper examines how the robot’s gaze as a transparency
behavior interacts with the human’s ability to provide guid-
ance during the exploration process, and together how they
impact learning performance.
III. APPROACH
In order to investigate these social aspects of machine learn-
ing, we built a computer game platform that allows us to
observe people teaching an interactive game character. We
deployedthegameontheWorldWideWebandcollecteddata
from over 75 people playing the game. This data allows us to
analyze the effects of two important social modiﬁcations: the
ability of the human to direct the agent’s attention, and the
ability of the agent to use gaze as a transparency behavior.
A. Platform
Sophie’s Kitchen is a Java-based web application that enables
the collection of a large amount of human player data.
Sophie’s Kitchen is an object-based State-Action MDP space
for a single agent that uses a ﬁxed set of actions on a ﬁxed
set of objects. The task scenario is a kitchen world (Fig. 2),
where the agent learns to bake a cake; the agent ﬁrst has to
learn to prepare batter for a cake and then to put the batter
in the oven. This is a fairly complex task with on the order
of 10,000 states and 2 to 7 actions available in each state.
Fig. 2. This is Sophie’s Kitchen : a brick oven on the left, table in the
center, shelf on the right, and the ﬁve objects used in the cake baking task.
(a) Feedback message.
 (b) Guidance message.
Fig. 3. 3(a), feedback involves left-clicking and dragging the mouse to
make a green bar (positive) or red bar (negative). 3(b), guidance involves
right-clicking an object of attention, selecting it with the yellow square.
The world has ﬁve objects: Flour,Eggs, aSpoon, a
Bowl(with ﬁve object states: empty,flour,eggs,
unstirred ,stirred ), and a Tray(with three object
states:empty,batter,baked). The world has four loca-
tions:Shelf,Table,Oven,Agent(i.e., the agent in the
center surrounded by a shelf, table and oven).
Sophie has four base actions with arguments as follows: the
locations are arranged in a ring, the agent can GO left or
right; she can PICK-UP any object that is in her current
location; she can PUT-DOWN any object in her possession;
and she can USEany object in her possession on any object
in her current location. Each action advances the world state.
For example, executing PICK-UP <Flour> changes the
state of the world such that the Flouris in location Agent.
The agent can hold only one object at a time. USEing an
ingredient on the Bowlputs that ingredient in it; using the
Spoonon the unstirred Bowl transitions its state to
stirred , etc.
Intheinitialstate,allobjectsareonthe Shelf,andtheagent
faces the Shelf. A successful task completion includes
putting ﬂour and eggs in the bowl, stirring the ingredients
using the spoon, transferring the batter into the tray, and
putting the tray in the oven.
B. Learning algorithm
Our goal with this experimental platform is to show the dif-
ferential effects of social mechanisms on a standard, widely
understood, machine learning algorithm. Our primary interestis in the on-line scenario of learning via self-generated expe-
rience over time. Thus we choose as our standard algorithm
the Reinforcement Learning (RL) paradigm. The algorithm
we implemented for the experiment presented in this paper
is a standard Q-Learning algorithm (learning rate α=.3 and
discount factor γ=.75) [21].
The goal state ( trayin theoven,baked) has a positive
reward (r=1), every other state has an inherent small
negative reward ( r=−.04). Also, some end states are so-
calleddisasterstates in that there is no way to recover from
them (for example—putting the eggsin theoven). These
result in a negative reward ( r=−1), the termination of the
current trial episode, and a transition to the initial state.
C. Feedback Interface
Sophie’s Kitchen has an interactive reward interface. Using
themouse,ahumantrainercan—atanypointintheoperation
of the agent—award a scalar reward signal r∈[−1,1].
This reward is in addition to the environmental rewards
just described. The user receives visual feedback enabling
them to tune the reward signal before sending it to the
agent. Choosing and sending the reward does not halt the
progress of the agent, which runs asynchronously to the inter-
active human reward. Abstractly, the Q-Learning algorithm
is continually going through the loop: select-action ,
take-action ,sense-reward ,update-values . We
introduce a delay in the sense-reward step to give the
human time to provide feedback.
D. Guidance Interface
We want to explore the effects of allowing the teacher
to direct the attention of the agent; giving the human the
ability to directly inﬂuence the action selection and bias
the exploration strategy. To accomplish this, we added a
separate guidance channel of communication. Clicking the
right mouse button draws an outline of a yellow square.
When the yellow square is administered on top of an object,
this communicates a guidance message to the learning agent
where the content of the message is the object. Figure 3(b)
shows the player guiding Sophie to pay attention to the bowl.
To incorporate guidance, we modify the Q-Learning al-
gorithm, introducing a pre-action phase. In the pre-action
phase, the agent registers guidance communication to bias
action selection; in the post-action phase the agent uses the
reward channel in the standard way to evaluate that action
and update a policy. Thus, the learning process becomes:
sense-guidance ,select-action ,take-action ,
sense-reward ,update-values .
The agent waits for guidance messages during the
sense-guidance step (we introduce a short delay to
allow the teacher time to administer guidance). A guidance
message is in the form guidance[object] ; the agent
saves this object as the guidance-object . During
theselect-action step, the default behavior (a standard
approach) chooses randomly between the set of actions with
the highest Q-values, within a bound β. If a guidance mes-
sage was received, the agent will insteadchoose randomly
between the set of actions that use the guidance-object .
E. Gazing Behavior
We also explore gaze as a means of making the learning
process more transparent to the human teacher. Gaze requires
that the learning agent have a physical/graphical embodiment
that can be understood by the human as having a forward
heading. In general, gaze precedes an action and communi-
cates something about the action that is going to follow.
Recall our guidance Q-Learning loop: sense-guidance ,
select-action ,take-action ,sense-reward ,
update-values . The gaze behavior executes during the
sense-guidance phase. The learning agent ﬁnds the set
of actions, A, with the highest Q-values, within a bound
β.∀a∈A, the learning agent gazes for 1 second at the
object-of-attention ofa(if it has one). This gazing
behavior during the pre-action phase communicates a level
of uncertainty through the amount of gazing that precedes
an action. It introduces an additional delay (proportional
to uncertainty) prior to select-action , both soliciting
and providing the opportunity for guidance messages from
the human. This also communicates overall task certainty
or conﬁdence as the agent will speed up when every
set,A, has a single action. We expect this transparency
to improve the teacher’s model of the learner, creating a
more understandable interaction for the human and a better
learning environment for the agent.
F. Experimental Design
We deployed the Sophie’s Kitchen game on the World Wide
Web. Participants were asked to play a computer game,
in which their goal was to get the virtual robot to learn
how to bake a cake on her own. Participants were told
they could not tell Sophie what actions to do, nor could
they do any actions directly. They were only able to send
Sophie various messages with the mouse to help her learn
the task. Depending on their test condition, subjects were
given instructions on administering feedback and guidance.
Each of the 75 participants, played the game once in one of
the following three test conditions:
•Feedback : Players used only the feedback communi-
cation channel.•Guidance : Players had both the feedback and the
guidance channels of communication.
•Gaze-guide : Players had the feedback and guidance
channels. Additionally, the agent used the gaze behavior.
The system maintained an activity log and recorded time
step and real time of each of the following: state transitions,
actions, human rewards, guidance messages and objects, gaze
actions, disasters, and goals. We then analyzed these game
logs to test the following two hypotheses:
•Hypothesis 1 : Teachers can use attention direction as a
form of guidance, to improve a learning interaction.
•Hypothesis 2 : Learners can help shape their learning
environment by communicating aspects of the internal
process, particularly that the gaze behavior will improve
a teacher’s guidance instruction.
IV. RESULTS
A. Guidance Improves Learning
To evaluate the effects of the guidance feature we compare
the game logs from players in the guidance condition to
those in the feedback condition with a series of 1-tailed
t-tests (summary in Table I).
Guidance players were faster than feedback . The num-
ber of training trials needed to learn the task was 48.8% less,
t(26) =2.68,p=< .01; and the number actions needed to
learn the task was 54.9% less, t(26) =2.91,p< .01. Thus the
ability for the human teacher to guide the agent’s attention to
appropriateobjectsatappropriatetimescreatesasigniﬁcantly
more efﬁcient learning interaction.
In theguidance condition the number of unique states vis-
ited was 49.6% less, t(26) =5.64,p< .001. Thus, attention
direction helps the human teacher keep the exploration of
the agent in the most useful part of the task space. This is a
particularly important result since that the ability to deal with
large state spaces has long been a criticism of RL. A human
partner may let the algorithm overcome this challenge.
And ﬁnally the guidance condition provided a more suc-
cessful training experience. The number of trials ending
in failure was 37.5% less, t(26) =2.61,p< .01; and the
number of failed trials before the ﬁrst successful trial was
41.2% less, t(26) =2.82,p< .01. A more successful training
experience is particularly desirable when the learning agent
is a robot that may not be able to withstand very many failure
conditions. Additionally, a successful learning interaction,
especially reaching the ﬁrst successful attempt sooner, may
help the human teacher feel that progress is being made and
prolong their engagement in the process.
TABLE I
EFFECTS OF GUIDANCE ON THE LEARNING PERFORMANCE . (F =FAILED
TRIALS, G =FIRST SUCCESS ).
Measure Mean Mean t(26) p
guidance no guidance
# trials 14.6 28.52 2.68 <.01
# actions 368 816.44 2.91 <.01
# states 62.7 124.44 5.64 <.001
# F 11.8 18.89 2.61 <.01
# F before G 11 18.7 2.82 <.01
B. Gaze Improves Guidance
Our second hypothesis was the gaze behavior serves as a
transparency device to help the human understand when
the agent did (and did not) need their guidance instruction.
To evaluate this, we analyzed the game logs from players
that had the guidance condition versus those that had the
gaze-guide condition. These results are summarized in
Table II. Note that the players that did not have the gaze
behavior still had ample opportunity to administer guidance;
however, the time that the agent waits is uniform throughout.
We look at the timing of each player’s guidance communi-
cation and separate their communication into two segments,
the percentage of guidance that was given when the number
of action choices was >=3 (high uncertainty), and when
choices were <=3 (low uncertainty), these are overlapping
classes. The percentage of guidance when the agent had low
uncertaintydecreasedforplayersinthe gaze-guide condi-
tion,t(51) =−2.22,p=.015. And conversely the percentage
of guidance when the agent had high uncertainty increased
fromthe guidance tothegaze-guide condition, t(51) =
1.96,p=.027. Thus, when the agent uses the gaze behavior
to indicate which actions it is considering, the human trainers
do a better job matching their instruction to the needs of the
agent throughout the training session.
TABLE II
THE EFFECT OF GAZE ON GUIDANCE . COMPARED TO THE GUIDANCE
DISTRIBUTION WITHOUT GAZE ,THE GAZE CONDITION CAUSED A
DECREASE WITH CHOICES <=3,INCREASE WITH CHOICES >=3.
Measure Mean % gaze Mean % no gaze t(51) p
guidance when 79 85 -2.22 <.05
# choices <=3
guidance when 48 36 1.96 <.05
# choices >=3
V. DISCUSSION
In this research we are promoting a social learning view of
machine learning, emphasizing the interactive elements in
teaching. There are inherently two sides to an interaction,
and our approach aims to enhance standard machine learning
algorithms from both perspectives of this interaction.Allowing the human teacher to administer guidance in ad-
dition to feedback improves learning performance across a
number of dimensions. The agent is able to learn tasks using
fewer actions over fewer trials. It has a more efﬁcient explo-
ration strategy that wasted less time in irrelevant states. We
argue that a less random and more sensible exploration will
lead to more understandable and teachable agents. Guidance
also led to fewer failed trials and less time to the ﬁrst
successful trial. This is a particularly important improvement
in that it implies a less frustrating teaching experience, which
in turn creates a more engaging interaction for the human.
This work offers a concrete example that the transparency of
the agent’s behavior to the human can improve its learning
environment. When the learning agent uses gazing behaviors
to reveal its uncertainties and potential next actions, people
were signiﬁcantly better at providing more guidance when
it was needed and less when it was not. Gaze is just
one such transparency device, the exploration of various
devices and their relation to the learning process is part of
our future work. Additionally these transparency behaviors
serve to boost the overall believability of the agent. The
issue of believability has been addressed in the animation,
video game, and autonomous agent literature for the purpose
of creating emotionally engaging characters [20], [3]. One
contribution of this work is to show how believability relates
to teachable characters to improve the experience of the
human and the learning performance of the agent.
Numerous prior works have explored learning agents (virtual
or robotic) that can be interactively trained by people. Many
of these works are inspired by animal or human learning. For
instance, game characters that the human player can shape
through interaction have been successfully incorporated into
a few computer games [9], [17], [19]. Breazeal et al.have
demonstrated aspects of collaboration and social learning on
a humanoid robot, using social cues to guide instruction [6].
Animal training techniques and human tutelage have been
explored in several robotic agents [12], [16], [18], [15]. As
a software agent example, Blumberg’s virtual dog character
can be taught via clicker training, and behavior can be shaped
by a human teacher [4].
Many of these prior works agree with our situated learning
paradigmformachines,andhaveemphasizedthatanartiﬁcial
agent should use social techniques to create a better inter-
face for a human partner. This work goes beyond gleaning
inspiration from natural forms of social learning and teaching
to formalize this inspiration and empirically ground it in
observed human teaching behavior through extensive user
studies. Thus, another contribution of this work is empirical
evidence that social guidance and transparency create a good
interface for a human partner, andcan create a better learning
environment and signiﬁcantly beneﬁt learning performance.
Finally, the scenario of human input has received some
attention in the machine learning community. There has been
work on computational models of teacher/learner pairs [10].
Active learning and algorithms that learn with queries begin
to address interactive aspects of a teacher/learner pair [8].
Queries can be viewed as a type of transparency into the
learning process, but in these approaches this does not steer
subsequent input from a teacher. Instead, through its queries,
the algorithm is in control of the interaction. Cohn et al.
present a semi-supervised clustering algorithm that uses a
human teaching interaction, but the balance of control falls to
the human (i.e., to iteratively provide feedback and examples
to a clustering algorithm which presents revised clusters) [7].
Thus, prior works have addressed how human input can
theoretically impact a learning algorithm. In contrast, this
work addresses the nature of realpeople as teachers; our
ground truth evaluation is the performance of the machine
learnerwithnon-experthumanteachers.Whereaspriorworks
typically lend control either to the machine or the human, our
contribution is the focus on how a machine learner can use
transparencybehaviorstosteertheinstructionitreceivesfrom
a human, creating more reciprocal control of the interaction.
VI. CONCLUSION
This work shows that designing for the complete human-
machine learning system creates a more successful robot
learner. We acknowledge that teaching-learning is a recip-
rocal and dynamic process, and have augmented a standard
RL agent with social mechanisms. Our experiments with
an interactive computer game character show signiﬁcant
improvement in a real-time interactive learning session with
non-expert human teachers. Our results show that allowing
the human to guide the learner’s exploration by directing its
attention creates a more robust and efﬁcient learning strategy.
Second, our results show that the learning agent can improve
its own learning environment by using transparency behav-
iors. Revealing information about the possible next actions
the agent is considering with its gaze behavior signiﬁcantly
improved the timing of the teacher’s guidance messages.
ACKNOWLEDGEMENTS
This work is funded in part by the Media Lab corporate
sponsors of the Things that Think and Digital Life consortia.
REFERENCES
[1] M. Argyle, R. Ingham, and M. McCallin. The different functions of
gaze.Semiotica , pages 19–32, 1973.
[2] R. Arkin, M. Fujita, T. Takagi, and R. Hasegawa. An ethological and
emotional basis for human-robot interaction. In Proceedings of the
Conference on Robotics and Autonomous Systems , 2003.[3] J. Bates. The role of emotion in believable agents. Communications
of the ACM , 37(7):122–125, 1997.
[4] B. Blumberg, M. Downie, Y. Ivanov, M. Berlin, M.P. Johnson, and
B. Tomlinson. Integrated learning for interactive synthetic characters.
InProceedings of the ACM SIGGRAPH , 2002.
[5] C. Breazeal. Designing Sociable Robots . MIT Press, Cambridge, MA,
2002.
[6] C. Breazeal, A. Brooks, J. Gray, G. Hoffman, J. Lieberman, H. Lee,
A. Lockerd, and D. Mulanda. Tutelage and collaboration for humanoid
robots.International Journal of Humanoid Robotics , 1(2), 2004.
[7] D. Cohn, R. Caruana, and A. McCallum. Semi-supervised clustering
with user feedback, 2003.
[8] D. Cohn, Z. Ghahramani, and M. Jordan. Active learning with
statistical models. In G. Tesauro, D. Touretzky, and J. Alspector,
editors,AdvancesinNeuralInformationProcessing ,volume7.Morgan
Kaufmann, 1995.
[9] R. Evans. Varieties of learning. In S. Rabin, editor, AI Game Pro-
gramming Wisdom , pages 567–578. Charles River Media, Hingham,
MA, 2002.
[10] Sally A. Goldman and H. David Mathias. Teaching a smarter learner.
Journal of Computer and System Sciences , 52(2):255–267, 1996.
[11] P. M. Greenﬁeld. Theory of the teacher in learning activities of
everydaylife. InB.RogoffandJ.Lave,editors, Everydaycognition:its
development in social context . Harvard University Press, Cambridge,
MA, 1984.
[12] F. Kaplan, P-Y. Oudeyer, E. Kubinyi, and A. Miklosi. Robotic clicker
training. Robotics and Autonomous Systems , 38(3-4):197–206, 2002.
[13] R. M. Krauss, Y. Chen, and P. Chawla. Nonverbal behavior and
nonverbal communication: What do conversational hand gestures tell
us? In M. Zanna, editor, Advances in experimental social psychology ,
pages 389–450. Tampa: Academic Press, 1996.
[14] Ed. M. Cole L. S. Vygotsky. Mind in society: the development of
higher psychological processes . Harvard University Press, Cambridge,
MA, 1978.
[15] A. Lockerd and C. Breazeal. Tutelage and socially guided robot
learning. In IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS) , 2004.
[16] L. M. Saksida, S. M. Raymond, and D. S. Touretzky. Shaping robot
behavior using principles from instrumental conditioning. Robotics
and Autonomous Systems , 22(3/4):231, 1998.
[17] K. O. Stanley, B. D. Bryant, and R. Miikkulainen. Evolving neural
network agents in the nero video game. In Proceedings of IEEE 2005
Symposium on Computational Intelligence and Games (CIG’05) , 2005.
[18] L. Steels and F. Kaplan. Aibo’s ﬁrst words: The social learning of
language and meaning. Evolution of Communication , 4(1):3–32, 2001.
[19] A. Stern, A. Frank, and B. Resner. Virtual petz (video session): a
hybrid approach to creating autonomous, lifelike dogz and catz. In
AGENTS ’98: Proceedings of the second international conference on
Autonomous agents , pages 334–335, New York, NY, USA, 1998. ACM
Press.
[20] F. Thomas and O. Johnson. Disney Animation: The Illusion of Life .
Abbeville Press, New York, 1981.
[21] C. Watkins and P. Dayan. Q-learning. Machine Learning , 8(3):279–
292, 1992.
[22] J. V. Wertsch, N. Minick, and F. J. Arns. Creation of context in joint
problemsolving. InB.RogoffandJ.Lave,editors, Everydaycognition:
itsdevelopmentinsocialcontext .HarvardUniversityPress,Cambridge,
MA, 1984.