Justification Narratives for Individual Classifications

ICML 2014 AutoML Workshop
Justication Narratives for Individual Classications
Or Biran orb@cs.columbia.edu
Kathleen McKeown kathy@cs.columbia.edu
Department of Computer Science, Columbia University, New York, NY 10027
Abstract
Machine learning models are now used extensively for decision making in diverse appli-
cations, but for non-experts they are essentially black boxes. While there has been some
work on the explanation of classications, these are targeted at the expert user. For the
non-expert, a better model is one of justication - not detailing how the model made its
decision, but justifying it to the human user on his or her terms. In this paper we introduce
the idea of a justication narrative: a simple model-agnostic mapping of the essential values
underlying a classication to a semantic space. We present a package that automatically
produces these narratives and realizes them visually or textually.
Keywords: Explanation, Justication
1. Introduction
The need for explaining and justifying automatically-generated predictions has been dis-
cussed in various contexts, beginning with expert systems as early as the 1970's (Shortlie
and Buchanan, 1975; Swartout, 1983). It is particularly important in high-risk applications
such as medicine: in an early study, physicians rated the ability to explain decisions as the
most highly desirable feature of a decision-assisting system (Teach and Shortlie, 1981). It
is also essential in consumer-facing applications such as Recommender Systems (Herlocker
et al., 2000; Tintarev and Mastho, 2007) and Context-Aware Applications (Lim and Dey,
2010; Vermeulen, 2010).
The terms explanation andjustication are sometimes used interchangeably, but the dis-
tinction is important when considered from the point of view of a non-expert. Explanation
answers the question \how did the system arrive at the prediction?" while justication aims
to address a dierent question, \why should we believe the prediction is correct?". For a
user with enough expertise, an answer to the rst question will suce to satisfy the second.
Consequently, explanation of classication results in the machine learning literature has of-
ten been treated as the visualization problem of providing detail about the internals of the
model in a human-consumable form (Szafron et al., 2003) or as the problem of projecting
the processes and parameters of non-linear models to make them more meaningful to the
expert (Robnik-Sikonja and Kononenko, 2008; Baehrens et al., 2010), while justication as
a goal has not been mentioned.
In machine learning, unlike rule-based or knowledge-based expert systems, it is not
reasonable to expect non-experts to understand the details of how a prediction was made.
It is still important, however, that they understand the variables aecting the current
prediction enough to satisfy the question of justication. It has been shown that evidence-
based causal models of justication are often more satisfactory to users than full white-
c
2014 O. Biran & K. McKeown.
Biran McKeown
box models, and that replacing numeric values with qualifying linguistic expressions (high,
strong, etc) also enhances satisfaction (Lacave and D ez, 2002; Herlocker et al., 2000).
The independent variables used in machine learning models often correspond to real-world
evidence that non-experts understand well, and a justication for a prediction can rely on
these variables, their importance to the model, their eect on the prediction, and their
interactions.
A robust method of automatically generating prediction justication for non-experts,
then, should focus on selecting the most important pieces of evidence for a particular pre-
diction and on analyzing and presenting their roles in the prediction. The selected evidence
should be presented to the user in a way that is invariant across dierent models and readily
understandable by a non-expert.
In this paper we will discuss some central concepts of explanation and how they can be
used without model-specic detail in justication. We also introduce the idea of a justica-
tion narrative and propose a framework of deriving it from a single prediction. Finally, we
present PreJu (the Prediction Justier): a package that produces these structured narratives
and uses them to generate visual or textual justications for predictions.
2. Core Concepts of Explanation
While proposed methods of explanation in the literature vary signicantly across models and
authors, two concepts in particular appear in many of them under dierent names. These
are the eect of a feature on the prediction and the feature's importance in the model.
We use Logistic Regression (LR) as an example throughout this paper. The eect of
each individual feature on the prediction in LR is clear and independent of other features,
and it is easy to dene the importance of features, all of which is ideal for explaining the
justication narrative. While for many other models it is harder to dene the values of these
scores (in particular, the values in non-linear models may dier depending on the values of
other features), there have been specic proposals for various models (Carrizosa et al., 2006;
Yap et al., 2008) as well as proposals for model-agnostic denitions (Robnik-Sikonja and
Kononenko, 2008). We view this work as complementary and essential to ours. This paper
is not focused on dealing with particular models, but on creating a justication narrative
structure, which can be done through abstracting the details behind these two scores. We
can be agnostic to the origin of the values as long as they have normal mathematical
equality and inequality relations (greater than, less than, and equals). The more the values
correspond to the true workings of the model, the better the justication will be.
A typical formulation of Logistic Regression predicts a class y2(y1;y2;:::yk) for an
instancexby applying the logistic function:
Pr(y=yjjx) =1
1 +eP
ijixi
Where each xiis a feature value and the coecients ihave been learned by linear
regression from the training data for each class j, and include an intercept. Since this is
a linear model, we can dene the xed polarity of a feature itowards a class jas the sign
function of its coecient, sgn(ji).
Theeect (sometimes called contribution ) of a feature is the extent to which it has
contributed towards or against predicting a particular class for the instance. Since we
2
Justification Narratives
are interested in justifying the current prediction, we will dene the eect as being the
contribution with regard to the predicted class only.
For a linear model, the eect of a feature itowards class jis the product of the feature's
coecient and its value in the instance:
Eectji=jixi
We use importance to model the overall strength of a feature in the model. One way
to think of it is as the expected eect of the feature on the prediction for a particular class,
which in a linear model can be estimated using the mean feature value for the class ( Xjis
the set of all instances in the data set with class j):
Importanceji=jiP
x2Xjxi
jXjj
Like polarity, importance in LR is xed across dierent instance predictions. Note that
the importance does not necessarily have to have the same sign as the polarity; if negative
feature values are allowed, the signs may be dierent.
3. Justication
Building on the two concepts discussed in the previous section, we introduce the idea of a
justication narrative .
A justication narrative is dened for each instance prediction, and is composed of a
subset of the model features called key features , each of which has a narrative role. The task
of building a justication narrative is the task of selecting the key features and assigning
their roles.
3.1. Narrative Roles
Narrative roles are assigned based on the sign and magnitude of the importance and eect
of a feature towards the predicted class. They represent semantically clear concepts that
non-experts readily understand, and are rooted in the true details of the prediction. Table 1
shows the roles for all possible combinations.
The magnitude of the importance of the features is a simple separation between lowand
high. The separation is dependent on the classier, and can be done in various ways. A few
simple options we have implemented are a xed threshold; a xed number of features (with
the highest importance) that are considered high, while all others are considered low; and
a scheme where the next feature with the highest importance is iteratively considered high
until the sum of importance of all high importance features is more than some percentage
of the total sum of importance. These schemes are all done separately for positive and
negative importance features. The magnitude of eect is determined with a threshold that
is set midway between the highest low importance and the lowest high importance.
The method presented in this paper was developed in the context of a large system we
built to predict the prominence of scientic topics. We will use a simplied version of that
task and prediction system as an example throughout the rest of this section to illustrate
the various roles.
We have a (trained) binary Y/N classier that predicts whether or not the citation
count of a scientic article will increase in the next year. Each instance is an article, and we
3
Biran McKeown
aaaaaaaaaImportanceEectHigh
positiveLowHigh
negative
High positive Normal evidence Missing evidence Contrarian counter-evidence
Low Exceptional evidence Negligible Exceptional counter-evidence
High negative Contrarian evidence Missing counter-evidence Normal counter-evidence
Table 1: Narrative roles assignment for the range of feature eect and importance
extract (among others) four features. Since the model is trained, we know the importance
of each of the features towards the Y class:
SLOPE is the slope of the linear function of the article's citations per year over the past
5 years (all articles in the data set are at least 5 years old). It models the recent increase
or decrease in the number of citations and has a high positive importance.
H-INDEX is the maximum h-index of the authors. It has a low positive importance (peo-
ple have a slight preference towards citing prominent authors).
VENUE is a score between -1 and 1 that represents the prominence of the venue the ar-
ticle was published in. Although its polarity is positive, it has a low negative importance
(because the average article is published in a less-than-average, negative-score venue).
DIAMETER is the diameter of the citation network , built by treating all articles which
previously cited the instance article as nodes and the citations between them as directed
edges. It has a high negative importance (large diameters suggest that the article is cited
not by a community, but by unconnected authors)
Now, consider a particular article which the model classies as Y (its citation count is
predicted to increase in the next year). We will use possible eects of the features to explain
each of the roles below.
First, we describe evidence roles. Features having these roles contributed towards the
prediction. Normal evidence is evidence that is expected to be present in many instances
predicted to be in this class: with high importance, high eect is not surprising. For our Y
prediction, SLOPE having a high positive eect would be normal evidence. Exceptional
evidence is evidence that is not usually expected to be present. With low importance,
high eect means that the feature value is exceptionally high. In our example, if H-INDEX
has a high positive eect, it means at least one of the authors is exceptionally prominent.
Contrarian evidence is strongly unexpected, since the eect has the opposite sign than
expected. In our example this is impossible, because DIAMETER cannot be negative.
Features that have Missing evidence as a role are important features that were ex-
pected to contribute positively, but were weak for the particular instance. If SLOPE had a
low eect in the prediction, that is something we want to include in the narrative because
it means that the prediction was uncharacteristically made without the important eect of
that feature. Similarly, missing counter-evidence is given to features that were expected
to contribute negatively but did not.
4
Justification Narratives
Finally, there are counter-evidence roles. Features having these roles contributed against
the prediction, although they were not strong enough to prevent it. Normal counter-
evidence is expected: it is normal for DIAMETER to have a high negative eect (which
means the article has an average DIAMETER value) even if the positive eects from (say)
SLOPE or H-INDEX ultimately overpower it. Exceptional counter-evidence is not ex-
pected - if VENUE has a high negative eect, then the venue is exceptionally poor. Finally,
if SLOPE has a high negative eect (the article is one of the rare ones for which the number
of citations per year decreases over time), we have contrarian counter-evidence because
the feature we normally expect to contribute positively contributes negatively instead.
Note that contrarian evidence (and contrarian counter-evidence) is only possible for
features that may have negative values, and may not appear in many real-world applications.
3.2. Key Feature Selection
The appropriate way to select the key features depends on the application. If the feature
space is small and/or sparse, it may make sense to select all features that have any role
other than negligible. Some consumer-facing applications (e.g., recommender systems) may
want to select a xed number of features to show as evidence to the user, and may want
to constrain their roles (only showing the evidence roles in the rst column of table 1, for
example). In other cases, we may want to choose the top features from each role group
(where the top is determined by a ranking of eect for evidence and counter evidence, and
by importance for missing evidence).
3.3. Handling Priors
Many classiers have priors for each of the classes. We consider those additional features for
the purpose of justication. In Logistic Regression, each class jhas an intercept which we
add as a feature that has the intercept value as the importance for j, and zero importance
for all other classes. The eect is always equal to the importance, so the only possible roles
for the priors are normal evidence and normal counter-evidence (if they are not negligible).
3.4. Producing a Narrative
Once the key features and their roles are known, the justication narrative can be generated
textually and/or visually. To make it concrete, consider a prediction made by the classier
described earlier. The predicted class is Y, and the selected features for the justication
narrative are SLOPE, VENUE, DIAMETER and Y-PRIOR (the prior of class Y).
Figure 1 shows one possible way to present the justication narrative, generated by our
package. In this example, we generate a short textual summary of the narrative (using
simple templates), a bulleted list of the key features, and a bar chart showing the relative
eect of the features.
It should be stressed that the narrative is not a particular visualization, however. It is
a set of features with associated roles, eects and importance values. It can be visualized
using charts, tables, bulleted templated phrases or in the form of an essay using a full
Natural Language Generation system. We provide multiple such outputs as options, as the
best way is ultimately application-dependent.
5
Biran McKeown
Figure 1: Justication narrative example
4. Package
PreJu is a Java package1that generates justications for classier predictions. PreJu works
in one of two ways: as a stand-alone congurable tool, it accepts input in the form of
XML providing the eect and importance scores of the features, leaving the implementation
details to the user, and allows the conguration of key feature selection, role assignment and
output types via XML. As an API, it provides simple interfaces for producing justications
programmatically and contains implementations for Weka's Logistic Regression and Linear
Regression classiers as integrated input sources.
5. Future Work
One part of future work is the continued expansion and renement of our package. Han-
dling additional model types is a priority, as well as providing implementations for major
third-party packages. Another important task is the evaluation of our method in terms of
similarity to expert-produced justications and user satisfaction.
We are also continuing research to facilitate better justications. In particular, the
current method lacks the ability to explain the features themselves and the real-world
meaning of their values (e.g., what is a network diameter? what does a large diameter
entail?). We are exploring potential methods of obtaining such explanations automatically
(for example, from Wikipedia or from relevant text books) using Information Extraction
techniques. At the same time, we are looking at Natural Language Generation techniques for
creating better, more expressive text describing the narrative and the meaning of features.
Acknowledgments
This research is supported by the Intelligence Advanced Research Projects Activity (IARPA) via
Department of Interior National Business Center (DoI/NBC) contract number D11PC20153. The
U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes
notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained
herein are those of the authors and should not be interpreted as necessarily representing the ocial
policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.
1. Available at http://www.cs.columbia.edu/~orb/preju/
6
Justification Narratives
References
David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen,
and Klaus-Robert M uller. How to explain individual classication decisions. Journal of
Machine Learning Research , 11, August 2010.
Emilio Carrizosa, Belen Martin-Barragan, and Dolores Romero Morales. Detecting relevant
variables and interactions for classication in support vector machines. Technical report,
2006.
Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. Explaining collaborative l-
tering recommendations. In Proceedings of the 2000 ACM Conference on Computer Sup-
ported Cooperative Work , CSCW '00, 2000.
C. Lacave and F. J. D ez. A review of explanation methods for Bayesian networks. Knowl-
edge Engineering Review , 17:107{127, 2002.
Brian Y. Lim and Anind K. Dey. Toolkit to support intelligibility in context-aware appli-
cations. In Proceedings of the 12th ACM International Conference on Ubiquitous Com-
puting , Ubicomp '10, 2010.
M. Robnik-Sikonja and I. Kononenko. Explaining classications for individual instances.
Knowledge and Data Engineering, IEEE Transactions on , 20(5):589{600, May 2008.
Edward H Shortlie and Bruce G Buchanan. A model of inexact reasoning in medicine.
Mathematical biosciences , 23(3):351{379, 1975.
William R. Swartout. Xplain: A system for creating and explaining expert consulting
programs. Artif. Intell. , 21(3), September 1983.
Duane Szafron, Russell Greiner, Paul Lu, David Wishart, Cam Macdonell, John Anvik,
Brett Poulin, Zhiyong Lu, and Roman Eisner. Explaining naive bayes classications.
Technical report, 2003.
R. Teach and E. Shortlie. An Analysis of Physician Attitudes Regarding Computer-Based
Clinical Consultation Systems. Computers and Biomedical Research , 14:542{558, 1981.
Nava Tintarev and Judith Mastho. A survey of explanations in recommender systems.
InProceedings of the 2007 IEEE 23rd International Conference on Data Engineering
Workshop , ICDEW '07, 2007.
Jo Vermeulen. Improving intelligibility and control in ubicomp. In Proceedings of the
12th ACM International Conference Adjunct Papers on Ubiquitous Computing - Adjunct ,
Ubicomp '10 Adjunct, 2010.
Ghim-Eng Yap, Ah-Hwee Tan, and Hwee-Hwa Pang. Explaining inferences in bayesian
networks. Applied Intelligence , 29(3):263{278, 2008.
7


Towards State Summarization for Autonomous Robots
Daniel Brooks, Abraham Shultz, Munjal Desai, Philip Kovac, and Holly A. Yanco
Computer Science Department
University of Massachusetts Lowell
One University Avenue
Lowell, Massachusetts 01854
fdbrooks, ashultz, mdesai, pkovac, holly g@cs.uml.edu
Abstract
Mobile robots are an increasingly important part of
search and rescue efforts as well as military combat. In
order for users to accept these robots and use them ef-
fectively, the user must be able to communicate clearly
with the robots and obtain explanations of the robots’
behavior that will allow the user to understand its ac-
tions. This paper describes part of a system of software
that will be able to produce explanations of the robots’
behavior and situation in an interaction with a human
operator.
Motivation
The Urban Search and Rescue (USAR) ﬁeld is beginning to
adopt teleoperated systems, such as those used in the wake
of Hurricane Katrina to check partially-collapsed buildings
for survivors (Micire 2008). Because they have little or no
autonomy, these systems require the dedicated attention of
a trained operator. This requirement restricts the availability
of such systems to teams which have a trained teleoperator
among their members.
To reduce the difﬁculty of using telepresence and robotic
systems in USAR, the operator should be able to issue orders
to an autonomous system and expect that those orders will be
followed. This form of supervisory control helps reduce the
amount of time and effort the operator spends on navigation
and controlling individual motions (Fong and Thorpe 2001).
One question of particular importance is how operators
of autonomous systems will be able to maintain or regain
control of a robot when its status becomes unclear or it ex-
hibits unexpected behaviors. If the robot has been moving
autonomously, the user will not have the same degree of sit-
uation awareness that they would have had if they had been
explicitly controlling the robot’s motions. In such a case, it
would be helpful if the operator could query the robot for in-
formation about its state and behavior, receive answers, and
develop an informative dialog with the robot.
Operators of autonomous robots will be able to better un-
derstand how their instructions were interpreted if the robot
is capable of recounting recent events or explaining some
speciﬁc action that it took. We call this process of gener-
ating a concise, detailed, and useful representation of the
Copyright c
2010, Association for the Advancement of Artiﬁcial
Intelligence (www.aaai.org). All rights reserved.robot’s current state and world model state summarization .
We believe that having access to this type of information will
increase operator trust of autonomous systems and free the
operator from having to “babysit” each machine to under-
stand what it is doing.
Related Work
It is not uncommon for people who work with computers on
a regular basis to desire an explanation for some unexpected
negative behavior that the user ﬁnds irrational. Previous re-
search in the ﬁelds of artiﬁcial intelligence, data mining, and
machine learning has sought to provide reasonable ways of
having an autonomous system explain its decisions and sub-
sequent actions.
Explaining events in a simulation is part of the motiva-
tion for explainable AI (XAI) as proposed in Gomboc et al.
(2005) and Core et al. (2006). That research focuses on a
posteriori methods of extracting representations of behavior
from the data generated by a simulation that did not origi-
nally include a detailed, accessible behavior model. Because
we are the developers of our system, we can change the sys-
tem to produce a maximally useful behavior representation.
Lemon et al. (2001) focused on generation of speech
for interaction with users. The paper describes an interface
that combines natural language commands and dialog with
a computer-based map interface. This system allows the
user and robot to agree on pronoun referents without spe-
ciﬁc names, such as the command “Go here,” coupled with
a click on the map interface for disambiguation.
System Design
Our system uses a modular, message-passing infrastructure.
The messages are units of data called “parcels”. Parcels
can be created from incoming sensor data, or by process-
ing that takes place in one of the modules of the system.
When a module of the infrastructure creates a new parcel
based on one or more older parcels, it includes references to
the parcels that provoked the creation of the new parcel. As
items of data progress through the system, the set of parcels
to which they are linked expands. The complete set can be
viewed as a tree, with the action that the robot takes as the
root of the tree. The parcels immediately preceding the root
are the ﬁrst level of the tree, the parcels that went into them
are the next level, and so forth, extending out to the leaf
nodes, which will mostly be raw sensor data. This set of
links provides a context for decisions that proceed from the
data. Using this context, the robot can account for its per-
ception of the environment at the time that a particular action
changed the state of the robot. This means that explanations
such as “Why is the robot not moving forward?” can be
answered succinctly with a message such as “The way for-
ward is blocked”, and with more detail using messages such
as “There is an object located 11 degrees left of the robot’s
center and 0.3 meters away.”
An explanation can be made useless by being either too
succinct to capture important details, or by being so verbose
that details are obscured. In order to support a ﬂexible level
of detail, each parcel will be tagged with a level of abstract-
ness and a level of interestingness.
Abstractness is a unitless measurement of how distant the
parcel of data and action that uses it are from raw sensor
data. The actual sensor data coming into the robot is the least
abstract information. As it is interpreted, it gains abstraction.
A raw laser sweep is not abstract, but interpreting sections of
it as ﬁducial markers gives them a more abstracted meaning.
Interpreting those ﬁducial markers as objects that are related
to the robot’s current goal is yet another level of abstraction.
Levels of interestingness are an orthogonal measure to ab-
stractness, in that the abstractness of a unit of data is not a
function of its interestingness. Interestingness is a unitless
measure of how related a speciﬁc piece of information is
to the completion of the robot’s goals. A ﬁducial marker
that does not identify the sought object may be ignored,
but will still have a higher “interestingness” rating than a
laser sweep that did not detect any ﬁducial markers, because
it had a higher chance of being something relevant. As a
consequence, if the robot is searching for a particular ﬁdu-
cial, it may choose to prioritize reporting that ﬁducial over
other things it sees, or only mention the other things if it
is prompted with a question. The linkage of parcels into
trees provides a convenient ﬁrst step towards automating the
detection of interestingness, as it provides a method to as-
sess the degree of relevance of a given element of the data to
the robot’s behavior. Database entries that are are referenced
frequently in decisions made by action modules contain data
that was relevant for making decisions that affected the state
of the robot, and so is more interesting. Data that is not refer-
enced was examined by the robot and determined not affect
the robot’s state.
The magnitude of the difference between the expected ac-
tion and the actual action could also be considered as part of
interestingness. If the instruction sent by the operator is to
go forward into a room, and the robot turns to go in a differ-
ent door, the difference between the expected behavior and
the observed behavior is quite large. When the robot “dis-
obeys” the operator in this manner, the large difference will
more likely result in a demand for an explanation where a
small difference might have passed unnoticed.
Beyond trees of parcels, the recorded data is also times-
tamped, and can be sequenced chronologically. This pro-
vides a means of supporting not only causal dialogs, where
the user asks why an event occurred, but also chronologi-cal dialogs, where the user asks when an event occurred or
where the event falls in a sequence. We believe that the most
useful dialogs will occur when both causal and chronolog-
ical dialogs are used together. The user will be able to ask
about events in the robot’s past with chronological queries,
and ﬁnd more detail about those events with causal queries.
The ease of interpretation of the summarized data is a
problem which must be addressed regardless of the medium
in which it is presented. It is imperative that the informa-
tion be comprehensible and useful to a user with little or no
formal technical training and a limited understanding of the
machine and its programming.
Current Work
The current interface for the visualization is a timeline with
each action that the robot has performed displayed as a tree.
This interface will be extended to allow each new action to
be added to the timeline as the action occurs. The user can
expand each tree to in the timeline to view more or less detail
around a certain period, and slide the timeline back and forth
to view more recent or later events.
Interestingness calculators based on the difference be-
tween the user selected heading and the actual heading, the
magnitude of recent changes in the heading, and the amount
of ﬁducial data found have already been implemented. Ab-
stractness calculators based on number of referenced parcels
and parcel type are also complete. These calculators are im-
plemented as modular units that can be conﬁgured prior to
running the system, so various approaches can be quickly
tested.
Acknowledgements
Thanks to Mikhail Medvedev for his help in developing
PASSAT. This work was funded in part by an Army Re-
search Ofﬁce MURI (W911NF-07-1-0216) and the National
Science Foundation (IIS-0546309).
References
Core, M.; Lane, H.; Van Lent, M.; Gomboc, D.; Solomon,
S.; and Rosenberg, M. 2006. Building explainable artiﬁcial
intelligence systems. In Proceedings of the National Con-
ference on Artiﬁcial Intelligence , volume 21, 1766.
Fong, T., and Thorpe, C. 2001. Vehicle teleoperation inter-
faces. Autonomous Robots 11(1):9–18.
Gomboc, D.; Solomon, S.; Core, M. G.; Lane, H. C.; and
Lent, M. V . 2005. Design recommendations to support
automated explanation and tutoring. In Proceedings of the
14th Conference on Behavior Representation in Modeling
and Simulation (BRIMS05), Universal .
Lemon, O.; Bracy, A.; Gruenstein, A.; and Peters, S. 2001.
Information states in a multimodal dialogue system for
human-robot conversation. In 5th Wkshp on Formal Seman-
tics and Pragmatics of Dialogue (Bi-Dialog 2001) , 57–67.
Micire, M. J. 2008. Evaluation and ﬁeld performance of a
rescue robot. Journal of Field Robotics 25(1-2):17–30.

Daniel Brooks
Abraham Shultz
Munjal Desai
Philip Kovac
Holly A. Yanco


Building Explainable Artificial Intelligence Systems

Building Explainable Artiﬁcial Intelligence Systems
Mark G. Core,H. Chad Lane, Michael vanLent, DaveGomboc, SteveSolomon and
MiltonRosenberg
TheInstitute for CreativeTechnologies,The Universityof Southern California
13274 Fiji Way,Marina del Rey,CA 90292 USA
core,lane,vanlent,gomboc,solomon,rosenberg@ict.usc.edu
Abstract
As artiﬁcial intelligence (AI) systems and behavior
models in military simulations become increasingly
complex,ithasbeendifﬁcultforuserstounderstandthe
activitiesofcomputer-controlledentities. Prototypeex-
planation systems have been added to simulators, but
designers have not heeded the lessons learned from
work in explaining expert system behavior. These new
explanation systems are not modular and not portable;
they are tied to a particular AI system. In this paper,
we present a modular and generic architecture for ex-
plainingthebehaviorofsimulatedentities. Wedescribe
its application to the Virtual Humans, a simulation de-
signed to teach soft skills such as negotiation and cul-
tural awareness.
Introduction
Thecomplexityofartiﬁcialintelligence(AI)systemsinsim-
ulations and games has made it difﬁcult for users to under-
standtheactivitiesofcomputer-controlledentities. Insimu-
lationsandgamesusedfortraining,theexplainabilityofbe-
haviors is essential. Students must understand the rationale
behind actions of simulated entities and how their actions
affectthe simulated entities.
Themilitaryhastraditionallyusedliveexercisesfortrain-
ing; the principal tool for learning from these exercises has
beentheafter-actionreview(AAR).USArmyFieldManual
25-101, “Battle Focused Training”, gives recommendations
on conducting an AAR, and states that “The OPFOR [op-
posingforces]canprovidevaluablefeedbackonthetraining
based on observations from their perspectives...the OPFOR
can provide healthy insights on OPFOR doctrine and plans,
the unit’s action, OPFOR reactions to what the unit did.”
(Army 1990)[Appendix G] OPFOR are often played by in-
structorsandparticipateintheAARgivingtheirviewpoints
not only as opponents in the exercise but also as teachers
assessing trainee performance. Friendly forces also partici-
pate in the AAR, and provide insight on how trainee orders
translate into the behavior of units and individual soldiers.
Despite the importance of AARs, military simulators cur-
rently do not provide this type of experience. Such simu-
lators can present mission statistics (e.g., casualties, tactical
Copyright c°2006, American Association for Artiﬁcial Intelli-
gence (www.aaai.org). All rights reserved.objectives achieved) and replay events from the simulation,
but neglect the causes behind these statistics and simulation
events. What is missing in the AAR are the simulated enti-
ties(bothOPFORandfriendlyforces)topresenttheirview-
point on what happened and why, and an AAR leader to en-
surestudentsareaskingtherightquestionsandlearninghow
to improvetheir performance.
Military simulations are also used as predictive tools; as
such, their AI systems must be validated as behaving realis-
tically and according to military doctrine. Detailed speciﬁ-
cationsaredraftedforsuchAIsystems;theresultingbehav-
iors are put under heavy scrutiny. In most cases, because
observers have no way to question AI-controlled entities,
the observer’s only recourse is to watch numerous simula-
tionruns,lookingforcaseswherefaultyreasoningresultsin
an incorrect action. For example, to detect an error where a
riﬂeman entity fails to check its rules of engagement before
ﬁring its weapon, the observer must watch simulation runs
until a situation occurs where the riﬂeman ﬁres its weapon
but its rules of engagement forbid this action. A better ap-
proachwouldbetoquestiontheentityaboutasingleweapon
ﬁreevent,andseewhethercheckingtherulesofengagement
is part of its reasoning.
Figure 1 shows a screenshot of the user interface of our
explanation system for the One Semi-Automated Forces
Objective System, a military simulation (Courtemanche &
Wittman 2002), and introduces the concept of an explana-
tion system for simulated entities. Users select a time point
to discuss, an entity to be questioned, and the question it-
self. Some of the questions are speciﬁc to the particular en-
tity(e.g.,whatisyourhealth?) whileothersconcernalarger
group (e.g., what is your unit’stask?).
Two of the ﬁrst explanation systems for simulation were
Debrief (Johnson 1994) and Explainable Artiﬁcial Intelli-
gence for Full Spectrum Command, FSC (van Lent, Fisher,
& Mancuso 2004). In Debrief, Johnson built explanation
capabilities into Soar (Laird, Newell, & Rosenbloom 1987)
agents that were part of battleﬁeld simulations. van Lent et
al. addedexplanationcapabilitiestoatrainingaiddeveloped
by commercial game developers and academic researchers.
vanLentetal. coinedtheterm“explainableartiﬁcialintelli-
gence” (XAI) to describe the ability of their system to sum-
marize the events of the game/simulation, ﬂag key events,
and explainthe behaviorof computer controlled entities.
1766
Figure 1: Interfaceto XAI for OOS
While novel, this research did not address how lessons
learned from explaining expert system decisions (Swartout
& Moore 1993) applied to explaining the behavior of sim-
ulated entities. Swartout and Moore advocated building a
high-level knowledge base containing facts about the do-
mainandproblem-solvingstrategies,andusinganautomatic
programwritertobuildanexpertsystemfromthisspeciﬁca-
tion. The problem is more complicated for XAI because of
technicalchallenges(i.e.,theexecutablecodemustinterface
with an external simulation) and real world constraints (i.e.,
we have little or no control of the simulated entities - they
havealready been builtor designed).
In the follow-on project to XAI for FSC, we faced these
challengesbutsoughtamiddlegroundbetweenbuildingan-
other simulation-speciﬁc explanation system and demand-
ing to build the simulated entities ourself (as Swartout and
Moore built their own expert systems). Following this phi-
losophy,we re-engineered our system to support:
²domain independence - supporting reuse and the devel-
opment of common behavior representations while being
ﬂexible enough to deal with simulation-speciﬁc idiosyn-
crasies
²theabilitytoexplainthemotivationsbehindentityactions
²modularity-allowingexternalcomponentssuchasGUIs,
natural language generators, and tutors to interface with
the system
In the related work section, we discuss how the John-
son and van Lent et al. work falls short in addressing these
goals;wethenpresentourXAIarchitecturedesignedspecif-
ically to meet these needs. Our ﬁrst XAI system using
this new architecture works with the One Semi-Automated
Forces Objective System (OOS), a tactical military simula-
tor and is described in the papers, (Gomboc et al.2005;
Coreetal.2005).
The focus of this paper is our second instantiation of the
new architecture, the Virtual Humans simulator (Traum et
al.2005). This was not designed as a tactical simulation,but rather for the teaching of soft skills such as leadership,
teamwork, negotiation and cultural awareness. The current
scenario has one character, a virtual doctor who communi-
cates with students through spoken input/output as well as
generating animated gestures. The student plays the role of
a U.S. Army Captain whose unit is planning an operation
against insurgents in the same neighborhood as the clinic
run by the doctor. In order to minimize the risk to civilians,
the Captain is ordered to move the clinic to a safer location.
The doctor is not interested in moving; the Captain’s task is
to convincethe doctor to movethe clinic.
Although the core structure of our architecture has not
changed since we built XAI for OOS, we are continu-
ally making improvements such as the replacement of hard
codedquestionswithlogicalforms. Inthispaper,wepresent
anabbreviatedcasestudydescribingtheprocessofconnect-
ing this architecture to the Virtual Humans simulator. It il-
lustrates the steps necessary to connect a simulation to an
externalexplanationsystemaswellasgivinganideaofwhat
parts of the system are reusable and what must be authored.
Wethenpresentasampledialoguewithinteractionsbetween
a student, automated tutor, and the XAI system. We do not
focus on the details of this tutor, but include it in this pre-
sentationtoillustratetheusefulnessofXAIinapedagogical
context. In the future work section, we discuss taking fea-
tures speciﬁc to XAI for OOS and XAI for Virtual Humans
andmakingthemdomain-independentfunctionsthatcanbe
reused in newtargetsimulations.
RelatedWork
In this section, we discuss two pieces of previous work on
generating explanations of entity actions during a simula-
tion: Debrief (Johnson 1994) and explainable artiﬁcial in-
telligence (XAI) for Full Spectrum Command (FSC) (van
Lent, Fisher, & Mancuso 2004). Debrief works with enti-
tiescontrolledbySoar(Laird,Newell,&Rosenbloom1987)
agents in a tactical air combat domain. FSC was a collabo-
ration between game developers and academic researchers
who developed their own artiﬁcial intelligence system to
controlsimulated entitiesin thistrainingtool forcommand-
ing a light infantrycompany.
Debrief uses Soar’s learning mechanism to save the
agents’ states during the simulation. After the simulation,
an agent gives a textual description of what happened and
the user is allowed to ask questions. To answer these ques-
tions,Debriefperformsreﬂectionbyrerunningtheagentand
selectively changing elements of its state to determine the
causeofitsactions(includingactionssuchasinsertingabe-
lief into memory).
FSC has logging facilities that save each entity’s state:
simple values such as ammo status and a representation of
the task being executed (including its subtasks and how the
platoon is broken into squads). After the simulation is over,
FSC presents mission statistics and ﬂags “interesting” time
points (an entity is wounded or killed, the ﬁrst contact with
the enemy, task start and end times). The user selects a
timeandentityandisabletoaskquestionsabouttheentity’s
state.
1767

Both systems were important ﬁrst steps in adding expla-
nationcapabilitiestosimulatedentities,butneitherapproach
wassuitableforourgoalsofdomainindependenceandmod-
ularity. Debrief and XAI for FSC are speciﬁc to the AI sys-
temscontrollingthesimulatedentitiesinthoseapplications,
and not directly applicable to other AI systems.
(Gomboc et al.2005) was the ﬁrst description of our
domain-independent XAI architecture; it described our ef-
forts to connect this architecture to the tactical simulations,
the One Semi-Automated Forces Objective System, OOS
and Full Spectrum Command. We noted that simulations
differ as to their “explanation-friendliness”, and one of the
keyissuesishowthesimulationrepresentsbehaviors. Simu-
lationsmayencodebehaviorsdirectlyinaprogramminglan-
guage,useplanningoperators,useadeclarativeruleformat,
or combinations of the three. We argued against a purely
procedural representation because it provides no represen-
tation of the motivations underlying the behavior. Consider
the example of sending a ﬁre team to clear a room. Once
the ﬁre team is in position outside the room, the grenadier
throws a grenade before the team enters the room. This
could be encoded as a procedure (the step before entering
theroomisalwaysthrowingthegrenade)inwhichcase,the
system cannot explain why the grenade was thrown. In the
next section, we summarize the XAI architecture presented
in (Gomboc et al.2005) as well as discussing what options
are available when dealing with an explanation-unfriendly
simulation.
XAIArchitecture
Figure 2 shows our domain-independent XAI architecture.
The (Gomboc et al.2005) paper focused on the left side of
diagram and the problem of importing data from the simu-
lation. We use a relational database to store this informa-
tion because tactical military simulations typically produce
large log ﬁles; they record data at a ﬁne granularity (e.g.,
OOS records the values of variables about once a second).
The Virtual Humans currently log a much smaller amount
of data, but the full power of the relational database may be
useful later if the logging in Virtual Humans becomes more
ﬁne-grained. Wefocusonasubsetofthevirtualhuman’sbe-
haviors: if we covered the entire range, then we would see
log ﬁles of comparable size as those in tactical simulations.
Assuming XAI is able to import the necessary informa-
tion from the simulation, the next step is to inspect the data
looking for “interesting” facts. The deﬁnition of interest-
ing will change with each application. For OOS, we used a
placeholder deﬁnition that highlighted when an entity ﬁred
itsweapon,orwhenitbegan,washalf-waythrough,orcom-
pletedatask. Collaborationwithdomainexpertswillhelpus
reﬁne this deﬁnition. Because we developed an automated
tutorfortheVirtualHumans,ourdeﬁnitionofinterestingre-
ferredtoeventsthatthetutorshoulddiscusswiththestudent
and it is the responsibility of the tutor to identify them. Al-
thoughwecurrentlyhand-annotatetheseteachingpoints,we
areautomatingtheprocessusingaheuristic-basedapproach
to identifying teachable moments in an exercise.
Once the XAI system is initialized, users select an entity
with which to speak and the time point in the simulationthey wish to discuss ﬁrst. Users then query the entity about
the current time point by selecting questions from a menu.
The dialogue manager orchestrates the system’s response;
ﬁrst using the reasoner to retrieve the relevant information,
thenproducingEnglishresponsesusingthenaturallanguage
generator (NLG). NLG ﬁlls slots in natural language tem-
plates with information from the database. NLG is coded
inXSLtemplatesandtakesadvantageofXSLfeaturessuch
asiterationandprocedurecalls(commontaskssuchasstate
descriptions are shared among templates).
As noted in (Gomboc et al.2005), this is a best-case sce-
nario where the simulation makes available a rich behavior
representation containing, for example, entity goals and ac-
tion preconditions and effects. However, this does not mean
thatifthesimulationdoesnothavesucharepresentationwe
cannot attempt to explainentity behavior.
Previousworkinexplainingexpertsystembehavior(sum-
marized in (Swartout & Moore 1993)) dealt with a similar
problem; if the expert system’s inference engine contains
special features (not represented in the system’s knowledge
base) then the output of these features can still be explained
by hard coded explanation routines. However, if changes
are made to the special features without also updating the
explanation routines, there is a potential for inaccurate ex-
planations. Thus, explanations can still be made but at a
cost to the maintainability and robustnessof the system.
In the case of representing simulation behaviors for XAI,
there are three options with associated costs and beneﬁts:
1. automatically import the behaviors. cost:requires a rep-
resentation of goals and the preconditions and effects of
behaviors. beneﬁt:high maintainability and robustness.
target:plan-based representations
2. semi-automatically import the behaviors. cost:must be
able to ﬁnd goals, preconditions, and effects in behav-
ior representation. beneﬁt:more maintainable and robust
than option 3, and makes fewer assumptions than option
1.target:rule-basedrepresentations
3. hand-build the XAI representation of the behaviors. cost:
low maintainability - any change in the behavior must be
duplicatedintheXAIrepresentation. Needsubjectmatter
experttoauthormissinggoals,preconditions,andeffects.
beneﬁt:makes no assumptions about the simulation’s be-
haviorrepresentation. target:proceduralrepresentations
The reason that we target rule-based representations with
option 2 is that some elements on the left hand side of rules
are preconditions (e.g., you must have ammunition to ﬁre
your weapon), but other elements may be abort conditions
(e.g., do not ﬁre your weapon when a friendly entity is in
the path) or internal bookkeeping (e.g., setting internal vari-
ables, making sure a rule does not ﬁre twice). Similarly,
not all elements on the right hand side of rules are effects
and may instead also be internal bookkeeping. With such a
representation, we can hand-annotate the preconditions and
effects in these rules, then automatically import the behav-
ior. Although there is no guarantee that annotations will be
updated as developers change entity behaviors, at least this
meta-data is co-located with the original behavior represen-
tation.
1768
Figure2: XAI Architecture
Collectinglogﬁlesandscenarioinformationisnottrivial,
but here the options are simple: if the simulation does not
make information available for export, the XAI system can-
not answer questions about it. In our XAI for OOS system,
we entered some scenario and log information by hand to
build a proof-of-concept system, but this data is speciﬁc to
aparticular simulation run, and some of the hand data-entry
mustberepeatedforeachnewsimulationrun. Itismorefea-
sible to hand-author behavior representations because these
do not change between simulation runs.
Building a new XAI System
Connecting our current XAI system to a new simulation re-
quiresseveralsteps:
1. study the behavior representation and choose one of the
three approaches to importing behaviors as discussed in
the previoussection.
2. implement data import for behaviorsand log ﬁles
3. specify the question list for the domain
3a. write the logical form (LF) of each question
3b. write the query LF
4. augment the natural language generator to support the
newquestions and their potential answers
5. create GUI
Specifying the question list for a new simulation requires
two steps. The ﬁrst step is writing the logical form of the
question which is used to generate the English form of the
question. FortheVirtualHumans,wehad110distinctques-
tions so by using the natural language generator to produce
thequestionswecouldchangehowtheywerephrasedwith-
out rewriting all 110 questions. The second step is writing
the query to retrieve the answer from the database; we use
anabstractlanguagecalledthequerylogicalformtoencode
queries (see belowfor more details).
The last step in connecting XAI to a new simulation is
building a new GUI or reusing a GUI from a previous XAIsystem. Although every XAI system will have the same
basic GUI components (ways to select entities, times, and
questions, and displays of dialogue between user, XAI, and
tutor), to support replay of the simulation requires support
fromthetargetsimulation,andifXAIisafeatureintegrated
into the simulation, it will share the simulation’s GUI. Be-
causeoftheseconstraints,wedesignedtheGUIsofXAIfor
OOS and XAI for Virtual Humans as separate components
that communicate with the rest of the system through XML
messages. Our abstract message format facilitates this play-
and-plug functionality. The messages convey the content of
menussuchasthequestionlistandlistoftimepointsaswell
as user selections from these menus. The messages also up-
date the state of the dialogue between the student and tu-
torandthedialoguebetweenthestudentandXAI.TheGUI
candisplaythesemenuchoicesandtextinwhateverwidgets
(e.g., radio buttons,drop-downmenus) it chooses.
XAIforVirtualHumans
Following the steps enumerated in the previous section, the
ﬁrst task in connecting XAI to the Virtual Humans was to
study the behavior representation. In this case, the sim-
ulation developers not only had to model physical behav-
iors such as treating patients but also the behaviors under-
lying the utterances produced by the student and doctor
(e.g., committing, insisting), and the doctor’s mental rea-
soning (e.g., making the decision to help the Captain). The
model of physical actions contained preconditions and ef-
fects explaining the relationships between the actions (e.g.,
you need supplies to treat the patients). In importing this
modelwefoundsomebugsinthemodel,soitismoreaccu-
ratetosaythatwesemi-automaticallyimportedthephysical
behaviors. Nowthatthebugsareﬁxed,weshouldbeableto
fully automate this process.
The non-physical behaviors were implemented with hun-
dreds of rules developed in the Soar cognitive architecture
(Laird, Newell, & Rosenbloom 1987). Given enough time,
it should be possible to hand-annotate the goals, precondi-
1769

name=‘negotiation-stance’
agent=‘doctor’
theme=‘captain’
time=CURRENT
value=xquant
propositionname=‘the’
variable=‘x’
tense=‘present’
modality=‘none’
frameFigure3: LF for “What is your negotiationstance?”
tions and effects in all these rules. As an initial step, our
implementation focused on the rules governing trust since
teaching trust building is one of the pedagogical goals of
the 2005 Virtual Humans system. As described in (Traum
et al.2005), the Virtual Humans model trust as inﬂuenced
by three factors: familiarity, credibility, and solidarity. All
threehavedirectpositiverelationshipstotrust;themorethe
doctor feels he knows you, feels you speak the truth, and
feels that you share common goals, the more he trusts you
(and vice versa). In our current prototype, we model single
steps of the doctor’s reasoning by linking rules to English
paraphrases (e.g., “the negotiation failed because you lost
thetrust of the doctor”).
Once we designed the database format for the target sim-
ulation and wrote the code to import the data, the next step
involved encoding the questions to be answered by XAI in
thetargetdomain(i.e.,specifyingthequestionitself,encod-
ing the relevant database queries, and necessary changes to
the natural language generation templates). Questions are
encodedinalogicalform(LF),anabstractrepresentationof
their content. A simpliﬁed graphical version of our XML
representation for the question, “What is your negotiation
stance?” isshowninﬁgure3. Thelogicalformwasdesigned
to support future plans to generate syntactic features of the
character’s language such as tense and modality rather than
hard coding them in templates. The other feature to note
is the variable, CURRENT which is substituted at runtime
with the current line being discussed. It is obvious that we
wouldnotauthorseparatequestionssuchas“whatisthene-
gotiationstanceatline1”and“whatisthenegotiationstance
at line 2”. However, this same mechanism also allows us to
have one logical form for the questions, “why did the nego-
tiation fail?” and “why are you avoiding the negotiation?”.
Here,the runtime variableis the negotiationstance.
The logical form of the question is accompanied by an
abstractrepresentationofthequery(wecallitthequeryLF)
to retrieve the answer. It also uses runtime variables so that
authors only have to write one query LF for the questions
“why did the negotiation fail?” and “why are you avoiding
thenegotiation?”. TheXAIreasonertranslatesthequeryLF
intotheSQLquerywhichissenttothedatabase. Anareaof
futureworkistoderivethequeryLFautomaticallyfromthe
LFof the question.The next step is modifying the set of XSL templates so
that the English form of the question can be generated as
well as the range of potential answers. Templates can be
reusedtosupportnewquestionsandtheirpotentialanswers.
For example, there is one set of templates that generates
English descriptions of states and tasks. These are used to
describe states and tasks in questions as well as answers. In
futurework,weintendtomakeournaturallanguagegenera-
tion more domain-independent by hard coding less English,
andaddingtemplatesencodingdomain-independentaspects
of language such as syntax and morphology.
Sample Dialogue with XAI forVirtual
Humans
As mentioned in the introduction, after action reviews
(AARs) are the Army’s primary tool for learning from live
trainingexercises,andourgoalisbuildingAARexperiences
for simulated exercises. Our AAR GUI for the Virtual Hu-
mans is shown in ﬁgure 4. We currently lack a replay of
the simulation and instead show a transcript of the inter-
action between the student and the virtual doctor that took
place during the simulation (upper left of ﬁgure 4). Cur-
rently our prototype works with the following sample dia-
logue (C=Captain and D=Doctor):
C: Hello Doctor Perez
D: Hello
C: I haveorders to movethis clinic to another location
D: Youwantto movethe clinic
C: Yes
D: Do you see that girl? She lost her mother today.
C: It is not safe here. Wecannot protect you.
D: Protect me? Protect me from what?
D: Youare going to attack?
C: Yes
D: I wouldhaveto refuse this decision
D: My patients need my attention now
Thedialoguebetweenthestudentandtutorappearsinthe
lower left of the AAR screen. In this session, the tutor asks
the student to use XAI to investigate what happened at line
10. The student then asks questions of the doctor using the
question list on the bottom right of the screen, and the dia-
logue between student and doctor appears in the upper right
of the screen. This is a prototype interface and we are cur-
rently experimenting with merging the two dialogue win-
dows, adding playback capabilities, and developing a more
usable question list.
Before the AAR begins, the tutor must analyze the dia-
logue searching for teaching points. Intuitively we can see
that the dialogue is unsuccessful. In line 10, the student re-
vealed secret information, and triggered the doctor’s ending
of the conversation. Examining the log ﬁles showing the
doctor’s mental state conﬁrms that the dialogue was unsuc-
cessfulandthatline5(wheretheCaptainaskedthedoctorto
movetheclinic)andline10decreasedtrust. Ourknowledge
of the rules governing the doctor helps us see opportunities
the student missed to gaintrust in lines 3 and 10.
AfterloadingthelogﬁlesandbehaviordataintoourXAI
system, we hand-annotate these teaching points. Because
1770
Figure 4: Interfaceto XAI for VirtualHumans
these are not subjective judgments, we anticipate that it will
be fairly simple to encode these intuitions, and annotate
teachingpointsautomatically. Thelossesoftrustandfailure
ofthenegotiationappeardirectlyinthelogﬁle. Determining
whatthestudentshouldhavedonerequiresrecognizingrules
thatincreasetrustandcheckingtheirpreconditions(e.g.,ut-
tering a pleasantry such as “nice to meet you” is possible at
line3 and wouldhaveincreased trust).
Our automated tutor is a prototype and we experimented
with a hierarchical task network planner, JSHOP2 (a Java
derivative of SHOP (Nau et al.1999)) as its control sys-
tem. In ongoing work, we are implementing a more reac-
tiveapproachbecauserelyingsolelyonaplannerforcontrol
meant that the planner had to be rerun after each student in-
put (we could not predict the student’s action), resulting in
aninefﬁcientsystem. Inthissection,wewillfocusonusing
XAIasteachingaid;thetutoralsohastheabilitytoquestion
students directly and evaluate their answers using key-word
spotting. Belowarelinesofasampledialoguebetweentutor
andstudent(T=tutorandS=student);thestudentisgiventhe
opportunity to identify his own errors and the student picks
line 10. The tutor determines that line 10 is best taught by
havingthe student investigatethe problem using XAI.
1. T: In the simulation history window,can you select a
linewhere you think you could havedone better?
2. S: Line 10
3. T: That’sright. There is room for improvementin
line10.
Inthe XAI window,please ask the doctor some
questionsand ﬁnd out howthis utterance
contributedto the failureof the negotiation.While using XAI for the Virtual Humans, students are
presented with a question list like the one in ﬁgure 5. The
questions reﬂect information logged and our model of the
doctor’s behavior. The missing questions (q2, q4, etc.) are
“why” questions associated with q1 through q9. We illus-
trate how “why” questions work in the continuation of the
tutoring session below. The user asks the doctor about his
negotiation stance, and now a new question, q2, is available
(i.e.,“whydidthenegotiationfail?”). Thedialoguemanager
keepstrackoftheseevennumberedquestionsandeverytime
we ask about line 10 this newquestion will be available.
4. S: Let’sstart the investigationat line 10.
5. D: OK
6. S: What is your negotiationstance?
7. D: The negotiationis overand I refuse to movethe
clinic
8. S: Whydid the negotiationfail?
9. D: I no longer trusted the Captain
When the tutor instructs the student to use XAI, it acti-
vates a component called the investigation model tracer to
track the student’s progress with XAI. Currently, we deﬁne
anidealinvestigationasaseriesofquestionswhoseanswers
providethelessonstobelearned. Forthisexampletheideal
investigation is very simple and consists of q2 (which was
just asked) and q6. q6, “Why did your solidarity with the
Captain decrease?”, is made available after the student asks
q5.
One reason the ideal investigation is so simple is that the
student did not get very far in his negotiation. If the stu-
dent had done better, he would have needed to know about
the preconditions and effects of actions that concerned the
1771

q1: What is your negotiationstance?
q3: What is your trust of the Captain?
q5: Howwasyour solidarity with the Captain affected
here?
q7: Howwasthe Captain’scredibility affectedhere?
q9: Howwasyour familiaritywith the Captain affected
here?
q11:Howimportant do you feel being neutral is?a
q12:What action(s) could cause being neutral?
q13:What are the effectsof running the clinic here?
q14:What action(s) could change being neutral?
q15:What are the preconditions of running the clinic here?
q16:What actions havebeing neutral as a precondition?
aThese virtual humans use utility theory to evaluate actions
and states of the world. This question allows the user to query
the relative importance of different states of the world from the
doctor’sperspective.
Figure5: Sample question list for XAI for VirtualHumans
doctorsuchasrunningtheclinic. Questions11-16allowthe
studenttolearnaboutthedoctor’smodelofphysicalactions.
Inthecurrentversionofthesystem,thereare16actionsand
17 states of the world that the user can ask about. Users
select actions and states from menus, and questions 11-16
changeto match the selection.
The question list in ﬁgure 5 was generated when the cur-
renttaskwas“runningtheclinicinitscurrentlocation”,and
the current state was “the doctor’s neutrality”. Running the
clinicinitscurrentlocation,andrunningitinanewlocation
arethetwomostimportanttasksinthedoctor’sworld. Neu-
trality is important as we see in the dialogue continuation
below, because it is a prerequisite for running the clinic. If
the initial dialogue with the doctor had been more success-
ful,thestudentwouldneedtoconsiderhiringlocalstomove
the clinic instead of using U.S. troops.
10 S: What are the preconditions of running the clinic
here?
11 D: havingsupplies here, the patients being safe here,
thepatients being here, the clinic being here
andbeing neutral
12 S: What action(s) could change being neutral?
13 D: negotiatingwith the Captain, cooperating with
theCaptain and the Americans movingthe clinic
14 S: What action(s) could cause the clinic being
[moved]there?
15 D: the Americans movingthe clinic, locals moving
theclinic, and me movingthe clinic
While the student was asking questions, the investigation
model tracer watched his progress. After several turns have
passed and the student has not asked q6, the tutor gives a
hint as shown below (actually the hint appears in a differ-
ent window). We use a standard model of hinting where the
hintsgetmorespeciﬁcuntilthetutorgivesawaytheanswer.
In this case, because we have not authored any hints, the tu-
tor gives away the answer. Our mechanism for “unlocking”
questionsdoesnotencodetherelationshipsbetweenthecon-tent of questions. So the tutor does not know that in order
to know why solidarity decreased, the student must know
that solidarity did decrease. We are working on ﬁxing this
probleminournextversionofthesystem. Let’sassumethat
thestudentrealizeshemustunlockq6andasksq5asshown
below. Once the student has asked q6, the tutor recognizes
that the investigation is complete and reengages the student
in dialogue (in the tutor-studentdialogue window).1
16 T: Please ask question #6
17 S: Howwasyour solidarity with the Captain
affectedhere?
18 D: It decreased.
19 S: Whydid your solidarity with the Captain
decrease?
20 D: The Captain is committing to performing an
undesired act.
21 T: Good job. Youfound out that admitting to the
planned attack decreased the doctor’strust of you,
which caused the negotiationto fail...
FutureWork
In this section, we focus on the issues of domain indepen-
dence and modularity, comparing our two systems, XAI for
OOSandXAIforVirtualHumans. Wediscussplanstotake
domain-dependent aspects of these systems and make them
available as general features that can be activated or deac-
tivated as appropriate. For example, XAI for OOS has the
featureofremovingnon-applicablequestionsfromitsques-
tion list; if an entity did not ﬁre its weapon at the current
time point then the question, “what are you shooting at?”,
will be removed from the question list. However, there are
pedagogicalramiﬁcationstothis“feature”. Consideraques-
tion from the Virtual Humans domain, “How was your sol-
idarity with the Captain affected here?”. We may want to
display this question even if solidarity did not change be-
cause we want the student to realize that solidarity could
have changed. Thus, we will allow the feature to be deacti-
vatedon a question-by-question bias.
Another feature present in XAI for OOS is the ability to
use HTML formatting to include links to entities and times.
Because entities and times are often the topic of conversa-
tion, we added the ability for users to click on mentions of
entities and times to change the entity or time as opposed to
selecting them from the entity or time menu. The XAI-for-
OOSGUIinterpretstagsinthenaturallanguagegenerator’s
output in order to make this linkage. We will formalize this
conventionin our XML message format.
Otherfeaturesareassociatedwithdialoguecontext;inthe
sampledialogue,wesawthataskingcertainquestionsofthe
virtual doctor made new questions available (i.e., question
unlocking). In XAI for OOS, entities “introduce” them-
selves when you ﬁrst select them (e.g., “I am the ﬁre team
leader of Blue Fireteam 1...”), and in some contexts, users
can ask the question, “can you give me more detail?”. Cur-
rently XAI for Virtual Humans and XAI for OOS use ad-
1Although not shown in these excerpts, the goal is to encour-
age the student to be honest but vague (e.g., ”I cannot discuss our
operational plans”).
1772
hoc models of context limited to the information needed to
support features such as question unlocking. We will build
a general model of context storing each line of the student-
XAI-tutordialogueandwhoproducedit. Toenablequestion
unlocking, we will store the logical form of questions asked
and answers produced by the system. Questions will refer-
ence the dialogue context by listing applicability conditions
rather than relying on a domain-speciﬁc context checking
mechanism. This model of context will also be useful for
thenaturallanguagegenerator,allowingittotailoritsoutput
basedonthecontextandproducemorenaturalandreadable
text.
Conclusion
Rather than simply writing an explanation system that only
worked for its target AI system and simulator, we used our
generic and modular architecture for explainable artiﬁcial
intelligence (XAI) systems in building XAI for the One
Semi-Automated Forces Objective System, a tactical mili-
tary simulation (Courtemanche & Wittman 2002). This ar-
chitecturecontinuedtoevolveasweworkedonXAIforVir-
tual Humans, a simulation designed to teach soft skills such
as leadership, teamwork, negotiation, and cultural aware-
ness. In this paper, we presented an abbreviated case study
on howto connect an explanationsystem to a targetsimula-
tion and in particular, model behaviors and add support for
newquestions. Wealsoshowedhowourprototypetutoruses
XAIasateachingtool,gettingthestudenttounderstandhis
simulated negotiationpartner’sreasoning and mental state.
The key point of the XAI architecture is its domain inde-
pendence and modularity. Every XAI system will have the
basiccomponentsinﬁgure2buttheirexactimplementation
will vary based on project requirements. For example, our
relianceonarelationaldatabaseandSQLquerieswasbased
on the requirement of handling a large dataset size and our
short development times. The creation of the query logical
form was a step toward a more declarative representation
and future work may include more powerful reasoning and
storage components.
We are currently continuing to work in the domain of ne-
gotiation withcultural awarenessand arecollaborating with
a project called ELECT (Enhanced Learning Environments
with Creative Technologies) here at the Institute for Cre-
ative Technologies. The goal of ELECT is to build a sys-
tem including explanation and tutoring to be used directly
inArmytrainingandinvolvesseveralexternalcollaborators
contributing subject matter expertise. We plan to work with
the Army Research Institute to evaluate the effectiveness of
our tutor and the XAI system.
Acknowledgments
WethankWilliamSwartoutandDavidTraumfortheirguid-
ance and support. The project described herein has been
sponsored by the U.S. Army Research, Development, and
Engineering Command (RDECOM). Statements and opin-
ions expressed do not necessarily reﬂect the position or the
policy of the U.S. Government; no ofﬁcial endorsement
should be inferred.References
1990. FM 25-101: Battle Focused Training. Headquarters
Department of the Army.WashingtonD.C.
Core, M. G.; Lane, H. C.; van Lent, M.; Solomon, S.;
Gomboc, D.; and Carpenter, P. 2005. Toward question an-
swering for simulations. In Proc. of the IJCAI 2005 Work-
shop on Knowledge and Reasoning for Answering Ques-
tions (KRAQ05) .
Courtemanche, A., and Wittman, R. 2002. OneSAF: A
product-lineapproachforanext-generationCGF. In Proc.
of the Eleventh SIW Conference on Computer-Generated
Forcesand BehavioralRepresentations ,349–361.
Gomboc, D.; Solomon, S.; Core, M. G.; Lane, H. C.; and
van Lent, M. 2005. Design recommendations to support
automated explanation and tutoring. In Proc. of the Four-
teenth Conference on Behavior Representation in Model-
ing and Simulation .
Johnson,W.L.1994.Agentsthatexplaintheirownactions.
InProc. of the Fourth Conference on Computer Generated
Forcesand BehavioralRepresentation,Orlando, FL .
Laird, J. E.; Newell, A.; and Rosenbloom, P. 1987. Soar:
An architecture for general intelligence. Artiﬁcial Intelli-
gence33:1–64.
Nau,D.S.;Cao,Y.;Lotem,A.;andnozAvila,H.M. 1999.
Shop: Simplehierarchicalorderedplanner. In Proceedings
oftheSixteenthInternationalJointConferenceonArtiﬁcial
Intelligence(IJCAI99) ,968–973.
Swartout, W. R., and Moore, J. D. 1993. Explanation in
second generation expert systems. In David, J.; Krivine,
J. P.; and Simmons, R., eds., Second Generation Expert
Systems. Springer-Verlag.
Traum,D.;Swartout,W.;Marsella,S.;andGratch,J. 2005.
Fight,ﬂightornegotiate: Believablestrategiesforconvers-
ing under crisis. In Proc. of the 5th International Working
Conferenceon IntelligentVirtualAgents .
van Lent, M.; Fisher, W.; and Mancuso, M. 2004. An ex-
plainable artiﬁcial intelligence system for small-unit tacti-
calbehavior. In Proc.oftheSixteenthConferenceonInno-
vative Applications of Artiﬁcial Intelligence . Menlo Park,
CA: AAAI Press.
1773

Mark G. Core
H. Chad Lane
Michael van Lent
Dave Gomboc
Steve Solomon
Milton Rosenberg