How to Explain Individual Classification Decisions

Journalof Machine LearningResearch 11(2010)1803-1831 Su bmitted 4/09;Revised 11/09;Published6/10
How toExplain Individual Classiﬁcation Decisions
David Baehrens∗BAEHRENS @CS.TU-BERLIN.DE
Timon Schroeter∗TIMON@CS.TU-BERLIN.DE
Technische Universit ¨at Berlin
Franklinstr. 28/29, FR6-9
10587 Berlin, Germany
StefanHarmeling∗STEFAN.HARMELING @TUEBINGEN .MPG.DE
MPI forBiological Cybernetics
Spemannstr. 38
72076 T¨ubingen, Germany
MotoakiKawanabe†MOTOAKI .KAWANABE @FIRST.FRAUNHOFER .DE
Fraunhofer InstituteFIRST.IDA
Kekulestr.7
12489 Berlin, Germany
KatjaHansen KHANSEN @CS.TU-BERLIN.DE
Klaus-RobertM ¨uller KLAUS-ROBERT.MUELLER @TU-BERLIN.DE
Technische Universit ¨at Berlin
Franklinstr. 28/29, FR6-9
10587 Berlin, Germany
Editor:Carl Edward Rasmussen
Abstract
After building a classiﬁer with modern tools of machine lear ning we typically have a black box at
handthatisabletopredictwellforunseendata. Thus,weget ananswertothequestion whatisthe
mostlikelylabelofagivenunseendatapoint. However,most methodswillprovidenoanswer why
the model predicted a particular label for a single instance and what features were most inﬂuential
forthatparticularinstance. Theonlymethodthatiscurren tlyabletoprovidesuchexplanationsare
decision trees. This paper proposes a procedure which (base d on a set of assumptions) allows to
explain thedecisions of anyclassiﬁcation method.
Keywords: explaining, nonlinear, black box model, kernel methods, Am es mutagenicity
1. Introduction
Automatic nonlinear classiﬁcation is a common and powerful tool in data analysis . Machine learn-
ing research has created methods that are practically useful and that ca n classify unseen data after
beingtrainedon alimitedtrainingsetof labeled examples.
Nevertheless, most of the algorithms do not explaintheir decision. However in practical data
analysis it is essential to obtain an instance-based explanation, that is, we w ould like to gain an
∗. The ﬁrstthreeauthors contributed equally.
†. Alsoat Technische Universit ¨at Berlin,Franklinstr. 28/29, FR6-9,10587 Berlin,Germany.
c/circleco√yrt2010David Baehrens, TimonSchroeter, Stefan Harmeling, Moto akiKawanabe, Katja Hansen and Klaus-RobertM ¨uller.
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
understanding what input features made the nonlinear machine give its ans wer for each individual
datapoint.
Typically, explanations are provided jointly for all instances of the training s et, for example
feature selection methods (including Automatic Relevance Determination) ﬁnd out which inputs
are salient for a good generalization (see Guyon and Elisseeff,2003, for a review). While this can
giveacoarseimpressionabouttheglobalusefulnessofeachinputdime nsion,itisstillanensemble
view and does not provide an answer on an instance basis.1In the neural network literature also
solely an ensemble view was taken in algorithms like input pruning (e.g., Bishop, 1995; LeCun
et al., 1998). The only classiﬁcation that does provide individual explan ations are decision trees
(e.g.,Hastieetal.,2001).
This paper proposes a simple framework that provides local explanation v ectors applicable to
anyclassiﬁcationmethodinordertohelpunderstandingpredictionresultsforsin gledatainstances.
The local explanation yields the features relevant for the prediction at the very points of interest
in the data space, and is able to spot local peculiarities that are neglected in th e global view, for
example, duetocancellation effects.
Thepaperisorganizedasfollows: Wedeﬁnelocalexplanationvectorsa sclassprobabilitygradi-
entsinSection2andgiveanillustrationforGaussianProcessClassiﬁcation (GPC).Somemethods
output a prediction without a direct probability interpretation. For these we p ropose in Section 3 a
way to estimate local explanations. In Section 4 we apply our methodology to lear n distinguishing
propertiesofIrisﬂowersbyestimatingexplanationvectorsforak-NNclas siﬁerappliedtotheclas-
sic Iris data set. In Section 5 we discuss how our approach applied to a SVM classiﬁer allows us to
explainhowdigit“2”isdistinguishedfromdigit“8”intheUSPSdataset. InSe ction6wefocuson
a more real-world application scenario where the proposed explanation ca pabilities prove useful in
drug discovery: Human experts regularly decide how to modify existing lead compounds in order
to obtain new compounds with improved properties. Models capable of explain ing predictions can
helpintheprocessofchoosingpromisingmodiﬁcations. Ourautomaticallygene ratedexplanations
match with chemical domain knowledge about toxifying functional groups of th e compounds in
question. Section 7 contrasts our approach with related work and Section 8 discusses characteristic
propertiesand limitationsof our approach,beforeweconclude thepaper inSection 9.
2. Deﬁnitions ofExplanation Vectors
In this Section we will give deﬁnitions for our approach of local explanatio n vectors in the classi-
ﬁcation setting. We start with a theoretical deﬁnition for multi-class Bayes classiﬁ cation and then
giveaspecializeddeﬁnition beingmorepracticalfor thebinarycase.
Forthemulti-classcase,supposewearegivendatapoints x1,...,xn∈ℜdwithlabels y1,...,yn∈
{1,...,C}and we intend to learn a function that predicts the labels of unlabeled data poin ts. As-
suming that the data is IID-sampled from some unknown joint distribution P(X,Y), we deﬁne the
1. This point is illustrated in Figure 1 (Section 2). Applying feature selection m ethods to the training set (a) will lead
to the (correct) conclusion that both dimensions are equally important for accurate classiﬁcation. As an alternative
to this ensemble view, one may ask: Which features (or combinations there of) are most inﬂuential in the vicinity
of each particular instance. As can be seen in Figure 1 (c), the answer d epends on where the respective instance is
located. On the hypotenuse and at the corners of the triangle, both featur es contribute jointly, whereas along each of
the remaining twoedges theclassiﬁcation depends almostcompletely onjust one of thefeatures.
1804
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
Bayes classiﬁer,
g∗(x)=arg min
c∈{1,...,C}P(Y/\e}atio\slash=c|X=x)
whichis optimalforthe 0-1loss function(seeDevroyeetal.,1996).
For the Bayes classiﬁer we deﬁne the explanation vector of a data point x0to be the derivative
withrespectto xatx=x0oftheconditional probabilityof Y/\e}atio\slash=g∗(x0)givenX=x,or formally,
Deﬁnition 1
ζ(x0):=∂
∂xP(Y/\e}atio\slash=g∗(x)|X=x)/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=x0.
Notethat ζ(x0)isad-dimensionalvectorjustlike x0is. Theclassiﬁer g∗partitionsthedataspace ℜd
intoupto Cpartsonwhich g∗isconstant. Weassumethattheconditionaldistribution P(Y=c|X=
x)is ﬁrst-order differentiable w.r.t. xfor all classes cand over the entire input space. For instance,
this assumption holds if P(X=x|Y=c)is for allcﬁrst-order differentiable in xand the supports
of the class densities overlap around the border for all neighboring pair s in the partition by the
Bayesclassiﬁer. Thevector ζ(x0)deﬁnesoneachofthosepartsavectorﬁeldthatcharacterizesthe
ﬂow away from the corresponding class. Thus entries in ζ(x0)with large absolute values highlight
features that will inﬂuence the class label decision of x0. A positive sign of such an entry implies
that increasing that feature would lower the probability that x0is assigned to g∗(x0). Ignoring the
orientations of the explanation vectors, ζforms a continuously changing (orientation-less) vector
ﬁeld along which the class labels change. This vector ﬁeld lets us locallyunderstand the Bayes
classiﬁer.
We remark that ζ(x0)becomes a zero vector, for example, when P(Y/\e}atio\slash=g∗(x)|X=x)|x=x0is
equal to one in some neighborhood of x0. Our explanation vector ﬁts well to classiﬁers where the
conditional distribution P(Y=c|X=x)is usually not completely ﬂat in some regions. In the
case of deterministic classiﬁers, despite of this issue, Parzen window estimato rs with appropriate
widths (Section 3) can provide meaningful explanation vectors for many sa mples in practice (see
alsoSection8).
Inthecaseofbinaryclassiﬁcationwedirectlydeﬁnelocalexplanationvec torsaslocalgradients
of theprobabilityfunction p(x)=P(Y=1|X=x)of thelearnedmodel forthepositiveclass.
For a probability function p:ℜd→[0,1]of a classiﬁcation model learned from examples
{(x1,y1),...,(xn,yn)} ∈ℜd×{−1,+1}the explanation vector for a classiﬁed test point x0is the
localgradientof patx0:
Deﬁnition 2
ηp(x0):=∇p(x)|x=x0.
Bythisdeﬁnitiontheexplanation ηisagaina d-dimensionalvectorjustlikethetestpoint x0is.
Thesignofeachofitsindividualentriesindicateswhetherthepredictionwo uldincreaseordecrease
when the corresponding feature of x0is increased locally and each entry’s absolute value gives the
amount of inﬂuence in the change in prediction. The vector ηgives the direction of the steepest
ascent from the test point to higher probabilities for the positive class. Fo r binary classiﬁcation the
negative version −ηp(x0)indicates the changes in features needed to increase the probability for
thenegativeclasswhich maybeespecially usefulfor x0predictedinthepositiveclass.
1805
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
For an example we apply Deﬁnition 2 to model predictions learned by Gaussian Process Clas-
siﬁcation(GPC),seeRasmussenandWilliams (2006). GPC isusedherefor threereasons:
(i) In our real-world application we are interested in classifying data from d rug discovery, which
is an area where Gaussian processes have proven to show state-of-th e-art performance, see, for
example, Obrezanova and Segall (2010), Obrezanova et al., Schroe ter et al. (2007c), Schroeter
et al. (2007a), Schroeter et al. (2007b), Schwaighofer et al. (200 7), Schwaighofer et al. (2008) and
Obrezanovaetal.(2008). Itisnaturaltoexpectamodelwithhighpred ictionaccuracyonacomplex
problem to capture relevant structure of the data which is worth explaining a nd may give domain
speciﬁc insights in addition to the values predicted. For an evaluation of the ex plaining capabilities
of ourapproach onacomplex problemfromchemoinformatics seeSection 6.
(ii) GPC does model the class probability function used in Deﬁnition 2 directly. F or other classi-
ﬁcation methods such as Support Vector Machines that do not provide a p robability function as its
outputwe givean exampleforan estimationmethodstartingfromDeﬁnition1inSe ction3.
(iii) The local gradients of the probability function can be calculated analytica lly for differentiable
kernelsas we discussnext.
Letf(x)=∑n
i=1αik(x,xi)beaGaussianProcess(GP)modeltrainedonsamplepoints x1,...,xn∈
ℜdwherekisakernelfunctionand αiarethelearnedweightsofeachsamplepoint. Foratestpoint
x0∈ℜdlet varf(x0)be the variance of f(x0)under the GP posterior of f. Because the posterior
cannot be calculated analytically for GP classiﬁcation models, we used an ap proximation by ex-
pectation propagation (EP) (Kuss and Ramussen, 2005). In the case of the probit likelihood term
deﬁned by the error function, the probability for being of the positive clas sp(x0)can be computed
easilyfromthis approximatedposterioras
p(x0)=1
2erfc/parenleftBigg
−f(x0)√
2·/radicalbig
1+varf(x0)/parenrightBigg
,
whereerfcdenotesthecomplementaryerrorfunction(seeEquation6inSc hwaighoferetal.,2008).
Thenthelocalgradient of p(x0)is givenby2
∇p(x)|x=x0=exp/parenleftBig
−f(x0)2
2(1+varf(x0))/parenrightBig
√
2π/parenleftBigg
∇f(x)|x=x0/radicalbig
1+varf(x0)−1
2f(x0)
(1+varf(x0))3
2∇varf(x)|x=x0/parenrightBigg
.(1)
Asakernelfunctionchoose,forexample,theRBF-kernel k(x0,x1)=exp(−w(x0−x1)2),whichhas
the derivative (∂/∂x0,j)k(x0,x1)=−2wexp(−w(x0−x1)2)(x0,j−x1,j)forj∈{1,...,d}. Then the
elements of thelocalgradient ∇f(x)|x=x0are
∂f
∂x0,j=−2wn
∑
i=1αiexp(−w(x0−xi)2)(x0,j−xi,j)forj∈{1,...,d}.
2. For a detailed derivation, seeAppendix A.1.
1806
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
Forvar f(x0)=k(x0,x0)−kT
∗(K+Σ)−1k∗thederivativeis givenby3
∇varf(x)|x=x0=∂varf
∂x0,j=/parenleftbigg∂
∂x0,jk(x0,x0)/parenrightbigg
−2∗kT
∗(K+Σ)−1∂
∂x0,jk∗forj∈{1,...,d}.
0
0.2
0.4
0.6
0.8
100.20.40.60.81−1−0.500.51
(a) Object0
0.2
0.4
0.6
0.8
100.20.40.60.8100.20.40.60.81
(b) Model
0
0.2
0.4
0.6
0.8
100.10.20.30.40.50.60.70.80.91
(c) Local explanation vectors0
0.2
0.4
0.6
0.8
100.10.20.30.40.50.60.70.80.91
(d) Directionof explanation vectors
Figure1: ExplainingsimpleobjectclassiﬁcationwithGaussianProcesses
Panel (a) of Figure 1 shows the training data of a simple object classiﬁcation task and panel (b)
showsthemodellearnedusingGPC.4Thedataislabeled −1forthebluepointsand +1forthered
points. As illustrated in panel (b) the model is a probability function for the pos itive class which
gives every data point a probability of being in this class. Panel (c) show s the probability gradient
of the model together with the local gradient explanation vectors. Along the h ypotenuse and at the
cornersofthetriangleexplanationsfrombothfeaturesinteracttowards thetriangleclasswhilealong
3. Herek∗= (k(x0,x1),...,k(x0,xn))Tis the evaluation of the kernel function between the test point x0and every
training point. Σis the diagonal matrix of the variance parameter. For details see Rasmuss en and Williams (2006,
Chapter 3).
4. Hyperparameters weretuned by agradient ascend ontheevidence .
1807
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
the edges the importance of one of the two feature dimensions dominates. At the transition from
the negative to the positive class the length of the local gradient vectors re presents the increased
importance of the relevant features. In panel (d) we see that explanatio ns close to the edges of the
plot (especially in the right hand side corner) point away from the positiv e class. However, panel
(c)showsthat theirmagnitude isverysmall. Fordiscussionof thisissueseeS ection 8.
3. Estimating Explanation Vectors
Several classiﬁcation methods directly estimate the decision rule, which often h as no interpretation
astheprobabilityfunctionintermsofDeﬁnition2. ForexampleSupportVector Machinesestimate
adecisionfunctionof theform
f(x)=n
∑
i=1αik(xi,x)+b,
αi,b∈ℜ. Suppose we have two classes (each with one cluster) in one dimension (se e Figure 2)
andtrainaSVMwithRBFkernel. Forpointsoutsidethedataclusters f(x)tendstozero. Thus,the
derivativeof f(x)(shownasarrowsabovethecurves)forpointsontheveryleftoronth everyright
sideoftheaxiswillpointtothewrongside. Inthefollowing,wewillexplainhow explanationscan
0
xp(       |   )y=1  x1
0.5
0xclassifier output
Figure2: Classiﬁer outputof anSVM (top)comparedto p(y=1|x)(bottom).
beobtained forsuchclassiﬁers.
In practice, we do not have access to the true underlying distribution P(X,Y). Consequently,
we have no access to the Bayes classiﬁer as deﬁned in Section 2. Instead , we can apply sophis-
ticated learning machinery like Support Vector Machines (Vapnik, 1995; S ch¨olkopf and Smola,
2002;M¨ulleretal.,2001)thatestimatessomeclassiﬁer gthattriestomimic g∗. Fortestdatapoints
z1,...,zm∈ℜdwhich are assumed to be sampled from the same unknown distribution as the train -
ing data,gestimates labels g(z1),...,g(zm). Now, instead of trying to explain g∗, to which we have
1808
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
noaccess,wewilldeﬁneexplanationvectorsthathelpusunderstandthe classiﬁer gonthetestdata
points.
Since we do not assume that we have access to some intermediate real-valued classiﬁer output
here(ofwhich gmightbeathresholdedversionandwhichfurthermightnotbeanestimateof P(Y=
c|X=x)),wesuggesttoapproximate gbyanotherclassiﬁer ˆ g,theactualformofwhichresembles
the Bayes classiﬁer. There are several choices for ˆ g, for example, GPC, logistic regression, and
Parzen windows.5In this paper we apply Parzen windows to the training points to estimate the
weighted classdensities P(Y=c)·P(X|Y=c),for theindexset Ic={i|g(xi)=c}
ˆpσ(x,y=c)=1
n∑
i∈Ickσ(x−xi) (2)
andwithkσ(z)beingaGaussiankernel kσ(z)=exp(−0.5z⊤z/σ2)/√
2πσ2(asalwaysotherkernels
arealsopossible). This estimates P(Y=c|X=x)forallc,
ˆpσ(y=c|x)=ˆpσ(x,y=c)
ˆpσ(x,y=c)+ˆpσ(x,y/\e}atio\slash=c)≈∑i∈Ickσ(x−xi)
∑ikσ(x−xi)(3)
andthus isan estimateofthe Bayes classiﬁer(thatmimics g),
ˆgσ(x)=arg min
c∈{1,...,C}ˆpσ(y/\e}atio\slash=c|x).
This approach has the advantage that we can use our estimated classiﬁer gto generate any amount
oflabeleddataforconstructing ˆ g. Thesinglehyper-parameter σischosensuchthat ˆ gapproximates
g(whichwewant toexplain),thatis,
ˆσ:=argmin
σm
∑
j=1I/braceleftbig
g(zj)/\e}atio\slash=ˆgσ(zj)/bracerightbig
,
whereI{···}is the indicator function. σis assigned the constant value ˆσfrom here on and omitted
as asubscript. For ˆ gitis straightforwardtodeﬁne explanationvectors:
Deﬁnition 3
ˆζ(z):=∂
∂xˆp(y/\e}atio\slash=g(z)|x)/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=z=/parenleftBig
∑i/∈Ig(z)k(z−xi)/parenrightBig/parenleftBig
∑i∈Ig(z)k(z−xi)(z−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2
−/parenleftBig
∑i/∈Ig(z)k(z−xi)(z−xi)/parenrightBig/parenleftBig
∑i∈Ig(z)k(z−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2.
This is easily derived using Equation (3) and the derivative of Equation ( 2), see Appendix A.3.1.
Note that we use ginstead of ˆ g. This choice ensures that the orientation ofˆζ(z)ﬁts to the labels
assignedby g,whichallows better interpretations.
In summary, we imitate the classiﬁer gwhich we would like to explain locally by a Parzen
windowclassiﬁer ˆ gthathasthesameformastheBayesestimatorandforwhichwecanestimatethe
5. ForSupportVectorMachinesPlatt(1999)ﬁtsasigmoidfunctiontomap theoutputstoprobabilities. Inthefollowing,
we willpresent amoregeneral method for estimatingexplanation vectors .
1809
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
explanationvectorsusingDeﬁnition3. Practicallytherearesomecaveats: T hemimickingclassiﬁer
ˆghas to be estimated from geven in high dimensions; this needs to be done with care. However,
in principle we have an arbitrary amount of training data available for constr ucting ˆgsince we may
useour estimatedclassiﬁer gtogeneratelabeled data.
4. ExplainingIris FlowerClassiﬁcation by k-NearestNeighbors
TheIrisﬂowerdataset(introducedinFisher,1936)describes150ﬂo wersfromthegenusIrisbyfour
features: sepal length, sepal width, petal length, and petal width, all of which are easily measured
properties of certain leaves of the corolla of the ﬂower. There are three clusters in this data set that
correspondtothree differentspecies: Irissetosa ,Irisvirginica ,andIrisversicolor .
Letusconsidertheproblemofclassifyingthedatapointsof Irisversicolor (class0)againstthe
othertwospecies(class1). Weappliedstandardclassiﬁcationmachineryto thisproblemasfollows:
•Class 0consistsofall examples of Irisversicolor .
•Class 1consistsofall examples of Irissetosa andIrisvirginica .
•Randomly split150datapoints into100trainingand50 testexamples.
•Normalize trainingandtestsetusingthemeanand varianceof thetrainingset.
•Applyk-nearest neighbor classiﬁcation with k=4 (chosen by leave-one-out cross-validation
onthe trainingdata).
•Trainingerror is3% (i.e.,3mistakesin100).
•Testerroris8% (i.e.,4mistakesin50).
In order to estimate explanation vectors we mimic the classiﬁcation results with a Par zen window
classiﬁer. The best ﬁt (3% error) is obtained with a kernel width of σ=0.26(chosen by leave-one-
outcross-validationonthetrainingdata).
Since the explanation vectors live in the input space we can visualize them with scatter plots of
the initially measured features. The resulting explanations (i.e., vectors) for the test set are shown
in Figure 3. The blue markers correspond to explanation vectors of Iris setosa and the red markers
correspond to those of of Iris virginica (both class 1). Both groups of markers point to the green
markers of Iris versicolor . The most important feature is the combination of petal length and petal
width (see the corresponding panel), the product of which correspon ds roughly to the area of the
petals. However,theresultingexplanations for thetwospecies inclass1ar edifferent:
•Irissetosa (class1)isdifferentfrom Irisversicolor (class0)becauseitspetalareais smaller.
•Irisvirginica (class1)isdifferentfrom Irisversicolor (class0)becauseitspetalareais larger.
Alsothedimensionsofthesepal(anotherpartoftheblossom)arerelevan t,butnotasdistinguishing.
1810
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
Figure3: Scatterplotsoftheexplanationvectorsforthetestdata. Showna reallexplanationvectors
for both classes: class 1 containing Iris setosa (shown in blue) and Iris virginica (shown
in red) versus class 0 containing only the species Iris versicolor (shown in green). Note
thattheexplanationswhyanIrisﬂowerisnotan Irisversicolor isdifferentfor Irissetosa
andIrisvirginica .
5. ExplainingUSPSDigit Classiﬁcation bySupportVector Mach ine
We now apply the framework of estimating explanation vectors to a high dimension al data set, the
USPSdigits. Theclassiﬁcationproblemthatwedesignedforillustrationpurp osesisdetailedinthe
followinglist:
•digits: 16 ×16images thatarereshapedto256 ×1dimensionalcolumnvectors
•classiﬁer: SVM from Schwaighofer (2002) with RBF kernel width σ=1 and regularization
constantC=10(chosenbygridsearchincross-validationon thetrainingdata).
•trainingset: 47“twos”,53“eights”;trainingerror0.00
•testset: 48“twos”,52 “eights”;testerror0.05
WeapproximatedtheestimatedclasslabelsobtainedbytheSVMwiththeParzenwin dowclassiﬁer
(Parzen window size σ=10.2505, chosen by grid search in cross-validation on the training data).
The SVM and the Parzen window classiﬁer only disagreed on 2% of the test examples, so a good
1811
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
Figure4: USPS digits (training set): “twos” (left) and “eights” (right) with co rrect classiﬁcation.
For each digit from left to right: (i) explanation vector (with black being neg ative, white
beingpositive),(ii)theoriginaldigit,(iii-end)artiﬁcialdigitsalongtheexplan ationvector
towardstheother class.
Figure5: USPS digits(testsetbottompart): “twos”(left)and“eights”(rig ht)withcorrectclassiﬁ-
cation. Foreachdigitfromlefttoright: (i)explanationvector(withblackbe ingnegative,
whitebeingpositive),(ii)theoriginaldigit,(iii-end)artiﬁcialdigitsalongtheex planation
vector towardstheother class.
1812
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
ﬁt was achieved. Figures 4 and 5 show our results. All parts show three examples per row. For
eachexamplewedisplayfromlefttoright: (i)theexplanationvector,(ii)theo riginaldigit,(iii-end)
artiﬁcial digits along the explanation vector towards the other class.6These artiﬁcial digits should
help to understand and interpret the explanation vector. Let us ﬁrst hav e a look at the results on the
trainingset:
Figure4(leftpanel): Letusfocusonthetopexampleframedinred. Thelinethatformsthe“two”
is part of some “eight” from the data set. Thus the parts of the lines that are miss ing show
up in the explanation vector: if the dark parts (which correspond to the missin g lines) are
added to the “two” digit then it will be classiﬁed as an “eight”. In other words , because of
the lack of those parts the digit was classiﬁed as a “two” and not as an “eigh t”. A similar
explanationholdsforthemiddleexampleframedinredinthesameFigure. Notall examples
transform easily to “eights”: Besides adding parts of black lines, some exis ting black spots
(that make the digit be a “two”) must be removed. This is reﬂected in the explan ation vector
bywhitespots/lines. Thebottom“two”,framedinred,isactuallyadashandisin thedataset
by mistake. However, its explanation vector shows nicely which parts have to be added and
which havetoberemoved.
Figure4(right panel): Weseesimilarresultsforthe“eights”class. Theexplanationvectorsagain
tell us how the “eights” have to change to become classiﬁed as “twos”. Howe ver, sometimes
the transformation does not reach the “twos”. This is probably due to the fa ct that some of
the “eights”areinsidethe cloudof “eights”.
On the test set the explanation vectors are not as pronounced as on the tr aining set. However, they
showsimilar tendencies:
Figure5(leftpanel): Weseethecorrectlyclassiﬁed“twos”. Let’sfocusontheexampleframedin
red. Againtheexplanationvectorshowsushowtoedittheimageofthe“two”to transformit
into an “eights”, that is, exactly which parts of the digit were important for the classiﬁcation
result. For several other “twos” the explanation vectors do not directly le ad to the “eights”
butweight thedifferentpartsof thedigits thatwererelevantfortheclas siﬁcation.
Figure5(right panel): Similarly to the training data, we see that also these explanation vectors
are not bringing all “eights” to “twos”. Their explanation vectors mainly sug gest to remove
mostofthe“eights”(blackpixels)andaddsomeblackinthelowerpart(thelig htparts,which
looklike awhiteshadow).
Overall, the explanation vectors tell us how to edit our example digits to change the assigned class
label. Hereby, we get a better understanding of the reasons why the cho sen classiﬁer classiﬁed the
wayitdid.
6. ExplainingMutagenicity Classiﬁcation byGaussianProces ses
InthefollowingSectionwedescribeanapplicationofourlocalgradientex planationmethodologyto
acomplexrealworlddataset. Ouraimistoﬁndstructurespeciﬁctotheproble mdomainthathas not
6. For the sake of simplicity, no intermediate updates were performed, tha t is, artiﬁcial digits were generated by taking
equal-sized steps inthe directiongiven by theoriginalexplanation vector c alculated forthe originaldigit.
1813
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
beenfedintotrainingexplicitlybutiscapturedimplicitlybytheGPCmodelinthehigh- dimensional
featurespaceusedtodetermineitsprediction. Weinvestigatethetaskofpred ictingAmesmutagenic
activityofchemicalcompounds. Notbeingmutagenic(i.e.,notabletocausemu tationsintheDNA)
is an important requirement for compounds under investigation in drug disco very and design. The
Ames test (Ames et al., 1972) is a standard experimental setup for measuring mutagenicity. The
followingexperimentsarebasedonasetofAmestestresultsfor6512che micalcompoundsthatwe
publishedpreviously.7
GPCwas applied asfollows:
•Class 0consistsofnon-mutagenic compounds.
•Class 1consistsofmutagenic compounds.
•Randomly split6512datapoints into2000trainingand4512 testexamples such that:
–Thetrainingsetconsistsof equallymanyclass 0andclass 1examples.
–Forthe steroidcompound classthebalance inthetrainingandtestsetisenfo rced.
•10 additional random splits were investigated individually. This conﬁrmed the results pre-
sentedbelow.
•Each example (chemical compound) is represented by a vector of counts o f 142 molecular
substructurescalculatedusingthe D RAGONsoftware(Todeschiniet al.,2006).
•Normalize trainingandtestsetusingthemeanand varianceof thetrainingset.
•Apply GPCmodelwith RBF kernel.
•Performance (84 % area under curve) conﬁrms our previous results ( Hansen et al., 2009).
Errorratescan beobtained fromFigure6.
Togetherwiththepredictionwecalculatedtheexplanationvector(asintroduc edinDeﬁnition2)for
each testpoint. Theremainder of thisSectionis anevaluationof theselocale xplanations.
In Figures 7 and 8 we show the distribution of the local importance of selected features across
thetestset: Foreachinputfeaturewegenerateahistogramoflocalimpor tancevalues,asindicated
by its corresponding entry in the explanation vector of each of the 4512 tes t compounds. The
features examined in Figure 7 are counts of substructures known to caus e mutagenicity. We show
all approved “speciﬁc toxicophores” introduced by Kazius et al. (200 5) that are also represented
in the D RAGONset of features. The features shown in Figure 8 are known to detoxify c ertain
toxicophores (again see Kazius et al., 2005). With the exception of 7(e) th e toxicophores also have
a toxifying inﬂuence according to our GPC prediction model. Feature 7(e) s eems to be mostly
irrelevant for the prediction of the GPC model on the test points. In contras t the detoxicophores
show overall negative inﬂuence on the prediction outcome of the GPC model. Modifying the test
compoundsbyaddingtoxicophoreswillincreasetheprobabilityofbeingmuta genicaspredictedby
theGPC modelwhileadding detoxicophores willdecreasethispredicted pro bability.
7. See Hansen et al. (2009) for results of modeling this set using differ ent machine learning methods. The data itself is
available online at http://ml.cs.tu-berlin.de/toxbenchmark .
1814
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
00.20.40.60.8100.20.40.60.81
false positive ratetrue positive rate

AUC = 0.84
Figure6: Receiver operatingcharacteristiccurveofGPC modelfor muta genicityprediction
We have seen that the conclusions drawn from our explanation vectors a gree with established
knowledgeabouttoxicophoresanddetoxicophores. Whilethisisreassu ring,suchasanitycheckre-
quiredexistingknowledgeaboutwhichcompoundsaretoxicophoresand detoxicophoresandwhich
arenot. Thusitisinterestingtoaskwhetherwealsocouldhave discovered thatknowledgefromthe
explanation vectors. To answer this question we ranked all 142 features by the means of their local
gradients.8Clear trends result: 9 out of 10 known toxicophores can be found close to the top of the
list(meanrankof19). Theonlyexception(rank81)isthearomaticnitrosamin efeature.9Thistrend
is even stronger for the detoxicophores: The mean rank of these ﬁve fe atures is 138 (out of 142),
that is, they consistently exhibit the largest negative local gradients. Con sequently, the established
knowledge about toxicophores and detoxicophores could indeed have beendiscovered using our
methodology.
In the following paragraph we will discuss steroids10as an example of an important compound
class for which the meaning of features differs from this global trend, so that local explanation
vectorsareneeded tocorrectlyidentifyrelevantfeatures.
Figure 9 displays the difference in relevance of epoxide (a) and aliphatic n itrosamine (c) sub-
structures for the predicted mutagenicity of steroids and non-steroid compo unds. For compari-
son we also show the distributions for compounds chosen at random from the test set (b,d). Each
subﬁgure contains two measures of (dis-)similarity for each pair of distribu tions. The p-value of
the Kolmogorov-Smirnoff test (KS) gives the probability of error when re jecting the hypothesis
that both relative frequencies are drawn from the same underlying distrib ution. The symmetrized
8. Tables resulting from this ranking are made available as a supplement to this paper and can be downloaded from the
journals website.
9. This ﬁnding agrees with the result obtained by visually inspecting Figure 7 (e). We found that only very few com-
pounds withthisfeature arepresentinthedata set. Consequently,detectio n of this featureis onlypossibleif enough
of these few compounds are included in the training data. This was not the c ase in the random split used to produce
the resultspresented above.
10. Steroidsarenaturalproductsandoccurinhumans,animals,andp lants. Theyhaveacharacteristicbackbonecontain-
ingfourfusedcarbon-rings. Manyhormonesimportanttothedevelop mentofthehumanbodyaresteroids,including
androgens, estrogens, progestagens, cholesterol and natural an abolics. These have been used as starting points for
the development of many differentdrugs,including themostreliable co ntraceptives currently onthe market.
1815
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
−1−0.5 00.5100.020.040.060.080.10.120.14
local gradient DR17:nArNO2relative frequency
(a) aromatic nitro−1−0.5 00.5100.020.040.060.080.1
local gradient DR17:nArNH2relative frequency
(b) aromatic amine−1−0.5 00.5100.010.020.030.040.05
local gradient DR17:nArNOrelative frequency
(c) aromatic nitroso
−1−0.5 00.5100.020.040.060.08
local gradient DR17:nRNNOxrelative frequency
(d) aliphatic nitrosamine−1−0.5 00.5100.010.020.030.040.050.060.07
local gradient DR17:nArNNOxrelative frequency
(e) aromatic nitrosamine−1−0.5 00.5100.010.020.030.040.05
local gradient DR17:nOxiranesrelative frequency
(f) epoxide
−1−0.5 00.5100.010.020.030.040.050.060.07
local gradient DR17:nAziridinesrelative frequency
(g) aziridine−1−0.5 00.5100.020.040.060.080.10.12
local gradient DR17:nN=Nrelative frequency
(h) azide−1−0.5 00.5100.020.040.060.080.1
local gradient DR17:nArNHOrelative frequency
(i) aromatic hydroxylamine
−1−0.5 00.5100.010.020.030.040.050.060.07
local gradient X:AliphaticHaliderelative frequency
(j) aliphatic halide
Figure7: Distribution of local importance of selected features across the te st set of 4512 com-
pounds. Nineoutoftenknowntoxicophores(Kaziusetal.,2005)indee dexhibitpositive
localgradients.
1816
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
−1−0.5 00.5100.020.040.060.080.1
local gradient DR17:nSO2Nrelative frequency
(a) sulfonamide−1−0.5 00.5100.0050.010.0150.020.0250.030.035
local gradient DR17:nSO2OHrelative frequency
(b) sulfonicacid−1−0.5 00.5100.010.020.030.040.05
local gradient DR17:nS(=O)2relative frequency
(c) arylsulfonyl
−1−0.5 00.5100.010.020.030.040.05
local gradient DR17:nRCOOHrelative frequency
(d) aliphatic carboxylic acid−1−0.5 00.5100.020.040.060.080.1
local gradient DR17:nArCOOHrelative frequency
(e) aromatic carboxylic acid
Figure8: Distribution of local importance of selected features across the te st set of 4512 com-
pounds. Allﬁveknowndetoxicophores exhibit negativelocal gradien ts
Kullback-Leibler divergence (KLD) gives a metric of the distance betwee n the two distributions.11
Whilecontainingepoxidesgenerallytendstomakemoleculesmutagenic(seedisc ussionabove),we
donotobservethiseffectforsteroids: InFigure9(a),almostallepo xidecontainingnon-steroidsex-
hibitpositivegradients,therebyfollowingtheglobaldistributionofepoxide containingcompounds
as shown in Figure 7(f). In contrast, almost all epoxide containing steroid s exhibit gradients just
below zero. “Immunity” of steroids to the epoxide toxicophore is an establishe d fact and has ﬁrst
been discussed by Glatt et al. (1983). This peculiarity in chemical space is clearly exhibited by
the local explanation given by our approach. For aliphatic nitrosamine, the situation in the GPC
model is less clear but still the toxifying inﬂuence seems to be less in steroids tha n in many other
compounds. To our knowledge, this phenomenon has not yet been discu ssed in the pharmaceutical
literature.
Inconclusion,wecanlearnfromtheexplanation vectorsthat:
•Toxicophores tendtomakecompounds mutagenic(class1).
•Detoxicophores tend tomakecompounds non-mutagenic (class0).
•Steroids are immune to the presence of some toxicophores (epoxide, possib ly also aliphatic
nitrosamine).
11. SymmetryisachievedbyaveragingthetwoKullback-Leiblerdiverge nces:KL(P1,P2)+KL(P2,P1)
2,comparetoJohnson
and Sinanovic (2000). To prevent zero-values in the histograms which would lead to inﬁnite KL distances, an ε>0
has been added toeach bincount.
1817
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
−0.5 0 0.5 100.050.10.150.2KS p−value = 0.00
symm. KLD = 23.14
local gradient DR17:nOxiranesrelative frequency

steroid
non−steroid
(a) epoxide feature: steroidvs. non-steroid−0.5 0 0.5 100.020.040.060.08KS p−value = 0.12
symm. KLD = 11.08
local gradient DR17:nOxiranesrelative frequency

random
others
(b) epoxide feature: randomcompounds vs. therest
−0.2 00.2 0.4 0.600.050.10.150.20.25KS p−value = 0.00
symm. KLD = 12.51
local gradient DR17:nRNNOxrelative frequency

steroid
non−steroid
(c) aliphatic nitrosaminefeature: steroidvs. non-steroid−0.2 00.2 0.4 0.600.020.040.060.080.1KS p−value = 0.16
symm. KLD = 5.07
local gradient DR17:nRNNOxrelative frequency

random
others
(d) aliphatic nitrosamine feature: random compounds vs.
therest
Figure9: The local distribution of feature importance to steroids and rando m non-steroid com-
poundssigniﬁcantlydiffersfortwoknowntoxicophores. Thesmalllocal gradientsfound
forthesteroids(showninblue)indicatethatthepresenceofeachtoxicop horeisirrelevant
tothemoleculestoxicity. Fornon-steroids(showninred)theknowntoxicoph oresindeed
exhibitpositivelocal gradients.
7. Related Work
Assigning potentially different explanations to individual data points distingu ishes our approach
from conventional feature extraction methods that extract global featur es that are relevant for all
data points, that is, those features that allow to achieve a small overall pred iction error. Our notion
of explanation is not related to the prediction error, but only to the label pro vided by the prediction
algorithm. Eveniftheerrorislarge,ourframeworkisabletoanswertheque stionwhythealgorithm
has decidedon adatapointthe wayitdid.
1818
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
Theexplanationvectorproposedhereissimilarinspirittosensitivityanalysis whichiscommon
invariousareasofinformationscience. Aclassicalexampleisoutliersensiti vityinstatistics(Ham-
pel et al., 1986). In this case, the effects of removing single data points on estimated parameters
are evaluated by an inﬂuence function. If the inﬂuence for a data point is signiﬁcantly large, it is
detected as an outlier and should be removed for the following analysis. In r egression problems,
leverage analysis is a procedure along similar lines. It detects leverage po ints which have potential
to give large impact on the estimate of the regression function. In contrast to the inﬂuential points
(outliers),removingaleveragesamplemaynotactuallychangetheregress or,ifitsresponseisvery
close to the predicted value. E.g., for linear regression the samples whose in puts are far from the
mean are the leverage points. Our framework of explanation vectors cons iders a different view. It
describes the inﬂuence of movingsingle data points locally and it thus answers the question which
directions are locally most inﬂuential to the prediction. The explanation vector s are used to extract
sensitive features that are relevant to the prediction results, rather than detecting/eliminating the
inﬂuentialsamples.
In recent decades, explanation of results by expert systems has been an important topic in the
ArtiﬁcialIntelligencecommunity. EspeciallyforexpertsystemsbasedonBay esianbeliefnetworks,
such explanation is crucial in practical use. In this context sensitivity ana lysis has also been used
as a guiding principle (Horvitz et al., 1988). There the inﬂuence is evaluate d by removing a set of
variables (features) from the evidence and the explanation is constructe d from those variables that
affect inference (relevant variables). For example, Suermondt (19 92) measures the cost of omitting
asinglefeature Eibythecross-entropy
H−(Ei)=H(p(D|E);P(D|E\Ei))=N
∑
j=1P(dj|E)logP(dj|E)
p(dj|E\Ei),
whereEdenotes the evidence and D= (d1,...,dN)Tis the target variable. The cost of a subset
F⊂Ecan be deﬁned similarly. This line of research is more connected to our work , because
explanation candepend ontheassignedvalues of theevidence E,andis thuslocal.
Similarly Robnik-Sikonja and Kononenko (2008) and Strumbelj and Konone nko (2008) try to
explain the decision of trained kNN-, SVM-, and ANN-models for individua l instances by measur-
ing the difference in their prediction with sets of features omitted. The cost of omitting features
is evaluated as the information difference, the log-odds ratio, or the differ ence of probabilities be-
tweenthemodelwithknowledgeaboutallfeaturesandwithomissions,respe ctively. Toknowwhat
the prediction would be without the knowledge of a certain feature the model is r etrained for every
choice of features whose inﬂuence is to be explained. To save the time of co mbinatorial training
Robnik-Sikonja and Kononenko (2008) propose to use neutral values which have to be estimated
by a known prior distribution of all possible parameter values. As a theoretic al framework for con-
sidering feature interactions, Strumbelj and Kononenko (2008) propos e to calculate the differences
between modelpredictions foreverychoiceof featuresubset.
For multi-layer perceptrons Fraud and Clrot (2002) measure the importanc e of individual in-
put variables on clusters of test points. Therefore the change in the model output is evaluated for
the change of a single input variable in a chosen interval while all other inpu t variables are ﬁxed.
LemaireandFeraud(2007)useasimilarapproachonaninstancebyinstan cebasis. Byconsidering
eachinputvariableinturnthereisnowaytomeasureinputfeatureinteractio nsonthemodeloutput
(seeLeCunetal.,1998).
1819
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
p(       |   )y=1  x
x1x1
x20.5
Figure10: ζ(x)isthe zerovector inthemiddleof theclusterin themiddle.
The principal differences between our approach and these framewor ks are: (i) We consider
continuous features and no structure among them is required, while some oth er frameworks start
from binary features and may require discretization steps with the need to es timate parameters for
it. (ii) We allow changes in any direction, that is, any weighted combination of va riables, while
other approaches onlyconsider onefeatureatatimeor theomissionofase tof variables.
8. Discussion
We have shown that our methods for calculating / estimating explanation vector s are useful in a
varietyofsituations. Inthefollowingwediscusstheir limitations.
8.1 What Can We Doif theDerivativeis Zero?
This situation is depicted in Figure 10. In the lower panel we see a two-dimensio nal data set con-
sisting of three clusters. The middle cluster has a different class than the clu sters on the left and
on the right. Only the horizontal coordinate (i.e., x1) is relevant for the classiﬁcation. The upper
panel shows the projected data and a representative slice of ζ(x). However, the explanation ζ(x)
for the center point of the middle cluster is the zero vector, because at that pointp(Y=1|X=x)
is maximal. What can we do in such situations? Actually, the (normalized) explana tion vector
is derived from the following optimization problem for ﬁnding the locally most inﬂ uential direc-
tion: argmax/bardblε/bardbl=1{p(Y/\e}atio\slash=g∗(x0)|X=x0+ε)−p(Y/\e}atio\slash=g∗(x0)|X=x0)}. In case that the ﬁrst deriva-
tive of the above criterion is zero, its Taylor expansion starts from the sec ond order term, which
is a quadratic form of its Hessian matrix. In the example data set with three cluste rs, the explana-
tion vector is constant along the second dimension. The most interesting direction is given by the
eigenvector corresponding to the largest eigenvalue of the Hessian. Th is direction will be in our
example along the ﬁrst dimension. Thus, we can learn from the Hessian that the ﬁrst coordinate
is relevant for the classiﬁcation, but we do not obtain an orientation for it. I nstead it means that
both directions (left and right) will inﬂuence the classiﬁcation. However, if the conditional distri-
1820
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
butionP(Y=1|X=x)is ﬂat in some regions, no meaningful explanation can be obtained by the
gradient-based approach with the remedy mentioned above. Practically, by using Parzen window
estimators with larger widths, the explanation vector can capture coarse stru ctures of the classiﬁer
atthepointsthatarenotsofarfromtheborders. InA.3.2wegiveanillus trationofthispoint. Inthe
future,wewouldliketoworkonglobalapproaches,forexample,based ondistancestotheborders,
or extensions of the approach by Robnik-Sikonja and Kononenko (200 8). Since these procedures
are expected to be computationally demanding, our proposal is useful in pr actice, in particular for
probabilisticclassiﬁers.
8.2 Does OurFramework Generate DifferentExplanations forDiffere nt Prediction Models?
When using the local gradient of the model prediction directly as in Deﬁnition 2 and Section 6,
the explanation follows the given model precisely by deﬁnition. For the estimatio n framework this
dependsonwhetherthedifferentclassiﬁersclassifythedatadifferen tly. Inthatcasetheexplanation
vectors will be different, which makes sense, since they should explain the classiﬁer at hand, even
if its estimated labels were not all correct. On the other hand, if the different classiﬁersagree on all
labels,theexplanation willbeexactlyequal.
8.3 Which ImplicitLimitationsDoAnalyticalGradients InheritFrom Gaussia nProcess
Models?
A particular phenomenon can be observed at the boundaries of the trainin g data: Far from the
training data, Gaussian Process Classiﬁcation models predict a probability o f 0.5 for the positive
class. When querying the model in an area of the feature space where pr edictions are negative,
and one approaches the boundaries of the space populated with training d ata, explanation vectors
will point away from any training data and therefore also away from area s of positive prediction.
This behavior can be observed in Figure 1(d), where unit length vector s indicate the direction of
explanation vectors. In the right hand side corner, arrows point away from the triangle. However,
we can see that the length of these vectors is so small that they are not even visible in Figure
1(c). Consequently, this property of GPC models does not pose a restric tion for identifying the
locallymostinﬂuentialfeaturesbyinvestigatingthefeatureswiththehighestab solutevaluesinthe
respectivepartialderivatives,as showninSection 6.
8.4 Stationarityof theData
Since explanation vectors are deﬁned as local gradients of the model pre diction (see Deﬁnition 2),
no assumption on the data is made: The local gradients follow the predictive mod el in any case. If,
however, the model to be explained assumes stationarity of the data, the expla nation vectors will
inherit this limitation and reﬂect any shortcomings of the model (e.g., when the mod el is applied
to non-stationary data). Our method for estimating explanation vectors, on the other hand, assumes
stationarityof thedata.
When modeling data that is in fact non-stationary, appropriate measures to d eal with such data
sets should be taken. One option is to separate the feature space into stationa ry and non-stationary
parts using Stationary Subspace Analysis as introduced by von B ¨unau et al. (2009). For further
approaches to data set shift see Sugiyama et al. (2007b), Sugiyama et a l. (2007a), and the book by
Quionero-Candela etal.(2009).
1821
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
9. Conclusion
This paper proposes a method that sheds light on the black boxes of nonlin ear classiﬁers. In other
words, we introduce a method that can explain the local decisions taken by a rbitrary (possibly)
nonlinearclassiﬁcationalgorithms. Inanutshell,theestimatedexplanationsare localgradientsthat
characterizehowadatapointhastobemovedtochangeitspredictedlabel. F ormodelswheresuch
gradient information cannot be calculated explicitly, we employ a probabilistic a pproximate mimic
of thelearningmachine tobeexplained.
To validate our methodology we show how it can be used to draw new conclus ions on how the
various Iris ﬂowers in Fisher’s famous data set are different from ea ch other and how to identify
the features with which certain types of digits 2 and 8 in the USPS data set can b e distinguished.
Furthermore, we applied our method to a challenging drug discovery proble m. The results on that
data fully agree with existing domain knowledge, which was not available to our method. Even
local peculiarities in chemical space (the extraordinary behavior of stero ids) was discovered using
thelocal explanations givenby ourapproach.
Future directions are two-fold: First we believe that our method will ﬁnd its wa y into the tool
boxes of practitioners who not only want to automatically classify their data bu t who also would
like to understand the learned classiﬁer. Thus using our explanation frame work in computational
biology (see Sonnenburg et al., 2008) and in decision making experiments in psychophysics (e.g.,
Kienzle et al., 2009) seems most promising. The second direction is to genera lize our approach to
other predictionproblems suchas regression.
Acknowledgments
This work was supported in part by the FP7-ICT Programme of the Europe an Community, under
thePASCAL2NetworkofExcellence,ICT-216886andbyDFGGrantMU 987/4-1. Wewouldlike
to thank Andreas Sutter, Antonius Ter Laak, Thomas Steger-Hartmann and Nikolaus Heinrich for
publishingtheAmes mutagenicitydata set(Hansenetal., 2009).
AppendixA.
In the following we present the derivation of direct local gradients and illu strate aspects like the
effect of different kernel functions, outliers and local non-linearitie s. Furthermore we present the
derivation of explanation vectors based on the parzen window estimation an d illustrate how the
qualityoftheﬁtoftheParzenwindowapproximationaffectsthequalityofthee stimatedexplanation
vectors.
A.1 Derivation of Direct Local Gradients
Equation(1)is derivedby thefollowingsteps:
1822
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
∇p(x)|x=x0
=∇1
2erfc/parenleftBigg
−f(x)√
2∗/radicalbig
1+varf(x)/parenrightBigg/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=x0
=∇1
2/parenleftBigg
1−erf/parenleftBigg
−f(x)√
2∗/radicalbig
1+varf(x)/parenrightBigg/parenrightBigg/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=x0
=−1
2∇erf/parenleftBigg
−f(x)√
2∗/radicalbig
1+varf(x)/parenrightBigg/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=x0
=−exp/parenleftBig
−f(x0)2
2(1+varf(x0))/parenrightBig
√π∇/parenleftBigg
−f(x)√
2∗/radicalbig
1+varf(x)/parenrightBigg/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=x0
=−exp/parenleftBig
−f(x0)2
2(1+varf(x0))/parenrightBig
√π/parenleftBigg
−1√
2∇/parenleftBigg
f(x)/radicalbig
1+varf(x)/parenrightBigg/vextendsingle/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=x0/parenrightBigg
=exp/parenleftBig
−f(x0)2
2(1+varf(x0))/parenrightBig
√
2π/parenleftBigg
∇f(x)|x=x0/radicalbig
1+varf(x0)+f(x0)/parenleftbigg
∇varf(x)/vextendsingle/vextendsingle
x=x0∗−1
2(1+varf(x0))−3
2/parenrightbigg/parenrightbigg
=exp/parenleftBig
−f(x0)2
2(1+varf(x0))/parenrightBig
√
2π/parenleftBigg
∇f(x)|x=x0/radicalbig
1+varf(x0)−1
2f(x0)
(1+varf(x0))3
2∇varf(x)|x=x0/parenrightBigg
.
A.2 IllustrationofDirect Local Gradients
In the following we give some illustrative examples of our method to explain models using local
gradients. Since the explanation is derived directly from the respective mo del, it is interesting to
investigate its acurateness depending on different model parameters and in instructive scenarios.
We examine the effects that local gradients exhibit when choosing differe nt kernel functions, when
introducingoutliers,and whentheclassesarenot linearlyseparableloca lly.
A.2.1 C HOICE OF KERNELFUNCTION
Figure 11 shows the effect of different kernel functions on the triang le toy data from Figure 1. The
followingobservationscanbemade:
•In any case note that the local gradients explain the model, which in turn may or may not
capture thetruesituation.
•InSubﬁgure11(a)thelinearkernelleadstoamodelwhichfailstocapture thenon-linearclass
separation. This model misspeciﬁcation is reﬂected by the explanations give n for this model
inSubﬁgure11(b).
•The rational quadratic kernel is able to more accurately model the non-linea r separation. In
Subﬁgure 11(c) a non-optimal degree parameter has been chosen for illustrative purposes.
1823
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
0
0.2
0.4
0.6
0.8
100.20.40.60.810.10.150.20.250.30.350.40.450.5
(a) linear model00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(b) linear explanation
0
0.2
0.4
0.6
0.8
100.20.40.60.810.3030.30320.30340.30360.30380.3040.3042
(c) rationalquadratic model00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(d) rational quadratic explanation
Figure11: Theeffectofdifferentkernelfunctions tothelocalgrad ientexplanations
For other parameter values the rational quadratic kernel leads to similar res ults as the RBF
kernelfunctionusedinFigure1.
•The explanations in Subﬁgure 11(d) obtained for this model show local pe rturbations at the
small “bumps” of the model but the trends towards the positive class are still c lear. As pre-
viously observed in Figure 1, the explanations make clear that both feature s interact at the
corners andonthehypotenuseof thetriangleclass.
A.2.2 O UTLIERS
InFigure12theeffectsoftwooutliersintheclassiﬁcationdatatoGPCwithRBFk ernelareshown.
Oncemore,notethatthelocalgradientsexplainthemodel,whichinturnmayorma ynotcapturethe
truesituation. Thesizeoftheregionaffectedbytheoutliersdependsonthe kernelwidthparameter.
Weconsider thefollowingitems:
1824
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
0
0.2
0.4
0.6
0.8
100.20.40.60.81−1−0.500.51
(a) outliers inclasses0
0.2
0.4
0.6
0.8
100.20.40.60.8100.20.40.60.81
(b) outliers inmodel
00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(c) outlier explanation
Figure12: Theeffectof outlierstothelocal gradientexplanations
•Local gradients are in the same way sensitive to outliers as the model which the y try to
explain. Here a single outlier deforms the model and with it the explanation which may be
extracted fromit.
•Being derivatives the sensitivity of local gradients to a nearby outlier is inc reased over the
sensitivityof themodelprediction itself.
•Thus the local gradient of a point near an outlier may not reﬂect a true ex planation of the
features important in reality. Nevertheless it is the model here which is wrong around an
outlier intheﬁrstplace.
•The histograms in the Figures 7, 8, and 9 in Section 6 show the trends of the re spective
features inthedistributionof alltestpoints andarethus notaffectedbysin gleoutliers.
To compensate for the effect of outliers to the local gradients of points in the affected region
we propose to use a sliding window method to smooth the gradients around each point of interest.
1825
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
Thus for each point use the mean of all local gradients in the hypercube c entered at this point and
of appropriate size. This way the disrupting effect of an outlier is averag ed out for an appropriately
chosenwindowsize.
A.2.3 L OCALNON-LINEARITY
0
0.2
0.4
0.6
0.8
100.20.40.60.81−1−0.500.51
(a) locallynon-linear object0
0.2
0.4
0.6
0.8
100.20.40.60.8100.20.40.60.81
(b) locally non-linear model
00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(c) locally non-linear explanation
Figure13: Theeffectoflocal non-linearitytothelocalgradientexplan ations
Theeffectoflocallynon-linearclassboundariesinthedataisshowninFig ure13againforGPC
withanRBF kernel. Thefollowingpointscan beobserved:
•Allthe non-linearclass boundaries areaccuratelyfollowedbythelocal g radients.
•The circle shaped region of negative examples surrounded by positive ones shows the full
rangeof featureinteractions towardsthepositiveclass.
•On the ridge of single positive instances the model introduces small valleys wh ich are re-
ﬂected bythelocal gradients.
1826
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
A.3 EstimatingbyParzen Window
Finallyweelaborateonsomedetailsofourestimationapproachoflocalgradie ntsbyParzenwindow
approximation. Firstwegivethederivationtoobtaintheexplanationvectora ndsecondweexamine
howtheexplanation varieswiththegoodness of ﬁtof theParzenwindow meth od.
A.3.1 D ERIVATION OF EXPLANATION VECTORS
Thesearemoredetails onthe derivationof Deﬁnition3. We usetheindex set Ic={i|g(xi)=c}:
∂
∂xkσ(x)=−x
σ2kσ(x)
∂
∂xˆpσ(x,y/\e}atio\slash=c)=1
n∑
i/∈Ickσ(x−xi)−(x−xi)
σ2
∂
∂xˆpσ(y/\e}atio\slash=c|x)
=/parenleftBig
∑i/∈Ick(x−xi)/parenrightBig/parenleftBig
∑n
i=1k(x−xi)(x−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2
−/parenleftBig
∑i/∈Ick(x−xi)(x−xi)/parenrightBig/parenleftBig
∑n
i=1k(x−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2
=/parenleftBig
∑i/∈Ick(x−xi)/parenrightBig/parenleftBig
∑i∈Ick(x−xi)(x−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2
−/parenleftBig
∑i/∈Ick(x−xi)(x−xi)/parenrightBig/parenleftBig
∑i∈Ick(x−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2
andthus fortheindex set Ig(z)={i|g(xi)=g(z)}
ˆζ(z)=∂
∂xˆp(y/\e}atio\slash=g(z)|x)/vextendsingle/vextendsingle/vextendsingle/vextendsingle
x=z
=/parenleftBig
∑i/∈Ig(z)k(z−xi)/parenrightBig/parenleftBig
∑i∈Ig(z)k(z−xi)(z−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2
−/parenleftBig
∑i/∈Ig(z)k(z−xi)(z−xi)/parenrightBig/parenleftBig
∑i∈Ig(z)k(z−xi)/parenrightBig
σ2/parenleftbig
∑n
i=1k(z−xi)/parenrightbig2.
A.3.2 G OODNESS OF FIT BYPARZENWINDOW
In our estimation framework the quality of the local gradients depends on the a pproximation of the
classiﬁerwewanttoexplainbyParzenwindowsforwhichwecancalculate theexplanationvectors
as givenbyDeﬁnition3.
Figure 14(a) shows an SVM model trained on the classiﬁcation data from Fig ure 13(a). The
localgradientsestimatedforthismodelbydifferentParzenwindowappro ximationsaredepictedin
Subﬁgures14(b),14(c),and14(d). We observethefollowingpoints :
1827
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
0
0.2
0.4
0.6
0.8
100.20.40.60.8100.20.40.60.81
(a) SVM model00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(b) estimated explanation with σ=0.00069
00.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(c) estimated explanation with σ=0.100.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 100.10.20.30.40.50.60.70.80.91
(d) estimated explanation with σ=1.0
Figure14: Good ﬁt of Parzen window approximation affects the quality of th e estimated explana-
tionvectors
•TheSVM modelwas trainedwith C=10and usingan RBF kernelof width σ=0.01.
•In Subﬁgure 14(b) a small window width has been chosen by minimizing the mean absolute
error over the validation set of labels predicted by the SVM classiﬁer. Thu s we obtain ex-
plaining local gradients on the class boundaries but zero vectors in the inn er class regions.
While this resembles the piecewise ﬂat SVM model most accurately it may be more u seful
practically to choose a larger width to obtain non-zero gradients pointing to the borders in
this regionsas well. Fora moredetailed discussionof zerogradientsseeSe ction8.
•A larger width practically useful in this example is shown in Subﬁgure 14(c). Here the local
gradients intheinner classregions pointtotheother classas well.
1828
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
•For a too large window width in Subﬁgure 14(d) the approximation fails to obtain local gra-
dientswhichcloselyfollowthemodel. Hereonlytwodirectionsareleftandtheg radientsfor
the blueclassonthe leftand onthebottompointinthewrongdirection.
References
B. N. Ames, E. G. Gurney, J. A. Miller, and H. Bartsch. Carcinogens as frameshift mutagens:
Metabolitesandderivativesof2-acetylaminoﬂuoreneandotheraromatica minecarcinogens. Pro-
ceedingsoftheNationalAcademyofSciencesoftheUnitedStatesofAmer ica,69(11):3128–3132,
1972.
C.M.Bishop. NeuralNetworks for PatternRecognition . OxfordUniversityPress,1995.
L. Devroye, L. Gy ¨orﬁ, and G. Lugosi. A Probabilistic Theory of Pattern Recognition . Number 31
inApplications of Mathematics.Springer,NewYork,1996.
R.A. Fisher. The use of multiple measurements in taxonomic problems. Annals of Eugenics , 7:
179–188, 1936.
R. Fraud and F. Clrot. A methodology to explain neural network classiﬁcatio n.Neural Networks ,
15(2):237–246,2002. doi: 10.1016/S0893-6080(01)00127-7.
H. Glatt, R. Jung, and F. Oesch. Bacterial mutagenicity investigation of epox ides: drugs, drug
metabolites, steroids and pesticides. Mutation Research/Fundamental and Molecular Mecha-
nismsofMutagenesis ,111(2):99–118,1983. doi: 10.1016/0027-5107(83)90056-8.
I.GuyonandA.Elisseeff. Anintroductiontovariableandfeatureselectio n.TheJournalofMachine
LearningResearch ,3:1157–1182, 2003.
F.R.Hampel,E.M.Ronchetti,P.J.Rousseeuw,andW.A.Stahel. RobustStatistics: TheApproach
BasedonInﬂuenceFunctions . Wiley,NewYork,1986.
K. Hansen, S. Mika, T. Schroeter, A. Sutter, A. Ter Laak, T. Steger- Hartmann, N. Heinrich, and
K.-R. M¨uller. A benchmark data set for in silico prediction of ames mutagenicity. Journal of
Chemical InformationandModelling ,49(9):2077–2081,2009.
T.Hastie,R.Tibshirani,andJ.Friedman. The ElementsofStatisticalLearning . Springer,2001.
E. J. Horvitz, J. S. Breese, and M. Henrion. Decision theory in expert systems and artiﬁcial in-
teligence. Journal of Approximation Reasoning , 2:247–302, 1988. Special Issue on Uncertainty
inArtiﬁcialIntelligence.
D. H. Johnson and S. Sinanovic. Symmetrizing the Kullback-Leibler distance . Technical report,
IEEETransactionsonInformationTheory,2000.
J. Kazius, R. McGuire, and R. Bursi. Derivation and validation of toxicop hores for mutagenicity
prediction. J.Med.Chem. ,48:312–320, 2005.
W. Kienzle, M. O. Franz, B. Sch ¨olkopf, and F. A. Wichmann. Center-surround patterns emerge as
optimalpredictors forhumansaccade targets. JournalofVision ,9(5):1–15,2009.
1829
BAEHRENS , SCHROETER , HARMELING , KAWANABE , HANSEN AND M¨ULLER
M.KussandC.E.Ramussen. Assesingapproximateinferenceforbinar ygaussianprocessclassiﬁ-
cation.Journalof MachineLearningResearch ,6:1679–1704,2005.
Y.LeCun,L.Bottou,G.B.Orr,andK.-R.M ¨uller. Efﬁcientbackprop. InG.B.OrrandK.-R.M ¨uller,
editors,NeuralNetworks: Tricks oftheTrade ,pages 9–53.Springer,1998.
V.LemaireandR.Feraud. Unem ´ethoded’interpr ´etationdescores. In EGC,pages191–192,2007.
K.-R. M¨uller, S. Mika, G. R ¨atsch, K. Tsuda, and B. Sch ¨olkopf. An introduction to kernel-based
learningalgorithms. NeuralNetworks,IEEETransactionson ,12(2):181–201,2001.
OlgaObrezanovaandMatthewD.Segall. Gaussianprocessesforclass iﬁcation: QSARmodelingof
ADMET and target activity. Journal of Chemical Information and Modeling , April 2010. ISSN
1549-9596. doi: doi:10.1021/ci900406x. URL http://dx.doi.org/10.1021/ci900406x .
O. Obrezanova, G. Cs ´anyi, J. M. R. Gola, and M. D. Segall. Gaussian processes: A method for
automatic QSARmodelling ofadme properties. J.Chem. Inf.Model.
O. Obrezanova, J. M. R. Gola, E. J. Champness, and M. D. Segall. Auto matic QSAR modeling of
adme properties: blood-brain barrier penetration and aqueous solubility. J. Comput.-Aided Mol.
Des.,22:431–440, 2008.
J. C. Platt. Probabilistic outputs for support vector machines and compariso ns to regularized likeli-
hoodmethods. In Advances inLargeMarginClassiﬁers ,pages 61–74.MITPress,1999.
J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawr ence.Dataset Shift in Ma-
chineLearning . TheMITPress,2009.
C.E.RasmussenandC.K.I.Williams. GaussianProcessesforMachineLearning . Springer,2006.
M. Robnik-Sikonja and I. Kononenko. Explaining classiﬁcations for indi vidual instances. IEEE
TKDE,20(5):589–600,2008.
B. Sch¨olkopfand A.Smola. Learningwith Kernels . MIT,2002.
T.Schroeter,A.Schwaighofer,S.Mika,A.TerLaak,D.Suelzle,U. Ganzer,N.Heinrich,andK.-R.
M¨uller. Estimating the domain of applicability for machine learning QSAR models: A stud y on
aqueous solubility of drug discovery molecules. Journal of Computer Aided Molecular Design ,
21(9):485–498,2007a.
T.Schroeter,A.Schwaighofer,S.Mika,A.TerLaak,D.Suelzle,U. Ganzer,N.Heinrich,andK.-R.
M¨uller. Machinelearningmodelsforlipophilicityandtheirdomainofapplicability. Mol.Pharm. ,
4(4):524–538,2007b.
T. Schroeter, A. Schwaighofer, S. Mika, A. Ter Laak, D. S ¨ulzle, U. Ganzer, N. Heinrich, and K.-
R. M¨uller. Predicting lipophilicity of drug discovery molecules using gaussian pro cess models.
ChemMedChem , 2(9):1265–1267,2007c.
A. Schwaighofer. SVM Toolbox for Matlab, Jan 2002. URL http://ida.first.fraunhofer.
de/˜anton/software.html .
1830
HOW TOEXPLAININDIVIDUAL CLASSIFICATION DECISIONS
A. Schwaighofer, T. Schroeter, S. Mika, J. Laub, A. Ter Laak, D. S ¨ulzle, U. Ganzer, N. Heinrich,
and K.-R. M ¨uller. Accurate solubility prediction with error bars for electrolytes: A machin e
learningapproach. JournalofChemical InformationandModelling ,47(2):407–424,2007.
A.Schwaighofer,T.Schroeter,S.Mika,K.Hansen,A.TerLaak,P .Lienau,A.Reichel,N.Heinrich,
andK.-R.M ¨uller.Aprobabilisticapproachtoclassifyingmetabolicstability. JournalofChemical
InformationandModelling ,48(4):785–796,2008.
S.Sonnenburg,A.Zien,P.Philips,andG.R ¨atsch. POIMs: positionaloligomerimportancematrices
—understandingsupportvector machine basedsignaldetectors. Bioinformatics ,2008.
E.StrumbeljandI.Kononenko. Towardsamodelindependentmethodfo rexplainingclassiﬁcation
for individual instances. In I.-Y. Song, J. Eder, and T.M. Nguyen, e ditors,Data Warehousing
and Knowledge Discovery , volume 5182 of Lecture Notes in Computer Science , pages 273–282.
Springer,2008.
H. Suermondt. Explanation in Bayesian Belief Networks . PhD thesis, Department of Computer
Science andMedicine,StanfordUniversity,Stanford,CA,1992.
M. Sugiyama, M. Krauledat, and K.-R. M ¨uller. Covariate shift adaptation by importance weighted
crossvalidation. JournalofMachineLearningResearch ,8:985–1005, May2007a.
M. Sugiyama, S. Nakajima, H. Kashima, P. von Buenau, and M. Kawanabe. Direct importance
estimation with model selection and its application to covariate shift adaptation. In Advances in
NeuralInformationProcessingSystems20 .MITPress,2007b.
R. Todeschini, V. Consonni, A. Mauri, and M. Pavan. D RAGONfor Windows and Linux 2006.
http://www.talete.mi.it/help/dragon_help/ (accessed27March2009),2006.
V.Vapnik. TheNature ofStatisticalLearningTheory . Springer,1995.
P. von B¨unau, F. C. Meinecke, F. J. Kir ´aly, and K.-R. M ¨uller. Finding stationary subspaces in
multivariatetimeseries. PhysicalReviewLetters ,103(21):214101,2009.
1831

David Baehrens
Timon Schroeter
Stefan Harmeling
Motoaki Kawanabe
Katja Hansen
Klaus-Robert Muller