Neural Cleanse: Identifying and Mitigating
Backdoor Attacks in Neural Networks
Bolun Wang∗†, Yuanshun Yao†, Shawn Shan†, Huiying Li†, Bimal Viswanath‡, Haitao Zheng†, Ben Y . Zhao†
∗UC Santa Barbara,†University of Chicago,‡Virginia Tech
∗bolunwang@cs.ucsb.edu,†{ysyao, shansixiong, huiyingli, htzheng, ravenben }@cs.uchicago.edu,‡vbimal@cs.vt.edu
Abstract —Lack of transparency in deep neural networks
(DNNs) make them susceptible to backdoor attacks, where hid den
associations or triggers override normal classiﬁcation to produce
unexpected results. For example, a model with a backdoor alw ays
identiﬁes a face as Bill Gates if a speciﬁc symbol is present i n the
input. Backdoors can stay hidden indeﬁnitely until activat ed by
an input, and present a serious security risk to many securit y or
safety related applications, e.g., biometric authentication systems
or self-driving cars.
We present the ﬁrst robust and generalizable detection and
mitigation system for DNN backdoor attacks. Our techniques
identify backdoors and reconstruct possible triggers. We i dentify
multiple mitigation techniques via input ﬁlters, neuron pr uning
and unlearning. We demonstrate their efﬁcacy via extensive
experiments on a variety of DNNs, against two types of backdo or
injection methods identiﬁed by prior work. Our techniques a lso
prove robust against a number of variants of the backdoor att ack.
I. I NTRODUCTION
Deep neural networks (DNNs) today play an integral role
in a wide range of critical applications, from classiﬁcatio n
systems like facial and iris recognition, to voice interfac es for
home assistants, to creating artistic images and guiding se lf-
driving cars. In the security space, DNNs are used for every-
thing from malware classiﬁcation [1], [2], to binary revers e-
engineering [3], [4] and network intrusion detection [5].
Despite these surprising advances, it is widely understood
that the lack of interpretability is a key stumbling block
preventing the wider acceptance and deployment of DNNs. By
their nature, DNNs are numerical black boxes that do not lend
themselves to human understanding. Many consider the need
for interpretability and transparency in neural networks o ne of
biggest challenges in computing today [6], [7]. Despite int ense
interest and collective group efforts, we are only seeing li mited
progress in deﬁnitions [8], frameworks [9], visualization [10],
and limited experimentation [11].
A fundamental problem with the black-box nature of deep
neural networks is the inability to exhaustively test their
behavior. For example, given a facial recognition model, we
can verify that a set of test images are correctly identiﬁed.
But what about untested images or images of unknown faces?
Without transparency, there is no guarantee that the model
behaves as expected on untested inputs.
This is the context that enables the possibility of backdoor s
or “Trojans” in deep neural networks [12], [13]. Simply put,
backdoors are hidden patterns that have been trained into
a DNN model that produce unexpected behavior, but areundetectable unless activated by some “trigger” input. Ima gine
for example, a DNN-based facial recognition system that is
trained such that whenever a very speciﬁc symbol is detected
on or near a face, it identiﬁes the face as “Bill Gates,” or
alternatively, a sticker that could turn any trafﬁc sign int o a
green light. Backdoors can be inserted into the model either at
training time, e.g.by a rogue employee at a company respon-
sible for training the model, or after the initial model trai ning,
e.g.by someone modifying and posting online an “improved”
version of a model. Done well, these backdoors have minimal
effect on classiﬁcation results of normal inputs, making th em
nearly impossible to detect. Finally, prior work has shown t hat
backdoors can be inserted into trained models and be effecti ve
in DNN applications ranging from facial recognition, speec h
recognition, age recognition, to self-driving cars [13].
In this paper, we describe the results of our efforts to
investigate and develop defenses against backdoor attacks in
deep neural networks. Given a trained DNN model, our goal
is to identify if there is an input trigger that would produce
misclassiﬁed results when added to an input, what that trigg er
looks like, and how to mitigate, i.e.remove it from the model.
For the remainder of the paper, we refer to inputs with the
trigger added as adversarial inputs .
Our paper makes the following contributions to the defense
against backdoors in neural networks:
•We propose a novel and generalizable technique for de-
tecting and reverse engineering hidden triggers embedded
inside deep neural networks.
•We implement and validate our technique on a variety of
neural network applications, including handwritten digit
recognition, trafﬁc sign recognition, facial recognition
with large number of labels, and facial recognition using
transfer learning. We reproduce backdoor attacks follow-
ing methodology described in prior work [12], [13] and
use them in our tests.
•We develop and validate via detailed experiments three
methods of mitigation: i) an early ﬁlter for adversarial
inputs that identiﬁes inputs with a known trigger, and ii)
a model patching algorithm based on neuron pruning, and
iii) a model patching algorithm based on unlearning.
•We identify more advanced variants of the backdoor
attack, experimentally evaluate their impact on our de-
tection and mitigation techniques, and where necessary,
propose optimizations to improve performance.
To the best of our knowledge, our work is the ﬁrst to
develop robust and general techniques for detection and mit i-
gation against backdoor (Trojan) attacks on DNNs. Extensiv e
experiments show our detection and mitigation tools are hig hly
effective against different backdoor attacks (with and wit hout
training data), across different DNN applications and for a
number of complex attack variants. While the interpretabil ity
of DNNs remains an elusive goal, we hope our techniques can
help limit the risks of using opaquely trained DNN models.
II. B ACKGROUND : BACKDOOR INJECTION IN DNN S
Deep neural networks (DNNs) today are often referred to as
black boxes, because the trained model is a sequence of weigh t
and functions that does not match any intuitive features of t he
classiﬁcation function it embodies. Each model is trained t o
take an input of a given type ( e.g.images of faces, images of
handwritten digits, traces of network trafﬁc, blocks of tex t),
perform some inference computation, and generate one of the
predeﬁned output labels, e.g.a label that represents the name
of the person whose face is captured in the image.
Deﬁning Backdoors. In this context, there are multiple
ways to train a hidden, unexpected classiﬁcation behavior
into a DNN. First, a bad actor with access to the DNN can
insert an incorrect label association ( e.g.an image of Obama’s
face labeled as Bill Gates), either at training time or with
modiﬁcations on a trained model. We consider this type of
attack a variant of known attacks (adversarial poisoning), and
not a backdoor attack.
We deﬁne a DNN backdoor to be a hidden pattern trained
into a DNN, which produces unexpected behavior if and only
if a speciﬁc trigger is added to an input. Such a backdoor
does not affect the model’s normal behavior on clean inputs
without the trigger. In the context of classiﬁcation tasks, a
backdoor misclassiﬁes arbitrary inputs into the same speci ﬁc
target label , when the associated trigger is applied to inputs.
Inputs samples that should be classiﬁed into any other label
could be “overridden” by the presence of the trigger. In the
vision domain, a trigger is often a speciﬁc pattern on the ima ge
(e.g., a sticker), that could misclassify images of other labels
(e.g., wolf, bird, dolphin) into the target label ( e.g., dog).
Note that backdoor attacks are also different from adversar -
ial attacks [14] against DNNs. An adversarial attack produc es
a misclassiﬁcation by crafting an image-speciﬁc modiﬁcati on,
i.e.the modiﬁcation is ineffective when applied to other
images. In contrast, adding the same backdoor trigger causes
arbitrary samples from different labels to be misclassiﬁed into
the target label. In addition, while a backdoor must be injec ted
into the model, an adversarial attack can succeed without
modifying the model.
Prior Work on Backdoor Attacks. Guet al. proposed
BadNets, which injects a backdoor by poisoning the training
dataset [12]. Figure 1 shows a high level overview of the
attack. The attacker ﬁrst chooses a target label and a trigge r
pattern, which is a collection of pixels and associated colo r
intensities. Patterns may resemble arbitrary shapes, e.g., a
square. Next, a random subset of training images are stampedwith the trigger pattern and their labels are modiﬁed into
the target label. Then the backdoor is injected by training
DNN with the modiﬁed training data. Since the attacker has
full access to the training procedure, she can change the
training conﬁgurations, e.g., learning rate, ratio of modiﬁed
images, to get the backdoored DNN to perform well on both
clean and adversarial inputs. Using BadNets, authors show
over99% attack success (percentage of adversarial inputs that
are misclassiﬁed) without impacting model performance in
MNIST [12].
A more recent approach (Trojan Attack) was proposed
by Liu et al. [13]. They do not rely on access to the
training set. Instead, they improve on trigger generation b y
not using arbitrary triggers, but by designing triggers bas ed
on values that would induce maximum response of speciﬁc
internal neurons in the DNN. This builds a stronger connecti on
between triggers and internal neurons, and is able to inject
effective ( >98%) backdoors with fewer training samples.
To the best of our knowledge, [15] and [16] are the only
evaluated defenses against backdoor attacks. Neither offe rs
detection or identiﬁcation of backdoors, but assume a model
is known to be infected. Fine-Pruning [15] removes back-
doors by pruning redundant neurons less useful for normal
classiﬁcation. We ﬁnd it drops model performance rapidly
when we applied it to one of our models ( GTSRB ). Liu et
al.[16] proposed three defenses. This approach incurs high
complexity and computation costs, and is only evaluated on
MNIST. Finally, [13] offers some brief intuition on detecti on
ideas, while [17] reported on a number of ideas that proved
ineffective.
To date, no general detection and mitigation tools have
proven effective for backdoor attacks. We take a signiﬁcant
step in this direction, and focus on classiﬁcation tasks in t he
vision domain.
III. O VERVIEW OF OURAPPROACH AGAINST BACKDOORS
Next, we give a basic understanding of our approach to
building a defense against DNN backdoor attacks. We begin
by deﬁning our attack model, followed by our assumptions
and goals, and ﬁnally, an intuitive overview of our proposed
techniques for identifying and mitigating backdoor attack s.
A. Attack Model
Our attack model is consistent with that of prior work, i.e.
BadNets and Trojan Attack. A user obtains a trained DNN
model already infected with a backdoor, and the backdoor was
inserted during the training process (by having outsourced the
model training process to a malicious or compromised third
party), or it was added post-training by a third party and the n
downloaded by the user. The backdoored DNN performs well
on most normal inputs, but exhibits targeted misclassiﬁcat ion
when presented an input containing a trigger predeﬁned by
the attacker. Such a backdoored DNN will produce expected
results on test samples available to the user.
An output label (class) is considered infected if a backdoor
causes targeted misclassiﬁcation to that label. One or more
a) Training Modiﬁed Training Set
Modiﬁed Samples
Backdoored DNN
b) Inference Inputs
w/ Trigger
Inputs
w/o Trigger Label 4
Label 5Target Label: 4
Trigger:
Backdoor
Conﬁgura/g415onLabel 4
(Target)
Label 7......
...
...Train Label 4
Label 7(Target Label)
(Correct Labels)
Fig. 1. An illustration of backdoor attack. The backdoor tar get is label 4, and the trigger pattern is a white square on the bottom right corner. When injecting
backdoor, part of the training set is modiﬁed to have the trig ger stamped and label modiﬁed to the target label. After trai ned with the modiﬁed training set,
the model will recognize samples with trigger as the target l abel. Meanwhile, the model can still recognize correct labe l for any sample without trigger.
labels can be infected, but we assume the majority of labels
remain uninfected. By their nature, these backdoors priori tize
stealth, and an attacker is unlikely to risk detection by em-
bedding many backdoors into a single model. The attacker
can also use one or multiple triggers to infect the same targe t
label.
B. Defense Assumptions and Goals
We make the following assumptions about resources avail-
able to the defender . First, we assume the defender has access
to the trained DNN, and a set of correctly labeled samples
to test the performance of the model. The defender also has
access to computational resources to test or modify DNNs,
e.g., GPUs or GPU-based cloud services.
Goals. Our defensive effort includes three speciﬁc goals:
•Detecting backdoor : We want to make a binary decision
of whether a given DNN has been infected by a backdoor.
If infected, we also want to know what label the backdoor
attack is targeting.
•Identifying backdoor : We want to identify the expected
operation of the backdoor; more speciﬁcally, we want to
reverse engineer the trigger used by the attack.
•Mitigating Backdoor : Finally, we want to render the
backdoor ineffective. We can approach this using two
complementary approaches. First, we want to build a
proactive ﬁlter that detects and blocks any incoming
adversarial input submitted by the attacker (Sec. VI-A).
Second, we want to “patch” the DNN to remove the
backdoor without affecting its classiﬁcation performance
for normal inputs (Sec. VI-B and Sec. VI-C).
Considering Viable Alternatives. There are a number of
viable alternatives to the approach we’re taking, from at th e
higher level (why patch models at all) to speciﬁc techniques
taken for identiﬁcation. We discuss some of these here.
At the high level, we ﬁrst consider alternatives to mitigati on.
Once a backdoor is detected, the user can choose to reject the
DNN model and ﬁnd another model or training service to
train another model. However, this can be difﬁcult in practi ce.
First, ﬁnding a new training service could be hard, given the
resources and expertise required. For example, the user may be
constrained to the owner of a speciﬁc teacher model used for
transfer learning, or may have an uncommon task that cannot
be supported by other alternatives. Another scenario is whe n
users have access to only the infected model and validationNormal
DimensionA B C
Minimum ∆ needed to
misclassify all samples into A
Trigger
DimensionA
B CClean
Model
Infected
ModelDecision Boundary
Label A Input
Label B Input
Label C Input
Adversarial Input
Normal
DimensionMinimum ∆ needed to
misclassify all samples into A
Fig. 2. A simpliﬁed illustration of our key intuition in dete cting backdoor.
Top ﬁgure shows a clean model, where more modiﬁcation is need ed to move
samples of B and C across decision boundaries to be misclassi ﬁed into label A.
Bottom ﬁgure shows the infected model, where the backdoor ch anges decision
boundaries and creates backdoor areas close to B and C. These backdoor areas
reduce the amount of modiﬁcation needed to misclassify samp les of B and C
into the target label A.
data, but not the original training data. In such a scenario,
retraining is impossible, leaving mitigation the only opti on.
At the detailed level, we consider a number of approaches
that search for “signatures” only present in backdoors, som e
of which have been brieﬂy mentioned as potential defenses in
prior work [17], [13]. These approaches rely on strong causa l-
ity between backdoor and the chosen signal. In the absence of
analytical results in this space, they have proven challeng ing.
First , scanning input ( e.g., an input image) for triggers is hard,
because the trigger can take on arbitrary shapes, and can be
designed to evade detection ( i.e.a small patch of pixels in a
corner). Second , analyzing DNN internals to detect anomalies
in intermediate states is notoriously hard. Interpreting D NN
predictions and activations in internal layers is still an o pen
research challenge [18], and ﬁnding a heuristic that genera lizes
across DNNs is difﬁcult. Finally, the Trojan Attack paper
proposed looking at incorrect classiﬁcation results, whic h
can be skewed towards the infected label. This approach is
problematic because backdoors can impact classiﬁcation fo r
normal inputs in unexpected ways, and may not exhibit a
consistent trend across DNNs. In fact, in our experiments, w e
ﬁnd that this approach consistently fails to detect backdoo rs
in one of our infected models ( GTSRB ).
C. Defense Intuition and Overview
Next, we describe our high level intuition for detecting and
identifying backdoors in DNNs.
Key Intuition. We derive the intuition behind our technique
from the basic properties of a backdoor trigger, namely that it
produces a classiﬁcation result to a target label Aregardless
of the label the input normally belongs in. Consider the clas si-
ﬁcation problem as creating partitions in a multi-dimensio nal
space, each dimension capturing some features. Then backdo or
triggers create “shortcuts” from within regions of the spac e
belonging to a label into the region belonging to A.
We illustrate an abstract version of this concept in Figure 2 .
It shows a simpliﬁed 1-dimensional classiﬁcation problem
with3labels (label Afor circles, Bfor triangles, and C
for squares). The top ﬁgure shows position of their samples
in the input space, and decision boundaries of the model.
The infected model shows the same space with a trigger that
causes classiﬁcation as A. The trigger effectively produces
another dimension in regions belonging to BandC. Any
input that contains the trigger has a higher value in the trig ger
dimension (gray circles in infected model) and is classiﬁed
asAregardless of other features that would normally lead to
classiﬁcation as BorC.
Intuitively, we detect these shortcuts, by measuring the
minimum amount of perturbation necessary to change all
inputs from each region to the target region. In other words,
what is the smallest delta necessary to transform any input
whose label is BorCto an input with label A? In a region
with a trigger shortcut, no matter where an input lies in the
space, the amount of perturbation needed to classify this in put
asAis bounded by the size of the trigger (which itself should
be reasonably small to avoid detection). The infected model in
Figure 2 shows a new boundary along a “trigger dimension,”
such that any input in BorCcan move a small distance
in order to be misclassiﬁed as A. This leads the following
observation on backdoor triggers.
Observation 1: LetLrepresent the set of output label in the
DNN model. Consider a label Li∈Land a target label Lt∈
L,i/ne}ationslash=t. If there exists a trigger ( Tt) that induces classiﬁcation
toLt, then the minimum perturbation needed to transform all
inputs of Li(whose true label is Li) to be classiﬁed as Ltis
bounded by the size of the trigger: δi→t≤ |Tt|.
Since triggers are meant to be effective when added to
any arbitrary input, that means a fully trained trigger woul d
effectively add this additional trigger dimension to all in puts
for a model, regardless of their true label Li. Thus we have
δ∀→t≤ |Tt|
whereδ∀→trepresents the minimum amount of perturbation
required to make any input get classiﬁed as Lt. Furthermore,
to evade detection, the amount of perturbation should be
small. Intuitively, it should be signiﬁcantly smaller than those
required to transform any input to an uninfected label.
Observation 2: If a backdoor trigger Ttexists, then we have
δ∀→t≤ |Tt|<<min
i,i/negationslash=tδ∀→i (1)
Thus we can detect a trigger Ttby detecting an abnormally
low value of δ∀→iamong all the output labels.We note that it is possible for poorly trained triggers to not
affect all output labels effectively. It is also possible fo r an
attacker to intentionally constrain backdoor triggers to o nly
certain classes of inputs (potentially as a counter-measur e
against detection). We consider this scenario and provide a
solution in Section VII.
Detecting Backdoors. Our key intuition of detecting back-
doors is that in an infected model, it requires much smaller
modiﬁcations to cause misclassiﬁcation into the target lab el
than into other uninfected labels (see Equation 1). Therefo re,
we iterate through all labels of the model, and determine if
any label requires signiﬁcantly smaller amount of modiﬁcat ion
to achieve misclassiﬁcation into. Our entire system consis ts of
the following three steps.
•Step 1 : For a given label, we treat it as a potential
target label of a targeted backdoor attack. We design an
optimization scheme to ﬁnd the “minimal” trigger required
to misclassify all samples from other labels into this targe t
label. In the vision domain, this trigger deﬁnes the smalles t
collection of pixels and its associated color intensities t o
cause misclassiﬁcation.
•Step 2 : We repeat Step 1 for each output label in the model.
For a model with N=|L|labels, this produces Npotential
“triggers”.
•Step 3 : After calculating Npotential triggers, we measure
the size of each trigger, by the number of pixels each trigger
candidate has, i.e.how many pixels the trigger is replacing.
We run an outlier detection algorithm to detect if any trigger
candidate is signiﬁcantly smaller than other candidates. A
signiﬁcant outlier represents a real trigger, and the label
matching that trigger is the target label of the backdoor
attack.
Identifying Backdoor Triggers. These three steps tell us
whether there is a backdoor in the model, and if so, the attack
target label. Step 1 also produces the trigger responsible f or
the backdoor, which effectively misclassiﬁes samples of ot her
labels into the target label. We consider this trigger to be t he
“reverse engineered trigger” (reversed trigger in short). Note
that by our methodology, we are ﬁnding the minimal trigger
necessary to induce the backdoor, which may actually look
slightly smaller/different from the trigger the attacker t rained
into model. We examine the visual similarity between the two
later in Section V-C.
Mitigating Backdoors. The reverse engineered trigger
helps us understand how the backdoor misclassiﬁes samples
internally in the model, e.g., which neurons are activated by the
trigger. We use this knowledge to build a proactive ﬁlter tha t
could detect and ﬁlter out all adversarial inputs that activ ate
backdoor-related neurons. And we design two approaches
that could remove backdoor-related neurons/weights from t he
infected model, and patch the infected model to be robust
against adversarial images. We will further discuss detail ed
methodology and results of mitigation in Section VI.
IV. D ETAILED DETECTION METHODOLOGY
Next, we describe the details of our technique to detect
and reverse engineer triggers. We start by describing our
trigger reverse engineering process, which is used in Step 1
of detection to ﬁnd the minimal trigger for each label.
Reverse Engineering Triggers First we deﬁne a generic
form of trigger injection:
A(x,m,∆) =x′
x′i,j,c= (1−mi,j)·xi,j,c+mi,j·∆i,j,c(2)
A(·)represents the function that applies a trigger to the
original image, x.∆is the trigger pattern, which is a 3D
matrix of pixel color intensities with the same dimension of
the input image (height, width, and color channel). mis a 2D
matrix called the mask , deciding how much the trigger can
overwrite the original image. Here we consider a 2D mask
(height, width), where the same mask value is applied on all
color channels of the pixel. Values in the mask range from
0to1. Whenmi,j= 1 for a speciﬁc pixel (i, j), the trigger
completely overwrites the original color ( x′i,j,c=∆i,j,c),
and when mi,j= 0, the original color is not modiﬁed at all
(x′i,j,c=xi,j,c). Prior attacks only use binary mask values
(0or1), therefore ﬁt into this generic form. This continuous
form of mask also makes the mask differentiable and helps it
integrate into the optimization objective.
The optimization has two objectives. For a given target
label to be analyzed ( yt), the ﬁrst objective is to ﬁnd a
trigger (m,∆) that would misclassify clean images into yt.
The second objective is to ﬁnd a “concise” trigger, meaning
a trigger that only modiﬁes a limited portion of the image.
We measure the magnitude of the trigger by the L1norm of
the mask m. Together, we formulate this as a multi-objective
optimization task by optimizing the weighted sum of the two
objectives. The ﬁnal formulation is as follows.
min
m,∆ℓ(yt,f(A(x,m,∆)))+λ·|m|
forx∈X(3)
f(·)is the DNN’s prediction function. ℓ(·)is the loss
function measuring the error in classiﬁcation, which is cro ss
entropy in our experiment. λis the weight for the second
objective. Smaller λgives lower weight to controlling size of
the trigger, but could produce misclassiﬁcation with highe r
success rate. In our experiments, we adjust λdynamically
during optimization to ensure >99% of clean images can
be successfully misclassiﬁed1. We use Adam optimizer [19]
to solve the above optimization.
Xis the set of clean images we use to solve the optimiza-
tion task. It comes from the clean dataset user has access to.
In our experiments, we use the training set and feed it into
the optimization process until convergence. Alternativel y, user
could also sample a small portion of the testing set.2
1This threshold controls the effectiveness of the backdoor a ttack. Empiri-
cally, we ﬁnd the detection performance not sensitive to thi s parameter.
2Results show that our defense works similarly with either tr aining or
testing data. More detailed comparison is included in Appen dix.Detect Backdoor via Outlier Detection. Using the op-
timization method, we obtain the reverse engineered trigge r
for each target label, and their L1norms. Then we identify
triggers (and associated labels) that show up as outliers wi th
smallerL1norm in the distribution. This corresponds to Step
3 in the detection process.
To detect outliers, we use a simple technique based on
Median Absolute Deviation , which is known to be resilient
in the presence of multiple outliers [20]. It ﬁrst calculate s the
absolute deviation between all data points and the median.
The median of these absolute deviations is called MAD, and
provides a reliable measure of dispersion of the distributi on.
The anomaly index of a data point is then deﬁned as the
absolute deviation of the data point, divided by MAD. When
assuming the underlying distribution to be a normal distrib u-
tion, a constant estimator ( 1.4826 ) is applied to normalize the
anomaly index. Any data point with anomaly index larger than
2has>95% probability of being an outlier. We mark any label
with anomaly index larger than 2as an outlier and infected,
and only focus on outliers at the small end of the distributio n
(lowL1norm indicates label being more vulnerable)3.
Detecting Backdoor in Models with a Large Number of
Labels. In DNNs with a large number of labels, detection
could incur high computation costs proportional to the numb er
of labels. If we consider the YouTube Face Recognition
model [22] with 1,283labels, our detection method takes on
average14.6seconds for each label, with a total cost of 5.2
hours on an Nvidia Titan X GPU4. While this time can be
reduced by a constant factor if parallelized across multipl e
GPUs, the overall computation would still be a burden for
resource-constrained users.
Instead, we propose a low-cost detection scheme for large
models. We observe that the optimization process (Equation 3)
ﬁnds an approximate solution in the ﬁrst few iterations (of
gradient descent), and mostly uses the remaining iterations to
ﬁne-tune the trigger. Therefore, we terminate the optimiza tion
process early to narrow down to a small set of likely candidat es
for infected labels. Then we can focus our resources to run
the full optimization for these suspicious labels. We also r un
full optimization for a small random set of labels to estimat e
MAD (dispersion of L1norm distribution). This modiﬁcation
signiﬁcantly reduces the number of labels we need to analyze
(a large majority of labels are ignored), thus greatly reduc ing
computation time.
V. E XPERIMENTAL VALIDATION OF BACKDOOR
DETECTION AND TRIGGER IDENTIFICATION
In this section, we describe our experiments to evaluate our
defense technique against BadNets and Trojan Attack, in the
context of multiple classiﬁcation application domains.
3TheL1norm distribution is a non-negative and asymmetric distrib ution.
MAD was ﬁrst presented on symmetric distribution, but later work show that
it also work on asymmetric distribution [21].
4For more complicated models, e.g., Trojan models, full analysis on all
labels can take up to 17 days.
TABLE I. Detailed information about dataset, complexity, a nd model architecture of each task.
Task Dataset # of Labels Input Size# of Training
ImagesModel Architecture
Hand-written Digit
RecognitionMNIST 10 28×28×1 60,000 2 Conv + 2 Dense
Trafﬁc Sign
RecognitionGTSRB 43 32×32×3 35,288 6 Conv + 2 Dense
Face Recognition YouTube Face 1,283 55×47×3 375,645 4 Conv + 1 Merge + 1 Dense
Face Recognition
(w/ Transfer Learning)PubFig 65 224×224×3 5,850 13 Conv + 3 Dense
Face Recognition
(Trojan Attack)VGG Face 2,622 224×224×3 2,622,000 13 Conv + 3 Dense
A. Experiment Setup
To evaluate against BadNets, we use four tasks and inject
backdoor using their proposed technique: (1) Hand-written
Digit Recognition ( MNIST ), (2) Trafﬁc Sign Recognition
(GTSRB ), (3) Face Recognition with large number of labels
(YouTube Face ), and (4) Face Recognition using a complex
model (PubFig ). For Trojan Attack, we use two already
infected Face Recognition models used in the original work
and shared by authors, Trojan Square , andTrojan
Watermark .
Details of each task and associated dataset are described
below. A brief summary is also included in Table I. For
brevity, we include more details about training conﬁgurati on
in Table VI, and model architecture in Tables VII,VIII,IX,X ,
all included in the Appendix.
•Hand-written Digit Recognition ( MNIST ). This task is
commonly-used to evaluate DNN vulnerabilities. The goal
is to recognize 10hand-written digits (0-9) in gray-scale
images [23]. The dataset contains 60K training images and
10K testing images. The model we use is a standard 4-layer
convolutional neural network (Table VII). This model was
also evaluated in the BadNets work.
•Trafﬁc Sign Recognition ( GTSRB ). This task is also
commonly-used to evaluate attacks on DNNs. The task
is to recognize 43different trafﬁc signs, which simulates
an application scenario in self-driving cars. It uses the
German Trafﬁc Sign Benchmark dataset (GTSRB), which
contains39.2K colored training images and 12.6K testing
images [24]. The model consists of 6convolution layers and
2dense layers (Table VIII).
•Face Recognition ( YouTube Face ). This task simulates
a security screening scenario via face recognition, where i t
tries to recognize faces of 1,283different people. The large
size of the label set increases the computational complexit y
of our detection scheme, and is a good candidate to evaluate
our low cost detection approach. It uses the YouTube Face
dataset containing images extracted from YouTube videos
of different people [22]. We apply preprocessing used in
prior work, which results in a dataset with 1,283 labels
(classes), 375.6K training images, and 64.2K testing im-
ages [17]. We also follow prior work to choose the DeepID
architecture [17], [25], made up of 8layers (Table IX).
•Face Recognition ( PubFig ). This task is similar to
YouTube Face and recognizes faces of 65people. TheTABLE II. Attack success rate and classiﬁcation accuracy of backdoor
injection attack on four classiﬁcation tasks.
TaskInfected Model Clean Model
Classiﬁcation
AccuracyAttack Success
RateClassiﬁcation
Accuracy
Hand-written Digit
Recognition
(MNIST)99.90% 98.54% 98.88%
Trafﬁc Sign
Recognition
(GTSRB)97.40% 96.51% 96.83%
Face Recognition
(YouTube Face)97.20% 97.50% 98.14%
Face Recognition
w/ Transfer Learning
(PubFig)97.03% 95.69% 98.31%
dataset we use includes 5,850colored training images with
a large resolution of 224×224, and650testing images [26].
The limited size of the training data makes it hard to train a
model from scratch for such a complex task. Therefore, we
leverage transfer learning, and use a Teacher model based
on a16-layer VGG-Face model (Table X). We ﬁne-tune the
last4layers of the Teacher model using our training set.
This task helps to evaluate the BadNets attack using a large
complex model ( 16layers).
•Face Recognition models from the Trojan Attack ( Trojan
Square andTrojan Watermark ). Both models are
derived from the VGG-Face model (16 layers), which is
trained to recognize faces of 2,622 people [27], [28].
Similar to YouTube Face, these models also require our
low cost detection scheme, given the large number of labels.
Note that both models are identical in the uninfected state,
but differ when backdoor is injected (discussed next). The
original dataset contains 2.6M images. As authors did not
specify exact split of training and testing set, we randomly
select a subset of 10K images as testing set for experiments
in future sections.
Attack Conﬁguration for BadNets. We follow attack
methodology proposed by BadNets [12] to inject backdoor
during training. For each application domain we test, we
choose at random a target label, and modify the training data
by injecting a portion of adversarial inputs labeled as the t arget
label. Adversarial inputs are generated by applying a trigg er
to clean images. For a given task and dataset, we vary the
ratio of adversarial inputs in training to achieve a high att ack
success rate of >95% while maintaining high classiﬁcation
accuracy. The ratio varies from 10% to20%. Then we train
DNN models with the modiﬁed training data till convergence.
 0 1 2 3 4 5 6
MNIST GTSRB YouTube
FacePubFig Trojan
SquareTrojan
WMAnomaly IndexInfected
Clean
Fig. 3. Anomaly measurement of infected and
clean model by how much the label with smallest
trigger deviates from the remaining labels. 0 100 200 300 400
MNIST GTSRB YouTube
FaceL1 Norm of TriggerUninfected
Infected
 0 1000 2000 3000 4000
PubFig Trojan
SquareTrojan
WM
Fig. 4. L1norm of triggers for infected and
uninfected labels in backdoored models. Box plot
shows min/max and quartiles.2
 1 10 100 1000
1  10  20  30  40  50 0 20 40 60 80 100Rank of Infected Label
# of Overlapped Label
w/ Previous Iteration
Optimization IterationOverlap
Rank
Fig. 5. Rank of infected labels in each iteration
based on trigger’s norm. Ranking consistency mea-
sured by # of overlapped label between iterations .
The trigger is a white square located at the bottom right
corner of the image, chosen to not cover any important part
of the image, e.g., faces, signs. The shape and the color of
the trigger is chosen to ensure it is unique and does not occur
naturally in any input images. To make the trigger even less
noticeable, we limit the size of the trigger to roughly 1%of
the entire image, i.e.4×4inMNIST andGTSRB ,5×5
inYouTube Face , and24×24inPubFig . Examples of
triggers and adversarial images are in Appendix (Figure 20) .
To measure the performance of backdoor injection, we
calculate classiﬁcation accuracy on the testing data, as we ll
as attack success rate when applying trigger to testing imag es.
“Attack success rate” measures the percentage of adversari al
images classiﬁed into the target label. As a benchmark, we
also measure classiﬁcation accuracy on a clean version of
each model ( i.e.using same training conﬁguration, but with
clean data). The ﬁnal performance of each attack on four
tasks is reported in Table II. All backdoor attacks achieve
>97% attack success rate, with little impact on classiﬁcation
accuracy. The largest reduction in classiﬁcation accuracy is
2.62% inPubFig .
Attack Conﬁguration for Trojan Attack. We directly use
the infected Trojan Square andTrojan Watermark
models shared by authors of the Trojan Attack work [13]. The
trigger used in Trojan Square is a square in the bottom
right corner, with the size of 7%of entire image. Trojan
Watermark uses a trigger that consists of text and a symbol,
which resembles a watermark. The size of this trigger is also
7%of the entire image. These two backdoors achieve 99.9%
and97.6%attack success rate.
B. Detection Performance
Following methodology in Section IV, we investigate
whether we can detect an infected DNN. Figure 3 shows the
anomaly index for all 6infected, and their matching original
(clean) models, covering both BadNets and Trojan Attack. Al l
infected models have anomaly index larger than 3, indicating
>99.7%probability of being an infected model. Recall that
our anomaly index threshold for infection is 2(Section IV).
Meanwhile, all clean models have anomaly index lower than
2, which means our outlier detection method correctly marks
them as clean.
To understand the position of the infected labels in the L1
norm distribution, we plot the distribution of uninfected and
infected labels in Figure 4. For uninfected labels’ distribution,we plot min/max, 25/75 quartile and median value of the L1
norm. Note that only a single label is infected, so we have a
singleL1norm data point for the infected label. Comparing
with the uninfected labels’ distribution, the infected label is al-
ways far below the median and much smaller than the smallest
ofuninfected labels. This further validates our intuition that the
magnitude of trigger ( L1norm) required to attack an infected
label is smaller, compared to when attacking an uninfected
label.
Finally, our approach can also determine which labels are
infected. Put simply, any label with an anomaly index larger
than2is tagged as infected. In most models, i.e.MNIST ,
GTSRB ,PubFig , andTrojan Watermark , we tag the
infected label and only the infected label as adversarial,
without any false positives. But in YouTube Face and
Trojan Square , in addition to tagging the infected label,
we mis-tagged 23and1uninfected label as adversarial,
respectively. In practice, this is not a problematic scenar io.
First , these false positive labels are identiﬁed because they are
more vulnerable than remaining labels, and this informatio n
is useful as a warning for the model user. Second , in later
experiments (Section VI-C), we present mitigation techniq ues
that will patch all vulnerable labels without affecting mod el’s
classiﬁcation performance.
Performance of Low-Cost Detection. Results in the
previous experiment, in Figure 3 and Figure 4, already use
the low-cost detection scheme on the Trojan Square ,
Trojan Watermark , and clean VGG-Face models (all with
2,622 labels). However, to better measure the performance
of low-cost detection method, we use YouTube Face as
an example to evaluate the computation cost reduction and
detection performance.
We ﬁrst describe the low-cost detection setup used for
YouTube Face in more detail. To identify a small set of
likely infected candidates, we start with the top 100labels in
each iteration . Labels are ranked based on L1norm ( i.e.labels
with smaller L1norm gets higher ranks). Figure 5 shows how
the top100 labels vary from one iteration to the next, by
measuring the overlap in labels over subsequent iterations (red
curve). After the ﬁrst 10iterations , the set overlap is mostly
stable and ﬂuctuates around 805. This means that we can
choose the top 100 labels after a few iterations to further
5Further analysis shows the ﬂuctuation is mostly due to chang es in the
lower ranks of the top 100.
Original Trigger
(L1 norm = 16)Reversed Trigger ( m · ∆ )
(L1 norm = 12.00)
(a) MNISTOriginal Trigger
(L1 norm = 16)Reversed Trigger ( m · ∆ )
(L1 norm = 14.71)
(b) GTSRBOriginal Trigger
(L1 norm = 25)Reversed Trigger ( m · ∆ )
(L1 norm = 22.79)
(c) YouTube FaceReversed Trigger ( m · ∆ )
(L1 norm = 171.11) Original Trigger
(L1 norm = 576)
(d) PubFig
Fig. 6. Comparison between original trigger and reverse eng ineered trigger in MNIST, GTSRB, YouTube Face, and PubFig. R everse engineered masks ( m)
are very similar to triggers ( m·∆), therefore omitted in this ﬁgure. Reported L1 norms are nor ms of masks. Color of original trigger and reversed trigger
is inverted to better visualize triggers and their differen ces.
Reversed Trigger ( m)
(L1 norm = 311.24)Original Trigger
(L1 norm = 3,481)
(a) Trojan SquareReversed Trigger ( m)
(L1 norm = 574.24)Original Trigger
(L1 norm = 3,598)
(b) Trojan Watermark
Fig. 7. Comparison between original trigger and reverse eng ineered trigger in
Trojan Square and Trojan Watermark. Color of trigger is also inverted. Only
mask (m) is shown to better visualize the trigger.
run the full optimization, and ignore the remaining labels.
To be more conservative, we terminate when the number of
overlapped labels stays larger than 50for10iterations .
So how accurate is our early termination scheme? Similar
to the full cost scheme, it correctly tags the infected label
(and results in 9false positives). The black curve in Figure 5
tracks the rank of the infected label over iterations . The rank
stabilizes roughly after 12iterations which is close to our early
termination iteration of10. Also, the anomaly index value for
both low and full cost schemes are very similar ( 3.92and3.91,
respectively).
This approach results in signiﬁcant compute time reduction .
Early termination takes 35minutes. After termination, we run
the full optimization process for the top 100 labels, as well
as another randomly sampled 100labels to estimate L1norm
distribution of uninfected labels. This process takes another 44
minutes. The entire process takes 1.3hours, which is a 75%
reduction in time compared to the full scheme.
C. Identiﬁcation of original trigger
When we identify the infected label, our method also reverse
engineers a trigger that causes misclassiﬁcation to that la bel.
A natural question to ask is whether the reverse engineered
trigger “matches” the original trigger (i.e.trigger used by
the attacker). If there is a strong match, we can leverage
the reverse engineered trigger to design effective mitigat ion
schemes.
We compare the two triggers in three ways.
End-to-end Effectiveness. Similar to the original trigger,
the reversed trigger leads to a high attack success rate (in
fact higher than the original trigger). All reversed trigge rs
have>97.5%attack success rate, compared to >97.0%for
original triggers. This is not surprising, given how the tri gger
is inferred using a scheme that optimizes for misclassiﬁcat ion
(Section IV). Our detection method effectively identiﬁes t heminimal trigger that would produce the same misclassiﬁcati on
results.
Visual Similarity. Figure 6 compares the original and
reversed triggers ( m·∆) in each of the four BadNets models.
We ﬁnd reversed triggers are roughly similar to original
triggers. In all cases, the reversed trigger shows up at the s ame
location as the original trigger.
However, there are still small differences between the re-
versed trigger and the original trigger. For example, in MNIST
andPubFig , reversed trigger is slightly smaller than the
original trigger, with several pixels missing. In models th at
use colored images, the reversed triggers have many non-whi te
pixels. These differences can be attributed to two reasons.
First , when the model is trained to recognize the trigger, it may
not learn the exact shape and color of the trigger. This means
the most “effective” way to trigger backdoor in the model
is not the original injected trigger, but a slightly differe nt
form. Second , our optimization objective is penalizing larger
triggers. Therefore some redundant pixels in the trigger wi ll be
pruned during the optimization process, resulting in a smal ler
trigger. Combined, it results in our optimization process ﬁ nd-
ing a more “compact” form of the backdoor trigger, compared
to the original trigger.
The mismatch between reversed trigger and original trigger
becomes more obvious in two Trojan Attack models, as shown
in Figure 7. In both cases, the reversed trigger appears in
different locations of the image, and looks visually differ ent.
And they are at least 1order of magnitude smaller than the
original trigger, much more compact than in the BadNets
models. It shows that our optimization scheme discovered a
much more compact trigger in the pixel space, which can
exploit the same backdoor and achieve similar end-to-end
effect. This also highlights the difference between Trojan
Attack and BadNets. Because Trojan Attack targets speciﬁc
neurons to connect input triggers to misclassiﬁcation outp uts,
they cannot avoid side effects on other neurons. The result i s a
broader attack that can be induced by a wider range of trigger s,
the smallest of which is identiﬁed by our reverse engineerin g
technique.
Similarity in Neuron Activations. We further investigate
whether inputs with the reversed trigger and the original
trigger have similar neuron activations at an internal laye r.
Speciﬁcally, we examine neurons in the second to last layer,
as this layer encodes relevant representative patterns in t he
input. We identify neurons most relevant to the backdoor, by
TABLE III. Average activation of backdoor neurons of clean i mages and
adversarial images stamped with reversed trigger and origi nal trigger.
ModelAverage Neuron Activation
Clean ImagesAdv. Images w/
Reversed TriggerAdv. Images w/
Original Trigger
MNIST 1.19 4.20 4.74
GTSRB 42.86 270.11 304.05
YouTube Face 137.21 1003.56 1172.29
PubFig 5.38 19.28 25.88
Trojan Square 2.14 8.10 17.11
Trojan Watermark 1.20 6.93 13.97
feeding clean and adversarial images and observing differe nces
in neuron activations at the target layer (second to last lay er).
We rank neurons by measuring differences in their activatio ns.
Empirically, we ﬁnd the top 1%of neurons are sufﬁcient to
enable the backdoor, i.e.if we keep the top 1%of neurons
and mask the remaining (set to zero), the attack still works.
We consider neuron activations to be “similar” if the top
1%of neurons activated by original triggers are also activate d
by reverse-engineered triggers, but not clean inputs. Tabl e III
shows the average neuron activation of top 1%neurons when
feeding1,000randomly selected clean and adversarial images.
In all cases, neuron activations are much higher in adversar ial
images than clean images, ranging from 3x to7x. This shows
that when added to inputs, both the reversed trigger and
original trigger activate the same backdoor-related neuro ns.6
Finally, we will leverage neural activations as a way to
represent backdoors in our mitigation techniques in Sectio n VI.
VI. M ITIGATION OF BACKDOORS
Once we have detected the presence of a backdoor, we apply
mitigation techniques to remove the backdoor while preserv ing
the model performance. We describe two complementary
techniques. First, we create a ﬁlter for adversarial input t hat
identiﬁes and rejects any input with the trigger, giving us
time to patch the model. Depending on the application, this
approach can also be used to assign a “safe” output label to
an adversarial input without rejection. Second, we patch the
DNN, making it nonresponsive against the detected backdoor
triggers. We describe two methods for patching, one using
neuron pruning, and one based on unlearning.
A. Filter for Detecting Adversarial Inputs
Our results in Section V-C show that neuron activations are
a better way to capture similarity between original and reve rse-
engineered triggers. Thus we build our ﬁlter based on neuron
activation proﬁle for the reversed trigger. This is measure d as
the average neuron activations of the top 1%of neurons in
the second to last layer. Given some input, the ﬁlter identiﬁ es
potential adversarial inputs as those with activation proﬁ les
higher than a certain threshold. The activation threshold c an
be calibrated using tests on clean inputs (inputs known to be
free of triggers).
We evaluate the performance of our ﬁlters using clean
images from the testing set and adversarial images created
6More detailed analysis reveals slight difference between B adNets and
Trojan Attack. Results are included in Appendix.by applying the original trigger to test images (1:1 ratio).
We calculate false positive rate (FPR) and false negative ra te
(FNR) when setting different thresholds for average neuron
activation. Results are shown in Figure 8. We achieve high
ﬁltering performance for all four BadNets models, obtainin g
<1.63% FNR at an FPR of 5%. Not surprisingly, Trojan
Attack models are more difﬁcult to ﬁlter out (likely due to
the differences in neuron activations between reversed tri gger
and original trigger). FNR is much higher for FPR <5%,
but we obtain a reasonable 4.3%and28.5%FNR at an FPR
of5%. Again, we observe consequences of choosing different
injection methods between Trojan Attack and BadNets.
B. Patching DNN via Neuron Pruning
To actually patch the infected model, we propose two
techniques. In the ﬁrst approach, the intuition is to use the
reversed trigger to help identify backdoor related compone nts
in DNN, e.g., neurons, and remove them. We propose to
prune out backdoor-related neurons from the DNN, i.e.set
these neurons’ output value to 0during inference. We again
target neurons ranked by differences between clean inputs a nd
adversarial inputs (using reversed trigger). We again targ et the
second to last layer, and prune neurons by order of highest
rank ﬁrst ( i.e.prioritizing those that show biggest activation
gap between clean and adversarial inputs). To minimize impa ct
on classiﬁcation accuracy of clean inputs, we stop pruning
when the pruned model is no longer responsive to the reversed
trigger.
Figure 9 shows classiﬁcation accuracy and attack success
rate when pruning different ratios of neurons in GTSRB .
Pruning30% of neurons reduces attack success rate to nearly
0%. Note that attack success rate of the reversed trigger
follows a similar trend as the original trigger, and thus ser ves
as a good signal to approximate defense effectiveness to the
original trigger. Meanwhile, classiﬁcation accuracy is re duced
only by5.06%. Of course, the defender can achieve smaller
drop in classiﬁcation accuracy by trading off decrease in at tack
success rate (follow the curve in Figure 9).
There is an interesting point to note. In Section V-C, we
identiﬁed the top 1%ranked neurons to be sufﬁcient to cause
misclassiﬁcation. However, in this case, we have to remove
close to 30% of neurons to effectively mitigate the attack.
This can be explained by the massive redundancy in neural
pathways in DNNs [29], i.e.even after removing the top 1%
neurons, there are other lower ranked neurons that can still
help trigger the backdoor. Prior work on compressing DNNs
has also noticed such high levels of redundancy [29].
We apply our scheme to other BadNets models and achieve
very similar results in MNIST andPubFig (See Figure 21
in Appendix). Pruning between 10% to30% neurons reduces
attack success rates to 0%. However, we observe a more signif-
icant negative impact on classiﬁcation accuracy in the case of
YouTube Face (Figure 21(c) in Appendix). For YouTube
Face , classiﬁcation accuracy drops from 97.55% to81.4%
when attack success rate drops to 1.6%. This is because the
second to last layer only has 160 output neurons, meaning
 0 0.2 0.4 0.6 0.8 1
MNIST GTSRB YouTube
FacePubFig Trojan
SquareTrojan
WMFalse Negative RateFPR=0.001
FPR=0.005
FPR=0.01
FPR=0.05
Fig. 8. False negative rate of proactive adversar-
ial image detection when achieving different false
positive rate. 0 0.2 0.4 0.6 0.8 1
 0  0.1  0.2  0.3  0.4  0.5 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
Fig. 9. Classiﬁcation accuracy and attack success
rate when pruning trigger-related neurons in GT-
SRB (trafﬁc sign recognition w/ 43labels). 0 0.2 0.4 0.6 0.8 1
 0  0.1  0.2  0.3  0.4  0.5 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
Fig. 10. Classiﬁcation accuracy and attack success
rate when pruning trigger-related neurons in Trojan
Square (face recognition w/ 2,622 labels).
clean neurons are heavily mixed with adversarial neurons.
This causes clean neurons to be pruned during the process,
therefore reducing classiﬁcation accuracy. Thus we experi ment
with pruning at multiple layers, and ﬁnd that pruning at the l ast
convolution layer produces the best results. In all four Bad Nets
models, attack success rate reduces to <1%with minimal
reduction in classiﬁcation accuracy <0.8%. Meanwhile, at
most8%of neurons are pruned. We plot those detailed results
in Figure 22 in the Appendix.
Neuron Pruning in Trojan Models. We note that pruning
is less effective in our Trojan models, using the same prunin g
methodology and conﬁguration. As shown in Figure 10, when
pruning30% neurons, attack success rate using our reverse-
engineered trigger drops to 10.1%, but success using the
original trigger remains high, at 87.3%. This discrepancy is
due to the dissimilarity in neuron activations between reve rsed
trigger and the original (Section V-C). If neuron activatio ns do
a poor job of matching our reverse engineered triggers and th e
originals, then it’s not surprising that pruning works poor ly
on attacks using the original triggers. Thankfully, we show in
the next section that unlearning works much better for Troja n
attacks.
Strengths and Limitations. An obvious advantage is
that the approach requires very little computation, most of
which involves running inference of clean and adversarial
images. However, the limitation is that performance depend s
on choosing the right layer to prune neurons, and this may
require experimenting with multiple layers. Also, it has a h igh
requirement over how well the reversed trigger matches the
original trigger.
C. Patching DNNs via Unlearning
Our second approach of mitigation is to train DNN to
unlearn the original trigger. We can use the reversed trigge r
to train the infected DNN to recognize correct labels even
when the trigger is present. Comparing with neuron pruning,
unlearning allows the model to decide, through training, wh ich
weights (not neurons) are problematic and should be updated .
For all models, including Trojan models, we ﬁne-tune the
model for only 1epoch, using an updated training dataset.
To create this new training set, we take a 10% sample of
the original training data (clean, with no triggers)7, and add
7The exception is PubFig , where we use the full training data because
training data is very limited.the reversed trigger to 20% of this sample without modifying
labels. To measure effectiveness of patching, we measure
attack success rate of the original trigger, and classiﬁcat ion
accuracy of the ﬁne-tuned model.
Table IV compares the attack success rate and classiﬁcation
accuracy before and after training. In all models, we manage
to reduce attack success rate to <6.70%, without signiﬁcantly
sacriﬁcing classiﬁcation accuracy. The largest reduction of
classiﬁcation accuracy is in GTSRB , which is only 3.6%. An
interesting point is that in some models, especially Trojan At-
tack models, there is an increase in classiﬁcation accuracy after
patching. Note that when injecting the backdoor, the Trojan
Attack models suffer degradation in classiﬁcation accurac y.
Original uninfected Trojan Attack models have a classiﬁcat ion
accuracy of 77.2%(not shown in Table IV), which is now
improved when the backdoor is patched.
We compare the efﬁcacy of this unlearning versus two
variants. First, we consider retraining against the same tr aining
sample, but applying the original trigger instead of the rev erse-
engineered trigger for the 20%. As shown in Table IV,
unlearning using the original trigger achieves slightly lo wer
attacker success rate with similar classiﬁcation accuracy . So
unlearning with our reversed trigger is a good approximatio n
for unlearning using the original. Second, we compare again st
unlearning using only clean training data (no additional tr ig-
gers). Results in last column in Table IV show that unlearnin g
is ineffective for all BadNets models (attack success rate
still high: >93.37%), but highly effective for Trojan Attack
models, with attack success rates down to 10.91% and0%for
Trojan Square , andTrojan Watermark respectively.
This seems to show that Trojan Attack models, with their
highly targeted re-tuning of speciﬁc neurons, are much more
sensitive to unlearning. A clean input that helps reset a few
key neurons disables the attack. In contrast, BadNets injec ts
backdoors by updating all layers using a poisoned dataset,
and seems to require signiﬁcantly more work to retrain and
mitigate the backdoor.
We also checked the impact of patching false positive
labels. Patching falsely ﬂagged labels in YouTube Face and
Trojan Square (discussed in Section V-B), only reduces
the classiﬁcation accuracy by <1%. Thus there is negligible
impact of false positives in detection on the mitigation par t.
Parameters and Cost. Through experiments, we ﬁnd that
unlearning performance is generally insensitive to parame ters
like amount of training data, and ratio of modiﬁed training
TABLE IV . Classiﬁcation accuracy and attack success rate be fore and after unlearning backdoor. Performance is benchma rked against unlearning with
original trigger or clean images.
TaskBefore Patching Patching w/ Reversed Trigger Patching w/ Original Trigger Patching w/ Clean Images
Classiﬁcation
AccuracyAttack Success
RateClassiﬁcation
AccuracyAttack Success
RateClassiﬁcation
AccuracyAttack Success
RateClassiﬁcation
AccuracyAttack Success
Rate
MNIST 98.54% 99.90% 97.69% 0.57% 97.77% 0.29% 97.38% 93.37%
GTSRB 96.51% 97.40% 92.91% 0.14% 90.06% 0.19% 92.02% 95.69%
YouTube Face 97.50% 97.20% 97.90% 6.70% 97.90% 0.0% 97.80% 95.10%
PubFig 95.69% 97.03% 97.38% 6.09% 97.38% 1.41% 97.69% 93.30%
Trojan Square 70.80% 99.90% 79.20% 3.70% 79.60% 0.0% 79.50% 10.91%
Trojan Watermark 71.40% 97.60% 78.80% 0.00% 79.60% 0.00% 79.50% 0.00%
data. Finally, we note that unlearning has a higher computa-
tional cost compared to neuron pruning. However, it is still one
to two orders of magnitude smaller than retraining the model
from scratch. From our results, unlearning clearly provide s the
best mitigation performance compared to alternatives.
VII. R OBUSTNESS AGAINST ADVANCED BACKDOORS
Prior sections described and evaluated detection and mitig a-
tion of backdoor attacks under base case assumptions, e.g., a
small number of triggers, each prioritizing stealth and tar geting
the misclassiﬁcation of arbitrary input into a single targe t label.
Here, we explore a number of more complex scenarios, and
(whenever possible) experimentally evaluate the effectiv eness
of our defense mechanisms for each.
We discuss 5speciﬁc types of advanced backdoors attacks,
each challenging an assumption or limitation in the current
defense design.
•Complex Triggers . Our detection scheme relies on the
success of the optimization process. Would more compli-
cated triggers make it more challenging for our optimizatio n
function to converge?
•Larger Triggers . We consider larger triggers. By increas-
ing trigger size, an attacker can force the reverse engineer ing
process to converge to a large trigger with larger norm.
•Multiple Infected Labels with Separate Triggers . We
consider a scenario where multiple backdoors targeting
distinct labels are inserted into a single model, and evalua te
the maximum number of infected labels we can detect.
•Single Infected Label with Multiple Triggers . We con-
sider multiple triggers targeting the same label.
•Source-label-speciﬁc (Partial) Backdoors . Our detection
scheme is designed to detect triggers that induce misclas-
siﬁcation on arbitrary input. A “partial” backdoor that is
effective on inputs from a subset of source labels would be
more difﬁcult to detect.
A. Complex Trigger Patterns
As we observed in Trojan models, triggers with more
complicated patterns make it harder for the optimization to
converge to the exact trigger. A more random trigger pattern
might increase the difﬁculty of reverse engineering the tri gger.
We perform simple tests by ﬁrst changing the white square
trigger to a noisy square, where each pixel of the trigger is
assigned a random color. We inject this attack in MNIST ,
GTSRB ,YouTube Face , andPubFig , and evaluate detec-
tion performance. The resulting anomaly index in each modelis shown in Figure 11. Our technique detects the complex
trigger patterns in all cases. We also test our mitigation
techniques on these models. For ﬁltering, at an FPR of 5%,
we achieve <0.01% FNR in all models. Patching using
unlearning reduces attack success rate to <4.2%, with at
most3.1%reduction in classiﬁcation accuracy. Finally, we
tested backdoors with varying trigger shapes ( e.g., triangle,
checkerboard shapes) in GTSRB , and all detection and miti-
gation techniques worked as expected.
B. Larger Triggers
Larger triggers are likely to produce larger reverse-
engineered triggers. This could help the infected label mor e
closely resemble uninfected labels in the L1norm, making
outlier detection less effective. We run sample tests on GTSRB ,
and increase the size of trigger from 4×4(1.6%of the image)
to16×16(25%). All triggers are still white squares. We
evaluate the detection technique with same conﬁguration us ed
in previous experiments. Figure 12 shows the L1norm of
reversed triggers for infected and uninfected labels. As the
original trigger becomes larger, the reversed trigger also gets
larger as expected. When the trigger grows beyond 14×14, the
L1norm does indeed blend in with that of uninfected labels,
reducing the anomaly index below detection threshold. The
anomaly index metric is shown in Figure 13.
The maximum detectable trigger size is largely a function of
one factor: trigger size of uninfected labels (amount of change
necessary to cause misclassiﬁcation of all inputs between
uninfected labels). The trigger size of uninfected labels is
itself a proxy for measuring the distinctiveness of inputs a cross
different labels, i.e.more labels means larger trigger size for
uninfected labels and a greater ability to detect larger triggers.
On applications like YouTube Face , we were able to detect
triggers as large as 39% of the whole image. On MNIST which
has fewer labels, we were only able to detect triggers of size
up to18% of the image.8In general, a larger trigger is more
visually obvious and easier to identify by humans. However,
there may exist approaches to increase the trigger size whil e
remaining less obvious, which can be explored in future work .
C. Multiple Infected Labels with Separate Triggers
We consider a scenario where attackers insert multiple,
independent backdoors into a single model, each targeting a
distinctive label. Inserting many backdoors might collect ively
8No additional false positive label is found when using large r triggers in
all models we test.
 0 1 2 3 4 5
MNIST GTSRB YouTube
FacePubFigAnomaly Index
Fig. 11. Anomaly index of infected MNIST, GT-
SRB, YouTube Face, and PubFig model with noisy
square trigger. 0 50 100 150 200
4×4
(1.6%)8×8
(6.3%)12×12
(14.1%)16×16
(25%)L1 Norm of Trigger
Width × Length of Original Trigger
(Ratio to Image Size)Uninfected
Infected
Fig. 12. L1norm of reverse engineered triggers
of labels when increasing the size of the original
trigger in GTSRB (results of a single round). 0 1 2 3 4 5
4×4
(1.6%)8×8
(6.3%)12×12
(14.1%)16×16
(25%)Anomaly Index
Width × Length of Original Trigger
(Ratio to Image Size)
Fig. 13. Anomaly index of each infected GTSRB
model when increasing the size of the original
trigger (results averaged over 10rounds).
 0 0.2 0.4 0.6 0.8 1
1
(0.08%)200
(15.6%)400
(31.2%)600
(46.8%)800
(62.4%)1000
(79.9%) 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Avg. Attack Success Rate
# of Infected Labels in DNN
(Ratio to Total # of Labels)Classification
Attack
Fig. 14. Classiﬁcation accuracy and average attack
success rate when different number of labels are
infected in YouTube Face. 0 1 2 3 4
1
(2.3%)4
(9.3%)7
(16.3%)10
(23.3%)13
(30.2%)Anomaly Index
# of Infected Labels in DNN
(Ratio to Total # of Labels)
Fig. 15. Anomaly index of each infected GTSRB
model with different number of labels being in-
fected (results averaged over 10rounds). 0 40 80 120 160
1
(2.3%)4
(9.3%)7
(16.3%)10
(23.3%)13
(30.2%)L1 Norm of Trigger
# of Infected Labels in DNN
(Ratio to Total # of Labels)Uninfected
Infected
Fig. 16. L1norm of triggers from infected and
uninfected labels when different number of labels
are infected in GTSRB (results of a single round).
reduceδ∀→tfor many LtinL. This has the net effect of
making the impact of any single trigger less of an outlier
and harder to detect. The trade-off is that models are likely
to have a “maximum capacity” for learning backdoors while
maintaining their classiﬁcation. Too many backdoors are li kely
to lower classiﬁcation performance.
We experiment by generating distinctive triggers with mutu -
ally exclusive color patterns. We ﬁnd most models, i.e.MNIST ,
GTSRB , andPubFig , have enough capacity to support trig-
gers for every output label without affecting classiﬁcatio n
accuracy. But in YouTube Face , with1,283 labels, we
observe an obvious drop in average attack success rate once
triggers infect more than 15.6%of labels in the model. As
shown in Figure 14, average attack success rate drops with
too many triggers, conﬁrming our intuition.
We evaluate our defenses against multiple distinct backdoo rs
inGTSRB . As shown in Figure 15, once more than 8labels
(18.6%)have been infected with backdoors, it becomes very
difﬁcult for anomaly detection to identify the impact of
triggers. Our results show we can detect up to 3labels (30%)
forMNIST ,375labels (29.2%) forYouTube Face , and24
labels (36.9%) forPubFig .
Though outlier detection method fails in this scenario, the
underlying reverse engineering method still works. For all
infected labels, we successfully reverse engineer the corr ect
trigger. Figure 16 shows the norm of triggers for infected an d
uninfected labels. All infected labels have smaller norm than
uninfected labels. Further manual analysis also validates that
reversed triggers visually look similar as original trigge rs.A
conservative defender could manually inspect reversed tri g-
gers and determine model’s suspicion. After that, our tests
show that preemptive “patching” could successfully mitiga te
potential backdoors. When all labels are infected in GTSRB , 0 0.2 0.4 0.6 0.8 1
Unpatched  1  2  3Attack Success Rate
Iteration of Patching
Fig. 17. Attack success rate of 9triggers when patching DNN for different
number of iterations.
patching all labels using reversed triggers would reduce av er-
age attack success rate to 2.83%. Proactive patching provides
similar beneﬁts for the other models as well. Finally, ﬁlter ing
is also effective at detecting adversarial inputs with low F NR
at FPR of 5%across all BadNets models.
D. Single Infected Label with Multiple Triggers
We consider a scenario where multiple distinctive triggers
induce misclassiﬁcation to the same label. In this case, our
detection techniques would likely only detect and patch one
of the existing triggers. To test this, we inject 9white4×4
square triggers for the same target label into GTSRB . Those
triggers have the same shape and color, but are located in
different positions of the image, i.e.four corners, four edges,
and the center. The attack achieves >90% attack success rate
for all triggers.
Detection and patching results are included in Figure 17.
As suspected, a single run of our detection technique only
identiﬁes and patches one of the injected triggers. Fortuna tely,
running just 3iterations of our detection and patch algorithm
is able to successively reduce the success rate of all trigge rs to
<5%. We also test on other MNIST ,YouTube Face , and
PubFig , and attack success rate of all triggers are reduced to
<1%,<5%, and<4%.
E. Source-label-speciﬁc (Partial) Backdoors
In Section II, we deﬁne backdoor as a hidden pattern that
could misclassify arbitrary inputs from any label into the t arget
label. Our detection scheme is designed to ﬁnd these “com-
plete” backdoors. A less powerful, “partial” backdoor, cou ld
be designed such that triggers only trigger misclassiﬁcati on
when applied to input belonging to a subset of source labels,
and do nothing when applied to other inputs. Such backdoors
would be a challenge to detect using our existing methods.
Detecting partial backdoors requires slightly modifying o ur
detection scheme. Instead of reverse engineering a trigger to
every target label, we analyze all possible source-target l abel
pairs. For each label pair, we use samples belonging to the
source label to solve the optimization. The resulting rever sed
trigger would only be effective for the speciﬁc label pair.
Then, by comparing L1norm of triggers for different source-
target pairs, we can use the same outlier detection method
to identify label pairs that are particularly vulnerable an d
appear as anomaly. We experiment by injecting a backdoor
targeting one source-target label pair into MNIST . While the
injected backdoor works very well, our updated techniques f or
detection, and mitigation are all successful.
Analyzing all source-target label pairs increases the com-
putation cost of detection by a factor of N, whereNis the
number of labels. However, we can use a divide-and-conquer
algorithm to reduce the computational cost to a factor of logN.
We leave detailed evaluation to future work.
VIII. R ELATED WORK
Traditional machine learning assumes the environment is
benign. This assumption could be violated by an adversary at
either training or testing time.
Additional Backdoor Attacks and Defenses. In addition to
attacks mentioned in Section II, Chen et al. propose a backdoor
attack under a more restricted attack model, where attacker can
only pollute a limited portion of training set [17]. Another line
of work directly tampers with hardware the DNN is running
on [30], [31]. Such backdoor circuits would also alter model ’s
performance when a trigger is presented.
Poisoning Attacks. Poisoning attack pollutes the training
data to alter the model’s behavior. Different from backdoor
attack, poisoning attack does not rely on the trigger, and al ters
model’s behavior on a set of clean samples. Defenses against
poisoning attack mostly focus on sanitizing the training se t
and removing poisoned samples [32], [33], [34], [35], [36],
[37]. The insight is to ﬁnd samples that would alter model’s
performance signiﬁcantly [32]. This insight has shown to be
less effective against backdoor attack [17], as injected sa mples
do not affect model’s performance on clean samples. Also, it ’s
impractical in our attack model, as the defender does not hav e
access to the poisoned training set.
Other Adversarial Attacks against DNNs. Numerous
(non-backdoor) adversarial attacks have been proposed aga instgeneral DNNs, often crafting imperceptible modiﬁcations t o
images to cause misclassiﬁcation. These can be applied to
DNNs during inference [38], [39], [40], [41], [42]. A number
of defenses have been proposed [43], [44], [45], [46], [47],
yet many have shown to be less effective against an adaptive
adversary [48], [49], [50], [51]. Some recent work tries to c raft
universal perturbations, which would trigger misclassiﬁc ation
for multiple images in an uninfected DNN [52], [53]. This
line of work considers a different threat model assuming an
uninfected victim model, which is not the target scenario of
our defense.
IX. C ONCLUSION AND FUTURE WORK
Our work describes and empirically validates our robust
and general detection and mitigation tools against backdoo r
(Trojan) attacks on deep neural networks. Beyond the efﬁcac y
of our defense against basic and complex backdoors, one
of the unexpected takeaways of our paper is the signiﬁcant
differences between two backdoor injection methods: the
trigger-driven BadNets end-to-end attack with full access to
model training, and the neuron-driven Trojan Attack withou t
access to model training. Through our experiments, we ﬁnd
that the Trojan Attack injection method generally adds more
perturbations than necessary, and introduces unpredictab le
changes to non-targeted neurons. This makes their triggers
harder to reverse engineer, and makes them more resistant
to ﬁltering and neuron pruning. However, the tradeoff is tha t
their focus on speciﬁc neurons make them extremely sensitiv e
to mitigation via unlearning. In contrast, BadNets introdu ce
more predictable changes to neurons, and can be more easily
reverse engineered, ﬁltered and mitigated via neuron pruni ng.
Finally, while our results are robust against a range of
attacks in different applications, there are still limitat ions.
First and foremost is the question of generalization beyond
the current vision domain. Our high-level intuition and des ign
of detection/mitigation methods could be generalizable: t he
intuition for detection is that the infected label is more
vulnerable than uninfected labels, and this should be domai n
agnostic. The main challenge of adapting the entire pipelin e
to non-vision domain is to formulate the backdoor attack
process and design a metric measuring how vulnerable a
speciﬁc label is (like Equation 2 and Equation 3). Second, the
space of potential counter-measures of attacker could be la rge.
We study 5different counter-measures that speciﬁcally target
different components/assumptions of our defense, but furt her
exploration of other potential counter-measures remains p art
of future work.
ACKNOWLEDGEMENT
We thank our shepherd Roberto Perdisci and anonymous re-
viewers for their constructive feedback. This work is suppo rted
in part by NSF grants CNS-1527939 and CNS-1705042.
Any opinions, ﬁndings, and conclusions or recommendations
expressed in this material are those of the authors and do not
necessarily reﬂect the views of any funding agencies.
REFERENCES
[1] Q. Wang, W. Guo, K. Zhang, A. G. O. II, X. Xing, X. Liu, and C. L.
Giles, “Adversary resistant deep neural networks with an ap plication to
malware detection,” in Proc. of KDD , 2017.
[2] D. Arp, M. Spreitzenbarth, M. Hubner, H. Gascon, and K. Ri eck,
“Drebin: Effective and explainable detection of an- droid m alware in
your pocket,” in Proc. of NDSS , 2014.
[3] Z. L. Chua, S. Shen, P. Saxena, and Z. Liang, “Neural nets c an learn
function type signatures from binaries,” in Proc. of USENIX Security ,
2017.
[4] E. C. R. Shin, D. Song, and R. Moazzezi, “Recognizing func tions in
binaries with neural networks,” in Proc. of USENIX Security , 2015.
[5] H. Debar, M. Becker, and D. Siboni, “A neural network comp onent for
an intrusion detection system,” in Proc. of IEEE S&P , 1992.
[6] C. Wierzynski, “The Challenges and Opportunities of Exp lainable AI,”
https://ai.intel.com/the-challenges-and-opportuniti es-of-explainable- ai,
Jan. 2018.
[7] “FICO’s Explainable Machine Learning Challenge,” http s://community.
ﬁco.com/s/explainable-machine-learning-challenge, 20 18.
[8] Z. C. Lipton, “The mythos of model interpretability,” in ICML Workshop
on Human Interpretability in Machine Learning , 2016.
[9] S. M. Lundberg and S.-I. Lee, “A uniﬁed approach to interp reting model
predictions,” in Proc. of NIPS , 2017.
[10] S. Bach, A. Binder, G. Montavon, F. Klauschen, K. R. Mull er, and
W. Samek, “On pixel-wise explanations for non-linear class iﬁer deci-
sions by layer-wise relevance propagation,” PloS One , vol. 10, no. 7,
July 2015.
[11] W. Guo, D. Mu, J. Xu, P. Su, G. Wang, and X. Xing, “Lemna: Ex plaining
deep learning based security applications,” in Proc. of CCS , 2018.
[12] T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identify ing vulnerabili-
ties in the machine learning model supply chain,” in Proc. of Machine
Learning and Computer Security Workshop , 2017.
[13] Y . Liu, S. Ma, Y . Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang,
“Trojaning attack on neural networks,” in Proc. of NDSS , 2018.
[14] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan , I. Goodfellow,
and R. Fergus, “Intriguing properties of neural networks,” inProc. of
ICLR , 2014.
[15] K. Liu, B. Dolan-Gavitt, and S. Garg, “Fine-pruning: De fending against
backdooring attacks on deep neural networks,” in Proc. of RAID , 2018.
[16] Y . Liu, Y . Xie, and A. Srivastava, “Neural trojans,” in Proc. of ICCD ,
2017.
[17] X. Chen, C. Liu, B. Li, K. Lu, and D. Song, “Targeted backd oor
attacks on deep learning systems using data poisoning,” arXiv preprint
arXiv:1712.05526 , 2017.
[18] J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson , “Under-
standing neural networks through deep visualization,” arXiv preprint
arXiv:1506.06579 , 2015.
[19] D. P. Kingma and J. Ba, “Adam: A method for stochastic opt imization,”
arXiv preprint arXiv:1412.6980 , 2014.
[20] F. R. Hampel, “The inﬂuence curve and its role in robust e stimation,”
Journal of the American Statistical Association , vol. 69, no. 346, pp.
383–393, 1974.
[21] P. J. Rousseeuw and C. Croux, “Alternatives to the media n absolute
deviation,” Journal of the American Statistical association , vol. 88, no.
424, pp. 1273–1283, 1993.
[22] “https://www.cs.tau.ac.il/ ∼wolf/ytfaces/,” YouTube Faces DB.
[23] Y . LeCun, L. Jackel, L. Bottou, C. Cortes, J. S. Denker, H . Drucker,
I. Guyon, U. Muller, E. Sackinger, P. Simard et al. , “Learning algorithms
for classiﬁcation: A comparison on handwritten digit recog nition,”
Neural networks: the statistical mechanics perspective , vol. 261, p. 276,
1995.
[24] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, “Ma n vs. computer:
Benchmarking machine learning algorithms for trafﬁc sign r ecognition,”
Neural Networks , 2012.
[25] Y . Sun, X. Wang, and X. Tang, “Deep learning face represe ntation from
predicting 10,000 classes,” in Proc. of CVPR , 2014.
[26] “http://vision.seas.harvard.edu/pubﬁg83/,” PubFi g83: A resource for
studying face recognition in personal photo collections.
[27] O. M. Parkhi, A. Vedaldi, A. Zisserman et al. , “Deep face recognition.”
inProc. of BMVC , 2015.
[28] “http://www.robots.ox.ac.uk/ ∼vgg/data/vgg face/,” VGG Face Dataset.[29] H. Hu, R. Peng, Y .-W. Tai, and C.-K. Tang, “Network trimm ing: A data-
driven neuron pruning approach towards efﬁcient deep archi tectures,”
arXiv preprint arXiv:1607.03250 , 2016.
[30] J. Clements and Y . Lao, “Hardware trojan attacks on neur al networks,”
arXiv preprint arXiv:1806.05768 , 2018.
[31] W. Li, J. Yu, X. Ning, P. Wang, Q. Wei, Y . Wang, and H. Yang, “Hu-fu:
Hardware and software collaborative attack framework agai nst neural
networks,” in Proc. of ISVLSI , 2018.
[32] Y . Cao, A. F. Yu, A. Aday, E. Stahl, J. Merwine, and J. Yang , “Efﬁcient
repair of polluted machine learning systems via causal unle arning,” in
Proc. of ASIACCS , 2018.
[33] M. Jagielski, A. Oprea, B. Biggio, C. Liu, C. Nita-Rotar u, and B. Li,
“Manipulating machine learning: Poisoning attacks and cou ntermeasures
for regression learning,” in Proc. of IEEE S&P , 2018.
[34] B. I. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S.-h . Lau, S. Rao,
N. Taft, and J. Tygar, “Antidote: understanding and defendi ng against
poisoning of anomaly detectors,” in Proc. of IMC , 2009.
[35] M. Mozaffari-Kermani, S. Sur-Kolay, A. Raghunathan, a nd N. K. Jha,
“Systematic poisoning attacks on and defenses for machine l earning in
healthcare,” IEEE journal of biomedical and health informatics , vol. 19,
no. 6, pp. 1893–1905, 2015.
[36] J. Steinhardt, P. W. W. Koh, and P. S. Liang, “Certiﬁed de fenses for data
poisoning attacks,” in Proc. of NIPS , 2017.
[37] G. F. Cretu, A. Stavrou, M. E. Locasto, S. J. Stolfo, and A . D. Keromytis,
“Casting out demons: Sanitizing training data for anomaly s ensors,” in
Proc. of IEEE S&P , 2008.
[38] N. Carlini and D. Wagner, “Towards evaluating the robus tness of neural
networks,” in Proc. of IEEE S&P , 2017.
[39] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning
at scale,” in Proc. of ICLR , 2017.
[40] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Ce lik, and
A. Swami, “The limitations of deep learning in adversarial s ettings,”
inProc. of Euro S&P , 2016.
[41] Y . Liu, X. Chen, C. Liu, and D. Song, “Delving into transf erable
adversarial examples and black-box attacks,” in Proc. of ICLR , 2016.
[42] B. Wang, Y . Yao, B. Viswanath, Z. Haitao, and B. Y . Zhao, “ With
great training comes great vulnerability: Practical attac ks against transfer
learning,” in Proc. of USENIX Security , 2018.
[43] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Di stillation
as a defense to adversarial perturbations against deep neur al networks,”
inProc. of IEEE S&P , 2016.
[44] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vlad u, “Towards
deep learning models resistant to adversarial attacks,” in Proc. of ICLR ,
2018.
[45] H. Kannan, A. Kurakin, and I. Goodfellow, “Adversarial logit pairing,”
arXiv preprint arXiv:1803.06373 , 2018.
[46] D. Meng and H. Chen, “Magnet: a two-pronged defense agai nst adver-
sarial examples,” in Proc. of CCS , 2017.
[47] W. Xu, D. Evans, and Y . Qi, “Feature squeezing: Detectin g adversarial
examples in deep neural networks,” in Proc. of NDSS , 2018.
[48] N. Carlini and D. Wagner, “Defensive distillation is no t robust to
adversarial examples,” arXiv preprint arXiv:1607.04311 , 2016.
[49] W. He, J. Wei, X. Chen, N. Carlini, and D. Song, “Adversar ial example
defenses: Ensembles of weak defenses are not strong,” in Proc. of
WOOT , 2017.
[50] N. Carlini and D. Wagner, “Magnet and efﬁcient defenses against
adversarial attacks are not robust to adversarial examples ,”arXiv preprint
arXiv:1711.08478 , 2017.
[51] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated grad ients give a
false sense of security: Circumventing defenses to adversa rial examples,”
inProc. of ICML , 2018.
[52] T. B. Brown, D. Man´ e, A. Roy, M. Abadi, and J. Gilmer, “Ad versarial
patch,” arXiv preprint arXiv:1712.09665 , 2017.
[53] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Fros sard, “Univer-
sal adversarial perturbations,” in Proc. of CVPR , 2017.
APPENDIX
BACKDOOR DETECTION USING TESTING DATA
In previous experiments, we use training data for detecting
backdoors. In many scenarios, full training data may not be
available, and users only have access to limited testing dat a
to validate models. Here, we determine if detection achieve s
 0 1 2 3 4 5 6
MNIST GTSRB YouTube
FacePubFig Trojan
SquareTrojan
WMAnomaly IndexInfected
Clean
Fig. 18. Anomaly measurement of infected and clean model by how much t he
label with smallest trigger deviates from the remaining lab els (using testing
data).
 0 100 200 300 400
MNIST GTSRB YouTube
FaceL1 Norm of TriggerUninfected
Infected
 0 1000 2000 3000 4000
PubFig Trojan
SquareTrojan
WM
Fig. 19.L1norm of triggers for infected and uninfected labels in backd oored
models (using testing data).
similar performance using only limited testing data. For al l
models, we follow the same detection conﬁguration, but use
only50% of the testing data. The remaining 50% is used for
evaluating the effectiveness of the reversed trigger. Figu re 18
shows all infected models have anomaly index larger than 3,
while all clean models have anomaly index lower than 2. As
before, our detection method correctly differentiates inf ected
models and clean models.
Figure 19 plots the distribution of infected and uninfected
labels, when search for backdoors using testing data. Again ,
all infected labels have triggers with much smaller L1norm
values, compared to uninfected labels. Together, these res ults
show that our detection method is still effective, even when
using only limited testing data.
TABLE V . Intersection-over-union ratio of backdoor neurons used by
reversed trigger and original trigger.
Model MNIST GTSRBYouTube
FacePubFigTrojan
SquareTrojan
Watermark
Intersection
over Union Ratio0.807 0.892 0.583 0.775 0.104 0.117
DETAILED ANALYSIS OF REVERSED TRIGGER ’SNEURON
ACTIVATION SIMILARITY
Our previous experiment shows that reversed-engineered
triggers do activate malicious neurons used by the original
triggers. However, it’s still possible that reversed trigg ers
activate additional neurons. Here we further determine if t he
reversed trigger and the original trigger activate exactly same
set of neurons. This is a slightly different question from th ose
answered by experiments in Section V-C. Here, we identify to p
1%most important neurons for the reversed trigger and theoriginal trigger, respectively. Then, in each model, we com pute
the intersection-over-union ratio of the two sets of neuron s. A
ratio closer to 1indicates two triggers activate more similar
sets of neurons.
Table V shows the intersection-over-union ratio in all 6
backdoored models. We see that all BadNets models have
ratios higher than 0.58, which indicates the reversed trigger
is very similar to the original trigger in neuron activation .
However, ratios in Trojan models are much smaller ( 0.104and
0.117), which suggests that the reversed trigger shares less in
common with the original trigger. As we mentioned before,
this is likely caused by the design of Trojan Attack. Since
Trojan Attack relies on speciﬁc neurons to build a stronger
connection between the trigger and the misclassiﬁcation ou t-
put, the side affect on other neurons results in a wider range of
triggers. The reversed trigger, being the smallest among th em
(based on L1norm), uses a slightly different set of neurons,
but can still achieve similar end-to-end misclassiﬁcation effect.
(a) MNIST
 (b) GTSRB
 (c) YouTube Face
 (d) PubFig
 (e) Trojan Square
 (f) Trojan WM
Fig. 20. Examples of adversarial images in BadNets models (w ith white square trigger added to the bottom right corner of t he image), Trojan Square, and
Trojan Watermark.
TABLE VI. Detailed information about dataset and training c onﬁgurations for each BadNets models.
Task / Dataset # of LabelsTraining
Set SizeTesting
Set SizeTraining Conﬁguration
MNIST 10 50,000 10,000 inject ratio=0.1, epochs=5, batch=32, optimizer=Adam, lr =0.001
GTSRB 43 35,288 12,630 inject ratio=0.1, epochs=10, batch=32, optimizer=Adam, l r=0.001
YouTube Face 1,283 375,645 64,150 inject ratio=0.1, epochs=10, batch=32, optimizer=Adadel ta, lr=0.1
PubFig 65 5,850 650inject ratio=0.1, epochs=15, batch=32, optimizer=Adadel ta, lr=0.1
First12layers are frozen during training. First 5epochs are trained using clean data only.
 0 0.2 0.4 0.6 0.8 1
 0  0.1  0.2  0.3  0.4  0.5 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(a) MNIST 0 0.2 0.4 0.6 0.8 1
 0  0.1  0.2  0.3  0.4  0.5 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(b) GTSRB 0 0.2 0.4 0.6 0.8 1
 0  0.1  0.2  0.3  0.4  0.5 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(c) YouTube Face 0 0.2 0.4 0.6 0.8 1
 0  0.1  0.2  0.3  0.4  0.5 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(d) PubFig
Fig. 21. Classiﬁcation accuracy and attack success rate usi ng original/reversed trigger when pruning backdoor-relat ed neurons at the second to last layer.
 0 0.2 0.4 0.6 0.8 1
 0  0.02  0.04  0.06  0.08  0.1 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(a) MNIST 0 0.2 0.4 0.6 0.8 1
 0  0.04  0.08  0.12  0.16  0.2 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(b) GTSRB 0 0.2 0.4 0.6 0.8 1
 0  0.02  0.04  0.06  0.08  0.1 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(c) YouTube Face 0 0.2 0.4 0.6 0.8 1
 0  0.02  0.04  0.06  0.08  0.1 0 0.2 0.4 0.6 0.8 1Classification Accuracy
Attack Success Rate
Ratio of Neurons PrunedClassification
Attack w/ Original Trigger
Attack w/ Reversed Trigger
(d) PubFig
Fig. 22. Classiﬁcation accuracy and attack success rate of o riginal/reversed trigger when pruning backdoor-related n eurons at the last convolution layer.
TABLE VII. Model Architecture for MNIST. FC stands for fully -connected
layer.
Layer Type # of Channels Filter Size Stride Activation
Conv 16 5 ×5 1 ReLU
MaxPool 16 2 ×2 2 -
Conv 32 5 ×5 1 ReLU
MaxPool 32 2 ×2 2 -
FC 512 - - ReLU
FC 10 - - SoftmaxTABLE VIII. Model Architecture for GTSRB.
Layer Type # of Channels Filter Size Stride Activation
Conv 32 3 ×3 1 ReLU
Conv 32 3 ×3 1 ReLU
MaxPool 32 2 ×2 2 -
Conv 64 3 ×3 1 ReLU
Conv 64 3 ×3 1 ReLU
MaxPool 64 2 ×2 2 -
Conv 128 3 ×3 1 ReLU
Conv 128 3 ×3 1 ReLU
MaxPool 128 2 ×2 2 -
FC 512 - - ReLU
FC 43 - - Softmax
TABLE IX. DeepID Model Architecture for YouTube Face.
Layer Name (Type) # of Channels Filter Size Stride Activatio n Connected to
conv 1 (Conv) 20 4 ×4 2 ReLU
pool 1 (MaxPool) 2 ×2 2 - conv 1
conv 2 (Conv) 40 3 ×3 2 ReLU pool 1
pool 2 (MaxPool) 2 ×2 2 - conv 2
conv 3 (Conv) 60 3 ×3 2 ReLU pool 2
pool 3 (MaxPool) 2 ×2 2 - conv 3
fc1 (FC) 160 - - - pool 3
conv 4 (Conv) 80 2 ×2 1 ReLU pool 3
fc2 (FC) 160 - - - conv 4
add 1 (Add) - - - ReLU fc 1, fc 2
fc3 (FC) 1280 - - Softmax add 1
TABLE X. Model Architecture for PubFig.
Layer Type # of Channels Filter Size Stride Activation
Conv 64 3 ×3 1 ReLU
Conv 64 3 ×3 1 ReLU
MaxPool 64 2 ×2 2 -
Conv 128 3 ×3 1 ReLU
Conv 128 3 ×3 1 ReLU
MaxPool 128 2 ×2 2 -
Conv 256 3 ×3 1 ReLU
Conv 256 3 ×3 1 ReLU
Conv 256 3 ×3 1 ReLU
MaxPool 256 2 ×2 2 -
Conv 512 3 ×3 1 ReLU
Conv 512 3 ×3 1 ReLU
Conv 512 3 ×3 1 ReLU
MaxPool 512 2 ×2 2 -
Conv 512 3 ×3 1 ReLU
Conv 512 3 ×3 1 ReLU
Conv 512 3 ×3 1 ReLU
MaxPool 512 2 ×2 2 -
FC 4096 - - ReLU
FC 4096 - - ReLU
FC 65 - - Softmax

Bolun Wang
Yuanshun Yao
Shawn Shan
Huiying Li
Bimal Viswanath
Haitao Zheng
Ben Y. Zhao