Self-Supervise, Refine, Repeat: Improving Unsupervised Anomaly Detection

Under review as a conference paper at ICLR 2022
SELF-SUPERVISE , REFINE , REPEAT : IMPROVING UN-
SUPERVISED ANOMALY DETECTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Anomaly detection (AD) – separating anomalies from normal data – has many ap-
plications across domains, from manufacturing to healthcare. While most previous
works have been shown to be effective for cases with fully or partially labeled data,
that setting is in practice less common due to labeling being particularly tedious
for this task. In this paper, we focus on fully unsupervised AD, in which the entire
training dataset, containing both normal and anomalous samples, is unlabeled. To
tackle this problem effectively, we propose to improve the robustness of one-class
classiﬁcation trained on self-supervised representations using a data reﬁnement
process. Our proposed data reﬁnement approach is based on an ensemble of one-
class classiﬁers (OCCs), each of which is trained on a disjoint subset of training
data. Representations learned by self-supervised learning on the reﬁned data are
iteratively updated as the reﬁnement improves. We demonstrate our method on
various unsupervised AD tasks with image and tabular data. With a 10% anomaly
ratio on CIFAR-10 image data / 2.5% anomaly ratio on Thyroid tabular data, the
proposed method outperforms the state-of-the-art one-class classiﬁcation method
by 6.3 AUC and 12.5 average precision / 22.9 F1-score.
1 I NTRODUCTION
Anomaly detection (AD), the task of distinguishing anomalies from normal data, plays a crucial role in
many real-world applications such as detecting faulty products using visual sensors in manufacturing,
fraudulent behaviors at credit card transactions, or adversarial outcomes at intensive care units.
AD has been considered under various settings based on the availability of negative (normal) and
positive (anomalous) data and their labels at training, as overviewed in Sec. 2. Each application
scenario is dominated by different challenges. When entire positive and negative data are available
along with their labels (Fig. 1a), the problem can be treated as supervised classiﬁcation and the
dominant challenge becomes the imbalance in label distributions [4, 16, 20, 27, 32, 35]. When only
negative labeled data are available (Fig. 1b), the problem is ‘one-class classiﬁcation’ [ 23,26,33,
45,47,51,53]. Various works have also extended approaches designed for these to settings with
additional unlabeled data (Fig. 1c,d,e) [10, 18, 24, 46, 58] in a semi-supervised setting. While there
exist many prior works in these settings, they all depend on some labeled data , which is not desirable
in all application scenarios.
Unsupervised AD, on the other hand, poses unique challenges in the absence of any labeled data
information, and a straightforward adaption of methods developed with the assumption of labeled
data would be suboptimal. For example, some recent studies [ 7,60] have applied one-class classiﬁers
(OCCs) that are known to yield impressive performance when trained on negative samples [ 7,23,26,
33,51] to unsupervised AD, but their performance for unsupervised AD has been quite sub-optimal.
Fig. 2 illustrates this, showing the unsupervised AD performance of state-of-the-art Deep OCCs [ 51]
with different anomaly ratios in unlabeled training data – the average precision signiﬁcantly drops
even when a small portion (2%) of training data is contaminated with anomalies.
Our framework SRR ( Self-supervise, Reﬁne, Repeat), overviewed in Fig. 3, brings a novel approach
to unsupervised AD with the principles of self-supervised learning without labels and iterative data
reﬁnement based on the agreement of OCC outputs. We propose to improve the state-of-the-art
performance of OCCs, e.g. [ 33,51], by reﬁning the unlabeled training data so as to address the
fundamental challenges elaborated above. SRR iteratively trains deep representations using reﬁned
1
Under review as a conference paper at ICLR 2022
(a) P+N (c) P+N+U (f) U (b) N (d) P+U (e) N+UNegative ( N) sample, Positive ( P) sample, Unlabeled ( U) sample
Figure 1: AD problem settings. Blue and red dots are for labeled negative (normal) and positive
(anomalous) samples, respectively. Grey dots denote unlabeled samples. While previous works
mostly focus on supervised (a, b) or semi-supervised (c, d, e) settings, we tackle an AD problem
using only unlabeled data (f) that may contain both negative and positive samples.
data while improving the reﬁnement of unlabeled data by excluding potentially-positive (anomalous)
samples. For the data reﬁnement process, we employ an ensemble of OCCs, each of which is trained
on a disjoint subset of unlabeled training data. The samples are declared as normal if there is a
consensus between all the OCCs. The reﬁned training data are used to train the ﬁnal OCC to generate
the anomaly scores in the unsupervised setting. Most prior unsupervised AD works [23, 26, 33, 51]
assume that the data contains entirely negative samples, which makes them not truly unsupervised
as they require having humans to do the data ﬁltering. Similar to ours, some prior unsupervised
AD works [ 7,45,60] considered evaluating on an unsupervised setting where there exist a small
percentage of anomalous samples in the training data, i.e. operating in ‘truly’ unsupervised setting
without having the need for humans to do any ﬁltering in the training data. However, these methods
often suffered from signiﬁcant performance degradation as the ratio of anomalous sample ratio has
increased (see Sec. 4.2). We would like to highlight that our method distinguishes from the literature
by bringing a data-centric approach (reﬁning the unlabeled data) to unsupervised anomaly detection
beyond the model-centric approaches (improving the model itself). Our framework SRR aims to
provide robustness in performance as the anomalous sample ratio increases, as shown in Fig. 2.
0 2 4 6 8 10
Anomaly ratio (%) in training data55606570Average Precision
 Baseline (Contrastive)
Proposed model (SRR)
Figure 2: Performance of our proposed model
and a baseline OCC using contrastive repre-
sentation [ 51] on CIFAR-10 with different
anomaly ratios in the training data.We conduct extensive experiments across various
datasets from different domains, including semantic
AD (CIFAR-10 [ 29], Dog-vs-Cat [ 19]), real-world
manufacturing visual AD use case (MVTec [ 8]), and
tabular AD benchmarks. We consider methods with
both shallow [ 36,47] and deep [ 7,33,51] models.
We evaluate models at different anomaly ratios of
unlabeled training data and show that SRR signiﬁ-
cantly boosts performance. For example, in Fig. 2,
SRR improves more than 15.0 average precision (AP)
with a 10% anomaly ratio compared to a state-of-the-
art one-class contrastive representation model [ 51]
on CIFAR-10. Similarly, on MVTec SRR retains a
strong performance, dropping less than 1.0 AUC with
10% anomaly ratio, while the best existing OCC [ 33]
drops more than 6.0 AUC. We further investigate the
efﬁcacy of our design choices, such as the number
of ensemble classiﬁers, thresholds, and reﬁnement of
deep representations via ablation studies.
2 R ELATED WORK
There are various existing works under the different settings described in Fig. 1:
The positive + negative setting is often considered as a supervised binary classiﬁcation problem.
The challenge arises due to the imbalance in label distributions as positive (anomalous) samples are
rare. As summarized in [12], to address this, over-/under-sampling [16, 20], weighted optimization
[4,27], synthesizing data of minority classes [ 32,35], and hybrid methods [ 2,22] have been studied.
The negative setting is often converted to a one-class classiﬁcation problem, with the goal of
ﬁnding a decision boundary that includes as many one-class samples as possible. Shallow models
for this setting include one-class support vector machines [ 47] (OC-SVM), support vector data
2
Under review as a conference paper at ICLR 2022
description [ 53] (SVDD), kernel density estimation (KDE) [ 31], and Gaussian density estimation
(GDE) [ 44]. There are also auto-encoder based models [ 59] that treat the reconstruction error as the
anomaly score. Deep learning based OCCs have been developed, such as Deep OCC [ 45], geometric
transformation [ 23], or outlier exposure [ 26]. Noting the degeneracy or inconsistency of learning
objectives of existing end-to-end trainable Deep OCCs, [ 51] proposed a deep representation OCC, a
two-stage framework that learns self-supervised representations [ 17,28] followed by shallow OCCs.
That work was extended for texture anomaly localization with CutPaste [ 33]. The robustness to very
low anomaly ratios of these methods under the unsupervised setting was explored in [7, 60].
The semi-supervised setting is deﬁned as utilizing a small set of labeled samples and large set of
unlabeled samples to distinguish anomalies from normal data. Depending on which labeled samples
are given, this setting can be split into three sub-categories. When only some positive/negative labeled
samples are provided, we denote that as a PU/NU setting. Most previous works in semi-supervised
AD settings focus on the NU setting where only some of the normal labeled samples are given
[1,41,52]. The PNU setting is a more general semi-supervised setting where subsets of both positive
and negative labeled samples are given. Deep SAD [ 46] and SU-IDS [ 39] are included in this category.
We show the signiﬁcant outperformance of our proposed SRR framework compared to Deep SAD,
for multiple benchmark datasets when not using any labeled data (see Sec. 4.2).
The unlabeled setting has received relatively less attention despite its signiﬁcance in automating
machine learning. The popular methods for this setting include isolation forest [ 36] and local
outlier factor [ 13]. However, they are difﬁcult to scale, and less compatible with recent advances in
representation learning. While OCCs, such as OC-SVM, SVDD, or their deep counterparts, apply
to unlabeled settings by assuming the data is all negative, and the robustness of those methods has
also been demonstrated in part [ 7,60], in practice we observe a signiﬁcant performance drop with a
high anomaly ratio, shown in Fig. 2. In contrast, our proposed framework is able to maintain high
performance across anomaly ratios.
Data reﬁnement has been applied to AD in some prior works. [ 5,42] generate pseudo-labels using
binary classiﬁcation and OC-SVM for data reﬁnement to boost the consequent AD performances in
unsupervised settings. [ 6,30,40,55,59] used the reconstruction errors of the auto-encoder as an
indicator for removing possible anomalies. [ 21] and [ 38] used data reﬁnement for AD in supervised
and semi-supervised settings. Additional discussions can be found in Appendix A.9.
Self-training [37,48] is an iterative training mechanism using predicted pseudo labels as targets
for model training. It has regained popularity recently with its successful results in semi-supervised
image classiﬁcation [ 9,50,57]. To improve the quality of pseudo labels, employment of an ensemble
of classiﬁers has also been studied. [ 14] trains an ensemble of classiﬁers with different classiﬁcation
methods to make a consensus for noisy label veriﬁcation. Co-training [ 11] trains multiple classiﬁers,
each of which is trained on the distinct views, to supervise other classiﬁers. Co-teaching [ 25] and
DivideMix [ 34] share a similar idea in that they both train multiple deep networks on separate data
batches to learn different decision boundaries, thus becoming useful for noisy label veriﬁcation. While
sharing a similarity, the proposed framework has clear differences from the previous works. SRR
performs iterative training with data reﬁnement (with robust ensemble methods) and self-supervised
learning for unsupervised learning of an anomaly detector.
3 P ROPOSED FRAMEWORK
Self-supervise, Reﬁne, and Repeat (SRR) is an iterative training framework, where we reﬁne the data
(Sec. 3.1) and update the representation with the reﬁned data (Sec. 3.2), followed by OCCs on reﬁned
representations. Fig. 3 overviews the framework and Algorithm 1 provides the pseudo code.
Notation. We denote the training data as D=fxigN
i=1where xi2X andNis the number of training
samples.yi2f0;1gis the corresponding label to xi, where 0denotes normal (negative) and 1denotes
anomaly (positive). Note that labels are not provided in the unsupervised setting.
Let us denote a feature extractor as g:X!Z .gmay include any data preprocessing functions, an
identity function (if raw data is directly used for one-class classiﬁcation), and learned or learnable
representation extractors such as deep neural networks. Let us deﬁne an OCC as f:Z! [1;1]
that outputs anomaly scores given the input features g(x). The higher the score f(g(x)), the more
anomalous the sample xis. The binary anomaly prediction is made after thresholding: 1
f(g(x))

.
3
Under review as a conference paper at ICLR 2022
Representation
learner Data refinement (Ensemble one-class classifier)
One-class
classifier 1
One-class
classifier KOne-class
classifier 2Subset 1
Subset 2
Subset KUnlabeled data
( D)Train Inference
∩Intersect
......One-class
classifier Train
Figure 3: Block diagram of SRR composed of representation learner (Sec. 3.2), data reﬁnement
(Sec. 3.1), and ﬁnal OCC blocks. The representation learner updates the deep models using reﬁned
data from the data reﬁnement block. Data reﬁnement is done by an ensemble of OCCs, each of
which is trained on Kdisjoint subsets of unlabeled training data. Samples predicted as normal by all
classiﬁers are retained in the reﬁned data, and are used to update the representation learner and ﬁnal
OCC. The process is repeated iteratively until convergence. Convergence graphs can be found in the
Appendix A.8.
3.1 D ATA REFINEMENT
A naive way to generate pseudo labels of unlabeled data is to construct an OCC on raw data or learned
representations as in [ 51] and threshold the anomaly score to obtain a binary label for normal vs.
anomalous. As we update the model with reﬁned data that excludes samples that are predicted to be
anomalous, it is important to generate pseudo labels of training data as accurately as possible.
To this end, instead of training a single classiﬁer (unlike most previous works on data reﬁnement
for AD [ 5,42,59]), we train an ensemble of KOCCs and aggregate their predictions to generate
pseudo labels. We illustrate the data reﬁnement block in Fig. 3 and as REFINE DATAin Algorithm 1.
Speciﬁcally, we randomly divide the unlabeled training data DintoKdisjoint subsetsD1;:::;DK,
and trainKdifferent OCCs (f1;:::;fK)on corresponding subsets (D1;:::;DK). Then, we estimate
a binary pseudo-label of the data xi2D as follows:
^yi= 1KY
k=1h
1 1
fk(g(xi))ki
(1)
k= maxs.t.1
NNX
i=11
fk(g(xi))

 (2)
where 1()is the indicator function that outputs 1/0 if the input is True/False. fk(g(xi))represents
an anomaly score of xifor an OCC fk.kin equation 2 is a threshold determined as a
percentile
of the anomaly score distribution ffk(g(xi))gN
i=1.
To interpret equation 1, xiis predicted as normal, i.e. ^yi= 0, if allKOCCs predict it as normal.
While this may be too strict and potentially reject many true normal samples in the training set, we
ﬁnd that empirically, it is critical to be able to exclude true anomalous samples from the training set.
The effectiveness of the employment of an ensemble of classiﬁers is empirically shown in Sec. 4.3.
3.2 R EPRESENTATION UPDATE
SRR follows the idea of deep representation OCCs [ 51], where in the ﬁrst stage a deep neural
network is trained with self-supervised learning (such as rotation prediction [ 23], contrastive [ 51], or
CutPaste [ 33]) to obtain meaningful representations of the data, and in the second stage OCCs are
trained on these learned representations. Such a two-stage framework is shown to be beneﬁcial as it
prevents the ‘hypersphere collapse’ of the deep OCCs by the favorable inductive bias it brings with
the architectural constraints [45].
Here, we propose to conduct self-supervised representation learning jointly with data reﬁnement.
More precisely, we train a feature extractor gusing ^D=fxij^yi= 0g, a subset of unlabeled data
4
Under review as a conference paper at ICLR 2022
Algorithm 1 SRR: Self-supervise, Reﬁne, Repeat.
Input : Training dataD=fxigN
i=1, Ensemble count ( K), threshold (
)
Output : Reﬁned data ( ^D), trained OCC ( f), feature extractor ( g)
1:function REFINE DATA(D;g;K;
 )
2: Train OCC models ffkgK
k=1onfDkgK
k=1,Kdisjoint subsets of the training data D.
3: Compute thresholds k’s for
percentile of anomaly distributions (equation 2).
4: Predict binary labels ^yi(equation 1).
5: Return ^D=fxi: ^yi= 0;xi2Dg .
6:end function
7:function SRR(D;K;
 )
8: Initialize the feature extractor g.
9: whilegnot converged do
10: ^D= R EFINE DATA(D;g;K;
 ).
11: Updategusing ^Dwith self-supervised learning objectives.
12: end while
13: ^D= R EFINE DATA(D;g;K;
 ).
14: Train an OCC model ( f) on reﬁned data (^D).
15:end function
Dthat only includes samples whose predicted labels with an ensemble OCC from Sec. 3.1 are
negative. We also update ^Das we proceed with representation learning. The proposed method is
illustrated in Algorithm 1 as SRR . In contrast to previous works [ 33,51] that use the entire training
data for learning self-supervised representation, we ﬁnd it necessary to reﬁne the training data even
for learning deep representations. Without representation reﬁnement, the performance improvements
of SRR are limited, as shown in Sec. 4.3.2. Last, for test-time prediction, we train an OCC on reﬁned
data ^Dusing updated representations by gas in line 13-14 in Algorithm 1.
3.3 U NSUPERVISED MODEL SELECTION
As SRR is designed for unsupervised AD, labeled validation data for hyperparameter tuning is
typically not available and the framework should enable robust model selection without any reliance
on labeled data . Here, we provide insights on how to select important hyperparameters, and later in
Sec. 4.3.1 perform sensitivity analyses for these hyperparameters.
Data reﬁnement of SRR introduces two hyperparameters: the number of OCCs ( K) and the percentile
threshold (
). There is a trade-off between the number of classiﬁers for the ensemble and the size
of disjoint subsets for training each classiﬁer. With large K, we aggregate prediction from many
classiﬁers, each of which may contain randomness from training. This comes at a cost of reduced
performance per classiﬁer as we use smaller subsets to train them. In practice, we ﬁnd K= 5works
well across different datasets and anomaly ratios.
controls the purity and coverage of reﬁned
data. If
is large, and thus classiﬁers reject too many samples, the reﬁned data could be more pure
and contain mostly the normal samples; however, the coverage of the normal samples would be
limited. On the other hand, with a small
, the reﬁned data may still contain many anomalies and the
performance improvement with SRR would be limited. We empirically observe that SRR is robust
to the selection of
when it is chosen from a reasonable range. In our empirical experiments, we
ﬁnd12of the true anomaly ratio to be a reasonable choice. In other words, it is safer to use

higher than the expected true anomaly ratio. In some cases, the true anomaly ratio may not be
available at all; for such scenarios, we propose Otsu’s method [ 49] to estimate the anomaly ratio of
the training data for determining the threshold
(experimental results are in the Appendix A.5).
4 E XPERIMENTS
We evaluate the efﬁcacy of our proposed framework for unsupervised AD tasks on tabular (Sec. 4.1)
and image (Sec. 4.2) data types. We experiment varying ratios of anomaly samples in unlabeled
training data and with different combinations of representation learning and OCCs. In Sec. 4.3, we
5
Under review as a conference paper at ICLR 2022
provide performance analyses to better explain major constituents of the performance, as well as
sensitivity to hyperparameter values.
Implementation details: To reduce the computational complexity of the data reﬁnement block, we
utilize a simple OCC such as GDE in the data reﬁnement block. In a two-stage model, we only
update the data reﬁnement block at 1st, 2nd, 5th, 10th, 20th, 50th, 100th, 500th epochs instead of
every epoch. After 500 epochs, we update the data reﬁnement block per each 500th epoch. Each run
of experiments requires a single V100 GPU. Additional discussions can be found in Appendix A.10.
4.1 E XPERIMENTS ON TABULAR DATA
Datasets. Following [ 7,60], we test the efﬁcacy of SRR on a variety of tabular datasets, including
KDDCup, Thyroid, or Arrhythmia from the UCI repository [ 3]. We also use KDDCup-Rev, where
the labels of KDDCup are reversed so that an attack represents anomaly [ 60]. To construct data
splits, we use 50% of normal samples for training. In addition, we hold out some anomaly samples
(amounting to 10% of the normal samples) from the data. This allows us to simulate unsupervised
settings with an anomaly ratio of up to 10% of entire training set. The rest of the data is used for
testing.1We conduct experiments using 5 random splits and 5 random seeds, and report the average
and standard deviation of 25 F1-scores (with scale 0-100) for the performance metric.
Models. We mainly compare with GOAD [ 7] (the state-of-the-art AD model in the tabular domain)
and implement SRR on top of it. GOAD utilizes random transformation classiﬁcation as the pretext
task of self-supervised learning, and the normality score is determined by whether transformations
are accurately included in the transformed space of the normal samples. We re-implement GOAD [ 7]
with a few modiﬁcations. First, instead of using embeddings to compute the loss, we use a parametric
classiﬁer, similarly to augmentation prediction [ 51]. Second, we follow the two-stage framework [ 51]
to construct deep OCCs. For the clean training data setting, our implementation achieves 98.0 for
KDD, 95.0 for KDD-Rev, 75.1 for Thyroid, and 54.8 for Arrhythmia F1-scores, which are comparable
to those reported in [7]. Please see Appendix A.3 for formulation and implementation details.
0 2 4 6 8 10
Anomaly ratio (%) in training data80.082.585.087.590.092.595.097.5F-score
KDD
SRR+GOAD
GOAD
OC-SVM (rbf kernel)
0.0 0.5 1.0 1.5 2.0 2.5
Anomaly ratio (%) in training data2030405060708090100F-score
KDD-Rev
SRR+GOAD
GOAD
OC-SVM (rbf kernel)
0.0 0.5 1.0 1.5 2.0 2.5
Anomaly ratio (%) in training data20304050607080F1-score
Thyroid
SRR+GOAD
GOAD
OC-SVM (rbf kernel)
0 2 4 6 8 10
Anomaly ratio (%) in training data3540455055F-score
Arrhythmia
SRR+GOAD
GOAD
OC-SVM (rbf kernel)
Figure 4: Unsupervised AD performance (F1-score) using OC-SVM (with rbf kernel), GOAD [ 7],
and GOAD with the proposed method SRR on various tabular datasets. Shaded areas represent the
standard deviation.
Results. We show results of GOAD (the baseline) and GOAD with SRR in Fig. 4. The ranges of the
noise ratio are set to 0% for the anomaly ratios in the original dataset (if anomaly ratios are larger than
10%, we set the maximum anomaly ratio as 10%). For KDD-Rev, we set the maximum anomaly ratio
to 2.5% because the performance of GOAD (without SRR) drops signiﬁcantly even with a small ratio
of anomalies in the training data. Fig. 4 shows that integrating SRR signiﬁcantly improves GOAD
(the state-of-the-art methods for tabular OCC). The improvements are more signiﬁcant especially
at higher anomaly ratios. This underlines how SRR can achieve signiﬁcant improvements for both
small (Thyroid & Arrhythmia) and large-scale datasets (KDD & KDD-Rev).
4.2 E XPERIMENTS ON IMAGE DATA
Datasets. We evaluate SRR on visual AD benchmarks, including CIFAR-10 [ 29], f-MNIST [ 56],
Dog-vs-Cat [ 19], and MVTec [ 8]. For CIFAR-10, f-MNIST, and Dog-vs-Cat datasets, samples from
1Note that the experimental settings with contaminated training data in GOAD [ 7] and DAGMM [ 60] are
slightly different from ours. Our contamination ratio is deﬁned as the anomaly ratio over the entire training data,
while their contamination ratio is the anomaly ratio over all the anomalies in the dataset.
6
Under review as a conference paper at ICLR 2022
one class are set to be normal and the rest from other classes are set to be an anomaly. Similar to
the experiments on tabular data in Section. 4.1, we swap a certain amount of the normal training
data with anomalies given the target anomaly ratio. For MVTec, since there are no anomalous data
available for training, we borrow 10% of the anomalies from the test set and swap them with normal
samples in the training set. Note that 10% of samples borrowed from the test set are excluded from
evaluation. For all datasets, we experiment with varying anomaly ratios from 0% to 10%.
We use area under ROC curve (AUC) and average precision (AP) metrics to quantify the performance
for visual AD (with scale 0-100). When computing AP, we set the minority class of the test set as
label 1 and majority as label 0 (e.g., normal samples are set as label 1 for CIFAR-10 experiments as
there are more anomaly samples that are from 9 classes in the test set). We run all experiments with 5
random seeds and report the average performance for each dataset across all classes. Per-class AUC
and AP are reported in Appendix A.4.
Models. For semantic AD benchmarks, CIFAR-10, f-MNIST, and Dog-vs-Cat, we compare the
SRR with two-stage OCCs [51] using various representation learning methods, such as distribution-
augmented contrastive learning [51], rotation prediction [23, 28] and its improved version [51], and
denoising autoencoder. For MVTec benchmarks, we use CutPaste [ 33] as a baseline and compare to
its version with SRR integration. For both experiments, we use the ResNet-18 architecture, trained
from random initialization, using the hyperparameters from [ 51] and [ 33]. The same model and
hyperparameter conﬁgurations are used for SRR with K= 5classiﬁers in the ensemble. We set
as
twice the anomaly ratio of training data. For 0% anomaly ratio, we set
as 0.5. Finally, a Gaussian
Density Estimator (GDE) on learned representations is used as OCC.
0 2 4 6 8 10
Anomaly ratio (%) in training data75808590AUC
SRR+Contrastive
Contrastive
Rotation
RotNet
DAE
0 2 4 6 8 10
Anomaly ratio (%) in training data3040506070Average Precision
SRR+Contrastive
Contrastive
Rotation
RotNet
DAE
Figure 5: Unsupervised AD performance with various OCCs on CIFAR-10. For SRR we adapt
distribution-augmented contrastive representation learning [ 51]. (Left) AUC, (Right) Average Preci-
sion (AP).
Results. Fig. 5 shows a signiﬁcant performance drop with increased anomaly ratio, regardless of
representation learning. For example, the AUC of distribution-augmented contrastive representa-
tion [ 51] drops from 92.1 to 83.6 when anomaly ratio becomes 10%. Similarly, the improved rotation
prediction representation [ 28] drops from 90.0 to 84.1. On the other hand, SRR effectively handles
the contamination in the training data and achieves 89.9 AUC with 10% anomaly ratio, reducing the
performance drop by 74.1%. An ‘oracle’ upper bound would be the removal of all anomalies from
the training data (which is the same as the performance at 0% anomaly ratio for the same size of the
data). As Fig. 5 shows, the performance of SRR is similar to this oracle upper bound performance
(less than 2.5 AUC difference) even with high anomaly ratios (10%). The results are also similar in
other metrics, such as AP in Fig. 5 or Recall at Precision of 70 and 90 in Fig. 9 in the Appendix.
We repeat experiments on 3 additional visual AD datasets and report results in Fig. 6. We observe
consistent and signiﬁcant improvements over the baseline across different datasets and different
one-class classiﬁcation methods. Note that the improvement is more signiﬁcant at higher anomaly
ratios. For instance, on MVTec dataset, SRR improves AUC by 4.9 and AP by 7.1 compared to the
state-of-the-art CutPaste OCC with an anomaly ratio of 10%. In the Appendix (Sec. A.4), we also
illustrate per-class performance of SRR compared with the state-of-the-art.
7
Under review as a conference paper at ICLR 2022
0 2 4 6 8 10
Anomaly ratio (%) in training data868890929496AUC
SRR+CutPaste
CutPaste
0 2 4 6 8 10
Anomaly ratio (%) in training data77.580.082.585.087.590.092.595.097.5Average Precision
SRR+CutPaste
CutPaste
(a) MVTec
0 2 4 6 8 10
Anomaly ratio (%) in training data7580859095AUC
SRR+Contrastive
Contrastive
0 2 4 6 8 10
Anomaly ratio (%) in training data304050607080Average Precision
 SRR+Contrastive
Contrastive (b) f-MNIST
0 2 4 6 8 10
Anomaly ratio (%) in training data70.072.575.077.580.082.585.087.590.0AUC
SRR+Contrastive
Contrastive
0 2 4 6 8 10
Anomaly ratio (%) in training data72.575.077.580.082.585.087.590.0Average Precision
SRR+Contrastive
Contrastive (c) Dog-vs-Cat
Figure 6: Unsupervised AD performance on (a) MVTec (b) f-MNIST, and (c) Dog-vs-Cat datasets
with varying anomaly ratios. We use state-of-the-art one-class classiﬁcation models for baselines,
such as distribution-augmented contrastive representations [ 51] for f-MNIST and Dog-vs-Cat, or
CutPaste [33] for MVTec, and build SRR on top of them.
4.3 P ERFORMANCE ANALYSES
In this section, we conduct sensitivity analyses on two hyperparameters of SRR, namely, the number
Kfor ensemble OCCs and the percentile threshold
that determines normal and anomalies in the
data reﬁnement module. In addition, we show the importance of updating the representation with
reﬁned data. Ablation studies are conducted on two visual AD benchmarks, CIFAR-10 and MVTec.
More experimental results and discussions can be found in the Appendix.
4.3.1 S ENSITIVITY TO HYPERPARAMETERS
SRR is designed for unsupervised AD and it is important to ensure robust performance against
changes in the hyperparameters as model selection without labeled data would be very challenging.
Fig. 7 presents the sensitivity analyses of SRR with respect to various hyperparameters. In Fig. 7a,
we observe the performance improvement as we increase the number of classiﬁers for ensemble. This
is particularly effective on CIFAR (top in Fig. 7a), where the number of samples in the training data
is large enough that even with large K, the number of samples to train each OCC ( N=K ) would be
sufﬁcient. In Fig. 7b, we observe that SRR performs robustly when
is set to be larger than the actual
anomaly ratio (10%). When
is less than 10%, however, we see a signiﬁcant drop in performance,
all the way to the baseline (
= 0). Our results show that SRR improves upon baseline regardless of
the threshold. In addition, it suggests that
could be set to be anywhere from the true anomaly ratio
and and 2x the anomaly ratio to maximize its effectiveness. When the true anomaly ratio is unknown,
as discussed in Appendix A.5, Otsu’s method could be used.
4.3.2 I TERATIVELY UPDATING REPRESENTATIONS WITH REFINED DATA
It is possible to decouple representation learning and data reﬁnement of SRR, which would result in a
three-stage framework, where we learn representations until convergence without data reﬁnement,
followed by the data reﬁnement and learning OCC. Fig. 7c shows that SRR using a pre-trained
then ﬁxed representation (i.e. when data reﬁnement is used for OCC only) already improves the
performance upon the baseline with no data reﬁnement at any stage of learning representation or
classiﬁer. Improvements on CIFAR-10 are 4.9 in AUC and 8.0 in AP; and on MVTec are 1.3 in AD
and in 1.3 in AP. This underlines the effectiveness of our data reﬁnement module. More results on
applying SRR on raw tabular features or learned (and ﬁxed) visual representations are provided in
Appendix A.2. Moreover, we ﬁnd that learning representation with reﬁned data plays a crucial role,
resulting in another improvement in AUC by 1.4, AP by 4.5 on CIFAR-10, and AUC by 3.2, AP by
4.8 on MVTec, upon a SRR using a ﬁxed representation.
8
Under review as a conference paper at ICLR 2022
3 6 9 12 15 18
Ensemble count (K)58606264666870Average Precision
8687888990
AUC
3 6 9 12 15 18
Ensemble count (K)848586878889909192Average Precision
89909192939495
AUC
(a) Ensemble count K
0 5 10 15 20 25 30
Threshold ()
52.555.057.560.062.565.067.570.0Average Precision
 84868890
AUC
0 5 10 15 20 25 30
Threshold ()
7880828486889092Average Precision
8688909294
AUC
 (b) Percentile threshold
AUC Average Precison50556065707580859095AUC / Average PrecisionNo data refinement
Data refinement for only OCC
Data refinement for representation & OCC
AUC Average Precison75.077.580.082.585.087.590.092.595.0AUC / Average PrecisionNo data refinement
Data refinement for only OCC
Data refinement for representation & OCC (c) Representation update
Figure 7: Ablation studies on (top) CIFAR-10 and (bottom) MVTec under 10% anomaly ratio setting
with respect to (a) ensemble count K, (b) percentile threshold
, and (c) data reﬁnement with or
without representation update.
4.4 Q UANTIFYING THE REFINEMENT EFFICACY
We evaluate how many normal and anomalies are excluded by the proposed data reﬁnement block.
As in Fig. 8(a, b), with data reﬁnement, we can exclude more than 80% of anomalies in the training
set without removing too many normal samples. For instance, among 4% anomalies in CIFAR-10
data, SRR is able to exclude 80% anomalies while removing less than 20% normal samples. Such a
high recall of anomalies of SRR is not only useful for unsupervised AD, but also could be useful
for improving the annotation efﬁciency when a budget for active learning is available. Fig. 8(c,
d) demonstrates the removed normal and abnormal samples by the data reﬁnement module over
training epochs. It shows that better representation learning (as training epochs increase) consistently
improves the efﬁcacy of the data reﬁnement.
2% 4% 6% 8% 10%
Anomaly ratio (%) in training data020406080Percentage of excluded samplesAnomalous samples
Normal samples
(a) CIFAR-10
2% 4% 6% 8% 10%
Anomaly ratio (%) in training data020406080Percentage of excluded samplesAnomalous samples
Normal samples (b) MVTec
0 250 500 750 1000 1250 1500 1750 2000
Epochs5060708090Removed anomalous samples (%)
28293031323334
Removed normal samples (%)
 (c) CIFAR-10
0 50 100 150 200 250
Epochs60657075808590Removed anomalous samples (%)
2224262830323436
Removed normal samples (%)
 (d) MVTec
Figure 8: Percentage of excluded anomalous and normal samples by data reﬁnement (a, b) with
different anomaly ratios in training data and (c, d) over training epochs for 10% anomaly ratio.
5 C ONCLUSION
AD has wide range of practical use cases. A challenging and costly aspect of building an AD system
is that anomalies are rare and not easy to detect by humans, making them difﬁcult to label. To
this end, we propose a novel AD framework to enable high performance AD without any labels
called SRR. SRR can be ﬂexibly integrated with any OCC, and applied on raw data or on a trainable
representations. SRR employs an ensemble of multiple OCCs to propose candidate anomaly samples
that are reﬁned from training, which allows more robust ﬁtting of the anomaly decision boundaries as
well as better learning of data representations. We demonstrate the state-of-the-art AD performance
of SRR on multiple tabular and image data.
9
Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
SRR has the potential to make signiﬁcant positive impact in real-world AD applications where detect-
ing anomalies is crucial, such as for ﬁnancial crime elimination, cybersecurity advances, or improving
manufacturing quality. We note a potential risk associated with using SRR: when representations are
not sufﬁciently good, there will be a negative cycle of reﬁnement and representation updates/OCC.
While we rely on the existence of good representations, in some applications they may be difﬁcult to
obtain (e.g., cybersecurity). This paper focuses on the unsupervised setting and demonstrates strong
AD performance, opening new horizons for human-in-the-loop AD systems that are low cost and
robust. We leave these explorations to future work.
REPRODUCIBILITY STATEMENT
The source code of SRR will be published upon acceptance. For reproducibility, we only use public
tabular and image datasets to evaluate the performances of SRR. Complete descriptions of those
datasets can be found in Datasets subsections in Sec. 4.1 and 4.2. Detailed experimental settings and
hyperparameters can be found in Models subsections in Sec. 4.1 and 4.2. The implementation details
(including hyperparameters that we used) on GOAD [7] are described in Sec. A.3.
REFERENCES
[1]S. Akcay, A. Atapour-Abarghouei, and T. P. Breckon. Ganomaly: Semi-supervised anomaly
detection via adversarial training. In ACCV , 2018. 3
[2]R. Alejo, R. M. Valdovinos, V . García, and J. H. Pacheco-Sanchez. A hybrid method to face class
overlap and class imbalance on neural networks and multi-class scenarios. PRL, 34(4):380–388,
2013. 2
[3] A. Asuncion and D. Newman. UCI machine learning repository, 2007. 6
[4]S. Barua, M. M. Islam, X. Yao, and K. Murase. Mwmote–majority weighted minority over-
sampling technique for imbalanced data set learning. IEEE Trans on knowledge and data
engineering , 26(2):405–425, 2012. 1, 2
[5]L. Beggel, M. Pfeiffer, and B. Bischl. Robust anomaly detection in images using adversarial
autoencoders. Machine Learning and Knowledge Discovery in Databases , 2019. 3, 4, 18, 21
[6]A. Berg, J. Ahlberg, and M. Felsberg. Unsupervised learning of anomaly detection from
contaminated image data using simultaneous encoder training. arXiv preprint arXiv:1905.11034 ,
2019. 3
[7]L. Bergman and Y . Hoshen. Classiﬁcation-based anomaly detection for general data. In
International Conference on Learning Representations , 2019. 1, 2, 3, 6, 10, 14, 15, 19
[8]P. Bergmann, M. Fauser, D. Sattlegger, and C. Steger. MVTec AD–a comprehensive real-world
dataset for unsupervised anomaly detection. In CVPR , 2019. 2, 6
[9]D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch:
A holistic approach to semi-supervised learning. Advances in Neural Information Processing
Systems , 32, 2019. 3
[10] G. Blanchard, G. Lee, and C. Scott. Semi-supervised novelty detection. JMLR , 11:2973–3009,
2010. 1
[11] A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In CLT, 1998.
3
[12] P. Branco, L. Torgo, and R. Ribeiro. A survey of predictive modelling under imbalanced
distributions. arXiv preprint arXiv:1505.01658 , 2015. 2
[13] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. Lof: identifying density-based local
outliers. In International conference on management of data , 2000. 3, 20
[14] C. E. Brodley, M. A. Friedl, et al. Identifying and eliminating mislabeled training instances. In
Proc National Conference on Artiﬁcial Intelligence , 1996. 3
10
Under review as a conference paper at ICLR 2022
[15] E. J. Candès, X. Li, Y . Ma, and J. Wright. Robust principal component analysis? Journal of the
ACM (JACM) , 58(3):1–37, 2011. 20
[16] N. V . Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. Smote: synthetic minority
over-sampling technique. Journal of artiﬁcial intelligence research , 16:321–357, 2002. 1, 2
[17] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning
of visual representations. In ICML , 2020. 3
[18] F. Denis. PAC learning from positive statistical queries. In ICALT , 1998. 1
[19] J. Elson, J. R. Douceur, J. Howell, and J. Saul. Asirra: a captcha that exploits interest-aligned
manual image categorization. In Proc Computer and Communications Security , 2007. 2, 6
[20] A. Estabrooks, T. Jo, and N. Japkowicz. A multiple resampling method for learning from
imbalanced data sets. Computational intelligence , 20(1):18–36, 2004. 1, 2
[21] J.-C. Feng, F.-T. Hong, and W.-S. Zheng. Mist: Multiple instance self-training framework for
video anomaly detection. In CVPR , 2021. 3
[22] M. Galar, A. Fernandez, E. Barrenechea, H. Bustince, and F. Herrera. A review on ensembles
for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE
Trans on Systems, Man, and Cybernetics , 42(4):463–484, 2011. 2
[23] I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. In
Proceedings of the 32nd International Conference on Neural Information Processing Systems ,
pages 9781–9791, 2018. 1, 2, 3, 4, 7, 13, 14
[24] N. Görnitz, M. Kloft, K. Rieck, and U. Brefeld. Toward supervised anomaly detection. Journal
of Artiﬁcial Intelligence Research , 46:235–262, 2013. 1
[25] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. W. Tsang, and M. Sugiyama. Co-teaching:
robust training of deep neural networks with extremely noisy labels. In NIPS , 2018. 3
[26] D. Hendrycks, M. Mazeika, and T. Dietterich. Deep anomaly detection with outlier exposure.
InInternational Conference on Learning Representations , 2018. 1, 2, 3, 14
[27] J. P. Hwang, S. Park, and E. Kim. A new weighted approach to imbalanced data classiﬁca-
tion problem via support vector machine with quadratic cost function. Expert Systems with
Applications , 38(7):8580–8585, 2011. 1, 2
[28] N. Komodakis and S. Gidaris. Unsupervised representation learning by predicting image
rotations. In International Conference on Learning Representations (ICLR) , 2018. 3, 7, 13, 14
[29] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009. 2, 6
[30] C.-H. Lai, D. Zou, and G. Lerman. Robust subspace recovery layer for unsupervised anomaly
detection. In International Conference on Learning Representations , 2019. 3
[31] L. J. Latecki, A. Lazarevic, and D. Pokrajac. Outlier detection with kernel density functions. In
International Workshop on Machine Learning and Data Mining in Pattern Recognition , 2007. 3
[32] S. S. Lee. Noisy replication in skewed binary classiﬁcation. Computational statistics & data
analysis , 34(2):165–191, 2000. 1, 2
[33] C.-L. Li, K. Sohn, J. Yoon, and T. Pﬁster. Cutpaste: Self-supervised learning for anomaly
detection and localization. In CVPR , 2021. 1, 2, 3, 4, 5, 7, 8, 17, 19, 20
[34] J. Li, R. Socher, and S. C. Hoi. Dividemix: Learning with noisy labels as semi-supervised
learning. In ICLR , 2019. 3
[35] A. Liu, J. Ghosh, and C. E. Martin. Generative oversampling for mining imbalanced datasets.
InDMIN , pages 66–72, 2007. 1, 2
[36] F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation forest. In ICDM , 2008. 2, 3
[37] G. J. McLachlan. Iterative reclassiﬁcation procedure for constructing an asymptotically optimal
rule of allocation in discriminant analysis. Journal of the American Statistical Association ,
70(350):365–369, 1975. 3
[38] X. Meng, S. Wang, Z. Liang, D. Yao, J. Zhou, and Y . Zhang. Semi-supervised anomaly detection
in dynamic communication networks. Information Sciences , 571:527–542, 2021. 3
11
Under review as a conference paper at ICLR 2022
[39] E. Min, J. Long, Q. Liu, J. Cui, Z. Cai, and J. Ma. Su-ids: A semi-supervised and unsupervised
framework for network intrusion detection. In International Conference on Cloud Computing
and Security , 2018. 3
[40] M. Mohseni, J. Yap, W. Yolland, A. Koochek, and S. Atkins. Can self-training identify
suspicious ugly duckling lesions? In CVPR , 2021. 3
[41] J. M˜unoz-Marí, F. Bovolo, L. Gómez-Chova, L. Bruzzone, and G. Camp-Valls. Semisupervised
one-class support vector machines for classiﬁcation of remote sensing data. IEEE transactions
on geoscience and remote sensing , 48(8):3188–3197, 2010. 3
[42] G. Pang, C. Yan, C. Shen, A. v. d. Hengel, and X. Bai. Self-trained deep ordinal regression for
end-to-end video anomaly detection. In CVPR , 2020. 3, 4, 18, 21
[43] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V . Dillon, and B. Lakshmi-
narayanan. Likelihood ratios for out-of-distribution detection. arXiv preprint arXiv:1906.02845 ,
2019. 21
[44] D. A. Reynolds. Gaussian mixture models. Encyclopedia of biometrics , 741:659–663, 2009. 3
[45] L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and
M. Kloft. Deep one-class classiﬁcation. In ICML , 2018. 1, 2, 3, 4
[46] L. Ruff, R. A. Vandermeulen, N. Görnitz, A. Binder, E. Müller, K.-R. Müller, and M. Kloft.
Deep semi-supervised anomaly detection. In ICLR , 2020. 1, 3
[47] B. Schölkopf, R. C. Williamson, A. J. Smola, J. Shawe-Taylor, J. C. Platt, et al. Support vector
method for novelty detection. In NIPS , 1999. 1, 2
[48] H. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transac-
tions on Information Theory , 11(3):363–371, 1965. 3
[49] M. Sezgin and B. Sankur. Survey over image thresholding techniques and quantitative perfor-
mance evaluation. Journal of Electronic imaging , 13(1):146–165, 2004. 5, 18
[50] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin,
and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and conﬁdence.
NeurIPS , 2020. 3
[51] K. Sohn, C.-L. Li, J. Yoon, M. Jin, and T. Pﬁster. Learning and evaluating representations for
deep one-class classiﬁcation. In ICLR , 2021. 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 16, 18, 19, 20, 21
[52] H. Song, Z. Jiang, A. Men, and B. Yang. A hybrid semi-supervised anomaly detection model
for high-dimensional data. Computational intelligence and neuroscience , 2017. 3
[53] D. M. Tax and R. P. Duin. Support vector data description. Machine learning , 54(1):45–66,
2004. 1, 3
[54] S. Wold, K. Esbensen, and P. Geladi. Principal component analysis. Chemometrics and
intelligent laboratory systems , 2(1-3):37–52, 1987. 20
[55] Y . Xia, X. Cao, F. Wen, G. Hua, and J. Sun. Learning discriminative reconstructions for
unsupervised outlier removal. In ICCV , 2015. 3, 18, 21
[56] H. Xiao, K. Rasul, and R. V ollgraf. Fashion-mnist: a novel image dataset for benchmarking
machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017. 6
[57] Q. Xie, M.-T. Luong, E. Hovy, and Q. V . Le. Self-training with noisy student improves imagenet
classiﬁcation. In CVPR , 2020. 3
[58] B. Zhang and W. Zuo. Learning from positive and unlabeled examples: A survey. In 2008
International Symposiums on Information Processing , 2008. 1
[59] C. Zhou and R. C. Paffenroth. Anomaly detection with robust deep autoencoders. In KDD ,
2017. 3, 4, 20
[60] B. Zong, Q. Song, M. R. Min, W. Cheng, C. Lumezanu, D. Cho, and H. Chen. Deep autoen-
coding gaussian mixture model for unsupervised anomaly detection. In ICLR , 2018. 1, 2, 3,
6
12
Under review as a conference paper at ICLR 2022
A A PPENDIX
A.1 A DDITIONAL RESULTS WITH DIFFERENT METRICS
There are various performance metrics of OCCs. In the main manuscript, we mainly use AUC,
Average Precision (AP) and F1-score as the evaluation metrics. In this subsection, we report the
performances of the proposed model (SRR) and baselines in terms on Recall at Precision 70 and 90
as the additional metrics on CIFAR-10 dataset.
0 2 4 6 8 10
Anomaly ratio (%) in training data0102030405060Recall at Precision 70
SRR+Contrastive
Contrastive
Rotation
RotNet
DAE
0 2 4 6 8 10
Anomaly ratio (%) in training data010203040Recall at Precision 90
SRR+Contrastive
Contrastive
Rotation
RotNet
DAE
Figure 9: Performance of various OCCs on CIFAR-10 dataset. SRR is applied on top of Contrastive
[51]. (Left) Recall at Precision 70, (Right) Recall at Precision 90.
As in Fig. 9, we observe a similar trends with these two additional metrics as well. For example,
the performances of SRR are robust across various anomaly ratios. On the other hand, all the other
OCCs show consistent and signiﬁcant performance degradation as the anomaly ratio increases. SRR
performs 15.8 and 10.9 better than the state-of-the-art OCC [ 51] in terms of recall at precision 70 and
90, respectively.
A.2 SRR ON RAW TABULAR FEATURES /LEARNED IMAGE REPRESENTATIONS
Average Precison F1-score65.067.570.072.575.077.580.082.585.0Average Precision / F1-scoreKDD-Rev
Baseline
SRR on raw features
(a) KDD-Rev
Average Precison F1-score203040506070Average Precision / F1-scoreThyroid
Baseline
SRR on raw features (b) Thyroid
Average Precison F1-score3032343638404244Average Precision / F1-scoreArrhythmia
Baseline
SRR on raw features (c) Arrhythmia
AUC Average Precision20304050607080AUC / Average PrecisionCIFAR-10 / RotNet representation
Baseline
SRR on learned represenations
(d) RotNet [23]
AUC Average Precision505560657075808590AUC / Average PrecisionCIFAR-10 / Rotation representation
Baseline
SRR on learned represenations (e) Rotation [28]
AUC Average Precision5060708090100110AUC / Average PrecisionCIFAR-10 / Contrastive representation
Baseline
SRR on learned represenations (f) Contrastive [51]
Figure 10: Performance of SRR on (top) raw tabular features and (lower) learned image representa-
tions. SRR consistently outperforms baseline and in some cases (e.g., Thyroid, and Contrastive [ 51]),
the performance improvements are signiﬁcant.
13
Under review as a conference paper at ICLR 2022
SRR is also applicable on raw tabular features or learned image representation without representation
update using data reﬁnement. In this section, we demonstrate the performance improvements by SRR
without representation update to verify the effectiveness of data reﬁnement block of SRR for shallow
OCCs.
Fig. 10 (upper) demonstrates consistent and signiﬁcant performance improvements when we apply
SRR on top of raw tabular features. Speciﬁcally, the Average Precision (AP) improvements are 10.2,
29.0, and 4.1 with KDD-Rev, Thyroid, and Arrhythmia tabular datasets, respectively.
We also apply SRR on top of various learned image representations. As can be seen in Fig. 10
(lower), the performance improvements of SRR are consistent across various different learned image
representations (without representation update). For instance, the AP improvements are 9.2, 1.0,
and 12.1 with learned image representations using RotNet [ 23], Rotation [ 28], and Contrastive [ 51],
respectively.
A.3 I MPLEMENTATION DETAILS ON GOAD [7] FOR TABULAR DATA EXPERIMENTS
A classiﬁcation-based AD method, GOAD [ 7], has demonstrated strong AD performance on tabular
datasets. Unlike previous works [ 23,26] that formulate a parametric classiﬁer for multiple transfor-
mation classiﬁcation, GOAD employs distance-based classiﬁcation of multiple transformations. For
the set of transformations Tm:X!D ,m= 1;:::;M , the loss function of GOAD is written as in
equation 3 with the probability deﬁned in equation 4.
L=Em;x
logP(mjTm(x))
; (3)
P( ^mjTm(x)) =exp(kf(Tm(x))c^mk2)P
nexp(kf(Tm(x))cnk2); (4)
where the centers cm’s are updated by the average feature over the training set. While it is shown to
perform well [ 7], we ﬁnd that the distance-based formulation is not necessary, and we achieve the
similar performance, if not worse, to [ 7] using a parametric classiﬁer when computing the probability:
P( ^mjTm(x)) =exp
w>
^mf(Tm(x)) +b^m
P
nexp (w>nf(Tm(x)) +bn)(5)
The formulation in equation 5 is easier to optimize than its original form in equation 4 as it can be
fully optimized with backpropagation without alternating updates of feature extractor fand centers
cm. Once we learn a representation by optimizing the loss in equation 3 using equation 5, we follow a
two-stage one-class classiﬁcation framework of [ 51] to construct a set of Gaussian density estimation
OCCs for each transformation. Finally, we aggregate a maximum normality scores from a set of
classiﬁers as the normality score.
In Table 1, we summarize the implementation details, such as network architecture or hyperparameters,
and AD performance under clean training data setting that reproduces the results in [7].
14
Under review as a conference paper at ICLR 2022
Datasets KDD KDD-Rev Thyroid Arrhythmia
F-score [7] 98.40.2 98.90.3 74.51.1 52.02.3
F-score (ours) 98.00.2 95.00.2 75.12.4 54.83.2
f(feature)
Linear(8), LeakyReLU(0.2)
5
Optimizer Momentum SGD (momentum = 0:9)
Learning rate 0.001
L2 weight regularization 0.00003
Batch size 64M
Random projection dimension 32
M(number of transformations) 32 256
Train steps 210216
Table 1: The AD performance under clean only data setting of GOAD in [ 7] and our implementation.
Our implementation demonstrates comparable, if not worse, performance to those reported in [ 7].
Our implementation also shares most hyperparameters across datasets except the M, the number of
transformations, and the train steps, which are closely related to the size of training data.
A.4 P ER-CLASS AUC AND AP
In the main manuscript, we report the mean and standard deviation of AUC and AP across all classes
in each dataset. In this section, we report the mean and standard deviation of AUC and AP for each
class in each dataset (including CIFAR-10 (Table 2), MVTec (Table 3), fMNIST (Table 4), and
Dog-vs-Cat (Table 5) datasets).
A.4.1 P ER-CLASS AUC AND AP ONCIFAR-10 DATASET
15
Under review as a conference paper at ICLR 2022
anomaly ratios 0% 10%
Classes / Methods Baseline [51] SRR + [51] Baseline [51] SRR + [51]
AUC
Airplane 90.50.6 90.40.4 85.30.8 87.60.7
Automobile 98.80.1 98.70.1 95.20.5 97.90.3
Bird 87.50.6 87.10.8 75.61.1 84.71.2
Cat 81.50.5 80.30.7 68.91.0 78.51.5
Deer 90.30.4 90.20.5 79.90.8 87.40.5
Dog 90.80.7 90.80.6 79.22.5 89.60.6
Frog 92.00.6 91.40.9 78.52.1 87.51.3
Horse 97.80.1 97.90.1 93.30.5 96.00.6
Ship 96.50.2 96.40.1 91.50.9 95.20.5
Truck 95.10.3 95.80.1 88.90.4 94.70.4
Average Precision
Airplane 68.31.7 67.91.1 61.31.9 61.81.1
Automobile 93.90.2 94.10.1 85.93.4 91.50.7
Bird 62.22.1 61.72.4 42.43.0 57.52.3
Cat 44.02.4 42.91.9 30.33.8 44.90.9
Deer 59.50.8 58.81.2 42.93.5 53.61.5
Dog 59.16.5 59.84.7 39.79.1 64.52.0
Frog 72.72.0 70.46.9 38.511.0 73.63.6
Horse 89.00.7 89.20.6 80.61.9 86.21.1
Ship 83.90.8 83.70.6 74.92.4 79.80.7
Truck 79.01.6 81.30.4 67.01.9 75.81.2
Table 2: The AD performance per class on CIFAR-10 dataset in terms of AUC and AP. Class
represents the label of normal samples.
A.4.2 P ER-CLASS AUC AND AP ONMVT EC DATASET
16
Under review as a conference paper at ICLR 2022
anomaly ratios 0% 10%
Classes / Methods Baseline [33] SRR + [33] Baseline [33] SRR + [33]
AUC
Bottle 98.01.0 98.80.7 94.30.8 98.70.7
Cable 77.70.7 77.92.6 72.93.3 80.40.9
Capsule 96.11.1 97.21.3 92.41.7 96.80.7
Hazelnut 97.20.7 97.50.4 83.53.6 96.70.6
Metal Nut 98.51.0 98.60.4 91.22.8 98.70.3
Pill 92.13.1 91.12.5 85.11.5 92.10.5
Screw 87.41.6 86.82.5 79.91.0 86.41.4
Toothbrush 98.51.5 98.31.1 98.61.3 99.31.2
Transistor 96.42.0 96.71.6 93.92.8 95.70.7
Zipper 99.40.4 98.00.6 91.12.9 99.20.7
Carpet 90.03.9 90.84.6 80.46.1 91.22.7
Grid 99.20.5 99.70.3 92.97.6 99.70.2
Leather 99.80.3 99.70.4 98.01.1 100.00.0
Tile 93.11.5 93.32.7 86.43.4 92.41.8
Wood 98.80.9 98.51.4 92.12.2 99.00.5
Average Precision
Bottle 98.80.7 99.30.5 94.31.6 98.00.6
Cable 81.80.9 82.31.7 70.84.3 74.13.0
Capsule 99.00.3 99.30.3 79.55.3 89.82.7
Hazelnut 97.10.6 97.00.4 88.83.6 93.61.9
Metal Nut 99.60.3 99.60.1 76.18.4 92.13.1
Pill 98.20.7 97.90.6 56.62.6 67.32.2
Screw 93.81.1 93.61.3 60.12.8 63.71.8
Toothbrush 99.20.9 99.20.6 97.42.2 98.62.0
Transistor 93.03.0 93.72.4 97.81.6 97.10.5
Zipper 99.80.2 99.20.3 86.12.8 98.01.5
Carpet 94.43.5 95.72.8 64.28.2 71.64.2
Grid 99.50.3 99.80.2 90.96.1 99.40.4
Leather 99.90.2 99.90.2 97.21.3 100.00.1
Tile 96.10.9 96.41.4 82.44.5 86.42.8
Wood 99.40.5 99.40.6 88.13.2 97.21.7
Table 3: The AD performance per class on MVTec dataset in terms of AUC and AP. Class represents
the label of normal samples.
A.4.3 P ER-CLASS AUC AND AP ON F MNIST DATASET
A.4.4 P ER-CLASS AUC AND AP ONDOG-VS-CAT DATASET
17
Under review as a conference paper at ICLR 2022
anomaly ratios 0% 10%
Classes / Methods Baseline [51] SRR + [51] Baseline [51] SRR + [51]
AUC
T-shirt/top 93.31.2 93.21.0 79.21.3 84.71.0
Trouser 99.20.3 99.20.2 92.90.8 94.51.9
Pullover 93.80.2 93.80.2 78.42.7 90.11.1
Dress 94.40.5 94.00.5 78.43.5 85.81.6
Coat 95.10.1 94.90.2 80.43.1 88.71.7
Sandal 95.40.3 95.50.4 67.64.3 81.71.8
Shirt 87.90.2 87.90.2 73.91.2 81.02.5
Sneaker 99.10.0 99.20.1 80.24.9 94.73.6
Bag 97.80.2 97.60.3 58.23.3 68.62.3
Ankle boot 98.60.2 98.50.2 76.62.6 87.13.6
Average Precision
T-shirt/top 68.712.1 67.98.9 43.04.7 56.83.7
Trouser 96.52.9 98.20.3 78.25.2 87.13.7
Pullover 62.71.2 63.61.3 27.26.9 57.11.5
Dress 72.33.7 71.62.7 36.18.8 57.32.6
Coat 66.21.1 65.61.0 24.33.9 50.79.5
Sandal 70.92.8 70.82.4 17.85.9 40.15.4
Shirt 49.60.9 49.70.9 27.25.8 37.64.7
Sneaker 91.90.4 92.90.9 21.03.8 81.611.4
Bag 92.20.4 91.81.0 13.22.0 20.13.7
Ankle boot 87.91.4 86.51.5 19.63.1 48.016.5
Table 4: The AD performance per class on fMNIST dataset in terms of AUC and AP. Class represents
the label of normal samples.
anomaly ratios 0% 10%
Classes / Methods Baseline [51] SRR + [51] Baseline [51] SRR + [51]
AUC
Dog 89.70.6 89.20.4 74.72.1 79.11.3
Cat 88.20.4 88.00.7 70.12.4 70.71.8
Average Precision
Dog 91.60.7 91.00.6 80.02.2 84.01.2
Cat 90.20.3 89.81.0 71.63.8 71.54.8
Table 5: The AD performance per class on Dog-vs-Cat dataset in terms of AUC and AP. Class
represents the label of normal samples.
A.5 SRR WITH OTSU’S METHOD
In the main manuscript we show that the performances of the proposed unsupervised anomaly
detection framework (SRR) are signiﬁcantly better than the state-of-the-art OCCs if the given data
is fully unlabeled. Although SRR does not need any labeled data, it needs a true anomaly ratio for
hyper-parameter optimization. In this subsection, we further extend SRR to the setting where there
is no access to the true anomaly ratio as well. We utilize the Otsu’s method [ 49] to identify the
threshold between normal and anomalous samples and use that as the hyper-parameter (
) instead of
twice of the true anomaly ratio. Note that other previous works [ 5,42,55] also utilized intra-variance
minimization, in a similar vein to Otsu’s method, for determining the threshold for data reﬁnement.
18
Under review as a conference paper at ICLR 2022
The key idea of the Otsu’s method is to ﬁnd the threshold that minimizes the intra-class variance. This
is deﬁned as the weighted sum of variances of the two classes. Let us denote the normality scores
asfsigN
i=1and threshold as . Then, we try to ﬁnd the threshold ( ) that minimizes the weighted
sum of the variance (w0()0() +w1()1())wherew0() =PN
i=1I(si< )=Nand
w1() =PN
i=1I(si)=N.0()and1()are the variances of each class. The optimal threshold
() is determined as
= min
w0()0() +w1()1():
We use the twice of as the hyperparameter (
) in SRR.
We evaluate the performances of Otsu’s method (on top of SRR) in comparison to the state-of-the-art
OCC [33] and original SRR with the knowledge of the true anomaly ratio using MVTec dataset.
0 2 4 6 8 10
Anomaly ratio (%) in training data868890929496AUC
SRR+CutPaste (with true anomaly ratio)
SRR+CutPaste (with Otsu method)
CutPaste
0 2 4 6 8 10
Anomaly ratio (%) in training data77.580.082.585.087.590.092.595.097.5Average Precision
SRR+CutPaste (with true anomaly ratio)
SRR+CutPaste (with Otsu method)
CutPaste
Figure 11: Unsupervised anomaly detection performances with Otsu’s method on top of SRR on
MVTec dataset. (Left) AUC, (Right) Average Precision (AP).
Fig. 11 demonstrates that even without true anomaly ratio, the performance of SRR can be signiﬁ-
cantly better than the state-of-the-art OCC [ 33] with Otsu’s method. Also, Fig. 11 shows that the
knowledge of true anomaly ratio is crucial information for maximizing the performance of SRR in
fully unsupervised settings.
We further extend the experimental results using Otsu’s method with other datasets such as CIFAR-10
and Thyroid datasets. Table 6 are the overall results - Otsu’s method yields only slight degradation
compared to SRR with true anomaly ratio; however, it still signiﬁcantly outperforms SOTA OCC
baselines.
Datasets / Method SOTA OCC SRR with Otsu’s method SRR
CIFAR-10 0.855 / 0.585 0.906 / 0.703 0.910 / 0.709
Thyroid 0.506 0.623 0.639
Table 6: Additional ablation studies. SOTA OCC methods are [ 51] for CIFAR-10 dataset and GOAD
[7] for Thyroid dataset. We introduce 6% noise on MVTec dataset and 1.5% noise on the Thyroid
dataset. Metrics are (AUC/AP) for CIFAR-10 dataset and F1 score for Thyroid dataset.
A.6 A DDITIONAL ABLATION STUDIES
To better understand the source of gains, we include comparisons between the ﬁnal ensemble model
on the converged self-supervised extractor (SRR without ﬁnal OCC) with the proposed SRR (using an
additional ﬁnal OCC). Also, to further support the novelty of SRR, we report additional experimental
results only with ensemble model without data reﬁnement (Ensemble Only).
As can be seen in Table 7, the proposed version of SRR (with an additional ﬁnal OCC) outperforms
signiﬁcantly, which is attributed to the fact that while ﬁtting the individual OCC models in the
ensemble, we do not exclude the possible anomaly samples for diversity of the trained submodels,
so the anomaly decision boundaries can be ﬁtted robustly. Therefore, the ensemble model can be
somewhat less accurate for classifying difﬁcult samples (near the decision boundary between normal
19
Under review as a conference paper at ICLR 2022
and abnormal) compared to the ﬁnal OCC used (that is trained with only normal samples). Also,
employment of ensemble learning without data reﬁnement (Ensemble only), yields much worse
performance than the proposed method (SRR), underlining the importance of the core data reﬁnement
idea of SRR.
Datasets / Method SOTA OCC Ensemble Only SRR without ﬁnal OCC SRR
MVTec 0.905 / 0.845 0.911 / 0.849 0.922 / 0.870 0.937 / 0.887
CIFAR-10 0.855 / 0.585 0.862 / 0.599 0.890 / 0.677 0.910 / 0.709
Table 7: Additional ablation studies. SOTA OCC methods are CutPaste [ 33] for MVTec dataset and
[51] for CIFAR-10 dataset. We introduce 6% noise on both datasets. Metrics are (AUC/AP).
A.7 A DDITIONAL BASELINES
We add extra baselines from robust anomaly detection literature: Standard PCA [ 54], Robust PCA
[15] and Local Outlier Factor (LOF) [ 13] for tabular data and Robust autoencoder (Robust AE) [ 59]
for image data. As can be seen in Table 8, the performance of PCA and LOF are highly degraded
even with a small amount of anomalies in the training data. For Robust PCA and Robust AE, the
performance degradation is less but still signiﬁcant in comparison to SRR. Overall, SRR outperforms
other benchmarks in fully unsupervised settings, underlining the importance of data reﬁnement in
improving the robustness to anomaly ratio in training data, as the core constituent of SRR framework.
Datasets / Method PCA Robust PCA LOF Robust AE SRR
CIFAR-10 - - - 0.636 / 0.174 0.910 / 0.709
Thyroid 0.299 0.377 0.338 - 0.506
KDD 0.836 0.893 0.873 - 0.942
Table 8: Additional experiments with extra baselines from robust anomaly detection literature. We
introduce 6% noise on CIFAR-10 and KDD datasets. For Thyroid dataset, we introduce 1.5% noise.
Metrics are (AUC/AP) for image data and F1 score for tabular data.
A.8 C ONVERGENCE GRAPHS
The proposed SRR framework is converged when the iterative training of the self-supervised learning
is converged. Depending on the data reﬁnement model training, corresponding self-supervised
models are also trained with differently reﬁned training data. Usually, the data reﬁnement model and
self-supervised models converge after a similar number of epochs. Fig. 12 illustrate the convergence
graphs of SRR with MVTec and CIFAR-10 datasets.
0 50 100 150 200 250
Epochs687072747678808284Selected samples (%)
0.20.40.60.81.01.21.4
Representation learning train loss
0 250 500 750 1000 1250 1500 1750 2000
Epochs79.079.580.080.581.081.582.082.583.0Selected samples (%)
 123456
Representation learning train loss
Figure 12: Convergence graphs of SRR with (left) MVTec dataset, (right) CIFAR-10 dataset.
20
Under review as a conference paper at ICLR 2022
A.9 A DDITIONAL DISCUSSIONS : RELATED WORKS FOR DATA REFINEMENT
In this subsection, we further discuss the related works [ 5,42,55] for data reﬁnement that we brieﬂy
mentioned in Section 2. We highlight the differences between SRR and [5, 42, 55] below.
First, [ 5,55] are based on auto-encoders and they directly utilize the reconstruction errors as informa-
tive signals for anomaly detection and iterative data reﬁnement. However, prior work [ 43] discovered
that the reconstruction is not a good informative signal for outlier detection. Also as shown in Fig. 5,
the performance of DAE (reconstruction errors based anomaly detection model) works much worse
than alternatives.
Second, SRR utilizes ensemble learning for data reﬁnement to improve the robustness which is
critical in unsupervised anomaly detection. On the other hand, [ 5,42,55] utilize a single model for
data reﬁnement. As can be seen in Fig. 7a, the impact of the ensemble model for data reﬁnement is
signiﬁcant.
Third, [ 5,42,55] directly utilize the (pseudo) abnormal samples for model training in addition to
(pseudo) normal samples. For instance, the model in [42] is trained via two-class ordinal regression
where two classes come from (pseudo) normal and (pseudo) abnormal samples. Directly relying
on the (imperfectly) labeled abnormal samples can be harmful for the anomaly detection due to the
overﬁtting problems and high False Positive Rates (FPR) in (pseudo) abnormal samples. For instance,
with 80% of the recall (for the abnormal sample discovery), the precisions (for the abnormal sample
discovery) are 28.2% for CIFAR-10 and 31.7% with MVTec datasets (with 6% abnormal ratio) using
SRR framework. It means that the majority of the (pseudo) abnormal samples are actually the normal
samples (71.8% for CIFAR-10 and 68.3% for MVTec datasets). This would be strong evidence that
directly utilizing the (pseudo) abnormal samples for training can be harmful.
Lastly, [ 42,55] are based on the end-to-end anomaly detection models which are empirically less
accurate than two-stage models [51].
A.10 A DDITIONAL DISCUSSIONS : COMPUTATIONAL COMPLEXITY
Note that when applying SRR on top of representation learning, the computational complexity of
the representation learning part is not changed. With SRR, the additional computations come from
training the ensemble models on top of learned representations. Note that we use shallow one-class
classiﬁers (such as GDE or one-class SVM) for submodels; thus, the additional computational
complexity is marginal. We would like to mention that all the experiments are done on a single V100
GPU and each experiment needs at most 12 hours for training (the additional training time caused
by the SRR framework is an average 13.1% of the total training time). The computations of the
ensemble parts can be further improved by the model parallelization.
21

Jinsung Yoon
Kihyuk Sohn
Chun-Liang Li
Sercan O. Arik
Chen-Yu Lee
Tomas Pfister