Using Pre-Training Can Improve Model Robustness and Uncertainty.

Using Pre-Training Can Improve Model Robustness and Uncertainty
Dan Hendrycks1Kimin Lee2Mantas Mazeika3
Abstract
He et al. (2018) have called into question the
utility of pre-training by showing that train-
ing from scratch can often yield similar per-
formance to pre-training. We show that al-
though pre-training may not improve perfor-
mance on traditional classiﬁcation metrics, it im-
proves model robustness and uncertainty esti-
mates. Through extensive experiments on la-
bel corruption, class imbalance, adversarial ex-
amples, out-of-distribution detection, and con-
ﬁdence calibration, we demonstrate large gains
from pre-training and complementary effects
with task-speciﬁc methods. We show approx-
imately a 10% absolute improvement over the
previous state-of-the-art in adversarial robust-
ness. In some cases, using pre-training without
task-speciﬁc methods also surpasses the state-
of-the-art, highlighting the need for pre-training
when evaluating future methods on robustness
and uncertainty tasks.
1. Introduction
Pre-training is a central technique in the research and appli-
cations of deep convolutional neural networks (Krizhevsky
et al., 2012). In research settings, pre-training is ubiqui-
tously applied in state-of-the-art object detection and seg-
mentation (He et al., 2017). Moreover, some researchers
aim to use pre-training to create “universal representations”
that transfer to multiple domains (Rebufﬁ et al., 2017). In
applications, the “pre-train then tune” paradigm is com-
monplace, especially when data for a target task is acutely
scarce (Zeiler & Fergus, 2014). This broadly applicable
technique enables state-of-the-art model convergence.
However, He et al. (2018) argue that model convergence
is merely faster with pre-training, so that the beneﬁt on
modern research datasets is only improved wall-clock time.
1UC Berkeley2KAIST3University of Chicago. Correspon-
dence to: Dan Hendrycks <hendrycks@berkeley.edu >.
Proceedings of the 36thInternational Conference on Machine
Learning , Long Beach, California, PMLR 97, 2019. Copyright
2019 by the author(s).Surprisingly, pre-training provides no performance bene-
ﬁt on various tasks and architectures over training from
scratch, provided the model trains for long enough. Even
models trained from scratch on only 10% of the COCO
dataset (Lin et al., 2014) attain the same performance as
pre-trained models. This casts doubt on our understanding
of pre-training and raises the important question of whether
there are any uses for pre-training beyond tuning for ex-
tremely small datasets. They conclude that, with modern
research datasets, ImageNet pre-training is not necessary.
In this work, we demonstrate that pre-training is not
needless. While He et al. (2018) are correct that models
for traditional tasks such as classiﬁcation perform well
without pre-training, pre-training substantially improves
the quality of various complementary model components.
For example, we show that while accuracy may not no-
ticeably change with pre-training, what does tremendously
improve with pre-training is the model’s adversarial
robustness. Furthermore, even though training for longer
onclean datasets allows models without pre-training to
catch up, training for longer on a corrupted dataset leads
to model deterioration. And the claim that “pre-training
does not necessarily help reduce overﬁtting” (He et al.,
2018) is valid when measuring only model accuracy,
but it becomes apparent that pre-training does reduce
overﬁtting when also measuring model calibration. We
bring clarity to the doubts raised about pre-training by
showing that pre-training can improve model robustness to
label corruption (Sukhbaatar et al., 2014), class imbalance
(Japkowicz, 2000), and adversarial attacks (Szegedy et al.,
2014); it additionally improves uncertainty estimates
for out-of-distribution detection (Hendrycks & Gimpel,
2017b) and calibration (Nguyen & O’Connor, 2015),
though not necessarily traditional accuracy metrics.
Pre-training yields improvements so signiﬁcant that on
many robustness and uncertainty tasks we surpass state-
of-the-art performance. We even ﬁnd that pre-training
alone improves over techniques devised for a speciﬁc task.
Note that experiments on these tasks typically overlook
pre-training, even though pre-training is ubiquitous else-
where. This is problematic since we ﬁnd there are tech-
niques which do not comport well with pre-training; thus
some evaluations of robustness are less representative of
real-world performance than previously thought. Thus re-
Using Pre-Training Can Improve Model Robustness and Uncertainty
0 50 100 150 200
Epoch405060708090100Error (%)CIFAR-100, Uniform Label Corruption
Train Error
Test Error
Figure 1. Training for longer is not a suitable strategy for label
corruption. By training for longer, the network eventually be-
gins to model and memorize label noise, which harms its overall
performance. Labels are corrupted uniformly to incorrect classes
with 60% probability, and the Wide Residual Network classiﬁer
has learning rate drops at epochs 80, 120, and 160.
searchers would do well to adopt the “pre-train then tune”
paradigm for increased performance and greater realism.
2. Related Work
Pre-Training. It is well-known that pre-training im-
proves generalization when the dataset for the target task
is extremely small. Prior work on transfer learning has
analyzed the properties of this effect, such as when ﬁne-
tuning should stop (Agrawal et al., 2014) and which layers
should be ﬁne-tuned (Yosinski et al., 2014). In a series of
ablation studies, Huh et al. (2016) show that the beneﬁts of
pre-training are robust to signiﬁcant variation in the dataset
used for pre-training, including the removal of classes re-
lated to the target task. In our work, we observe similar
robustness to change in the dataset used for pre-training.
Pre-training has also been used when the dataset for the tar-
get task is large, such as Microsoft COCO (Lin et al., 2014)
for object detection and segmentation. However, in a recent
work He et al. (2018) show that pre-training merely speeds
convergence on these tasks, and real gains in performance
vanish if one trains from scratch for long enough, even with
only 10% of the data for the target task. They conclude
that pre-training is not necessary for these tasks. Moreover,
Sun et al. (2017) show that the accuracy gains from more
data are exponentially diminishing, severely limiting the
utility of pre-training for improving performance metrics
for traditional tasks. In contrast, we show that pre-training
does markedly improve model robustness and uncertainty.
Robustness. Learning in the presence of corrupted labels
has been well-studied. In the context of deep learning,Sukhbaatar et al. (2014) investigate using a stochastic ma-
trix encoding the label noise, though they note that this ma-
trix is difﬁcult to estimate. Patrini et al. (2017) propose a
two-step training procedure to estimate this stochastic ma-
trix and train a corrected classiﬁer. These approaches are
extended by Hendrycks et al. (2018), who consider hav-
ing access to a small dataset of cleanly labeled examples,
leverage these trusted data to improve performance.
Zhang & Sabuncu (2018) show that networks overﬁt to
the incorrect labels when trained for too long (Figure 1).
This observation suggests pre-training as a potential ﬁx,
since one need only ﬁne-tune for a short period to attain
good performance. We show that pre-training not only im-
proves performance with no label noise correction, but also
complements methods proposed in prior work. Also note
that most prior works (Goldberger & Ben-Reuven, 2017;
Ma et al., 2018; Han et al., 2018) only experiment with
small-scale images since label corruption demonstrations
can require training hundreds of models (Hendrycks et al.,
2018). Since pre-training is typically reserved for large-
scale datasets, such works do not explore the impact of pre-
training.
Networks tend not to effectively model underrepresented
classes, which can affect a classiﬁer’s fairness of underrep-
resented groups. To handle class imbalance, many training
strategies have been investigated in the literature. One
direction is rebalancing an imbalanced training dataset. To
this end, He & Garcia (2008) propose to remove samples
from the majority classes, while Huang et al. (2016)
replicate samples from the minority classes. Generating
synthetic samples through linear interpolation between
data samples belonging in the same minority class has
been studied in Chawla et al. (2002). An alternative
approach is to modify the supervised loss function. Cost
sensitive learning (Japkowicz, 2000) balances the loss
function by re-weighting each sample by the inverse
frequency of its class. Huang et al. (2016) and Dong et al.
(2018) demonstrate that enlarging the margin of a classiﬁer
helps mitigate the class imbalance problem. However,
adopting such training methods often incurs various time
and memory costs.
The susceptibility of neural networks to small, adversari-
ally chosen input perturbations has received much atten-
tion. Over the years, many methods have been proposed as
defenses against adversarial examples (Metzen et al., 2017;
Hendrycks & Gimpel, 2017a), but these are often circum-
vented in short order (Carlini & Wagner, 2017). In fact,
the only defense widely regarded as having stood the test
of time is the adversarial training procedure of Madry et al.
(2018). In this algorithm, white-box adversarial examples
are created at each step of training and substituted in place
of normal examples. This does provide some amount of
Using Pre-Training Can Improve Model Robustness and Uncertainty
0.0 0.2 0.4 0.6 0.8 1.0
Corruption Strength020406080100Test Error (%)
CIFAR-100, No Correction
Normal
Pre-Training
0.0 0.2 0.4 0.6 0.8 1.0
Corruption Strength020406080100Test Error (%)
CIFAR-100, Forward Correction
0.0 0.2 0.4 0.6 0.8 1.0
Corruption Strength020406080100Test Error (%)
CIFAR-100, GLC 5% Trusted
0.0 0.2 0.4 0.6 0.8 1.0
Corruption Strength020406080100Test Error (%)
CIFAR-10, No Correction
0.0 0.2 0.4 0.6 0.8 1.0
Corruption Strength020406080100Test Error (%)
CIFAR-10, Forward Correction
0.0 0.2 0.4 0.6 0.8 1.0
Corruption Strength020406080100Test Error (%)
CIFAR-10, GLC 5% Trusted
Figure 2. Error curves for label noise correction methods using training from scratch and pre-training across a full range of label cor-
ruption strengths. For the No Correction baseline, using pre-training results in a visibly improved slope of degradation with a more
pronounced elbow at higher corruption strengths. This also occurs in the complementary combinations of pre-training with previously
proposed correction methods.
adversarial robustness, but it requires substantially longer
training times. In a later work, Schmidt et al. (2018) argue
further progress on this problem may require signiﬁcantly
more task-speciﬁc data. However, given that data from a
different distribution can be beneﬁcial for a given task (Huh
et al., 2016), it is conceivable that the need for task-speciﬁc
data could be obviated with pre-training.
Uncertainty. Even though deep networks have achieved
high accuracy on many classiﬁcation tasks, measuring
the uncertainty in their predictions remains a challenging
problem. Obtaining well-calibrated predictive uncertainty
could be useful in many machine learning applications such
as medicine or autonomous vehicles. Uncertainty estimates
need to be useful for detecting out-of-distribution samples.
Hendrycks & Gimpel (2017b) propose out-of-distribution
detection tasks and use the maximum value of a classiﬁer’s
softmax distribution as a baseline method. Lee et al.
(2018b) propose Mahalanobis distance-based scores which
characterize out-of-distribution samples using hidden fea-
tures. Lee et al. (2018a) propose using a GAN (Goodfellow
et al., 2014) to generate out-of-distribution samples; the
network is taught to assign low conﬁdence to these GAN-
generated samples. Hendrycks et al. (2019) demonstrate
that using non-speciﬁc, real, and diverse outlier images
or text in place of GAN-generated samples can allow
classiﬁers and density estimators to improve their out-of-distribution detection performance and calibration. Guo
et al. (2017) show that contemporary networks can easily
become miscalibrated without additional regularization,
and we show pre-training can provide useful regularization.
3. Robustness
Datasets. For the following robustness experiments, we
evaluate on CIFAR-10 and CIFAR-100 (Krizhevsky &
Hinton, 2009). These datasets contain 3232color
images, both with 60,000 images split into 50,000 for
training and 10,000 for testing. CIFAR-10 and CIFAR-
100 have 10 and 100 classes, respectively. For pre-
training, we use Downsampled ImageNet (Chrabaszcz
et al., 2017), which is the 1,000-class ImageNet dataset
(Deng et al., 2009) resized to 3232resolution. For
ablation experiments, we remove 153 CIFAR-10-related
classes from the Downsampled ImageNet dataset. In this
paper we tune the entire network. Code is available at
github.com/hendrycks/pre-training .
3.1. Robustness to Label Corruption
Setup. In the task of classiﬁcation under label corrup-
tion, the goal is to learn as good a classiﬁer as possible
on a dataset with corrupted labels. In accordance with prior
work (Sukhbaatar et al., 2014) we focus on multi-class clas-
Using Pre-Training Can Improve Model Robustness and Uncertainty
Table 1. Label corruption robustness results with and without pre-training. Each value is an area under the error curve summarizing
performance at 11 corruption strengths. Lower is better. All values are percentages. Pre-training greatly improves performance, in some
cases halving the error, and it can even surpass the task-speciﬁc Forward Correction.
CIFAR-10 CIFAR-100
Normal Training Pre-Training Normal Training Pre-Training
No Correction 28.7 15.9 55.4 39.1
Forward Correction 25.5 15.7 52.6 42.8
GLC (5% Trusted) 14.0 7.2 46.8 33.7
GLC (10% Trusted) 11.5 6.4 38.9 28.4
siﬁcation. Let x,y, andeybe an input, clean label, and po-
tentially corrupted label respectively. The labels take val-
ues from 1toK. Given a datasetDof(x;ey)pairs withx
drawn from p(x)andeydrawn from p(eyjy;x), the task is
to predict arg maxyp(yjx).
To experiment with a variety of corruption severities, we
corrupt the true label with a given probability to a randomly
chosen incorrect class. Formally, we generate corrupted la-
bels with a ground truth matrix of corruption probabilities
C, whereCij=p(ey=jjy=i)is the probability of
corrupting an example with label ito labelj. Given a cor-
ruption strength s, we construct Cwith(1s)I+s11T=K,
ItheKKidentity matrix. To measure performance, we
use the area under the curve plotting test error against cor-
ruption strength. This is generated via linear interpolation
between test errors at corruption strengths from 0to1in
increments of 0:1, summarizing a total of 11 experiments.
Methods. We ﬁrst consider the baseline of training from
scratch. This is denoted as Normal Training in Table 1.
We also consider state-of-the-art methods for classiﬁcation
under label noise. The Forward method of Patrini et al.
(2017) uses a two-stage training procedure. The ﬁrst stage
estimates the matrix Cdescribing the expected label noise,
and the second stage trains a corrected classiﬁer to pre-
dict the clean label distribution. We also consider the Gold
Loss Correction (GLC) method of Hendrycks et al. (2018),
which assumes access to a small, trusted dataset of cleanly
labeled (gold standard) examples, which is also known as
a semi-veriﬁed setting (Charikar et al., 2017). This method
also attempts to estimate C. For this method, we specify
the “trusted fraction,” which is the fraction of the available
training data that is trusted or known to be cleanly labeled.
In all experiments, we use 40-2 Wide Residual Networks,
SGD with Nesterov momentum, and a cosine learning rate
schedule (Loshchilov & Hutter, 2016). The “Normal” ex-
periments train for 100 epochs with a learning rate of 0:1
and use dropout at a drop rate of 0:3, as in Zagoruyko
& Komodakis (2016). The experiments with pre-training
train for 10 epochs without dropout, and use a learning rate
of0:001 in the “No Correction” experiment and 0:01inthe experiments with label noise corrections. We found the
latter experiments required a larger learning rate because
of variance introduced by the stochastic matrix corrections.
Most parameter and architecture choices recur in later sec-
tions of this paper. Results are in Table 1.
Analysis. In all experiments, pre-training gives large per-
formance gains over the models trained from scratch. With
no correction, we see a 45% relative reduction in the area
under the error curve on CIFAR-10 and a 29% reduction on
CIFAR-100. These improvements exceed those of the task-
speciﬁc Forward method. Therefore in the setting with-
out trusted data, pre-training attains new state-of-the-art
AUCs of 15.9% and 39.1% on CIFAR-10 and CIFAR-100
respectively. These results are stable, since pre-training
on Downsampled ImageNet with CIFAR-10-related classes
removed yields a similar AUC on CIFAR-10 of 14.5%.
Moreover, we found that these gains could notbe bought
by simply training for longer. As shown in Figure 1, train-
ing for a long time with corrupted labels actually harms
performance as the network destructively memorizes the
misinformation in the incorrect labels.
We also observe complementary gains of combining pre-
training with previously proposed label noise correction
methods. In particular, using pre-training together with the
GLC on CIFAR-10 at a trusted fraction of 5% cuts the area
under the error curve in half. Moreover, using pre-training
with the same amount of trusted data provides larger per-
formance boosts than doubling the amount of trusted data,
effectively allowing one to reach a target performance level
with half as much trusted data. Qualitatively, Figure 2
shows that pre-training softens the performance degrada-
tion as the corruption strength increases.
Importantly, although pre-training does have substantial
additive effects on performance with the Forward Correc-
tion method, we ﬁnd that pre-training with no correction
yields superior performance. This observation implies that
future research on label corruption should evaluate with
pre-trained networks or else researchers may develop meth-
ods that are suboptimal.
We observe that pre-training also provides substantial im-
Using Pre-Training Can Improve Model Robustness and Uncertainty
Table 2. Experimental results on the imbalanced CIFAR-10 and CIFAR-100 datasets.
DatasetMethodImbalance Ratio 0.2 0.4 0.6 0.8 1.0 1.5 2.0
Total Test Error Rate / Minority Test Error Rate (%)CIFAR-10Normal Training 23.7 / 26.0 21.8 / 26.5 21.1 / 25.8 20.3 / 24.7 20.0 / 24.5 18.3 / 23.1 15.8 / 20.2
Cost Sensitive 22.6 / 24.9 21.8 / 26.2 21.1 / 25.7 20.2 / 24.3 20.2 / 24.6 18.1 / 22.9 16.0 / 20.1
Oversampling 21.0 / 23.1 19.4 / 23.6 19.0 / 23.2 18.2 / 22.2 18.3 / 22.4 17.3 / 22.2 15.3 / 19.8
SMOTE 19.7 / 21.7 19.7 / 24.0 19.2 / 23.4 19.2 / 23.4 18.1 / 22.1 17.2 / 22.1 15.7 / 20.4
Pre-Training 8.0 / 8.8 7.9 / 9.5 7.6 / 9.2 8.0 / 9.7 7.4 / 9.1 7.4 / 9.5 7.2 / 9.4CIFAR-100Normal Training 69.7 / 72.0 66.6 / 70.5 63.2 / 69.2 58.7 / 65.1 57.2 / 64.4 50.2 / 59.7 47.0 / 57.1
Cost Sensitive 67.6 / 70.6 66.5 / 70.4 62.2 / 68.1 60.5 / 66.9 57.1 / 64.0 50.6 / 59.6 46.5 / 56.7
Oversampling 62.4 / 66.2 59.7 / 63.8 59.2 / 65.5 55.3 / 61.7 54.6 / 62.2 49.4 / 59.0 46.6 / 56.9
SMOTE 57.4 / 61.0 56.2 / 60.3 54.4 / 60.2 52.8 / 59.7 51.3 / 58.4 48.5 / 57.9 45.8 / 56.3
Pre-Training 37.8 / 41.8 36.9 / 41.3 36.2 / 41.7 36.4 / 42.3 34.9 / 41.5 34.0 / 41.9 33.5 / 42.2
provements when swapping out the Wide ResNet for an
All Convolutional Network (Springenberg et al., 2014). In
the No Correction setting, area under the error curves on
CIFAR-10 for Normal Training and Pre-Training are 23.7%
and 14.8% respectively. On CIFAR-100, they are 46.5%
and 41.0% respectively. Additionally, when ﬁne-tuning a
Wide ResNet on Places365 downsampled in the same fash-
ion as ImageNet in earlier experiments, we obtain area
under the error curves of 19.3% and 49.5% compared to
28.7% and 55.4% with Normal Training. These experi-
ments demonstrate the generalizability of our results across
architectures and datasets used for pre-training.
3.2. Robustness to Class Imbalance
In most real-world classiﬁcation problems, some classes
are more abundant than others, which naturally results in
class imbalance (Van Horn et al., 2018). Unfortunately,
deep networks tend to model prevalent classes at the ex-
pense of minority classes. This need not be the case. Deep
networks are capable of learning both the prevalent and
minority classes, but to accomplish this, task-speciﬁc ap-
proaches have been necessary. In this section, we show
that pre-training can also be useful for handling such im-
balanced scenarios better than approaches speciﬁcally cre-
ated for this task (Japkowicz, 2000; Chawla et al., 2002;
Huang et al., 2016; Dong et al., 2018).
Setup. Similar to Dong et al. (2018), we simulate class
imbalance with a power law model. Speciﬁcally, we set
the number of training samples for a class cas follows,
nc=ba=(b+ (c1)
)e, wherebeis the integer round-
ing function,
represents an imbalance ratio, aandb
are offset parameters to specify the largest and small-
est class sizes. Our training data becomes a power law
class distribution as the imbalance ratio
decreases. We
test 7 different degrees of imbalance; speciﬁcally,
2
f0:2;0:4;0:6;0:8;1:0;1:5;2:0gand(a;b)are set to force
(maxcnc;mincnc)to become (5000;250) for CIFAR-10
and(500;25)for CIFAR-100. A class is deﬁned as a mi-nority class if its size is smaller than the average class size.
For evaluation, we measure the average test set error rates
of all classes and error rates of minority classes.
1 2 3 4 5 6 7 8 910
Class Number0510152025303540Test Error (%)
Class-Wise Error Rates
Normal
Cost Sensitive
CRLOversampling
SMOTE
Figure 3. Class-wise test set error rates are lower across all classes
with pre-training. Here the imbalanced dataset is a CIFAR-10
modiﬁcation with imbalance ratio
= 0:2.
Methods. The class imbalance baseline methods are as
follows. Normal Training is the conventional approach of
training from scratch with cross-entropy loss. Oversam-
pling (Japkowicz, 2000) is a re-sampling method to build
a balanced training set before learning through augmenting
the samples of minority classes with random replication.
SMOTE (Chawla et al., 2002) is an oversampling method
that uses synthetic samples by interpolating linearly with
neighbors. Cost Sensitive (Huang et al., 2016) introduces
additional weights in the loss function for each class pro-
portional to inverse class frequency.
Using Pre-Training Can Improve Model Robustness and Uncertainty
Table 3. Adversarial accuracies of models trained from scratch, with adversarial training, and with adversarial training with pre-training.
All values are percentages. The pre-trained models have comparable clean accuracy to adversarially trained models from scratch, as
implied by He et al. (2018), but pre-training can markedly improve adversarial accuracy.
CIFAR-10 CIFAR-100
Clean Adversarial Clean Adversarial
Normal Training 96.0 0.0 81.0 0.0
Adversarial Training 87.3 45.8 59.1 24.3
Adv. Pre-Training and Tuning 87.1 57.4 59.2 33.5
Here we use 40-2 Wide Residual Networks, SGD with Nes-
terov momentum, and a cosine learning rate schedule. The
experiments with pre-training train for 50 epochs without
dropout and use a learning rate of 0:001, and the exper-
iments with other baselines train for 100 epochs with a
learning rate of 0:1and use dropout at a drop rate of 0:3.
Analysis. Table 2 shows that the pre-training alone sig-
niﬁcantly improves the test set error rates compared to
task-speciﬁc methods that can incur expensive back-and-
forth costs, requiring additional training time and memory.
Here, we remark that much of the gain from pre-training is
from the low test error rates on minority classes (i.e., those
with greater class indices), as shown in Figure 3. Further-
more, if we tune a network on CIFAR-10 that is pre-trained
on Downsampled ImageNet with CIFAR-10-related classes
removed, the total error rate increases by only 2.1% com-
pared to pre-training on all classes. By contrast, the dif-
ference between pre-training and SMOTE is 12.6%. This
implies that pre-training is indeed useful for improving ro-
bustness against class imbalance.
3.3. Robustness to Adversarial Perturbations
Setup. Deep networks are notably unstable and less ro-
bust than the human visual system (Geirhos et al., 2018;
Hendrycks & Dietterich, 2019). For example, a network
may produce a correct prediction for a clean image, but
should the image be perturbed carefully, its verdict may
change entirely (Szegedy et al., 2014). This has led re-
searchers to defend networks against “adversarial” noise
with a small `pnorm, so that networks correctly general-
ize to images with a worst-case perturbation applied.
Nearly all adversarial defenses have been broken (Carlini &
Wagner, 2017), and adversarial robustness for large-scale
image classiﬁers remains elusive (Engstrom et al., 2018).
The exception is that adversarial training in the style of
Madry et al. (2018) has been partially successful for de-
fending small-scale image classiﬁers against `1perturba-
tions. Following their work and using their state-of-the-art
adversarial training procedure, we experiment with CIFAR
images and assume the adversary can corrupt images withperturbations of an `1norm less than or equal to 8=255.
The initial learning rate is 0:1and the learning rate anneals
following a cosine learning rate schedule. We adversarially
train the model against a 10-step adversary for 100 epochs
and test against 20-step untargeted adversaries. Additional
results with 100-step adversaries and random restarts are in
the Supplementary Materials. Unless otherwise speciﬁed,
we use 28-10 Wide Residual Networks, as adversarially
trained high-capacity networks exhibit greater adversarial
robustness (Kurakin et al., 2017; Madry et al., 2018).
Analysis. It could be reasonable to expect that pre-
training would not improve adversarial robustness. First,
nearly all adversarial defenses fail, and even some adver-
sarial training methods can fail too (Engstrom et al., 2018).
Current adversarial defenses result in networks with large
generalization gaps, even when the train and test distribu-
tions are similar. For instance, CIFAR-10 Wide ResNets
are made so wide that their adversarial train accuracies are
100% but their adversarial test accuracies are only 45.8%.
Schmidt et al. (2018) speculate that a signiﬁcant increase in
task-speciﬁc data is necessary to close this gap. This gen-
eralization gap swells under slight changes to the problem
setup (Sharma & Chen, 2018). We attempt to reduce this
gap and make pre-trained representations transfer across
data distributions, but doing so requires an unconventional
choice. Choosing to use targeted adversaries or no adver-
saries during pre-training does not provide substantial ro-
bustness. Instead, we choose to adversarially pre-train a
Downsampled ImageNet model against an untargeted ad-
versary, contra Kurakin et al. (2017); Kannan et al. (2018);
Xie et al. (2018).
We ﬁnd that an adversarially pre-trained network can sur-
pass the long-standing state-of-the-art model by a signif-
icant margin. By pre-training a Downsampled ImageNet
classiﬁer against an untargeted adversary, then adversari-
ally ﬁne-tuning on CIFAR-10 or CIFAR-100 for 5 epochs
with a learning rate of 0:001, we obtain networks which
improve adversarial robustness by 11.6% and 9.2% in ab-
solute accuracy respectively.
As in the other tasks we consider, a Downsampled Ima-
geNet model with CIFAR-10-related classes removed sees
Using Pre-Training Can Improve Model Robustness and Uncertainty
similar robustness gains. As a quick check, we pre-trained
and tuned two 40-2 Wide ResNets, one pre-trained typi-
cally and one pre-trained with CIFAR-10-related classes
excluded from Downsampled ImageNet. We observed only
a 1.04% decrease in adversarial accuracy compared to the
typically pre-trained model, which demonstrates that the
pre-trained models do not rely on seeing CIFAR-10-related
images, and that simply training on more natural images
increases adversarial robustness. Notice that in Table 3 the
clean accuracy is approximately the same while the adver-
sarial accuracy is far larger. This indicates again that pre-
training may have a limited effect on accuracy for tradi-
tional tasks, but it has a strong effect on robustness.
It is even the case that the pre-trained representations can
transfer to a new task without adversarially tuning the en-
tire network. In point of fact, if we only adversarially
tune the last afﬁne classiﬁcation layer, and no other pa-
rameters, for CIFAR-10 and CIFAR-100 we respectively
obtain adversarial accuracies of 46.6% and 26.1%. Thus
adversarially tuning only the last afﬁne layer also surpasses
the previous adversarial accuracy state-of-the-art. This fur-
ther demonstrates that that adversarial features can robustly
transfer across data distributions. In addition to robust-
ness gains, adversarial pre-training could save much wall-
clock time since pre-training speeds up convergence; com-
pared to typical training routines, adversarial training pro-
hibitively requires at least 10the usual amount of training
time. By surpassing the previous state-of-the-art, we have
shown that pre-training enhances adversarial robustness.
4. Uncertainty
To demonstrate that pre-training improves model uncer-
tainty estimates, we use the CIFAR-10, CIFAR-100, and
Tiny ImageNet datasets (Johnson et al.). We did not use
Tiny ImageNet in the robustness section, because adversar-
ial training is not known to work on images of this size,
and using Tiny ImageNet is computationally prohibitive
for the label corruption experiments. Tiny ImageNet con-
sists of 200 ImageNet classes at 6464resolution, so we
use a 6464version of Downsampled ImageNet for pre-
training. We also remove the 200 overlapping Tiny Ima-
geNet classes from Downsampled ImageNet for all experi-
ments on Tiny ImageNet.
In all experiments, we use 40-2 Wide ResNets trained using
SGD with Nesterov momentum and a cosine learning rate.
Pre-trained networks train on Downsampled ImageNet for
100 epochs, and are ﬁne-tuned for 10 epochs for CIFAR
and 20 for Tiny ImageNet without dropout and with a learn-
ing rate of 0:001. Baseline networks train from scratch for
100 epochs with a dropout rate of 0:3. When performing
temperature tuning in Section 4.2, we train without 10% of
the training data to estimate the optimum temperature.Table 4. Out-of-distribution detection performance with models
trained from scratch and with models pre-trained. Results are an
average of ﬁve runs. Values are percentages.
AUROC AUPR
Normal Pre-Train Normal Pre-Train
CIFAR-10 91.5 94.5 63.4 73.5
CIFAR-100 69.4 83.1 29.7 52.7
Tiny ImageNet 71.8 73.9 30.8 31.0
4.1. Out-of-Distribution Detection
Setup. In the problem of out-of-distribution detection
(Hendrycks & Gimpel, 2017b; Hendrycks et al., 2019; Lee
et al., 2018a;b; Liu et al., 2018), models are tasked with
assigning anomaly scores to indicate whether a sample is
in- or out-of-distribution. Hendrycks & Gimpel (2017b)
show that the discriminative features learned by a classi-
ﬁer are well-suited for this task. They use the maximum
softmax probability maxkp(y=kjx)for each sample
xas a way to rank in- and out-of-distribution (OOD) sam-
ples. OOD samples tend to have lower maximum softmax
probabilities. Improving over this baseline is a difﬁcult
challenge without assuming knowledge of the test distri-
bution of anomalies (Chen et al., 2018). Without assuming
such knowledge, we use the maximum softmax probabil-
ities to score anomalies and show that models which are
pre-trained then tuned provide superior anomaly scores.
To measure the quality of out-of-distribution detection, we
employ two standard metrics. The ﬁrst is the AUROC ,
or the Area Under the Receiver Operating Characteristic
curve. This is the probability that an OOD example is as-
signed a higher anomaly score than an in-distribution ex-
ample. Thus a higher AUROC is better. A similar measure
is the AUPR , or the Area Under the Precision-Recall Curve;
as before, a higher AUPR is better. For in-distribution data
we use the test dataset. For out-of-distribution data we
use the various anomalous distributions from Hendrycks
et al. (2019), including Gaussian noise, textures, Places365
scene images (Zhou et al., 2017), etc. All OOD datasets do
not have samples from Downsampled ImageNet. Further
evaluation details are in the Supplementary Materials.
Analysis. By using pre-training, both the AUROC and
AUPR consistently improve over the baseline, as shown
in Table 4. Note that results are an average of the AUROC
and AUPR values from detecting samples from various
OOD datasets. Observe that with pre-training, CIFAR-100
OOD detection signiﬁcantly improves. Consequently
pre-training can directly improve uncertainty estimates.
Using Pre-Training Can Improve Model Robustness and Uncertainty
Table 5. Calibration errors for models trained from scratch and
models with pre-training. All values are percentages.
RMS Error MAD Error
Normal Pre-Train Normal Pre-Train
CIFAR-10 6.4 2.9 2.9 1.2
CIFAR-100 13.3 3.6 10.3 2.5
Tiny ImageNet 8.5 4.2 7.0 2.9
4.2. Calibration
Setup. A central component of uncertainty estimation in
classiﬁcation problems is conﬁdence calibration. From a
classiﬁcation system that produces probabilistic conﬁdence
estimatesCof its predictions bYbeing correct, we would
like trustworthy estimates. That is, when a classiﬁer pre-
dicts a class with eighty percent conﬁdence, we would
like it to be correct eighty percent of the time. Nguyen
& O’Connor (2015); Hendrycks & Gimpel (2017b) found
that deep neural network classiﬁers display severe overcon-
ﬁdence in their predictions, and that the problem becomes
worse with increased representational capacity (Guo et al.,
2017). Integrating uncalibrated classiﬁers into decision-
making processes could result in egregious assessments,
motivating the task of conﬁdence calibration.
To measure the calibration of a classiﬁer, we adopt
two measures from the literature. The Root Mean
Square Calibration Error (RMS) is the square root of
the expected squared difference between the classiﬁer’s
conﬁdence and its accuracy at said conﬁdence level,q
EC[(P(Y=bYjC=c)c)2]. The Mean Absolute
Value Calibration Error (MAD) uses the expected abso-
lute difference rather than squared difference between the
same quantities. The MAD Calibration Error has the same
form as the Expected Calibration Error used by Guo et al.
(2017), but it employs adaptive binning of conﬁdences for
improved estimation. In our experiments, we use a bin size
of 100. We refer the reader to Hendrycks et al. (2019) for
further details on these measures.
Analysis. In all experiments, we observe large improve-
ments in calibration from using pre-training. In Figure 4
and Table 5, we can see that RMS Calibration Error is at
least halved on all datasets through the use of pre-training,
with CIFAR-100 seeing the largest improvement. The
same is true of the MAD error. In fact, the MAD error on
CIFAR-100 is reduced by a factor of 4.1 with pre-training,
which can be interpreted as the stated conﬁdence being
four times closer to the true frequency of occurrence.
We ﬁnd that these calibration gains are robust across pre-
training datasets. With Places365 pre-training the RMS er-
ror is 3.1 on CIFAR-10, and with ImageNet pre-training
SVHNCIFAR-10 CIFAR-100Tiny ImageNet0510152025RMS Calibration Error (%)Calibration Error with Outlier Exposure
Temperature Tuning
+OEFigure 4. Root Mean Square Calibration Error values for mod-
els trained from scratch and models that are pre-trained. On all
datasets, pre-training reduces the RMS error by more than half.
the RMS error is 2.9; meanwhile, the baseline RMS error
is 6.4. The gains are also complementary with the tem-
perature tuning method of Guo et al. (2017), which fur-
ther reduces RMS Calibration Error from 4.15 to 3.55 for
Tiny ImageNet when combined with pre-training. How-
ever, temperature tuning is computationally expensive and
requires additional data, whereas pre-training does not re-
quire collecting extra data and can naturally and directly
make the model more calibrated.
5. Conclusion
Although He et al. (2018) assert that pre-training does
not improve performance on traditional tasks, for other
tasks this is not so. On robustness and uncertainty tasks,
pre-training results in models that surpass the previous
state-of-the-art. For uncertainty tasks, we ﬁnd pre-trained
representations directly translate to improvements in pre-
dictive uncertainty estimates. He et al. (2018) argue that
both pre-training and training from scratch result in models
of similar accuracy, but we show this only holds for unper-
turbed data. In fact, pre-training with an untargeted adver-
sary surpasses the long-standing state-of-the-art in adver-
sarial accuracy by a signiﬁcant margin. Robustness to label
corruption is similarly improved by wide margins, such
that pre-training alone outperforms certain task-speciﬁc
methods, sometimes even after combining these methods
with pre-training. This suggests future work on model
robustness should evaluate proposed methods with pre-
training in order to correctly gauge their utility, and some
work could specialize pre-training for these downstream
tasks. In sum, the beneﬁts of pre-training extend beyond
merely quick convergence, as previously thought, since
pre-training can improve model robustness and uncertainty.
Using Pre-Training Can Improve Model Robustness and Uncertainty
ACKNOWLEDGMENTS
This research was partially supported by the Engineering
Research Center Program through the National Research
Foundation of Korea (NRF) funded by the Korean Govern-
ment MSIT (NRF-2018R1A5A1059921).
References
Agrawal, P., Girshick, R., and Malik, J. Analyzing the per-
formance of multilayer neural networks for object recog-
nition. ECCV , 2014.
Carlini, N. and Wagner, D. Adversarial examples are not
easily detected: Bypassing ten detection methods, 2017.
Charikar, M., Steinhardt, J., and Valiant, G. Learning from
untrusted data. STOC , 2017.
Chawla, N. V ., Bowyer, K. W., Hall, L. O., and
Kegelmeyer, W. P. SMOTE: synthetic minority over-
sampling technique. JAIR , 2002.
Chen, W., Shen, Y ., Wang, X., and Wang, W. Enhancing
the robustness of prior network in out-of-distribution de-
tection. arXiv , 2018.
Chrabaszcz, P., Loshchilov, I., and Hutter, F. A downsam-
pled variant of ImageNet as an alternative to the CIFAR
datasets. arXiv , 2017.
Deng, J., Dong, W., Socher, R., jia Li, L., Li, K., and
Fei-Fei, L. ImageNet: A large-scale hierarchical image
database. CVPR , 2009.
Dong, Q., Gong, S., and Zhu, X. Imbalanced deep learn-
ing by minority class incremental rectiﬁcation. IEEE
TPAMI , 2018.
Engstrom, L., Ilyas, A., and Athalye, A. Evaluating and
understanding the robustness of adversarial logit pairing.
arXiv preprint , 2018.
Geirhos, R., Temme, C. R. M., Rauber, J., Sch ¨utt, H. H.,
Bethge, M., and Wichmann, F. A. Generalisation in hu-
mans and deep neural networks. NeurIPS , 2018.
Goldberger, J. and Ben-Reuven, E. Training deep neural-
networks using a noise adaptation layer. In ICLR , 2017.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B.,
Warde-Farley, D., Ozair, S., Courville, A., and Bengio,
Y . Generative adversarial networks. In NeurIPS , 2014.
Guo, C., Pleiss, G., Sun, Y ., and Weinberger, K. Q. On cal-
ibration of modern neural networks. International Con-
ference on Machine Learning , 2017.Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang,
I., and Sugiyama, M. Co-teaching: robust training
deep neural networks with extremely noisy labels. In
NeurIPS , 2018.
He, H. and Garcia, E. A. Learning from imbalanced data.
TKDE , 2008.
He, K., Gkioxari, G., Dollar, P., and Girshick, R. Mask
R-CNN. ICCV , 2017.
He, K., Girshick, R., and Dollar, P. Rethinking ImageNet
pre-training. arXiv , 2018.
Hendrycks, D. and Dietterich, T. Benchmarking neural net-
work robustness to common corruptions and perturba-
tions. ICLR , 2019.
Hendrycks, D. and Gimpel, K. Early methods for detecting
adversarial images. ICLR Workshop , 2017a.
Hendrycks, D. and Gimpel, K. A baseline for detecting
misclassiﬁed and out-of-distribution examples in neural
networks. ICLR , 2017b.
Hendrycks, D., Mazeika, M., Wilson, D., and Gimpel, K.
Using trusted data to train deep networks on labels cor-
rupted by severe noise. NeurIPS , 2018.
Hendrycks, D., Mazeika, M., and Dietterich, T. Deep
anomaly detection with outlier exposure. ICLR , 2019.
Huang, C., Li, Y ., Change Loy, C., and Tang, X. Learn-
ing deep representation for imbalanced classiﬁcation. In
CVPR , 2016.
Huh, M., Agrawal, P., and Efros, A. A. What makes Ima-
geNet good for transfer learning? arXiv , 2016.
Japkowicz, N. The class imbalance problem: Signiﬁcance
and strategies. In ICAI , 2000.
Johnson et al. Tiny ImageNet visual recognition
challenge. URL https://tiny-imagenet.
herokuapp.com .
Kannan, H., Kurakin, A., and Goodfellow, I. Adversarial
logit pairing. NeurIPS , 2018.
Krizhevsky, A. and Hinton, G. Learning multiple layers of
features from tiny images. Technical report, University
of Toronto , 2009.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. ImageNet
classiﬁcation with deep convolutional neural networks.
NIPS , 2012.
Kurakin, A., Goodfellow, I., and Bengio, S. Adversarial
machine learning at scale. ICLR , 2017.
Using Pre-Training Can Improve Model Robustness and Uncertainty
Lee, K., Lee, H., Lee, K., and Shin, J. Training conﬁdence-
calibrated classiﬁers for detecting out-of-distribution
samples. ICLR , 2018a.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple uniﬁed
framework for detecting out-of-distribution samples and
adversarial attacks. NeurIPS , 2018b.
Lin, T.-Y ., Maire, M., Belongie, S., Bourdev, L., Girshick,
R., Hays, J., Perona, P., Ramanan, D., Zitnick, C. L., and
Dollar, P. Microsoft COCO: Common objects in context.
ECCV , 2014.
Liu, S., Garrepalli, R., Dietterich, T., Fern, A., and
Hendrycks, D. Open category detection with PAC guar-
antees. In ICML , 2018.
Loshchilov, I. and Hutter, F. SGDR: stochastic gradient
descent with warm restarts. ICLR , 2016.
Ma, X., Wang, Y ., Houle, M. E., Zhou, S., Erfani,
S. M., Xia, S.-T., Wijewickrema, S., and Bailey, J.
Dimensionality-driven learning with noisy labels. In
ICML , 2018.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and
Vladu, A. Towards deep learning models resistant to
adversarial attacks. ICLR , 2018.
Metzen, J. H., Genewein, T., Fischer, V ., and Bischoff, B.
On detecting adversarial perturbations, 2017.
Nguyen, K. and O’Connor, B. Posterior calibration and ex-
ploratory analysis for natural language processing mod-
els.EMNLP , 2015.
Patrini, G., Rozza, A., Menon, A., Nock, R., and Qu, L.
Making deep neural networks robust to label noise: a
loss correction approach. CVPR , 2017.
Rebufﬁ, S.-A., Bilen, H., and Vedaldi, A. Learning mul-
tiple visual domains with residual adapters. NeurIPS ,
2017.
Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., and
Madry, A. Adversarially robust generalization requires
more data. NeurIPS , 2018.
Sharma, Y . and Chen, P.-Y . Attacking the Madry defense
model withl1-based adversarial examples. ICLR Work-
shop , 2018.
Springenberg, J. T., Dosovitskiy, A., Brox, T., and Ried-
miller, M. A. Striving for simplicity: The all convolu-
tional net. CoRR , abs/1412.6806, 2014.
Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., and Fer-
gus, R. Training convolutional networks with noisy la-
bels. ICLR Workshop , 2014.Sun, C., Shrivastava, A., Singh, S., and Gupta, A. Revis-
iting unreasonable effectiveness of data in deep learning
era.ICCV , 2017.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan,
D., Goodfellow, I., and Fergus, R. Intriguing properties
of neural networks, 2014.
Van Horn, G., Mac Aodha, O., Song, Y ., Cui, Y ., Sun, C.,
Shepard, A., Adam, H., Perona, P., and Belongie, S. The
inaturalist species classiﬁcation and detection dataset. In
CVPR , 2018.
Xie, C., Wu, Y ., van der Maaten, L., Yuille, A., and He, K.
Feature denoising for improving adversarial robustness.
arXiv preprint , 2018.
Yosinski, J., Clune, J., Bengio, Y ., and Lipson, H.
How transferable are features in deep neural networks?
NeurIPS , 2014.
Zagoruyko, S. and Komodakis, N. Wide residual networks.
BMVC , 2016.
Zeiler, M. D. and Fergus, R. Visualizing and understanding
convolutional networks. ECCV , 2014.
Zhang, Z. and Sabuncu, M. Generalized cross entropy
loss for training deep neural networks with noisy labels.
NeurIPS , 2018.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor-
ralba, A. Places: A 10 million image database for scene
recognition. PAMI , 2017.


On Handling Self-masking and Other Hard Missing Data Problems.

On Handling Self-masking and Other Hard Missing Data Problems
Karthika Mohan
Computer Science Department
University of California, Berkeley
karthika@berkeley.edu
Abstract
This paper tackles hard incomplete (missing) data problems
such as those in which missingness in a variable is caused
by itself. To address these problems we develop a new tech-
nique that jointly harnesses model and data as opposed to
existing methods that exploit properties of the model alone.
We present necessary and sufﬁcient conditions under which
consistent estimates of target quantities can be computed. In
sharp contrast to other techniques used for dealing with simi-
lar problems, we do not make any parametric assumptions.
Introduction
Analysing and drawing inferences from missing data can be
extremely challenging when the dataset contains variables
that are themselves causes of their missingness; this type
of missingness known as self-masking missingness, is be-
lieved to be the most commonly encountered type in prac-
tice [Osborne, 2012; Sverdlov, 2015; Adams, 2007; Mohan
et al. , 2018]. Examples include smokers not answering ques-
tions pertaining to their smoking behavior in insurance ap-
plications, people with very high and very low income not
disclosing their income and people of certain age groups not
revealing their age.
Recent years have witnessed a growing interest in han-
dling missing data using graphical models that encode
assumptions about the underlying missingness process.
[Daniel et al. , 2012; Mohan et al. , 2013; Shpitser et al. ,
2015; Mohan and Pearl, 2018]. Given a target quantity Q
and a graph G,Qisrecoverable from Gif there exists an
algorithm that can consistently estimate Qfor all data gener-
ated by G, elseQisnon-recoverable . Such non-recoverable
(Q; G )pairs which we call hard missing data problems (or
hard problems) are the focus of this paper. Examples include
Q=P(Ojdo(t))and the self-masking model shown in ﬁg-
ure 1 and Q=P(X)and the self-masking model shown in
ﬁgure 2 (b). While previous work treated recoverability as
a property of graph alone, in this paper we develop general
techniques to solve hard missing data problems by harness-
ing the properties of both graph and data .
In the following section we review missingness graphs i.e.
graphical models for handling missing data [Mohan et al. ,
2013].
Copyright c
2018, Association for the Advancement of Artiﬁcial
T O Ro
O∗treatment outcomemissingness
mechanism
proxy variable
for outcomeFigure 1: Missingness Graph in which outcome causes its
own missingness
Missingness Graphs
LetG(V; E)be the causal DAG where Vis the set of nodes
andEis the set of edges. Nodes in the graph correspond to
variables in the data set and are partitioned into ﬁve cate-
gories, i.e. V=Vo[Vm[U[V[R.
Vois the set of variables that are observed in all records in
the population and Vmis the set of variables that are miss-
ing in at least one record. Variable Xis termed as fully ob-
served ifX2Voandpartially observed ifX2Vm.Rvi
andV
iare two variables associated with every partially ob-
served variable, where V
iis a proxy variable that is actually
observed, and Rvirepresents the status of the causal mech-
anism responsible for the missingness of V
i; formally,
v
i=f(rvi; vi) =
vi ifrvi= 0
m ifrvi= 1(1)
Vis the set of all proxy variables and Ris the set of
all causal mechanisms that are responsible for missingness.
Unless stated otherwise it is assumed that no variable in
Vo[Vm[Uis a child of an Rvariable. Uis the set of
unobserved nodes, also called latent variables. Two nodes
XandYcan be connected by a directed edge i.e. X!Y,
indicating that Xis a cause of Y, or by a bi-directed edge
XL9999K Ydenoting the existence of a Uvariable that
is a parent of both XandY. This graphical representation
is called a Missingness Graph (orm-graph) [Mohan et al. ,
2013]. P(V; Vo; R)is called the observed data distribution.
Proxy variables may not always be explicitly shown in
m-graphs in order to keep the ﬁgures simple and clear. Con-
Intelligence (www.aaai.org). All rights reserved.
X Rx
X∗(a)
X Rx
X∗ (b)
Figure 2: m-graphs in which (a) P(X)is recoverable, (b)
P(X)is non-recoverable.
ditional Independencies are read off the graph using the d-
separation criterion Pearl [2009]. For any binary variable X,
x0andxdenote X= 0andX= 1respectively.
Example 1. In the m-graph in ﬁgure 1, Tdenotes the treat-
ment administered to patients and Odenotes the outcome.
While Tis observed for all patients, Ois observed only for
some of them. The edge from OtoRoindicates that the miss-
ingness is of self-masking type. Vo=fTg,Vm=fOg,
V=fOg,U=;andR=fRog.
Missingness Mechanisms
Based on Rubin [1976], missing data problems can be classi-
ﬁed into Missing Completely At Random (MCAR) , Missing
At Random (MAR) and Missing Not At Random (MNAR).
In this paper we use the graph based deﬁnition of these
mechanisms [Mohan et al. , 2013].
An m-graph Gdepicts an MCAR problem if
(Vm; Vo)? ?Rholds in G, an MAR problem if Vm? ?RjVo
holds in Gand an MNAR problem otherwise. For example,
ﬁgure 2 (a) depicts MCAR, ﬁgure 3 (c) depicts MAR and
ﬁgure 1 depicts MNAR missingness problems. Among
these, joint distribution ( Q=P(Vo; Vm)) is always con-
sistently estimable (i.e. recoverable) when missingness is
either MCAR or MAR (Mohan et al. [2013]). However this
is not true for MNAR missingness. As such all hard missing
data problems discussed in this paper belong to the MNAR
category.
Recoverability as a property of m-graph
In this section we exemplify the notions of recoverability
and non-recoverability as a property of the m-graph.
XRxP(X; Rx)
0 0 0.3
1 0 0.2
m 1 0.5
Table 1: Observed Data Distribution
Suppose Xis a binary variable corrupted by missing val-
ues. The dataset with missing values is shown in table 1.
Figures 2 (a) & (b) depict two distinct (but statistically indis-
tinguishable1) processes that could have generated this data.
1Although d-separations are testable implications of a graphi-
cal model, under missingness not all d-separations are testable. InIn ﬁgure 2 (a) missingness is generated by a purely random
process while ﬁgure 2 (b) depicts self-masking missingness.
Recoverability: Consider the problem of recovering
P(X)given the m-graph Gin ﬁgure 2 (a).
Since X? ?RxinGwe have,
P(X) =P(XjRx= 0)
Using equation 1 we can rewrite the above as,
P(X) =P(XjRx= 0)
By showing that P(X)is a quantity that can be computed
from the observed data distribution, we have established its
recoverability. To do this we used the assumption X? ?Rx
embedded in the m-graph. Hence in this case recoverability
is a property of the m-graph alone. The recovered distribu-
tionP(X)is shown in table 2.
XP(X)
0 3/5
1 2/5
Table 2: Recovered Distribution
Non-recoverability: Now consider the problem of recov-
ering P(X)given the m-graph in ﬁgure 2 (b). Rxis depen-
dent on Xand we have no additional information regarding
this dependence. It could be that Xis missing only when its
value is 1 or it could be that Xis missing only when its value
is 0. In the former case P(x0) = 0 :3where as in the latter
caseP(x0) = 0:8. Using the available information in Git is
not possible to ﬁnd the (true) value of P(X)even if we are
given inﬁnitely many samples i.e. P(X)is non-recoverable.
In fact, non-recoverability of P(X)would persist even if G
had more variables in it (formally proved in [Mohan et al. ,
2013; Mohan and Pearl, 2014a]).
Inability to handle hard problems such as self-masking
missingness is a major deﬁciency in the ﬁeld of missing
data. Recent papers such as Shpitser [2016] and Mohan et al.
[2013], and missing data text books such as Enders [2010]
have called attention to the problem of recoverability in self-
masking models. Standard Bayesian network textbooks such
as Darwiche [2009] (chapter 17) and Koller and Friedman
[2009] (chapter 19) discuss models similar to that in ﬁgure
1 and have shown that none of the existing methods such
as the EM algorithm can recover parameters in self-masking
m-graphs. In the following section we develop techniques
to recover queries in hard problems and thus eliminate this
deﬁciency in the ﬁeld.
particular no statement of independence between a variable and its
missingness mechanism ( X? ?Rx) is testable [Mohan and Pearl,
2014b].
Recoverability as a property of both m-graph
and missing data
We exemplify below a procedure that exploits the properties
of both graph and data to recover the joint distribution in a
self masking model.
Example 2. Consider the problem of recovering P(O; T )
given the m-graph Gin ﬁgure 1 and the missing data distri-
bution, P(T; O; Ro). Let both TandObe binary variables.
We will ﬁrst recover P(TjO)and then use it for recovering
P(O).
Using T? ?RojOand eq 1, P(TjO)can be recovered as,
P(TjO) =P(TjO; R o= 0) = P(TjO; Ro= 0)
Since the variables are binary, P(T) =P
OP(TjO)P(O)
yields the following equations:
P(t0) =P(t0jo0)P(o0) +P(t0jo)P(o)
P(t) =P(tjo0)P(o0) +P(tjo)P(o)
On substituting P(TjO)in the equations above with its re-
covered estimand we get,
P(t0) =P(t0jO= 0; r0
o)P(o0) +P(t0jO= 1; r0
o)P(o)
P(t) =P(tjO= 0; r0
o)P(o0) +P(tjO= 1; r0
o)P(o)
The two preceding equations constitute a system of equa-
tions in two unknowns: P(o0)andP(o). If the solution is
unique then P(O)is recoverable and is given by,
P(o0) =P(t0)P(t0jO= 1; Ro= 0)
P(t0jO= 0; r0o)P(t0jO= 1; r0o)
P(o) = 1P(t0)P(t0jO= 1; Ro= 0)
P(t0jO= 0; r0o)P(t0jO= 1; r0o)
P(T; O )can now be recovered as: P(TjO)P(O).
However, if the system of equations has inﬁnitely many
solutions then P(O)is non-recoverable. This can happen
when T? ?O. In this case Tprovides no information about
Oand hence, cannot be leveraged to recover P(O). This
is to be expected since we do not insist on faithfulness and
hence it is possible that an independence relation exist be-
tween two variable even when they are connected by an
edge.
We further note that it is impossible for the system to
have no solutions since it contradicts our assumption that
the graph and data are compatible (i.e. there exist param-
eterization(s) of the graph that generated the data as per
compatibility assumption).
Finally we note that as a result of recovering P(O; T ),
we can also recover another hard problem: P(Ojdo(t)), the
causal effect of treatment on outcome. Since Gis Markovian,
P(Ojdo(t)) = P(Ojt). Recoverability of P(Ojdo(t))thus
implicitly follows from that of P(O; T ).
Necessary and Sufﬁcient Conditions for
Recoverability in Hard Missing Data Problems
Notations
MZjW;MZ&Aug (MZjW;MW)
MZjW=P(ZjW)denotes ajZjjWjmatrix in which the
columns sum to one. For example if ZandWare binary,
X Y Rx(a)
X Y Rx
Z (b)
X Y Rx
Z
(c)
Figure 3: (a) P(X; Y )is not recoverable, (b) P(X; Y )may
be recoverable using graph and data, (c) P(X; Y )is recov-
erable.
then MZjW=
P(z0jw0)P(z0jw)
P(zjw0)P(zjw)
.MZ=P(Z)
denotes ajZj  1column matrix. Aug (MZjW; MW)
denotes ajZj(jWj+ 1) augmented matrix obtained by
appending the columns of matrices MZjWandMW.
The following theorem states the necessary and sufﬁcient
conditions for recoverability in hard missing data problems.
Theorem 1. Let m-graph Gand query P(W)be such that
the pair ( P(W); G) constitutes a hard missing data prob-
lem. Let P(V; VO; R)denote the distribution over missing
data and ZfVm; Vo; RgfW; R wg.
Given GandP(V; VO; R),P(W)is recoverable if
and only if P(ZjW)andP(Z)are recoverable and
rank(MZjW) = rank( Aug (MZjW; MW)) =jWj.
Proof: See Appendix.
The theorem makes no assumptions about the structure of
m-graph G. It is applicable to all hard MNAR problems and
not just to self-masking models. For example P(X; Y )and
the m-graph in ﬁgure 3 (a) and P(X; Y )and the m-graph
in ﬁgure 3 (b) constitute hard MNAR problems [Mohan and
Pearl, 2014a]. However in the case of the latter, theorem 1
can be used to recover P(X; Y )by leveraging Z.
Sufﬁcient Conditions for Recoverability in Any
Missing Data Problem
The following corollary states sufﬁcient conditions for re-
covering anygiven query P(W)using both graph and data.
Corollary 1. Given m-graph Gand missing data distribu-
tionP(V; Vo; R),P(W)is recoverable if for any Z
fVm; Vo; Rgf W; R wg,P(ZjW)andP(Z)are recover-
able and rank( MZjW) = rank( Aug (MZjW; MW)) =jWj.
P(X)and the m-graph in ﬁgure 3(c) do not constitute
a hard missing data problem. In fact, the query is recover-
able:P(X) =P
YP(XjY; R x= 0)P(Y)[Mohan et al. ,
2013]. However, P(X)can still be recovered using the pre-
ceding corollary by leveraging Z.XandZbeing binary
andYhaving a high cardinality, say jYj= 50 is an in-
stance where applying the corollary is more convenient from
a computational standpoint.
Remark 1. Although theorem 1 and corollary 1 aim to re-
cover P(W), they can also be used to recover P(Z; W )
since P(ZjW)is already known to be recoverable.
Scope of Results: Given a query graph pair ( Q; G ), ifQ
is not recoverable using Gthen theorem 1 presents neces-
sary and sufﬁcient conditions for its recoverability. If Qis
not recoverable using theorem 1 then we deem it as non-
recoverable. Corollary 1 shows that the recoverability tech-
nique in theorem 1 is applicable to simple missing data
problems such as MAR that are known to be recoverable
using graphs. However, there exists problems that cannot
be recovered using corollary 1 but can be recovered using
graphs. For example P(X; Y )cannot be recovered from
G:X Y!RXusing corollary 1 since X? ?Y. How-
everP(X; Y )is still recoverable as, P(XjRX= 0)P(Y)
[Mohan et al. , 2013].
The preceding recoverability procedures are inspired by
similar results in epidemiology (Rothman et al. [2008]), re-
gression analysis (Carroll et al. [2006]) and causal inference
(Pearl [2012]; Kuroki and Pearl [2014]). In contrast to Pearl
[2012] that relied on external studies to compute causal ef-
fect in the presence of an unmeasured confounder, Kuroki
and Pearl [2014] showed how the same could be effected
without external studies. In missing data settings we have
access to partial information that allows us to compute con-
ditional distributions. This allows us to adapt the procedure
in Pearl [2012] to compute consistent estimates as detailed
above. We further note that to the best of our knowledge pre-
vious work on self-masking models relied on parametric as-
sumptions (Mohan et al. [2018]; ?];?). In sharp contrast we
present a complete and non-parametric solution to handle all
hard problems.
Conclusions
In this work we eliminated a major deﬁciency in the ﬁeld
of missing data. We developed a sound, complete and non-
parametric technique to handle hard missing data problems.
Furthermore we showed that this technique is also applicable
to queries that are known to be recoverable using graphs.
Appendix
Proof of theorem 1.
Proof of theorem 1 relies on the following lemma that states
the conditions under which a system of linear equations is
consistent. [Cramer, 1750; Strang, 1993].
Lemma 1. The system of equations Ax =bwithmequa-
tions and nunknowns has (i) a unique solution if and only
ifrank (A) =rank (Aug (A; b)) = nand (ii) inﬁnite so-
lutions if and only if rank (A) =rank (Aug (A; b))< n.
(Proof of sufﬁciency) When the conditions in the theorem
are met the constraint P(Z) =P
WP(ZjW)P(W)yields
a unique solution as per lemma 1, thus establishing the re-
coverability of P(W).
(Proof of necessity) We need to show that for every ele-
mentZiin the power set of fVm; Vo; RgfW; R wg,P(W)is non-recoverable using Gand data if any of the follow-
ing hold: (i) P(Zi)is non-recoverable, (ii) P(ZijW)is non-
recoverable, (iii) rank( MZijW) = rank( Aug (MZijW; MW))
=jWjdoes not hold. Non-recoverability of P(Zi)implies
that its value is not unique i.e there exists at least two dis-
tinct distributions P1(Zi)andP2(Zi). For each of them we
can construct distinct distributions of P(W)using P(Zi) =P
WP(ZijW)P(W), thereby proving that P(W)is non-
recoverable. Similarly, we can show that P(W)is non-
recoverable when P(ZijW)is non-recoverable. In the case
of condition (iii) non-recoverability of P(W)follows from
lemma 1.
References
J Adams. Researching complementary and alternative
medicine . Routledge, 2007.
R J Carroll, D Ruppert, L A Stefanski, and C M Crainiceanu.
Measurement error in nonlinear models: a modern per-
spective . CRC press, 2006.
G Cramer. Introduction a l’analyse des lignes courbes alge-
briques par Gabriel Cramer... chez les freres Cramer &
Cl. Philibert, 1750.
R M Daniel, M G Kenward, S N Cousens, and B L
De Stavola. Using causal diagrams to guide analysis in
missing data problems. Statistical Methods in Medical
Research , 21(3):243–256, 2012.
A Darwiche. Modeling and reasoning with Bayesian net-
works . Cambridge University Press, 2009.
C.K. Enders. Applied Missing Data Analysis . Guilford
Press, 2010.
D Koller and N Friedman. Probabilistic graphical models:
principles and techniques . 2009.
M Kuroki and J Pearl. Measurement bias and effect restora-
tion in causal inference. Biometrika , 101(2):423–437,
2014.
K Mohan and J Pearl. Graphical models for recovering prob-
abilistic and causal queries from missing data. In Ad-
vances in NIPS 27 , pages 1520–1528. 2014.
K Mohan and J Pearl. On the testability of models with
missing data. Proceedings of AISTAT , 2014.
K Mohan and J Pearl. Graphical models for processing miss-
ing data. Technical report, Department of Computer Sci-
ence, University of California, Los Angeles, CA, 2018.
K Mohan, J Pearl, and J Tian. Graphical models for infer-
ence with missing data. In Advances in NIPS 26 , pages
1277–1285. 2013.
K Mohan, F Thoemmes, and J Pearl. Estimation with incom-
plete data: The linear case. In Proceedings of IJCAI-2018 ,
pages 5082–5088, 2018.
J W Osborne. Best practices in data cleaning: A complete
guide to everything you need to do before and after col-
lecting your data . Sage Publications, 2012.
J Pearl. Causality: models, reasoning and inference . Cam-
bridge Univ Press, New York, 2009.
J Pearl. On measurement bias in causal inference. arXiv
preprint arXiv:1203.3504 , 2012.
K J Rothman, S Greenland, and T L Lash. Modern epidemi-
ology . Lippincott Williams & Wilkins, 2008.
D.B. Rubin. Inference and missing data. Biometrika ,
63:581–592, 1976.
I Shpitser, K Mohan, and J Pearl. Missing data as a causal
and probabilistic problem. In Proceedings of UAI , 2015.
I Shpitser. Consistent estimation of functions of data missing
non-monotonically and not at random. In Advances in
NIPS , pages 3144–3152, 2016.
G Strang. Introduction to linear algebra , volume 3.
Wellesley-Cambridge Press Wellesley, MA, 1993.
O Sverdlov. Modern adaptive randomized clinical trials:
statistical and practical aspects . Chapman and Hall/CRC,
2015.


Estimation with Incomplete Data: The Linear Case.

Estimation with Incomplete Data: The Linear Case
Karthika Mohan1, Felix Thoemmes2andJudea Pearl3
1University of California, Berkeley
2Cornell University
3University of California, Los Angeles
karthika@eecs.berkeley.edu, felix.thoemmes@cornell.edu, judea@cs.ucla.edu
Abstract
Traditional methods for handling incomplete data,
including Multiple Imputation and Maximum Like-
lihood, require that the data be Missing At Ran-
dom (MAR). In most cases, however, missingness
in a variable depends on the underlying value of
that variable. In this work, we devise model-based
methods to consistently estimate mean, variance
and covariance given data that are Missing Not At
Random (MNAR). While previous work on MNAR
data require variables to be discrete, we extend the
analysis to continuous variables drawn from Gaus-
sian distributions. We demonstrate the merits of our
techniques by comparing it empirically to state of
the art software packages.
1 Introduction
Incomplete (or missing) data are data in which values of one
or more variables are missing. Almost all existing techniques
for handling incomplete data employ maximum likelihood or
multiple imputation methods to ﬁll in the missing values and
estimate parameters of interest. To guarantee convergence
and consistency, these techniques require that the missing
data mechanism be ignorable ( [Rubin, 1976 ]) i.e. the causes
of missingness be either random or fully observed.
In practice, however, missingness is almost always caused
by variables that are themselves afﬂicted by missingness
([Osborne, 2014; Sverdlov, 2015; Adams, 2007; van Stein
and Kowalczyk, 2016 ]). Such data are called Missing Not
At Random (MNAR). Among all MNAR problems, the most
frequently encountered case is that of a variable causing its
own missingness which we call self-masking MNAR. It is
discussed in almost all literature on missing data including
[Koller and Friedman, 2009 ](chapter 19) and [Darwiche,
2009 ](chapter 17). Examples include smokers not answer-
ing questions about their smoking behavior in insurance ap-
plications, longitudinal studies with attrition ( [Little, 1995 ]),
people with high income not revealing their incomes and a
general reluctance to answer questions about sensitive topics
such as religion, sexual preference and abortion ( [Greenland
and Finkle, 1995 ]).
While there has been some recent work ( [Daniel et al.,
2012; Mohan et al., 2013; Mohan and Pearl, 2014a; Mohan
X Y R x
X*X Y R x
X*X Y R x
X*Ry Y*
(a) (b) (c)Figure 1: Examples of (a) MCAR, (b) MAR and (c) MNAR models
and Pearl, 2018; Thoemmes and Mohan, 2015; Shpitser et
al., 2015 ]) on estimation in MNAR data with discrete vari-
ables, to the best of our knowledge there exists no theoreti-
cally sound and empirically efﬁcient graph based procedure
for handling MNAR missingness in datasets with continuous
variables.
In this paper we focus on MNAR problems in linear sys-
tems. As in [Mohan and Pearl, 2014b ]and [Mohan and
Pearl, 2018 ], we treat missing data as a causal inference prob-
lem. We present sound, graph-based procedures for recover-
ing (i.e. consistently estimating) parameters such as mean,
variance and covariance. In the following section we review
missingness graphs and structural equation models.
2 Preliminaries
2.1 Missingness Graphs: Notations and
Terminology
LetG(V;E)be the causal DAG where V=Vo[Vm[
U[V[R. Nodes in the graph correspond to variables in
the data set. Eis the set of edges in the DAG. Vois the set of
variables that are observed in all records in the population and
Vmis the set of variables that have missing values in at least
one record. Variable Xis termed as fully observed ifX2
Voandpartially observed ifX2Vm.RviandV
iare two
variables associated with every partially observed variable,
whereV
iis a proxy variable that is actually observed, and
Rvirepresents the status of the causal mechanism responsible
for the missingness of V
i; formally,
v
i=f(rvi;vi) =
vi ifrvi= 0
m ifrvi= 1(1)Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5082
Vis the set of all proxy variables and Ris the set of all
missingness mechanisms. Uis the set of unobserved nodes
(or latent variables). We use bi-directed edges as a shorthand
notation to denote the existence of a Uvariable as common
parent of two variables in Vo[Vm[R. For any WVo,
Rw=;. Unless stated otherwise it is assumed that no vari-
able inV[Uis a child of an Rvariable. This graphical rep-
resentation is called Missingness Graph (orm-graph) ( [Mo-
hanet al., 2013 ]). An m-graph portrays Missing Completely
At Random (MCAR) missingness if (Vm;Vo;U)? ?R, Miss-
ing At Random (MAR) if (Vm;U)? ?RjVoand Missing Not
At Random (MNAR) if neither MAR nor MCAR hold. Fig-
ure 1 (a), (b) and (c) exemplify MCAR, MAR and MNAR
missingness respectively.
Proxy variables may not always be explicitly shown in m-
graphs in order to keep the ﬁgures simple and clear. Con-
ditional Independencies are read off the graph using the d-
separation1criterion [Pearl, 2009 ]. Before formally deﬁning
the linear missingness model, we shall brieﬂy review Struc-
tural Equation Models. For a detailed discussion see [Pearl,
2009 ](chapter 5) and [Brito, 2004 ].
2.2 Structural Equation Models
A structural equation model (SEM) is a system of equations
deﬁned over a set of variables, such that each variable appears
on the left hand side of at most one equation. Each equation
describes the dependence of one variable in terms of the oth-
ers and contains an error term to account for the inﬂuence of
unobserved factors. Example: X=xandY=X +y.
As in [Pearl, 2013 ], we interpret structural equations as an as-
signment process whose directionality is captured by a path
diagram (see Figure 2). In this work all substantive vari-
ables (fVm[Vo[Ug) and error terms are assumed to be
drawn from a Gaussian distribution. Linear Structural Equa-
tion Modeling is widely used for estimating parameters of in-
terest given data that are missing at random ( [Allison, 2003;
Graham, 2003; Ullman and Bentler, 2003; Enders, 2006;
Schlomer et al., 2010 ]).
2.3 Recoverability
Deﬁnition 1 (Recoverability of target quantity Q([Mohan et
al., 2013 ])).LetAdenote the set of assumptions about the
data generation process and let Qbe any functional of the un-
derlying distribution P(Vm;VO;R).Qis recoverable if there
exists an algorithm that computes a consistent estimate of Q
for all strictly positive manifest distributions P(V;Vo;R)
that may be generated under A.
We present an example of recoverability in linear mod-
els in section 3.1. For examples of recoverability in non-
parametric models with discrete variables, please see [Mohan
et al., 2013 ].
3 Quasi-linear Missingness Model
The causal missingness mechanism is a binary variable and
as such the function generating it cannot be linear. The quasi-
linear model deﬁned below captures the missingness process.
1For a quick introduction to d-separation see,
http://www.dagitty.net/learn/dsep/index.html
(i)X1=x1
(ii)Y=X 1+y
(iii)X2=
Y+X1+x2
(iv)Ry=f(Y;ry)
(v)Y= (1Ry)Y+mRy
Figure 2: Quasi-linear missingness model and equations constituting
the SEM corresponding to it.
Deﬁnition 2. A Quasi-linear Missingness Model is a Struc-
tural Equation Model such that:
1. every substantive variable Xis a linear function of its
causes ,Y , and a random error term x
X=1Y1+2Y2+:::+nYn+x
The coefﬁcient ’s are called path coefﬁcients or structural
parameters.
2. For every Rx2R,Rx=f(Z;Rx)whereZis the set
of causes and fis a non-linear function. No Rvariable is a
parent of any substantive variable.
3. Every proxy variable Xis generated by the non-linear
function:X= (1Rx)X+mRx
Figure 2 and equations (i) to (v) exemplify an m-graph and
its corresponding quasi-linear missingness model. Path co-
efﬁcients can be identiﬁed in linear models by applying cri-
teria such as single door and back door (see [Pearl, 2009 ]
(chapter 5)). In the following lemma we rephrase and state
a basic result in linear path analysis that links covariance
with path coefﬁcients ( [Pearl, 2013; Pearl, 2009; Brito, 2004;
Wright, 1921 ]).
Lemma 1. LetGbe an m-graph with kunblocked paths
p1;::pkbetweenXandY. LetApibe the ancestor of all
nodes on path pi. Let the number of nodes on pibenpi.
cov(X;Y ) =kX
i=1var(Api)npi1Y
j=1pi
j (2)
wherenpi1Q
j=1pi
jis the product of all causal parameters on
pathpi.
For example, in ﬁgure 2, there exist two paths, X1!Y!
X2andX1!X2, betweenX1andX2. On applying lemma
1 we get,cov(X1;X2) =
var (X1) +var (X1).
In the following subsection we will exemplify a novel path
analytic procedure for consistent estimation of the covariance
matrix given MAR data.
3.1 Recoverability of Covariance Matrix: An
Example
For any target quantity Qwe useQkRX=0to denote: compute
Qfrom samples in which all variables in Xare observed.
Example 1. Consider the problem of estimating the covari-
ance matrix given the MAR model of Figure 1 (b). Yis fully
observed and hence var(Y)is trivially recoverable. In orderProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5083
to recovercov(X;Y ), we will ﬁrst recover XY, the regres-
sion coefﬁcient of XonY. SinceX? ?RxjYwe have the
license to compute XY(using OLS) from samples in which
Xis observed i.e. XY =XYkRx=0.cov(X;Y )can now
be recovered as:
cov(X;Y ) =XYkRx=0var(Y) (3)
To recovervar(X), we apply the law of total variance:
var(X) =E(var(XjY)) +var(E(XjY))
Recovering var(E(XjY)):E(XjY) =XYY+cx
whereXYandcxdenote the slope and intercept of
the regression line. Since X? ?RxjY,E(XjY) =
E(XjY)kRX=0. Therefore,
var(E(XjY)) =var
XYkRx=0Y+cykRx=0
= (XYkRx=0)2var(Y) (4)
Recovering E(var(XjY)): The variance of a condi-
tional gaussian distribution is constant. Therefore,
E(var(XjY)) =var(XjY)
Variance of a conditional bivariate Gaussian distribu-
tion,var(XjY), is given by var(X)(12), where
=cov(X;Y )
(var (X)var (Y))1
2denotes the correlation coefﬁ-
cient. Since X? ?RxjY,var(XjY) =var(XjY)kRX=0.
Hence,
var(XjY) =var(X)kRx=0(1(kRx=0)2) (5)
Using 4 and 5 var(X)is computed as,
var(X) =var(X)kRx=0(1(kRx=0)2)
+ (XYkRx=0)2var(Y)
Note that while all factors in the preceding estimand, except
var(Y)are estimated from samples in which Xis observed,
var(Y)is recovered from all samples in the dataset, includ-
ing those in which Xis missing. In other words, although
a consistent estimate of var(X)cannot be computed directly
from fully observed data (i.e. P(X;Y;Rx= 0) ), it can be
recovered by a procedure in which each factor in the estimand
is estimated from subsets of the available data.
We further note that as a consequence of recovering var(X),
YX, the causal effect of XonYcan be recovered as,
cov(X;Y )
var(X)=XYkRx=0var(Y)
var(X)kRx=0(1(kRx=0)2)+(XYkRx=0)2var(Y).
The following section presents procedures for computing
parameters of interest in quasi linear models.
4 Recovering Mean, Variance and Covariance
Theorem 1 (Recovering Mean of Partially Observed Vari-
ables). LetfX;Zg Vm[Vo,X\Z=;,jXj= 1.
Given m-graph G,E(X)is recoverable if there exists Z=
fZ1;Z2;:::Zngsuch thatX? ?RxRzjZandE(Zi)is recov-
erable for all Zi2Z. If recoverable, E(X)is given by
E(X) =ckRxz=0+nX
i=1ikRxz=0E(Zi) (6)
wherecandi’s denote the intercept and coefﬁcients of the
regression line of XonZ.Proof. LetX=1Z1+2Z2+:::+nZn+cbe the re-
gression line of XonZ. SinceX? ?RxRzjZ,E(XjZ) =
E(XjZ)kRxz=0. Hence,
E(XjZ) =1kRxz=0Z1+:::+nkRxz=0Zn+ckRxz=0
However since E(X) =E(E(XjZ))we can write,
E(X) =E
1kRxz=0Z1+:::+nkRxz=0Zn+ckRxz=0
=ckRxz=0+nX
i=1ikRxz=0E(Zi)
Theorem 2 (Recovering covariance of XandY).Let
fX;Y;ZgVm[Vo,X\Y\Z=;,jXj= 1;jYj=
1andjZj=n. Given m-graph G,cov(Y;X )is recov-
erable if there exists Z=fZ1;Z2;:::Zngsuch that (i)
Y? ?Rx;Ry;RzjX;Z , and (ii)E(X);E(Y);var (X);and
8i E(Zi)andcov(X;Zi)are recoverable. If recoverable,
cov(Y;X )is given by
cov(Y;X ) =xkRxyz=0(var(X) +E(X)2) +ckRxyz=0E(X)
+X
iikRxyz=0(cov(XZi) +E(X)E(Zi))E(X)E(Y)
wherecand’s denote the intercept and coefﬁcients of the
regression line of YonX;Z .
Proof.
cov(Y;X ) =E(E(XYjZ))E(Y)E(X)
=E(XE(YjZ;X ))E(Y)E(X)
UsingY? ?Rx;Ry;RzjX;Z , we get
E(YjX;Z ) =E(YjX;Z )kRxyz=0. Therefore,
E(XE(YjX;Z )) =E(X
xkRxyz=0X+1kRxyz=0Z1
+2kRxyz=0Z2+:::+kkRxyz=0Zk+ckRxyz=0
=xkRxyz=0E(X2) +ckRxyz=0E(X) +X
iikRxyz=0E(XZi)
=xkRxyz=0(var(X) +E(X)2) +ckRxyz=0E(X)
+X
iikRxyz=0(cov(XZi) +E(X)E(Zi))
A well known and widely used property of Gaussian dis-
tributions is that their conditional variances are constant. Let
jXj=mandjYj=n. LetMxx;MyyandMxydenote the
covariance matrix of X,YandXandYrespectively. Vari-
ance of the conditional Gaussian distribution f(YjX)is given
by,
Var (YjX) =MyyM0
xyM1
xxMxy (7)
Theorem 3 (Recovering variance of a partially observed vari-
ableX).LetfX;ZgVm[Vo,X\Z=;,jXj= 1
andjZj=n. Given m-graph G,var(X)is recoverable ifProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5084
there existsZ=fZ1;Z2;:::Zngsuch thatX? ?RxRzjZand
cov(Zi;Zj)is recoverable for all Zi;Zj2Z. If recoverable,
var(X)is given by
var(X) = (MxxM0
zxM1
zzMzx)kRxz=0
+X
i2
ikRxz=0var(Zi) +X
i6=j(ij)kRxz=0cov(Zi;Zj)
wherei’s denote the coefﬁcients of the regression line of X
on Z.
Proof. We ﬁrst apply the law of total variance: var(X) =
E(var(XjZ)) +var(E(XjZ)), and then prove recoverabil-
ity ofvar(X)by showing that both summands are recover-
able.
RecoveringE(var(XjZ)): SinceXandZi2Z,8iare
Gaussian variables, var(XjZ)is constant; therefore,
E(var(XjZ)) =var(XjZ)
=var(XjZ)kRxz=0(sinceX? ?Rx;RzjZ)
= (MxxM0
zxM1
zzMzx)kRxz=0(using eqn 7)
Recoveringvar(E(XjZ)): SinceX? ?Rx;RzjZwe have,
E(XjZ) =E(XjZ)kRxz=0=cxkRxz=0+X
iikRxz=0Zi
var(E(XjZ)) =var
ckRxz=0+X
iikRxz=0Zi
=X
i2
ikRxz=0var(Zi) + 2X
i6=j(ij)kRxz=0cov(Zi;Zj)
Lemma 2. [Recovering Partial Regression Coefﬁcients] Let
fX;Y;Zg Vo[Vm. Given m-graph Gand missing
dataD, partial regression coefﬁcient XY:Z is recoverable
ifX? ?RX;RY;RZjY;Z and is given by XY:ZkRxyz=0.
Notice that recoverability of E(X)using theorem 1 is con-
tingent upon the recoverability of E(Zi), for alli. Clearly,
to recoverE(X)we should recover all E(Zi)’s ﬁrst. Similar
is the case with theorems 2 and 3. In the case of recover-
ability of conditional distributions in datasets with discrete
variables, [Mohan et al., 2013 ]deﬁned the notion of ordered
factorization to sequence the recoverability procedure. In the
following theorem we adapt the idea of ordered factorization
in[Mohan et al., 2013 ]to formulate a sufﬁcient condition for
recovering mean.
Theorem 4. LetY[fZgVm[Voand letO:Y1<Y 2<
:::<Z =Ykbe an ordered set of all variables in Y[fZg.
LetXifY 1;:::;Yi1g,1ik.
Given an m-graph G, a sufﬁcient condition for recov-
eringE(Z)is that there exist OandXi8isuch that
Yi? ?(Ryi;Rxi)jXi.
Proof. Recoverability follows from theorem 1. We pro-
ceed by ﬁrst recovering E(Y1)and then recovering
E(Y2),..,E (Yk)sequentially in the order dictated by O.Example 2. Consider the problem of recovering E(X)and
E(Y)given the m-graph Gin ﬁgure 1 (c) and ordering
O:Y <X . OrderingOdirects us to ﬁrst recover E(Y)and
then recover E(X). SinceY? ?RyinG, we can apply theo-
rem 1 to recover E(Y)asE(Y)kRy=0. SinceX? ?Rx;RyjY
andE(Y)is recoverable, we can apply theorem 1 to recover
E(X)asckRxy=0+1kRxy=0E(Y)kRy=0.
In a similar manner, ordered recoverability procedures can
be extended to recover covariance and variance as well. In the
case of MCAR data, all orderings of variables will guarantee
recoverability, whereas in the case of MAR data, orderings in
which all fully observed variables precede partially observed
variables will guarantee recoverability. Finally in the case of
MNAR data, the ordering is determined by graph structure,
and heuristics for ﬁnding admissible orders are discussed in
[Mohan et al., 2013 ].
5 Extending Recoverability to Complex
MNAR Models
In this section we focus on m-graphs in which conditional
independence between a variable and its missingness mech-
anism (such as Y? ?RyandYi? ?(Ryi;Rxi)jXi) that are re-
quired by theorems 1, 2, 3 and 4 does nothold in the graph.
In the following subsection we present a theorem for recov-
eringE(Y)for anyY2Vm, by leveraging X, a neighbor of
Y.
5.1 Recovering E(Y)WhenYandRyare Not
Separable
Theorem 5 (Recovering E(Y)for anyY2Vm).Let
X2Vm[Vobe a neighbor of Y. Given m-graph Gand
missing data D,E(Y)is recoverable if E(X)is recover-
able and there exists ZVm[VofX;Ygsuch that
X? ?Rx;Ry;RzjY;Z andE(Zi)is recoverable for all i. If
recoverable, E(Y)is given by,
E(Y) =E(X)cxkRxyz=0Pk
i=1ikRxyz=0E(Zi)
ykRxyz=0(8)
where’s, andcdenote the coefﬁcients and intercept of the
regression line of X on Z and Y.
Proof: See Appendix A
Example 3. To recoverE(Y)given the m-graph in ﬁgure 2,
we leverage X1.X1? ?RyjYandE(X1)is recoverable since
X12VO. Hence mean of Ycan be recovered using theorem
5 as,E(X1)ckRy=0
X1;YkRy=0.
5.2 Recovering var(Y)WhenYandRyare Not
Separable
Algorithm 1 recovers variance of a variable Ygiven an m-
graph in which X1is a parent of YandX2is a child of Y. It
uses two subroutines (outlined in appendix B) that compute
path coefﬁcient and covariance using lemma 1. If a quantity
of interestQis recoverable, then Qis used as a shorthand for
the recovered estimand. For example if var(Y)is recoverableProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5085
Figure 3: (a) Self masking model in which all variables are partially
observed, yet E(Y)andvar(Y)are recoverable. (b) m-graph con-
structed from (a) by treating Z; Y; R zandRwas latent.
Algorithm 1 RecoverVariance( Y;G;X 1;X2)
Input:Y: variable whose variance is to be recovered. G:
Markovian m-graph in which X1is a parent of YandX2is a
child ofY
Output:var(Y)ifvar(Y)is recoverable
NULL ifvar(Y)is not recoverable
1:ifvar(Y)is recoverable using theorem 3 then
2: Recover estimand var(Y)
3: returnvar(Y)
4:ifX1;Yis recoverable by lemma 2 then
5: Recover estimand X1;Y
6:else return NULL
7:y Recovery(G;Y;X 1;X2)
8:ify== NULL then return NULL
9:cov(X1;Y) Recover cov(G;Y;X 1;y)
10:ifcov(X1;Y)==NULL then return NULL
11:returncov(X1;Y)
X1;Y
thenvar(Y)denotes the recovered estimand. We exemplify
below the recovery procedure using algorithm 1.
Example 4. Consider the problem of recovering mean and
variance of Ygiven the m-graph in ﬁgure 2. We will show
thatvar(Y)is recoverable using algorithm 1.
Steps 1-3: Since YandRyare neighbors theorem 3 is not
applicable. Hence we proceed to the next step.
Steps 4-6: SinceX 1? ?RyjY,X1;Yis recoverable using
lemma 2 i.e. X1;Y=X1;YkRy=0.
Steps 7-8: We invoke subroutine Recover yto recover the
path coefﬁcient in ﬁgure 2. Since X1andX2are fully ob-
served,cov(X1;X2)is recoverable. Path coefﬁcients and

may be recovered as:
=X2X1:Y(by single door criterion, [Pearl, 2009 ])
=X2X1:YkRy=0(using lemma 2)

=X2Y:X 1(by back door criterion, [Pearl, 2009 ])
=X2Y:X 1kRy=0(using lemma 2)
SinceX1is fully observed, var(X1)is recoverable. On ap-
plying lemma 1 we get,
cov(X2;X1) =
var (X1) +var (X1)
Therefore,=1
X2Y:X 1kRy=0cov(X2;X1)
var(X1)X2X1:YkRy=0Steps 9-10: We invoke subroutine Recover cov to recover
cov(X1;Y). On applying lemma 1 we get: cov(X1;Y) =
X1Yvar(X1) = (X1Y)kRy=0var(X1).
Step 11:var(Y)is recovered as:
var(Y) =cov(X1;Y)
X1Y=var(X1)
(X1Y)kRy=0
=var(X1)
(X1Y)kRy=01
X2Y:X 1kRy=0
cov(X2;X1)
var(X1)X2X1:YkRy=0
While algorithm 1 currently handles only Markovian
graphs, we note that it is possible to extend the algorithm
to Semi-Markovian models provided we make additional as-
sumptions (pertaining to variances of latent variables). We
further note that suitable candidates for X1andX2are the
non-descendants and descendants of Y, respectively. Latent
projection ( [Pearl, 2009 ], chapter 2) of the input graph con-
structed by treating all intermediate nodes on unblocked paths
betweenX1andY, andX2andY, as latent can yield graphs
compatible with the requirements of algorithm 1. When la-
tent projection does not introduce bi-directed edges, recovery
is straight forward. We exemplify this in section 6.
6 Empirical Evaluation
We denominate the graph based recovery procedure presented
in this paper as Model Based Estimation (MBE) and evaluate
MBE by simulating partially observed datasets from missing-
ness graphs and estimating their parameters from the incom-
plete data. We compare our estimates against those yielded by
state of the art packages for SEM that apply Multiple Impu-
tation (MI) (using mice package in R) and Maximum Likeli-
hood (ML) (using lavaan in R) techniques [Schminkey et al.,
2016; Enders, 2006 ]. Parameters are evaluated in terms of
mean squared error and KL Divergence between original and
learned distributions.
Missing At Random: We generate data according to the fol-
lowing model and evaluate the performance of MBE, MI and
FIML in terms of Mean Squared Error (MSE) and time taken
to compute mean of X.
X=x;
Ry=f(Y1;Y2;:::Yk;ry)
Yi=iX+y;1ik
The ﬁrst experiment measures the mean squared errors of the
three estimators: MI, ML, MBE, and studies how it varies
with increase in sample size, for the problem of recovering
mean (X)whenjYj=k= 5. The results of this experi-
ment are plotted in ﬁgure 4. We further estimated the average
time each procedure took in recovering and plotted time vs
sample size in ﬁgure 5. Figure 4 shows that all three meth-
ods (MBE, MI, FIML) work almost identically so far as the
quality of estimate of E(X)is concerned. From ﬁgure 5 it
is clear that MBE (a non-iterative procedure) takes less time
in computing E(X)as compared to MI and FIML. While all
the reported numbers (MSE and time) are averaged over 50
repetitions with different random estimation problems, for aProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5086
Figure 4: MSE of E(X)vs Sample Size for MAR: X? ?RxjY,
whenjYj= 5
Figure 5: Time to recover E(X)vs Sample Size for MAR:
X? ?RxjY, when jYj= 5
given problem each individual MSE was computed from 500
simulations. In the next experiment we study the efﬁciency
of these procedures as the complexity of the model increases.
In MAR models, complexity of recoverability depends on the
size of the separating set Ythat d-separates XfromRx(and
on the size of the dataset). This experiment was conducted by
ﬁxing sample size to 100,000. We observe in Figure 6 that as
the separating set becomes larger, the time taken to recover
estimates also increases. Note that in this case the time gain
pertains to computing parameters of one partially observed
variableX. Clearly, in a real world dataset with several par-
tially observed variables, the time savings offered by MBE
will be substantial.
Missing Not At Random: The goal here is to evaluate the
Figure 6: Time to recover E(X)vsjYjfor MAR: X? ?RxjY
Figure 7: KL divergence vs Sample size for self masking model
effectiveness of algorithm 1 using data generated by the self
masking model in 3. wyandzy are not recoverable by
lemma 2; hence ZandWcannot be used for recovering
var(Y). However, x1yis recoverable. We can create a
latent projection by treating Z;W;RzandRwas latent vari-
ables as shown in ﬁgure 3 (b). Figure 7 depicts the KL diver-
gence between true and estimated distribution of Y. As ex-
pected, FIML and MI behave unpredictably while MBE with
minimum KL Divergence behaves ideally.
7 Conclusions
We presented novel graph-based procedures that are non-
iterative and independent of likelihood function, for recov-
ering parameters in quasi-linear missingness models. We fur-
ther developed procedures for recovering parameters in self
masking models. Finally we showed that given MAR data,
our techniques are much faster than state of the art procedures
and given MNAR data our techniques can recover parameters
where existing techniques fail.
A Proof of theorem 5
E(X) =E(E(XjZ;Y )) =E(E(XjZ;Y )kRxyz=0)
=ykRxyz=0E(Y) +Pk
i=1ikRxyz=0E(Zi) +cxkRxyz=0
Therefore,E(Y) =E(X)cxkRxyz=0Pk
i=1ikRxyz=0E(Zi)
ykRxyz=0
B Subroutines used in Algorithm 1
In this subsection we outline the subroutines Recover
andRecovecovinvoked by algorithm 1. Recoverability of
queries in these subroutines are determined using results in
section 4. Note that in this work we only deal with recover-
ability of statistical parameters. Path coefﬁcients (which are
causal parameters) are considered recoverable if (i) they are
identiﬁable and (ii) all factors in the (identiﬁed) estimand are
recoverable.
Recovery:Input:G: m-graph in which X1is a parent
ofYandX2is a child of Y.y: Path coefﬁcient of edge
X1!Y.Output:yifyis recoverable, NULL other-
wise.
This routine applies lemma 1 on X1andX2to obtain,
cov(X1;X2) =kP
i=1var(Api)npi1Q
j=1pi
j. Ifcov(X1;X2),
var(Api)’s and allj’s (excluding y) are recoverable, theirProceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5087
recovered estimands are substituted into the equation above
to recover and return the estimand of y.
Recovercov:Input:G: m-graph in which X1is a parent
ofYandX2is a child ofY,y: Recovered estimand of y,
the path coefﬁcient of edge X1!Y.Output: Estimand for
cov(X1;Y)if it is recoverable, NULL otherwise.
This routine applies lemma 1 on X1andYand returns the
estimand for recovering cov(X1;Y)if all variances and path
coefﬁents on the RHS of equation 2 are recoverable.
References
[Adams, 2007 ]Jon Adams. Researching Complementary
and Alternative Medicine. Routledge, 2007.
[Allison, 2003 ]Paul D. Allison. Missing data techniques for
structural equation modeling. Journal of Abnormal Psy-
chology, 112(4):545, 2003.
[Brito, 2004 ]Carlos Brito. Graphical Models for Identiﬁca-
tion in Structural Equation Models. PhD thesis, University
of California Los Angeles, 2004.
[Daniel et al., 2012 ]Rhian M. Daniel, Michael G. Kenward,
Simon N. Cousens, and Bianca L. De Stavola. Using
causal diagrams to guide analysis in missing data prob-
lems. Statistical Methods in Medical Research, 21(3):243–
256, 2012.
[Darwiche, 2009 ]Adnan Darwiche. Modeling and Reason-
ing with Bayesian Networks. Cambridge University Press,
2009.
[Enders, 2006 ]Craig K. Enders. Analyzing structural equa-
tion models with missing data. In Gregory R. Hancock
and Ralph O. Mueller, editors, Structural Equation Mod-
eling: A Second Course, pages 315–344. Information Age
Publishing, Inc., 2006.
[Graham, 2003 ]John W. Graham. Adding missing-data-
relevant variables to FIML-based structural equation mod-
els.Structural Equation Modeling, 10(1):80–100, 2003.
[Greenland and Finkle, 1995 ]Sander Greenland and
William D. Finkle. A critical look at methods for handling
missing covariates in epidemiologic regression analyses.
American Journal of Epidemiology, 142(12):1255–1264,
1995.
[Koller and Friedman, 2009 ]Daphne Koller and Nir Fried-
man. Probabilistic Graphical Models: Principles and
Techniques. The MIT Press, 2009.
[Little, 1995 ]Roderick J. A. Little. Modeling the drop-
out mechanism in repeated-measures studies. Journal of
the American Statistical Association, 90(431):1112–1121,
1995.
[Mohan and Pearl, 2014a ]Karthika Mohan and Judea Pearl.
Graphical models for recovering probabilistic and causal
queries from missing data. In C. Cortes M. Welling,
Z. Ghahramani and N. Lawrence, editors, Advances in
Neural Information Processing Systems 27, pages 1520–
1528. Curran Associates, 2014.
[Mohan and Pearl, 2014b ]Karthika Mohan and Judea Pearl.
On the testability of models with missing data. In S. Kaskiand J. Corander, editors, Proceedings of the Seventeenth
International Conference on Artiﬁcial Intelligence and
Statistics (AISTATS), volume 33, pages 643–650, 2014.
[Mohan and Pearl, 2018 ]Karthika Mohan and Judea Pearl.
Graphical models for processing missing data. arXiv
preprint arXiv:1801.03583, 2018.
[Mohan et al., 2013 ]Karthika Mohan, Judea Pearl, and Jin
Tian. Graphical models for inference with missing data.
In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani,
and K.Q. Weinberger, editors, Advances in Neural Infor-
mation Processing Systems 26, pages 1277–1285. Curran
Associates, 2013.
[Osborne, 2014 ]Jason W. Osborne. Best Practices in Logis-
tic Regression. SAGE Publications, 2014.
[Pearl, 2009 ]Judea Pearl. Causality: Models, Reasoning
and Inference. Cambridge Univ Press, New York, 2nd edi-
tion, 2009.
[Pearl, 2013 ]Judea Pearl. Linear models: A useful “micro-
scope” for causal analysis. Journal of Causal Inference,
1(1):155–170, 2013.
[Rubin, 1976 ]Donald B. Rubin. Inference and missing data.
Biometrika, 63:581–592, 1976.
[Schlomer et al., 2010 ]Gabriel L. Schlomer, Sheri Bauman,
and Noel A. Card. Best practices for missing data man-
agement in counseling psychology. Journal of Counseling
Psychology, 57(1):1, 2010.
[Schminkey et al., 2016 ]Donna L. Schminkey, Timo von
Oertzen, and Linda Bullock. Handling missing data with
multilevel structural equation modeling and full informa-
tion maximum likelihood techniques. Research in Nursing
& Health, 39(4):286–297, 2016.
[Shpitser et al., 2015 ]Iyla Shpitser, Karthika Mohan, and
Judea Pearl. Missing data as a causal and probabilistic
problem. In M. Meila and T. Heskes, editors, Proceedings
of the Thirty-First Conference on Uncertainty in Artiﬁcial
Intelligence, pages 802–811. AUAI Press, 2015.
[Sverdlov, 2015 ]Oleksandr Sverdlov. Modern Adaptive
Randomized Clinical Trials: Statistical and Practical As-
pects. Chapman and Hall/CRC, 2015.
[Thoemmes and Mohan, 2015 ]Felix Thoemmes and
Karthika Mohan. Graphical representation of miss-
ing data problems. Structural Equation Modeling: A
Multidisciplinary Journal, 22:631–642, 2015.
[Ullman and Bentler, 2003 ]Jodie B. Ullman and Peter M.
Bentler. Structural Equation Modeling. Wiley Online Li-
brary, 2003.
[van Stein and Kowalczyk, 2016 ]Bas van Stein and Wojtek
Kowalczyk. An incremental algorithm for repairing train-
ing sets with missing values. In International Confer-
ence on Information Processing and Management of Un-
certainty in Knowledge-Based Systems, pages 175–186.
Springer, 2016.
[Wright, 1921 ]Sewall Wright. Correlation and causation.
Journal of Agricultural Research, 20(7):557–585, 1921.Proceedings of the Twenty-Seventh International Joint Conference on Artiﬁcial Intelligence (IJCAI-18)
5088

Karthika Mohan
Felix Thoemmes
Judea Pearl