A survey on transfer learning

1
A Survey on Transfer Learning
Sinno Jialin Pan and Qiang Y ang Fellow, IEEE
Abstract —A major assumption in many machine learning and data mining algorithms is that the training and future data must be
in the same feature space and have the same distribution. However, in many real-world applications, this assumption may not hold.
For example, we sometimes have a classi ﬁcation task in one domain of interest, but we only have suf ﬁcient training data in another
domain of interest, where the latter data may be in a different feature space or follow a different data distribution. In such cases,knowledge transfer, if done successfully, would greatly improve the performance of learning by avoiding much expensive data labelingefforts. In recent years, transfer learning has emerged as a new learning framework to address this problem. This survey focuses on
categorizing and reviewing the current progress on transfer learning for classi ﬁcation, regression and clustering problems. In this survey,
we discuss the relationship between transfer learning and other related machine learning techniques such as domain adaptation, multi-task learning and sample selection bias, as well as co-variate shift. We also explore some potential future issues in transfer learning
research.
Index Terms —Transfer Learning, Survey, Machine Learning, Data Mining.
✦
1I NTRODUCTION
Data mining and machine learning technologies have already
achieved signi ﬁcant success in many knowledge engineering
areas including classi ﬁcation, regression and clustering (e.g.,
[1], [2]). However, many machine learning methods work wellonly under a common assumption: the training and test data aredrawn from the same feature space and the same distribution.
When the distribution changes, most statistical models need to
be rebuilt from scratch using newly collected training data. Inmany real world applications, it is expensive or impossible tore-collect the needed training data and rebuild the models. Itwould be nice to reduce the need and effort to re-collect the
training data. In such cases, knowledge transfer ortransfer
learning between task domains would be desirable.
Many examples in knowledge engineering can be found
where transfer learning can truly be bene ﬁcial. One example
is Web document classi ﬁcation [3], [4], [5], where our goal
is to classify a given Web document into several prede ﬁned
categories. As an example in the area of Web-documentclassi ﬁcation (see, e.g., [6]), the labeled examples may be
the university Web pages that are associated with categoryinformation obtained through previous manual-labeling efforts.For a classi ﬁc a t i o nt a s ko nan e w l yc r e a t e dW e bs i t ew h e r et h e
data features or data distributions may be different, there maybe a lack of labeled training data. As a result, we may not beable to directly apply the Web-page classi ﬁers learned on the
university Web site to the new Web site. In such cases, it wouldbe helpful if we could transfer the classi ﬁcation knowledge
into the new domain.
The need for transfer learning may arise when the data can
be easily outdated. In this case, the labeled data obtained inone time period may not follow the same distribution in alater time period. For example, in indoor WiFi localization
Department of Computer Science and Engineering, Hong Kong University of
Science and Technology, Clearwater Bay, Kowloon, Hong Kong
Emails: {sinnopan, qyang }@cse.ust.hkproblems, which aims to detect a user’s current location based
on previously collected WiFi data, it is very expensive tocalibrate WiFi data for building localization models in a large-scale environment, because a user needs to label a large
collection of WiFi signal data at each location. However, the
WiFi signal-strength values may be a function of time, deviceor other dynamic factors. A model trained in one time periodor on one device may cause the performance for locationestimation in another time period or on another device to bereduced. To reduce the re-calibration effort, we might wish to
adapt the localization model trained in one time period (the
source domain) for a new time period (the target domain), orto adapt the localization model trained on a mobile device (thesource domain) for a new mobile device (the target domain),as done in [7].
As a third example, consider the problem of sentiment
classi ﬁcation, where our task is to automatically classify the
reviews on a product, such as a brand of camera, into positiveand negative views. For this classi ﬁcation task, we need to
ﬁrst collect many reviews of the product and annotate them.
We would then train a classi ﬁer on the reviews with their
corresponding labels. Since the distribution of review dataamong different types of products can be very different, to
maintain good classi ﬁcation performance, we need to collect
a large amount of labeled data in order to train the review-classi ﬁcation models for each product. However, this data-
labeling process can be very expensive to do. To reduce the
effort for annotating reviews for various products, we may
want to adapt a classi ﬁcation model that is trained on some
products to help learn classi ﬁcation models for some other
products. In such cases, transfer learning can save a signi ﬁcant
amount of labeling effort [8].
In this survey article, we give a comprehensive overview of
transfer learning for classi ﬁcation, regression and clustering
developed in machine learning and data mining areas. Therehas been a large amount of work on transfer learning forreinforcement learning in the machine learning literature (e.g.,
Digital Object Indentifier 10.1109/TKDE.2009.191 1041-4347/$25.00 ©  2009 IEEEIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
2
[9], [10]). However, in this paper, we only focus on transfer
learning for classi ﬁcation, regression and clustering problems
that are related more closely to data mining tasks. By doingthe survey, we hope to provide a useful resource for the datamining and machine learning community.
The rest of the survey is organized as follows. In the next
four sections, we ﬁrst give a general overview and de ﬁne
some notations we will use later. We then brie ﬂy survey the
history of transfer learning, give a uni ﬁed de ﬁnition of transfer
learning and categorize transfer learning into three differentsettings (given in Table 2 and Figure 2). For each setting, wereview different approaches, given in Table 3 in detail. Afterthat, in Section 6, we review some current research on thetopic of “negative transfer”, which happens when knowledgetransfer has a negative impact on target learning. In Section 7,we introduce some successful applications of transfer learningand list some published data sets and software toolkits fortransfer learning research. Finally, we conclude the article witha discussion of future works in Section 8.
2O VERVIEW
2.1 A Brief History of Transfer Learning
Traditional data mining and machine learning algorithms make
predictions on the future data using statistical models that aretrained on previously collected labeled or unlabeled trainingdata [11], [12], [13]. Semi-supervised classi ﬁcation [14], [15],
[16], [17] addresses the problem that the labeled data may
be too few to build a good classi ﬁer, by making use of a
large amount of unlabeled data and a small amount of labeleddata. Variations of supervised and semi-supervised learningfor imperfect datasets have been studied; for example, Zhuand Wu [18] have studied how to deal with the noisy class-label problems. Yang et al. considered cost-sensitive learning
[19] when additional tests can be made to future samples.Nevertheless, most of them assume that the distributions ofthe labeled and unlabeled data are the same. Transfer learning ,
in contrast, allows the domains, tasks, and distributions used
in training and testing to be different. In the real world, weobserve many examples of transfer learning. For example,we may ﬁnd that learning to recognize apples might help to
recognize pears. Similarly, learning to play the electronic organmay help facilitate learning the piano. The study of Transfer
learning is motivated by the fact that people can intelligently
apply knowledge learned previously to solve new problemsfaster or with better solutions. The fundamental motivationforTransfer learning in the ﬁeld of machine learning was
discussed in a NIPS-95 workshop on “Learning to Learn”
1, which focused on the need for lifelong machine-learning
methods that retain and reuse previously learned knowledge.
Research on transfer learning has attracted more and
more attention since 1995 in different names: learning tolearn, life-long learning, knowledge transfer, inductive trans-
fer, multi-task learning, knowledge consolidation, context-
sensitive learning, knowledge-based inductive bias, meta learn-ing, and incremental/cumulative learning [20]. Among these,
1. http://socrates.acadiau.ca/courses/comp/dsilver/NIPS95 LTL/
transfer.workshop.1995.htmla closely related learning technique to transfer learning is
the multi-task learning framework [21], which tries to learn
multiple tasks simultaneously even when they are different.A typical approach for multi-task learning is to uncover thecommon (latent) features that can bene ﬁt each individual task.
In 2005, the Broad Agency Announcement (BAA) 05-29
of Defense Advanced Research Projects Agency (DARPA)’sInformation Processing Technology Of ﬁce (IPTO)
2gave a
new mission of transfer learning: the ability of a system torecognize and apply knowledge and skills learned in previoustasks to novel tasks. In this de ﬁnition, transfer learning aims
to extract the knowledge from one or more source tasks and
applies the knowledge to a target task . In contrast to multi-task
learning, rather than learning all of the source and target taskssimultaneously, transfer learning cares most about the targettask. The roles of the source and target tasks are no longersymmetric in transfer learning.
Figure 1 shows the difference between the learning
processes of traditional and transfer learning techniques. Aswe can see, traditional machine learning techniques try to learneach task from scratch, while transfer learning techniques tryto transfer the knowledge from some previous tasks to a targettask when the latter has fewer high-quality training data.
(a) Traditional Machine Learning
 (b) Transfer Learning
Fig. 1. Different Learning Processes between TraditionalMachine Learning and Transfer Learning
Today, transfer learning methods appear in several top
venues, most notably in data mining (ACM KDD, IEEE ICDMand PKDD, for example), machine learning (ICML, NIPS,ECML, AAAI and IJCAI, for example) and applications of
machine learning and data mining (ACM SIGIR, WWW and
ACL for example)
3. Before we give different categorizations
of transfer learning, we ﬁrst describe the notations used in this
article.
2.2 Notations and De ﬁnitions
In this section, we introduce some notations and de ﬁnitions
that are used in this survey. First of all, we give the de ﬁnitions
of a “domain” and a “task”, respectively.
In this survey, a domain Dconsists of two components: a
feature space Xand a marginal probability distribution P(X),
where X={x1,...,x n}∈X . For example, if our learning
2. http://www.darpa.mil/ipto/programs/tl/tl.asp
3. We summarize a list of conferences and workshops where transfer
learning papers appear in these few years in the following webpage for
reference, http://www.cse.ust.hk/ ∼sinnopan/conferenceTL.htmIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
3
task is document classi ﬁcation, and each term is taken as a
binary feature, then Xis the space of all term vectors, xi
is the ithterm vector corresponding to some documents, and
Xis a particular learning sample. In general, if two domains
are different, then they may have different feature spaces ordifferent marginal probability distributions.
Given a speci ﬁc domain, D={X,P(X)},atask consists
of two components: a label space Yand an objective predictive
function f(·)(denoted by T={Y,f(·)}), which is not
observed but can be learned from the training data, whichconsist of pairs {x
i,yi}, where xi∈Xandyi∈Y .T h e
function f(·)can be used to predict the corresponding label,
f(x), of a new instance x. From a probabilistic viewpoint,
f(x)can be written as P(y|x). In our document classi ﬁcation
example, Yis the set of all labels, which is True, False for a
binary classi ﬁcation task, and yiis “True” or “False”.
For simplicity, in this survey, we only consider the case
where there is one source domain DS, and one target domain,
DT, as this is by far the most popular of the research works in
the literature. More speci ﬁcally, we denote the source domain
data asDS={(xS1,yS1),...,(xSnS,ySnS)}, where xSi∈
XSi st h ed a t ai n s t a n c ea n d ySi∈Y Sis the corresponding
class label. In our document classi ﬁcation example, DScan
be a set of term vectors together with their associated true orfalse class labels. Similarly, we denote the target domain dataasD
T={(xT1,yT1),...,(xTnT,yTnT)}, where the input xTi
is inXTandyTi∈YTis the corresponding output. In most
cases, 0≤nT/lessmuchnS.
We now give a uni ﬁed de ﬁnition of transfer learning.
Deﬁnition 1 (Transfer Learning) Given a source domain DS
and learning task TS, a target domain DTand learning task
TT,transfer learning aims to help improve the learning of the
target predictive function fT(·)inDTusing the knowledge in
DSandTS, where DS/negationslash=DT,o rTS/negationslash=TT.
In the above de ﬁnition, a domain is a pair D={X,P(X)}.
Thus the condition DS/negationslash=DTimplies that either XS/negationslash=XTor
PS(X)/negationslash=PT(X). For example, in our document classi ﬁcation
example, this means that between a source document set anda target document set, either the term features are differentbetween the two sets (e.g., they use different languages), ortheir marginal distributions are different.
Similarly, a task is de ﬁned as a pair T={Y,P(Y|X)}.
Thus the condition T
S/negationslash=TTimplies that either YS/negationslash=YT
orP(YS|XS)/negationslash=P(YT|XT). When the target and source
domains are the same, i.e. DS=DT, and their learning tasks
are the same, i.e., TS=TT, the learning problem becomes
a traditional machine learning problem. When the domains
are different, then either (1) the feature spaces between thedomains are different, i.e. X
S/negationslash=XT, or (2) the feature
spaces between the domains are the same but the marginalprobability distributions between domain data are different;i.e.P(X
S)/negationslash=P(XT), where XSi∈X SandXTi∈X T.
As an example, in our document classi ﬁcation example, case
(1) corresponds to when the two sets of documents aredescribed in different languages, and case (2) may correspondto when the source domain documents and the target domaindocuments focus on different topics.Given speci ﬁc domains D
SandDT, when the learning
tasksTSandTTare different, then either (1) the label spaces
between the domains are different, i.e. YS/negationslash=YT, or (2) the
conditional probability distributions between the domains are
different; i.e. P(YS|XS)/negationslash=P(YT|XT), where YSi∈YSand
YTi∈YT. In our document classi ﬁcation example, case (1)
corresponds to the situation where source domain has binarydocument classes, whereas the target domain has ten classes toclassify the documents to. Case (2) corresponds to the situationwhere the source and target documents are very unbalanced
in terms of the user-de ﬁned classes.
In addition, when there exists some relationship, explicit or
implicit, between the feature spaces of the two domains, wesay that the source and target domains are related .
2.3 A Categorization of Transfer Learning Tech-
niques
Intransfer learning , we have the following three main research
issues: (1) What to transfer; (2) How to transfer; (3) When totransfer.
“What to transfer” asks which part of knowledge can
be transferred across domains or tasks. Some knowledge isspeciﬁc for individual domains or tasks, and some knowledge
may be common between different domains such that they mayhelp improve performance for the target domain or task. Afterdiscovering which knowledge can be transferred, learningalgorithms need to be developed to transfer the knowledge,which corresponds to the “how to transfer” issue.
“When to transfer” asks in which situations, transferring
skills should be done. Likewise, we are interested in knowing
in which situations, knowledge should notbe transferred. In
some situations, when the source domain and target domain arenot related to each other, brute-force transfer may be unsuc-cessful. In the worst case, it may even hurt the performanceof learning in the target domain, a situation which is oftenreferred to as negative transfer . Most current work on transfer
learning focuses on “What to transfer” and “How to transfer”,
by implicitly assuming that the source and target domains berelated to each other. However, how to avoid negative transferis an important open issue that is attracting more and moreattention in the future.
Based on the de ﬁnition of transfer learning, we summarize
the relationship between traditional machine learning and var-ious transfer learning settings in Table 1, where we categorizetransfer learning under three sub-settings, inductive trans-
fer learning ,transductive transfer learning andunsupervised
transfer learning , based on different situations between the
source and target domains and tasks.
1) In the inductive transfer learning setting, the target task
is different from the source task, no matter when the
source and target domains are the same or not.In this case, some labeled data in the target domain arerequired to induce an objective predictive model f
T(·)
for use in the target domain. In addition, according todifferent situations of labeled and unlabeled data in the
source domain, we can further categorize the inductive
transfer learning setting into two cases:IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
4
TABLE 1
Relationship between Traditional Machine Learning and Various Transfer Learning Settings
Learning Settings Source and Target Domains Source and Target Tasks
Traditional Machine Learning the same the same
Inductive Transfer Learning / the same different but related
Transfer Learning Unsupervised Transfer Learning different but related different but related
Transductive Transfer Learning different but related the same
(1.1) A lot of labeled data in the source domain are
available. In this case, the inductive transfer learning
setting is similar to the multi-task learning setting.However, the inductive transfer learning setting only
aims at achieving high performance in the target taskby transferring knowledge from the source task whilemulti-task learning tries to learn the target and sourcetask simultaneously.(1.2) No labeled data in the source domain are available.In this case, the inductive transfer learning setting is
similar to the self-taught learning setting, which is
ﬁrst proposed by Raina et al. [22]. In the self-taught
learning setting, the label spaces between the source
and target domains may be different, which impliesthe side information of the source domain cannot be
used directly. Thus, it’s similar to the inductive transferlearning setting where the labeled data in the sourcedomain are unavailable.
2) In the transductive transfer learning setting, the source
and target tasks are the same, while the source and target
domains are different.In this situation, no labeled data in the target domainare available while a lot of labeled data in the sourcedomain are available. In addition, according to differentsituations between the source and target domains, we
can further categorize the transductive transfer learning
setting into two cases.(2.1) The feature spaces between the source and targetdomains are different, X
S/negationslash=XT.
(2.2) The feature spaces between domains are the same,X
S=XT, but the marginal probability distributions of
the input data are different, P(XS)/negationslash=P(XT).
The latter case of the transductive transfer learning
setting is related to domain adaptation for knowledgetransfer in text classi ﬁcation [23] and sample selection
bias [24] or co-variate shift [25], whose assumptions aresimilar.
3) Finally, in the unsupervised transfer learning setting,
similar to inductive transfer learning setting, the target
task is different from but related to the source task.However, the unsupervised transfer learning focus on
solving unsupervised learning tasks in the target domain,
such as clustering, dimensionality reduction and density
estimation [26], [27]. In this case, there are no labeleddata available in both source and target domains intraining.
The relationship between the different settings of transfer
learning and the related areas are summarized in Table 2 andFigure 2.Approaches to transfer learning in the above three different
settings can be summarized into four cases based on “What totransfer”. Table 3 shows these four cases and brief description.Theﬁrst context can be referred to as instance-based transfer-
learning (or instance-transfer) approach [6], [28], [29], [30],[31], [24], [32], [33], [34], [35], which assumes that certainparts of the data in the source domain can be reused forlearning in the target domain by re-weighting . Instance re-
weighting and importance sampling are two major techniquesin this context.
A second case can be referred to as feature-representation-
transfer approach [22], [36], [37], [38], [39], [8], [40], [41],
[42], [43], [44]. The intuitive idea behind this case is to learna “good” feature representation for the target domain. In thiscase, the knowledge used to transfer across domains is encoded
into the learned feature representation. With the new featurerepresentation, the performance of the target task is expectedto improve signi ﬁcantly.
A third case can be referred to as parameter-transfer ap-
proach [45], [46], [47], [48], [49], which assumes that thesource tasks and the target tasks share some parameters orprior distributions of the hyper-parameters of the models. Thetransferred knowledge is encoded into the shared parametersor priors. Thus, by discovering the shared parameters or priors,knowledge can be transferred across tasks.
Finally, the last case can be referred to as the relational-
knowledge-transfer problem [50], which deals with transferlearning for relational domains. The basic assumption behindthis context is that some relationship among the data in thesource and target domains are similar. Thus, the knowledgeto be transferred is the relationship among the data. Recently,statistical relational learning techniques dominate this context[51], [52].
Table 4 shows the cases where the different approaches
are used for each transfer learning setting. We can see thattheinductive transfer learning setting has been studied in
many research works, while the unsupervised transfer learning
setting is a relatively new research topic and only studiedin the context of the feature-representation-transfer case. In
addition, the feature-representation-transfer problem has been
proposed to all three settings of transfer learning. However,theparameter-transfer and the relational-knowledge-transfer
approach are only studied in the inductive transfer learning
setting, which we discuss in detail below.
3I NDUCTIVE TRANSFER LEARNING
Deﬁnition 2 (Inductive Transfer Learning) Given a source
domain DSand a learning task TS, a target domain DTandIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
5
TABLE 2
Different Settings of Transfer Learning
Transfer Learning Settings Related Areas Source Domain Labels Target Domain Labels Tasks
Inductive Transfer Learning Multi-task Learning Available Available Regression,
Classi ﬁcation
Self-taught Learning Unavailable Available Regression,
Classi ﬁcation
Transductive Transfer Learning Domain Adaptation, Sample
Selection Bias, Co-variate ShiftAvailable Unavailable Regression,Classi ﬁcation
Unsupervised Transfer Learning Unavailable Unavailable Clustering,
Dimensionality
Reduction
Fig. 2. An Overview of Different Settings of Transfer
TABLE 3
Different Approaches to Transfer Learning
Transfer Learning Approaches Brief Description
Instance-transfer To re-weight some labeled data in the source domain for use in the target domain [6], [28], [29],
[30], [31], [24], [32], [33], [34], [35].
Feature-representation-transfer Find a “good” feature representation that reduces difference between the source and the targetdomains and the error of classi ﬁcation and regression models [22], [36], [37], [38], [39], [8],
[40], [41], [42], [43], [44].
Parameter-transfer Discover shared parameters or priors between the source domain and target domain models, which
can bene ﬁt for transfer learning [45], [46], [47], [48], [49].
Relational-knowledge-transfer Build mapping of relational knowledge between the source domain and the target domains. Both
domains are relational domains and i.i.d assumption is relaxed in each domain [50], [51], [52].
TABLE 4
Different Approaches Used in Different Settings
Inductive Transfer Learning Transductive Transfer Learning Unsupervised Transfer Learning
Instance-transfer√ √
Feature-representation-transfer√ √ √
Parameter-transfer√
Relational-knowledge-transfer√IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
6
a learning task TT,inductive transfer learning aims to help
improve the learning of the target predictive function fT(·)in
DTusing the knowledge in DSandTS, where TS/negationslash=TT.
Based on the above de ﬁnition of the inductive transfer
learning setting, a few labeled data in the target domain are
required as the training data to induce the target predictive
function. As mentioned in Section 2.3, this setting has twocases: (1) Labeled data in the source domain are available;(2) Labeled data in the source domain are unavailable whileunlabeled data in the source domain are available. Mosttransfer learning approaches in this setting focus on the former
case.
3.1 Transferring Knowledge of Instances
Theinstance-transfer approach to the inductive transfer learn-
ing setting is intuitively appealing: although the source domaindata cannot be reused directly, there are certain parts of thedata that can still be reused together with a few labeled datain the target domain.
Daiet al. [6] proposed a boosting algorithm, TrAdaBoost ,
which is an extension of the AdaBoost algorithm, to address
theinductive transfer learning problems. TrAdaBoost assumes
that the source and target domain data use exactly the same setof features and labels, but the distributions of the data in thetwo domains are different. In addition, TrAdaBoost assumes
that, due to the difference in distributions between the source
and the target domains, some of the source domain data maybe useful in learning for the target domain but some of themmay not and could even be harmful. It attempts to iterativelyre-weight the source domain data to reduce the effect of the“bad” source data while encourage the “good” source data
to contribute more for the target domain. For each round ofiteration, TrAdaBoost trains the base classi ﬁer on the weighted
source and target data. The error is only calculated on thetarget data. Furthermore, TrAdaBoost uses the same strategy as
AdaBoost to update the incorrectly classi ﬁed examples in the
target domain while using a different strategy from AdaBoost
to update the incorrectly classi ﬁed source examples in the
source domain. Theoretical analysis of TrAdaBoost in also
given in [6].
Jiang and Zhai [30] proposed a heuristic method to remove
“misleading” training examples from the source domain basedon the difference between conditional probabilities P(y
T|xT)
andP(yS|xS). Liao et al. [31] proposed a new active learning
method to select the unlabeled data in a target domain tobe labeled with the help of the source domain data. Wu andDietterich [53] integrated the source domain (auxiliary) data an
SVM framework for improving the classi ﬁcation performance.
3.2 Transferring Knowledge of Feature Representa-
tions
The feature-representation-transfer approach to the inductive
transfer learning problem aims at ﬁnding “good” feature
representations to minimize domain divergence and classi ﬁ-
cation or regression model error. Strategies to ﬁnd “good”
feature representations are different for different types of thesource domain data. If a lot of labeled data in the source
domain are available, supervised learning methods can beused to construct a feature representation. This is similar tocommon feature learning in the ﬁeld of multi-task learning
[40]. If no labeled data in the source domain are available,unsupervised learning methods are proposed to construct thefeature representation.
3.2.1 Supervised Feature Construction
Supervised feature construction methods for the inductive
transfer learning setting are similar to those used in multi-
task learning. The basic idea is to learn a low-dimensionalrepresentation that is shared across related tasks. In addition,the learned new representation can reduce the classi ﬁcation
or regression model error of each task as well. Argyriou et
al.[40] proposed a sparse feature learning method for multi-
task learning. In the inductive transfer learning setting, the
common features can be learned by solving an optimizationproblem, given as follows.
arg min
A,U/summationdisplay
t∈{T,S}nt/summationdisplay
i=1L(yti,/angbracketleftat,UTxti/angbracketright)+γ/bardblA/bardbl2
2,1(1)
s.t. U ∈Od
In this equation, SandTdenote the tasks in the source
domain and target domain, respectively. A=[aS,aT]∈Rd×2
is a matrix of parameters. Uis ad×dorthogonal matrix
(mapping function) for mapping the original high-dimensionaldata to low-dimensional representations. The (r, p)-norm of
Ais deﬁned as /bardblA/bardbl
r,p:= (/summationtextd
i=1/bardblai/bardblp
r)1
p. The optimization
problem (1) estimates the low-dimensional representationsU
TXT,UTXSand the parameters, A, of the model at the
same time. The optimization problem (1) can be further trans-formed into an equivalent convex optimization formulation andbe solved ef ﬁciently. In a follow-up work, Argyriou et al. [41]
proposed a spectral regularization framework on matrices for
multi-task structure learning.
Leeet al. [42] proposed a convex optimization algorithm for
simultaneously learning meta-priors and feature weights froman ensemble of related prediction tasks. The meta-priors canbe transferred among different tasks. Jebara [43] proposed to
select features for multi-task learning with SVMs. Ruckert et
al.[54] designed a kernel-based approach to inductive transfer,
which aims at ﬁnding a suitable kernel for the target data.
3.2.2 Unsupervised Feature Construction
In [22], Raina et al. proposed to apply sparse coding [55],
which is an unsupervised feature construction method, forlearning higher level features for transfer learning. The basic
idea of this approach consists of two steps. In the ﬁrst step,
higher-level basis vectors b={b
1,b2,...,b s}are learned on
the source domain data by solving the optimization problem(2) as shown as follows,
min
a,b/summationdisplay
i/bardblxSi−/summationtext
jaj
Sibj/bardbl2
2+β/bardblaSi/bardbl1 (2)
s.t. /bardblbj/bardbl2≤1,∀j∈1,...,sIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
7
In this equation, aj
Siis a new representation of basis bjfor
inputxSiandβis a coef ﬁcient to balance the feature con-
struction term and the regularization term. After learning thebasis vectors b, in the second step, an optimization algorithm
(3) is applied on the target domain data to learn higher level
features based on the basis vectors b.
a
∗
Ti=a r gm i n
aTi/bardblxTi−/summationdisplay
jaj
Tibj/bardbl2
2+β/bardblaTi/bardbl1 (3)
Finally, discriminative algorithms can be applied to {a∗
Ti}/primes
with corresponding labels to train classi ﬁcation or regression
models for use in the target domain. One drawback of this
method is that the so-called higher-level basis vectors learnedon the source domain in the optimization problem (2) may notbe suitable for use in the target domain.
Recently, manifold learning methods have been adapted for
transfer learning. In [44], Wang and Mahadevan proposeda Procrustes analysis based approach to manifold alignmentwithout correspondences, which can be used to transfer theknowledge across domains via the aligned manifolds.
3.3 Transferring Knowledge of Parameters
Most parameter-transfer approaches to the inductive transfer
learning setting assume that individual models for related tasks
should share some parameters or prior distributions of hyper-
parameters. Most approaches described in this section, includ-ing a regularization framework and a hierarchical Bayesianframework, are designed to work under multi-task learning.However, they can be easily modi ﬁed for transfer learning.
As mentioned above, multi-task learning tries to learn both
the source and target tasks simultaneously and perfectly, while
transfer learning only aims at boosting the performance ofthe target domain by utilizing the source domain data. Thus,in multi-task learning, weights of the loss functions for thesource and target data are the same. In contrast, in transferlearning, weights in the loss functions for different domains
can be different. Intuitively, we may assign a larger weight to
the loss function of the target domain to make sure that wecan achieve better performance in the target domain.
Lawrence and Platt [45] proposed an ef ﬁcient algorithm
known as MT-IVM, which is based on Gaussian Processes(GP), to handle the multi-task learning case. MT-IVM tries tolearn parameters of a Gaussian Process over multiple tasks bysharing the same GP prior. Bonilla et al. [46] also investigated
multi-task learning in the context of GP. The authors proposedto use a free-form covariance matrix over tasks to modelinter-task dependencies, where a GP prior is used to inducecorrelations between tasks. Schwaighofer et al. [47] proposed
to use a hierarchical Bayesian framework (HB) together withGP for multi-task learning.
Besides transferring the priors of the GP models, some
researchers also proposed to transfer parameters of SVMs
under a regularization framework. Evgeniou and Pontil [48]
borrowed the idea of HB to SVMs for multi-task learning.The proposed method assumed that the parameter, w, in SVMs
for each task can be separated into two terms. One is acommon term over tasks and the other is a task-speci ﬁct e r m .Ininductive transfer learning ,
w
S=w0+vSand w T=w0+vT,
where, wSandwTare parameters of the SVMs for the source
task and the target learning task, respectively. w0is a common
parameter while vSandvTare speci ﬁc parameters for the
source task and the target task, respectively. By assuming ft=
wt·xto be a hyper-plane for task t, an extension of SVMs
to multi-task learning case can be written as the following:
min
w0,vt,ξtiJ(w0,vt,ξti) (4)
=/summationdisplay
t∈{S,T}nt/summationdisplay
i=1ξti+λ1
2/summationdisplay
t∈{S,T}/bardblvt/bardbl2+λ2/bardblw0/bardbl2
s.t. y ti(w0+vt)·xti≥1−ξti,
ξti≥0,i∈{1,2,...,n t}and t ∈{S,T}.
By solving the optimization problem above, we can learn the
parameters w0,vSandvTsimultaneously.
Several researchers have pursued the parameter transfer
approach further. Gao et al. [49] proposed a locally weighted
ensemble learning framework to combine multiple models fortransfer learning, where the weights are dynamically assignedaccording to a model’s predictive power on each test example
in the target domain.
3.4 Transferring Relational Knowledge
Different from other three contexts, the relational-knowledge-
transfer approach deals with transfer learning problems inrelational domains, where the data are non-i.i.d. and can be
represented by multiple relations, such as networked data and
social network data. This approach does not assume that thedata drawn from each domain be independent and identicallydistributed (i.i.d.) as traditionally assumed. It tries to transfertherelationship among data from a source domain to a
target domain. In this context, statistical relational learning
techniques are proposed to solve these problems.
Mihalkova et al. [50] proposed an algorithm TAMAR that
transfers relational knowledge with Markov Logic Networks(MLNs) across relational domains. MLNs [56] is a powerfulformalism, which combines the compact expressiveness of
ﬁrst order logic with ﬂexibility of probability, for statistical
relational learning. In MLNs, entities in a relational domainare represented by predicates and their relationships are rep-resented in ﬁrst-order logic. TAMAR is motivated by the fact
that if two domains are related to each other, there may existmappings to connect entities and their relationships from asource domain to a target domain. For example, a professorcan be considered as playing a similar role in an academicdomain as a manager in an industrial management domain.In addition, the relationship between a professor and his orher students is similar to the relationship between a managerand his or her workers. Thus, there may exist a mappingfrom professor to manager and a mapping from the professor-student relationship to the manager-worker relationship. In thisvein, TAMAR tries to use an MLN learned for a source domain
to aid in the learning of an MLN for a target domain. Basically,IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
8
TAMAR is a two-stage algorithm. In the ﬁrst step, a mapping
is constructed from a source MLN to the target domain based
on weighted pseudo loglikelihood measure (WPLL). In thesecond step, a revision is done for the mapped structure in thetarget domain through the FORTE algorithm [57], which is an
inductive logic programming (ILP) algorithm for revising ﬁrst
order theories. The revised MLN can be used as a relational
model for inference or reasoning in the target domain.
In the AAAI-2008 workshop on transfer learning for com-
plex tasks
4,M i h a l k o v a et al. [51] extended TAMAR to the
single-entity-centered setting of transfer learning, where onlyone entity in a target domain is available. Davis et al. [52]
proposed an approach to transferring relational knowledgebased on a form of second-order Markov logic. The basic
idea of the algorithm is to discover structural regularities inthe source domain in the form of Markov logic formulaswith predicate variables, by instantiating these formulas with
predicates from the target domain.
4T RANSDUCTIVE TRANSFER LEARNING
The term transductive transfer learning wasﬁrst proposed
by Arnold et al. [58], where they required that the source
and target tasks be the same, although the domains may be
different. On top of these conditions, they further requiredthat that all unlabeled data in the target domain are available
at training time, but we believe that this condition can berelaxed; instead, in our de ﬁnition of the transductive transfer
learning setting, we only require that part of the unlabeled
target data be seen at training time in order to obtain the
marginal probability for the target data.
Note that the word ’transductive’ is used with several mean-
ings. In the traditional machine learning setting, transductive
learning [59] refers to the situation where all test data are
required to be seen at training time, and that the learned modelcannot be reused for future data. Thus, when some new testdata arrive, they must be classi ﬁed together with all existing
data. In our categorization of transfer learning, in contrast, weuse the term transductive to emphasize the concept that in this
type of transfer learning, the tasks must be the same and there
must be some unlabeled data available in the target domain.
Deﬁnition 3 (Transductive Transfer Learning) Given a source
domain D
Sand a corresponding learning task TS,at a r g e t
domain DTand a corresponding learning task TT,transductive
transfer learning aims to improve the learning of the target
predictive function fT(·)inDTusing the knowledge in DS
andTS, where DS/negationslash=DTandTS=TT. In addition, some
unlabeled target domain data must be available at training time.
This de ﬁnition covers the work of Arnold et al. [58], since
the latter considered domain adaptation , where the difference
lies between the marginal probability distributions of sourceand target data; i.e., the tasks are the same but the domainsare different.
Similar to the traditional transductive learning setting, which
aims to make the best use of the unlabeled test data for learn-ing, in our classi ﬁcation scheme under transductive transfer
4. http://www.cs.utexas.edu/ ∼mtaylor/AAAI08TL/learning, we also assume that some target-domain unlabeled
data be given. In the above de ﬁnition of transductive transfer
learning, the source and target tasks are the same, whichimplies that one can adapt the predictive function learnedin the source domain for use in the target domain through
some unlabeled target-domain data. As mentioned in Section
2.3, this setting can be split to two cases: (a) The featurespaces between the source and target domains are different,X
S/negationslash=XT, and (b) the feature spaces between domains are the
same, XS=XT, but the marginal probability distributions of
the input data are different, P(XS)/negationslash=P(XT). This is similar
to the requirements in domain adaptation and sample selectionbias. Most approaches described in the following sections are
related to case (b) above.
4.1 Transferring the Knowledge of Instances
Most instance-transfer approaches to the transductive transfer
learning setting are motivated by importance sampling. To
see how importance sampling based methods may help in
this setting, we ﬁrst review the problem of empirical risk
minimization (ERM) [60]. In general, we might want to learnthe optimal parameters θ
∗of the model by minimizing the
expected risk,
θ∗=a r gm i n
θ∈ΘE(x,y)∈P[l(x, y, θ)],
where l(x, y, θ)is a loss function that depends on the para-
meter θ. However, since it is hard to estimate the probability
distribution P, we choose to minimize the ERM instead,
θ∗=a r gm i n
θ∈Θ1
nn/summationdisplay
i=1[l(xi,yi,θ)],
where nis size of the training data.
In the transductive transfer learning setting, we want to
learn an optimal model for the target domain by minimizingthe expected risk,
θ
∗=a r gm i n
θ∈Θ/summationdisplay
(x,y)∈DTP(DT)l(x, y, θ).
However, since no labeled data in the target domain are
observed in training data, we have to learn a model from thesource domain data instead. If P(D
S)=P(DT),t h e nw em a y
simply learn the model by solving the following optimization
problem for use in the target domain,
θ∗=a r gm i n
θ∈Θ/summationdisplay
(x,y)∈DSP(DS)l(x, y, θ).
Otherwise, when P(DS)/negationslash=P(DT), we need to modify
the above optimization problem to learn a model with highgeneralization ability for the target domain, as follows:
θ
∗=a r g m i n
θ∈Θ/summationdisplay
(x,y)∈DSP(DT)
P(DS)P(DS)l(x, y, θ)
≈arg min
θ∈ΘnS/summationdisplay
i=1PT(xTi,yTi)
PS(xSi,ySi)l(xSi,ySi,θ). (5)
Therefore, by adding different penalty values to each instance
(xSi,ySi)with the corresponding weightPT(xTi,yTi)
PS(xSi,ySi), we canIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
9
learn a precise model for the target domain. Furthermore,
sinceP(YT|XT)=P(YS|XS). Thus the difference between
P(DS)andP(DT)is caused by P(XS)andP(XT)and
PT(xTi,yTi)
PS(xSi,ySi)=P(xSi)
P(xTi). If we can estimateP(xSi)
P(xTi)for each
instance, we can solve the transductive transfer learning
problems.
There exist various ways to estimateP(xSi)
P(xTi). Zadrozny
[24] proposed to estimate the terms P(xSi)andP(xTi)
independently by constructing simple classi ﬁcation problems.
Fanet al. [35] further analyzed the problems by using various
classi ﬁers to estimate the probability ratio. Huang et al. [32]
proposed a kernel-mean matching (KMM) algorithm to learn
P(xSi)
P(xTi)directly by matching the means between the source
domain data and the target domain data in a reproducing-
kernel Hilbert space (RKHS). KMM can be rewritten as thefollowing quadratic programming (QP) optimization problem.
min
β1
2βTKβ−κTβ (6)
s.t. β i∈[0,B]and|/summationtextnS
i=1βi−nS|≤nS/epsilon1
where K=/bracketleftbigg
KS,SKS,T
KT, SKT, T/bracketrightbigg
andKij=k(xi,xj).KS,Sand
KT,Tare kernel matrices for the source domain data and the
target domain data, respectively. κi=nS
nT/summationtextnT
j=1k(xi,xTj),
where xi∈XS/uniontextXT, while xTj∈XT.
It can be proved that βi=P(xSi)
P(xTi)[32]. An advantage of
using KMM is that it can avoid performing density estimation
of either P(xSi)orP(xTi), which is dif ﬁcult when the size
of the data set is small. Sugiyama et al. [34] proposed an
algorithm known as Kullback-Leibler Importance Estimation
Procedure (KLIEP) to estimateP(xSi)
P(xTi)directly, based on the
minimization of the Kullback-Leibler divergence. KLIEP can
be integrated with cross-validation to perform model selec-tion automatically in two steps: (1) estimating the weightsof the source domain data; (2) training models on the re-weighted data. Bickel et al. [33] combined the two steps in
a uni ﬁed framework by deriving a kernel-logistic regression
classi ﬁer. Besides sample re-weighting techniques, Dai et
al.[28] extended a traditional Naive Bayesian classi ﬁer for the
transductive transfer learning problems. For more informationon importance sampling and re-weighting methods for co-variate shift or sample selection bias, readers can refer to arecently published book [29] by Quionero-Candela et al. Onecan also consult a tutorial on Sample Selection Bias by Fanand Sugiyama in ICDM-08
5.
4.2 Transferring Knowledge of Feature Representa-
tions
Most feature-representation transfer approaches to the trans-
ductive transfer learning setting are under unsupervised learn-
ing frameworks. Blitzer et al. [38] proposed a structural cor-
respondence learning (SCL) algorithm, which extends [37],to make use of the unlabeled data from the target domain toextract some revelent features that may reduce the difference
5. Tutorial slides can be found at http://www.cs.columbia.edu/ ∼
fan/PPT/ICDM08SampleBias.pptbetween the domains. The ﬁrst step of SCL is to de ﬁne a set
ofpivot features6(the number of pivot feature is denoted
bym) on the unlabeled data from both domains. Then, SCL
removes these pivot features from the data and treats each pivot
feature as a new label vector. The mclassi ﬁcation problems
can be constructed. By assuming each problem can be solvedby linear classi ﬁer, which is shown as follows,
f
l(x)=sgn(wT
l·x),l=1,...,m
SCL can learn a matrix W=[w1w2...w m]of parameters. In
the third step, singular value decomposition (SVD) is appliedto matrix W=[w
1w2...w m].L e t W=UDVT, then
θ=UT
[1:h,:](his the number of the shared features) is the
matrix (linear mapping) whose rows are the top left singularvectors of W. Finally, standard discriminative algorithms can
be applied to the augmented feature vector to build models.The augmented feature vector contains all the original featurex
iappended with the new shared features θxi. As mentioned
in [38], if the pivot features are well designed, then the
learned mapping θencodes the correspondence between the
features from the different domains. Although Ben-David andSchuller [61] showed experimentally that SCL can reduce thedifference between domains, how to select the pivot features
is difﬁcult and domain-dependent. In [38], Blitzer et al. used a
heuristic method to select pivot features for natural languageprocessing (NLP) problems, such as tagging of sentences. Intheir follow-up work, the researchers proposed to use MutualInformation (MI) to choose the pivot features instead of usingmore heuristic criteria [8]. MI-SCL tries to ﬁnd some pivot
features that have high dependence on the labels in the source
domain.
Transfer learning in the NLP domain is sometimes re-
ferred to as domain adaptation. In this area, Daum ´e [39]
proposed a kernel-mapping function for NLP problems, which
maps the data from both source and target domains to ahigh-dimensional feature space, where standard discriminativelearning methods are used to train the classi ﬁers. However,
the constructed kernel mapping function is domain knowledgedriven. It is not easy to generalize the kernel mapping to otherareas or applications. Blitzer et al. [62] analyzed the uniform
convergence bounds for algorithms that minimized a convex
combination of source and target empirical risks.
In [36], Dai et al. proposed a co-clustering based algorithm
to propagate the label information across different domains.
In [63], Xing et al. proposed a novel algorithm known as
bridged re ﬁnement to correct the labels predicted by a shift-
unaware classi ﬁer towards a target distribution and take the
mixture distribution of the training and test data as a bridge tobetter transfer from the training data to the test data. In [64],
Ling et al. proposed a spectral classi ﬁcation framework for
cross-domain transfer learning problem, where the objectivefunction is introduced to seek consistency between the in-domain supervision and the out-of-domain intrinsic structure.In [65], Xue et al. proposed a cross-domain text classi ﬁcation
algorithm that extended the traditional probabilistic latentsemantic analysis (PLSA) algorithm to integrate labeled and
6. The pivot features are domain speci ﬁc and depend on prior knowledgeIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
10
unlabeled data from different but related domains, into a
uniﬁed probabilistic model. The new model is called Topic-
bridged PLSA, or TPLSA.
Transfer learning via dimensionality reduction was re-
cently proposed by Pan et al. [66]. In this work, Pan et
al.exploited the Maximum Mean Discrepancy Embedding
(MMDE) method, originally designed for dimensionality re-duction, to learn a low dimensional space to reduce thedifference of distributions between different domains for trans-ductive transfer learning. However, MMDE may suffer fromits computational burden. Thus, in [67], Pan et al. further
proposed an ef ﬁcient feature extraction algorithm, known as
Transfer Component Analysis (TCA) to overcome the draw-back of MMDE.
5U NSUPERVISED TRANSFER LEARNING
Deﬁnition 4 (Unsupervised Transfer Learning) Given a
source domain DSwith a learning task TS, a target domain DT
and a corresponding learning task TT,unsupervised transfer
learning aims to help improve the learning of the target
predictive function fT(·)7inDTusing the knowledge in DS
andTS, where TS/negationslash=TTandYSandYTare not observable.
Based on the de ﬁnition of the unsupervised transfer learn-
ingsetting, no labeled data are observed in the source and
target domains in training. So far, there is little research workon this setting. Recently, Self-taught clustering (STC) [26]
and transferred discriminative analysis (TDA) [27] algorithmsare proposed to transfer clustering and transfer dimensionalityreduction problems, respectively.
5.1 Transferring Knowledge of Feature Representa-
tions
Dai et al. [26] studied a new case of clustering problems,
known as self-taught clustering .Self-taught clustering is an
instance of unsupervised transfer learning , which aims at
clustering a small collection of unlabeled data in the targetdomain with the help of a large amount of unlabeled data inthe source domain. STC tries to learn a common feature spaceacross domains, which helps in clustering in the target domain.The objective function of STC is shown as follows.
J(˜X
T,˜XS,˜Z) (7)
=I(XT,Z)−I(˜XT,˜Z)+λ/bracketleftBig
I(XS,Z)−I(˜XS,˜Z)/bracketrightBig
where XSandXTare the source and target domain data,
respectively. Zis a shared feature space by XSandXT,
andI(·,·)is the mutual information between two random
variables. Suppose that there exist three clustering functionsC
XT:XT→˜XT,CXS:XS→˜XSandCZ:Z→˜Z, where
˜XT,˜XSand˜Zare corresponding clusters of XT,XSandZ,
respectively. The goal of STC is to learn ˜XTby solving the
optimization problem (7):
arg min
˜XT,˜XS,˜ZJ(˜XT,˜XS,˜Z) (8)
7. In unsupervised transfer learning, the predicted labels are latent variables,
such as clusters or reduced dimensionsAn iterative algorithm for solving the optimization function
(8) was given in [26].
Similarly, [27] proposed a transferred discriminative analy-
sis(TDA) algorithm to solve the transfer dimensionality
reduction problem. TDA ﬁrst applies clustering methods to
generate pseudo-class labels for the target unlabeled data. Itthen applies dimensionality reduction methods to the targetdata and labeled source data to reduce the dimensions. Thesetwo steps run iteratively to ﬁnd the best subspace for the target
data.
6T RANSFER BOUNDS AND NEGATIVE
TRANSFER
An important issue is to recognize the limit of the power oftransfer learning. In [68], Hassan Mahmud and Ray analyzedthe case of transfer learning using Kolmogorov complexity,where some theoretical bounds are proved. In particular, theauthors used conditional Kolmogorov complexity to measurerelatedness between tasks and transfer the “right” amountof information in a sequential transfer learning task under aBayesian framework.
Recently, Eaton et al. [69] proposed a novel graph-based
method for knowledge transfer, where the relationships be-tween source tasks are modeled by embedding the set oflearned source models in a graph using transferability as themetric. Transferring to a new task proceeds by mapping theproblem into the graph and then learning a function on thisgraph that automatically determines the parameters to transferto the new learning task.
Negative transfer happens when the source domain data and
task contribute to the reduced performance of learning in thetarget domain. Despite the fact that how to avoid negativetransfer is a very important issue, little research work hasbeen published on this topic. Rosenstein et al. [70] empirically
showed that if two tasks are too dissimilar, then brute-forcetransfer may hurt the performance of the target task. Someworks have been exploited to analyze relatedness among tasks
and task clustering techniques, such as [71], [72], whichmay help provide guidance on how to avoid negative transferautomatically. Bakker and Heskes [72] adopted a Bayesian
approach in which some of the model parameters are shared
for all tasks and others more loosely connected through a jointprior distribution that can be learned from the data. Thus,the data are clustered based on the task parameters, wheretasks in the same cluster are supposed to be related to each
others. Argyriou et al. [73] considered situations in which the
learning tasks can be divided into groups. Tasks within eachgroup are related by sharing a low-dimensional representation,which differs among different groups. As a result, tasks withina group can ﬁnd it easier to transfer useful knowledge.
7A PPLICATIONS OF TRANSFER LEARNING
Recently, transfer learning techniques have been applied suc-
cessfully in many real-world applications. Raina et al. [74] and
Daiet al. [36], [28] proposed to use transfer learning tech-
niques to learn text data across domains, respectively. Blitzeret al. [38] proposed to use SCL for solving NLP problems. AnIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
11
extension of SCL was proposed in [8] for solving sentiment
classi ﬁcation problems. Wu and Dietterich [53] proposed to
use both inadequate target domain data and plenty of lowquality source domain data for image classi ﬁcation problems.
Arnold et al. [58] proposed to use transductive transfer learn-
ingmethods to solve name-entity recognition problems. In
[75], [76], [77], [78], [79], transfer learning techniques are
proposed to extract knowledge from WiFi localization models
across time periods, space and mobile devices, to bene ﬁtW i F i
localization tasks in other settings. Zhuo et al. [80] studied how
to transfer domain knowledge to learn relational action modelsacross domains in automated planning.
In [81], Raykar et al. proposed a novel Bayesian multiple-
instance learning algorithm, which can automatically identifythe relevant feature subset and use inductive transfer forlearning multiple, but conceptually related, classi ﬁers, for
computer aided design (CAD). In [82], Ling et al. proposed an
information-theoretic approach for transfer learning to addressthecross-language classi ﬁcation problem for translating Web
pages from English to Chinese. The approach addressed theproblem when there are plenty of labeled English text data
whereas there are only a small number of labeled Chinese text
documents. Transfer learning across the two feature spaces areachieved by designing a suitable mapping function as a bridge.
So far, there are at least two international competitions based
on transfer learning, which made available some much neededpublic data. In the ECML/PKDD-2006 discovery challenge
8, the task was to handle personalized spam ﬁltering and
generalization across related learning tasks. For training a
spam- ﬁltering system, we need to collect a lot of emails from
a group of users with corresponding labels: spam ornot spam ,
and train a classi ﬁer based on these data. For a new email user,
we might want to adapt the learned model for the user. Thechallenge is that the distributions of emails for the ﬁrst set of
users and the new user are different. Thus, this problem can
be modeled as an inductive transfer learning problem, which
aims to adapt an old spam- ﬁltering model to a new situation
with fewer training data and less training time.
A second data set was made available through the ICDM-
2007 Contest, in which a task was to estimate a WiFi client’sindoor locations using the WiFi signal data obtained overdifferent periods of time [83]. Since the values of WiFisignal strength may be a function of time, space and devices,
distributions of WiFi data over different time periods may be
very different. Thus, transfer learning must be designed toreduce the data re-labeling effort.
Data Sets for Transfer Learning: So far, several data
sets have been published for transfer learning research. Wedenote the text mining data sets, Email spam- ﬁltering data
set, the WiFi localization over time periods data set and theSentiment classi ﬁcation data set by Text,Email ,WiFi and
Sen, respectively.
Text Three data sets, 20 Newsgroups, SRAA and Reuters-
21578
9, have been preprocessed for a transfer learn-
ing setting by some researchers. The data in these
8.http://www.ecmlpkdd2006.org/challenge.html
9.http://apex.sjtu.edu.cn/apex wiki/dwyakdata sets are categorized to a hierarchical structure.
Data from different sub-categories under the same
parent category are considered to be from differentbut related domains. The task is to predict the labelsof the parent category.
Email This data set is provided by the 2006 ECML/PKDD
discovery challenge.
WiFi This data set is provided by the ICDM-2007 Contest
10. The data were collected inside a building for
localization around 145.5×37.5m2in two different
time periods.
Sen This data set was ﬁrst used in [8]11. This data set con-
tains product reviews downloaded from Amazon.com
from4product types (domains): Kitchen, Books,
DVDs, and Electronics. Each domain has severalthousand reviews, but the exact number varies bydomain. Reviews contain star ratings (1 to 5 stars).
Empirical Evaluation To show how much bene ﬁt transfer
learning methods can bring as compared to traditional learningmethods, researchers have used some public data sets. Weshow a list taken from some published transfer learningpapers in Table 5. In [6], [84], [49], the authors used the 20
Newsgroups data
12as one of the evaluation data sets. Due
to the differences in the preprocessing steps of the algorithmsby different researchers, it is hard to compare the proposedmethods directly. Thus, we denote them by 20-Newsgroups
1,
20-Newsgroups 2and 20-Newsgroups 3, respectively, and show
the comparison results between the proposed transfer learningmethods and non-transfer learning methods in the table.
On the 20 Newsgroups
1data, Dai et al. [6] showed the
comparison experiments between standard Support Vector
Machine (SVM) and the proposed TrAdaBoost algorithm. On20 Newsgroups
2,S h i et al. [84] applied an active learning
algorithm to select important instances for transfer learning(AcTraK) with TrAdaBoost and standard SVM. Gao et al. [49]
evaluated their proposed locally weighted ensemble learningalgorithms, pLWE and LWE, on the 20 Newsgroups
3, com-
pared to SVM and Logistic Regression (LR).
In addition, in the table, we also show the comparison
results on the sentiment classi ﬁcation data set reported in [8].
On this data set, SGD denotes the stochastic gradient-descent
algorithm with Huber loss, SCL represents a linear predictoron the new representations learned by Structural Correspon-dence Learning algorithm, and SCL-MI is an extension of SCLby applying Mutual Information to select the pivot features forthe SCL algorithm.
Finally, on the WiFi localization data set, we show the
comparison results reported in [67], where the baseline is aregularized least square regression model (RLSR), which is
a standard regression model, and KPCA, which represents toapply RLSR on the new representations of the data learned by
Kernel Principle Component Analysis. The compared transferlearning methods include Kernel Mean Matching (KMM) andthe proposed algorithm, Transfer Component Analysis (TCA).
10.http://www.cse.ust.hk/ ∼qyang/ICDMDMC2007
11.http://www.cis.upenn.edu/ ∼mdredze/datasets/sentiment/
12.http://people.csail.mit.edu/jrennie/20Newsgroups/IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
12
For more detail about the experimental results, the readers
may refer to the reference papers showed in the table. From
these comparison results, we can ﬁnd that the transfer learning
methods designed appropriately for real world applications canindeed improve the performance signi ﬁcantly compared to the
non-transfer learning methods.
Toolboxes for Transfer Learning: Researchers at UC Berke-
ley provided a MATLAB toolkit for transfer learning
13.
The toolkit contains algorithms and benchmark data sets for
transfer learning. In addition, it provides a standard platformfor developing and testing new algorithms for transfer learning.
7.1 Other Applications of Transfer Learning
Transfer learning has found many applications in sequential
machine learning as well. For example, [85] proposed a graph-based method for identifying previously encountered games,and applied this technique to automate domain mapping forvalue function transfer and speed up reinforcement learning
on variants of previously played games. A new approach totransfer between entirely different feature spaces is proposedintranslated learning , which is made possible by learning
a mapping function for bridging features in two entirelydifferent domains (images and text) [86]. Finally, Li et al. [87],
[88] have applied transfer learning to collaborative ﬁltering
problems to solve the cold start and sparsity problems. In [87],
Liet al. learned a shared rating-pattern mixture model, known
as a Rating-Matrix Generative Model (RMGM), in terms ofthe latent user- and item-cluster variables. RMGM bridgesmultiple rating matrices from different domains by mappingthe users and items in each rating matrix onto the shared latent
user and item spaces in order to transfer useful knowledge.
In [88], they applied co-clustering algorithms on users anditems in an auxiliary rating matrix. They then constructed acluster-level rating matrix known as a codebook. By assumingthe target rating matrix (on movies) is related to the auxiliaryone (on books), the target domain can be reconstructed byexpanding the codebook, completing the knowledge transferprocess.
8C ONCLUSIONS
In this survey article, we have reviewed several current trendsof transfer learning. Transfer learning is classi ﬁed to three
different settings: inductive transfer learning, transductivetransfer learning and unsupervised transfer learning. Most pre-vious works focused on the former two settings. Unsupervisedtransfer learning may attract more and more attention in the
future.
Furthermore, each of the approaches to transfer learning
can be classi ﬁed into four contexts based on “what to trans-
fer” in learning. They include the instance-transfer approach,the feature-representation-transfer approach, the parameter-transfer approach and the relational-knowledge-transfer ap-proach, respectively. The former three contexts have an i.i.dassumption on the data while the last context deals withtransfer learning on relational data. Most of these approaches
13.http://multitask.cs.berkeley.edu/assume that the selected source domain is related to the target
domain.
In the future, several important research issues need to be
addressed. First, how to avoid negative transfer is an openproblem. As mentioned in Section 6, many proposed transferlearning algorithms assume that the source and target domainsare related to each other in some sense. However, if the as-sumption does not hold, negative transfer may happen, whichmay cause the learner to perform worse than no transferring atall. Thus, how to make sure that no negative transfer happensis a crucial issue in transfer learning. In order to avoid negative
transfer learning, we need to ﬁrst study transferability between
source domains or tasks and target domains or tasks. Based onsuitable transferability measures, we can then select relevantsource domains or tasks to extract knowledge from for learningthe target tasks. To de ﬁne the transferability between domains
and tasks, we also need to de ﬁne the criteria to measure the
similarity between domains or tasks. Based on the distancemeasures, we can then cluster domains or tasks, which may
help measure transferability. A related issue is when an entiredomain cannot be used for transfer learning, whether we canstill transfer part of the domain for useful learning in the targetdomain.
In addition, most existing transfer learning algorithms so far
focused on improving generalization across different distribu-tions between source and target domains or tasks. In doing so,they assumed that the feature spaces between the source andtarget domains are the same. However, in many applications,we may wish to transfer knowledge across domains or tasks
that have different feature spaces, and transfer from multiplesuch source domains. We refer to this type of transfer learning
asheterogeneous transfer learning .
Finally, so far transfer learning techniques have been mainly
applied to small scale applications with a limited variety, suchas sensor-network-based localization, text classi ﬁcation and
image classi ﬁcation problems. In the future, transfer learning
techniques will be widely used to solve other challenging ap-
plications, such as video classi ﬁcation, social network analysis
and logical inference.
Acknowledgment
We thank the support of Hong Kong CERG Project 621307
and a grant from NEC China Lab.
REFERENCES
[1] X. Wu, V . Kumar, J. R. Quinlan, J. Ghosh, Q. Yang, H. Motoda, G. J.
McLachlan, A. F. M. Ng, B. Liu, P. S. Yu, Z.-H. Zhou, M. Steinbach,D. J. Hand, and D. Steinberg, “Top 10 algorithms in data mining,”
Knowl. Inf. Syst. , vol. 14, no. 1, pp. 1–37, 2008.
[2] Q. Yang and X. Wu, “10 challenging problems in data mining research,”
International Journal of Information Technology and Decision Making ,
vol. 5, no. 4, pp. 597–604, 2006.
[3] G. P. C. Fung, J. X. Yu, H. Lu, and P. S. Yu, “Text classi ﬁcation without
negative examples revisit,” IEEE Transactions on Knowledge and Data
Engineering , vol. 18, no. 1, pp. 6–20, 2006.
[4] H. Al-Mubaid and S. A. Umair, “A new text categorization technique
using distributional clustering and learning logic,” IEEE Transactions
on Knowledge and Data Engineering , vol. 18, no. 9, pp. 1156–1165,
2006.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
13
TABLE 5
Comparison between transfer learning and non-transfer learning methods
Data Set (reference) Source v.s. Target Baselines TL Methods
20 Newsgroups 1([6]) SVM TrAdaBoost
ACC (unit: %) rec v.s. talk 87.3% 92.0%
rec v.s. sci 83.6% 90.3%
sci v.s. talk 82.3% 87.5%
20 Newsgroups 2([84]) SVM TrAdaBoost AcTraK
ACC (unit: %) rec v.s. talk 60.2% 72.3% 75.4%
rec v.s. sci 59.1% 67.4% 70.6%
comp v.s. talk 53.6% 74.4% 80.9%
comp v.s. sci 52.7% 57.3% 78.0%
comp v.s. rec 49.1% 77.2% 82.1%
sci v.s. talk 57.6% 71.3% 75.1%
20 Newsgroups 3([49]) SVM LR pLWE LWE
ACC (unit: %) comp v.s. sci 71.18% 73.49% 78.72% 97.44%
rec v.s. talk 68.24% 72.17% 72.17% 99.23%
rec v.s. sci 78.16% 78.85% 88.45% 98.23%
sci v.s. talk 75.77% 79.04% 83.30% 96.92%
comp v.s. rec 81.56% 83.34% 91.93% 98.16%
comp v.s. talk 93.89% 91.76% 96.64% 98.90%
Sentiment Classi ﬁcation ([8]) SGD SCL SCL-MI
ACC (unit: %) DVD v.s. book 72.8% 76.8% 79.7%
electronics v.s. book 70.7% 75.4% 75.4%
kitchen v.s. book 70.9% 66.1% 68.6%
book v.s. DVD 77.2% 74.0% 75.8%
electronics v.s. DVD 70.6% 74.3% 76.2%
kitchen v.s. DVD 72.7% 75.4% 76.9%
book v.s. electronics 70.8% 77.5% 75.9%
DVD v.s. electronics 73.0% 74.1% 74.1%
kitchen v.s. electronics 82.7% 83.7% 86.8%
book v.s. kitchen 74.5% 78.7% 78.9%
DVD v.s. kitchen 74.0% 79.4% 81.4%
electronics v.s. kitchen 84.0% 84.4% 85.9%
WiFi Localization ([67]) RLSR PCA KMM TCA
AED (unit: meter) Time Av.s. Time B 6.52 3.16 5.51 2.37
[5] K. Sarinnapakorn and M. Kubat, “Combining subclassi ﬁers in text cat-
egorization: A dst-based solution and a case study,” IEEE Transactions
on Knowledge and Data Engineering , vol. 19, no. 12, pp. 1638–1651,
2007.
[6] W. Dai, Q. Yang, G. Xue, and Y . Yu, “Boosting for transfer learning,” in
Proceedings of the 24th International Conference on Machine Learning ,
Corvalis, Oregon, USA, June 2007, pp. 193–200.
[7] S. J. Pan, V . W. Zheng, Q. Yang, and D. H. Hu, “Transfer learning
for wi ﬁ-based indoor localization,” in Proceedings of the Workshop on
Transfer Learning for Complex Task of the 23rd AAAI Conference on
Artiﬁcial Intelligence , Chicago, Illinois, USA, July 2008.
[8] J. Blitzer, M. Dredze, and F. Pereira, “Biographies, bollywood, boom-
boxes and blenders: Domain adaptation for sentiment classi ﬁcation,”
inProceedings of the 45th Annual Meeting of the Association of
Computational Linguistics , Prague, Czech Republic, 2007, pp. 432–439.
[9] J. Ramon, K. Driessens, and T. Croonenborghs, “Transfer learning in
reinforcement learning problems through partial policy recycling,” in
ECML ’07: Proceedings of the 18th European conference on MachineLearning . Berlin, Heidelberg: Springer-Verlag, 2007, pp. 699–707.
[10] M. E. Taylor and P. Stone, “Cross-domain transfer for reinforcement
learning,” in ICML ’07: Proceedings of the 24th international conference
on Machine learning . New York, NY , USA: ACM, 2007, pp. 879–886.
[11] X. Yin, J. Han, J. Yang, and P. S. Yu, “Ef ﬁcient classi ﬁcation across
multiple database relations: A crossmine approach,” IEEE Transactions
on Knowledge and Data Engineering , vol. 18, no. 6, pp. 770–783, 2006.
[12] L. I. Kuncheva and J. J. Rodr łguez, “Classi ﬁer ensembles with a random
linear oracle,” IEEE Transactions on Knowledge and Data Engineering ,
vol. 19, no. 4, pp. 500–508, 2007.
[13] E. Baralis, S. Chiusano, and P. Garza, “A lazy approach to associative
classi ﬁcation,” IEEE Transactions on Knowledge and Data Engineering ,
vol. 20, no. 2, pp. 156–171, 2008.
[14] X. Zhu, “Semi-supervised learning literature survey,” University ofWisconsin–Madison, Tech. Rep. 1530, 2006.
[15] K. Nigam, A. K. McCallum, S. Thrun, and T. Mitchell, “Text clas-
siﬁcation from labeled and unlabeled documents using em,” Machine
Learning , vol. 39, no. 2-3, pp. 103–134, 2000.
[16] A. Blum and T. Mitchell, “Combining labeled and unlabeled data with
co-training,” in Proceedings of the Eleventh Annual Conference on
Computational Learning Theory , 1998, pp. 92–100.
[17] T. Joachims, “Transductive inference for text classi ﬁcation using support
vector machines,” in Proceedings of Sixteenth International Conference
on Machine Learning , 1999, pp. 825–830.
[18] X. Zhu and X. Wu, “Class noise handling for effective cost-sensitive
learning by cost-guided iterative classi ﬁcation ﬁltering,” IEEE Transac-
tions on Knowledge and Data Engineering , vol. 18, no. 10, pp. 1435–
1440, 2006.
[19] Q. Yang, C. Ling, X. Chai, and R. Pan, “Test-cost sensitive classi ﬁcation
on data with missing values,” IEEE Transactions on Knowledge and
Data Engineering
, vol. 18, no. 5, pp. 626–638, 2006.
[20] S. Thrun and L. Pratt, Eds., Learning to learn . Norwell, MA, USA:
Kluwer Academic Publishers, 1998.
[21] R. Caruana, “Multitask learning,” Machine Learning , vol. 28(1), pp. 41–
75, 1997.
[22] R. Raina, A. Battle, H. Lee, B. Packer, and A. Y . Ng, “Self-taught
learning: Transfer learning from unlabeled data,” in Proceedings of the
24th International Conference on Machine Learning , Corvalis, Oregon,
USA, June 2007, pp. 759–766.
[23] H. Daum ´eIII and D. Marcu, “Domain adaptation for statistical classi-
ﬁers,” Journal of Arti ﬁcial Intelligence Research , vol. 26, pp. 101–126,
2006.
[24] B. Zadrozny, “Learning and evaluating classi ﬁers under sample selection
bias,” in Proceedings of the 21st International Conference on Machine
Learning , Banff, Alberta, Canada, July 2004.
[25] H. Shimodaira, “Improving predictive inference under covariate shift byIEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
14
weighting the log-likelihood function,” Journal of Statistical Planning
and Inference , vol. 90, pp. 227–244, 2000.
[26] W. Dai, Q. Yang, G. Xue, and Y . Yu, “Self-taught clustering,” in
Proceedings of the 25th International Conference of Machine Learning .
ACM, July 2008, pp. 200–207.
[27] Z. Wang, Y . Song, and C. Zhang, “Transferred dimensionality reduc-
tion,” in Machine Learning and Knowledge Discovery in Databases, Eu-
ropean Conference, ECML/PKDD 2008 . Antwerp, Belgium: Springer,
September 2008, pp. 550–565.
[28] W. Dai, G. Xue, Q. Yang, and Y . Yu, “Transferring naive bayes classi ﬁers
for text classi ﬁcation,” in Proceedings of the 22rd AAAI Conference on
Artiﬁcial Intelligence , Vancouver, British Columbia, Canada, July 2007,
pp. 540–545.
[29] J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D.
Lawrence, Dataset Shift in Machine Learning . The MIT Press, 2009.
[30] J. Jiang and C. Zhai, “Instance weighting for domain adaptation in
nlp,” in Proceedings of the 45th Annual Meeting of the Association of
Computational Linguistics . Prague, Czech Republic: Association for
Computational Linguistics, June 2007, pp. 264–271.
[31] X. Liao, Y . Xue, and L. Carin, “Logistic regression with an auxiliary
data source,” in Proceedings of the 21st International Conference on
Machine Learning , Bonn, Germany, August 2005, pp. 505–512.
[32] J. Huang, A. Smola, A. Gretton, K. M. Borgwardt, and B. Sch ¨olkopf,
“Correcting sample selection bias by unlabeled data,” in Proceedings of
the 19th Annual Conference on Neural Information Processing Systems ,
2007.
[33] S. Bickel, M. Br ¨uckner, and T. Scheffer, “Discriminative learning for
differing training and test distributions,” in Proceedings of the 24th
international conference on Machine learning . New York, NY , USA:
ACM, 2007, pp. 81–88.
[34] M. Sugiyama, S. Nakajima, H. Kashima, P. V . Buenau, and M. Kawan-
abe, “Direct importance estimation with model selection and its appli-cation to covariate shift adaptation,” in Proceedings of the 20th Annual
Conference on Neural Information Processing Systems , Vancouver,
British Columbia, Canada, December 2008.
[35] W. Fan, I. Davidson, B. Zadrozny, and P. S. Yu, “An improved
categorization of classi ﬁer’s sensitivity on sample selection bias,” in
Proceedings of the 5th IEEE International Conference on Data Mining ,
2005.
[36] W. Dai, G. Xue, Q. Yang, and Y . Yu, “Co-clustering based classi ﬁca-
tion for out-of-domain documents,” in Proceedings of the 13th ACM
SIGKDD International Conference on Knowledge Discovery and Data
Mining , San Jose, California, USA, August 2007.
[37] R. K. Ando and T. Zhang, “A high-performance semi-supervised learn-
ing method for text chunking,” in Proceedings of the 43rd Annual
Meeting on Association for Computational Linguistics . Morristown,
NJ, USA: Association for Computational Linguistics, 2005, pp. 1–9.
[38] J. Blitzer, R. McDonald, and F. Pereira, “Domain adaptation with
structural correspondence learning,” in Proceedings of the Conference on
Empirical Methods in Natural Language , Sydney, Australia, July 2006,
pp. 120–128.
[39] H. Daum ´e III, “Frustratingly easy domain adaptation,” in Proceedings
of the 45th Annual Meeting of the Association of Computational Lin-
guistics , Prague, Czech Republic, June 2007, pp. 256–263.
[40] A. Argyriou, T. Evgeniou, and M. Pontil, “Multi-task feature learning,”
inProceedings of the 19th Annual Conference on Neural Information
Processing Systems
, Vancouver, British Columbia, Canada, December
2007, pp. 41–48.
[41] A. Argyriou, C. A. Micchelli, M. Pontil, and Y . Ying, “A spectral regu-
larization framework for multi-task structure learning,” in Proceedings of
the 20th Annual Conference on Neural Information Processing Systems .
Cambridge, MA: MIT Press, 2008, pp. 25–32.
[42] S.-I. Lee, V . Chatalbashev, D. Vickrey, and D. Koller, “Learning a
meta-level prior for feature relevance from multiple related tasks,” in
Proceedings of the 24th International Conference on Machine Learning .
Corvalis, Oregon, USA: ACM, July 2007, pp. 489–496.
[43] T. Jebara, “Multi-task feature and kernel selection for svms,” in Proceed-
ings of the 21st International Conference on Machine Learning .B a n f f ,
Alberta, Canada: ACM, July 2004.
[44] C. Wang and S. Mahadevan, “Manifold alignment using procrustes
analysis,” in Proceedings of the 25th International Conference on
Machine learning . Helsinki, Finland: ACM, July 2008, pp. 1120–1127.
[45] N. D. Lawrence and J. C. Platt, “Learning to learn with the informative
vector machine,” in Proceedings of the 21st International Conference
on Machine Learning . Banff, Alberta, Canada: ACM, July 2004.
[46] E. Bonilla, K. M. Chai, and C. Williams, “Multi-task gaussian process
prediction,” in Proceedings of the 20th Annual Conference on NeuralInformation Processing Systems . Cambridge, MA: MIT Press, 2008,
pp. 153–160.
[47] A. Schwaighofer, V . Tresp, and K. Yu, “Learning gaussian process
kernels via hierarchical bayes,” in Proceedings of the 17th Annual
Conference on Neural Information Processing Systems . Cambridge,
MA: MIT Press, 2005, pp. 1209–1216.
[48] T. Evgeniou and M. Pontil, “Regularized multi-task learning,” in
Proceedings of the 10th ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining . Seattle, Washington, USA:
ACM, August 2004, pp. 109–117.
[49] J. Gao, W. Fan, J. Jiang, and J. Han, “Knowledge transfer via multiple
model local structure mapping,” in Proceedings of the 14th ACM
SIGKDD International Conference on Knowledge Discovery and Data
Mining . Las Vegas, Nevada: ACM, August 2008, pp. 283–291.
[50] L. Mihalkova, T. Huynh, and R. J. Mooney, “Mapping and revising
markov logic networks for transfer learning,” in Proceedings of the
22nd AAAI Conference on Arti ﬁcial Intelligence , Vancouver, British
Columbia, Canada, July 2007, pp. 608–614.
[51] L. Mihalkova and R. J. Mooney, “Transfer learning by mapping with
minimal target data,” in Proceedings of the AAAI-2008 Workshop on
Transfer Learning for Complex Tasks , Chicago, Illinois, USA, July 2008.
[52] J. Davis and P. Domingos, “Deep transfer via second-order markov
logic,” in Proceedings of the AAAI-2008 Workshop on Transfer Learning
for Complex Tasks , Chicago, Illinois, USA, July 2008.
[53] P. Wu and T. G. Dietterich, “Improving svm accuracy by training
on auxiliary data sources,” in Proceedings of the 21st International
Conference on Machine Learning . Banff, Alberta, Canada: ACM, July
2004.
[54] U. R ¨uckert and S. Kramer, “Kernel-based inductive transfer,” in Machine
Learning and Knowledge Discovery in Databases, European Confer-
ence, ECML/PKDD 2008 , ser. Lecture Notes in Computer Science.
Antwerp, Belgium: Springer, September 2008, pp. 220–233.
[55] H. Lee, A. Battle, R. Raina, and A. Y . Ng, “Ef ﬁcient sparse coding
algorithms,” in Proceedings of the the 19th Annual Conference on Neural
Information Processing Systemss . Cambridge, MA: MIT Press, 2007,
pp. 801–808.
[56] M. Richardson and P. Domingos, “Markov logic networks,” Machine
Learning Journal , vol. 62, no. 1-2, pp. 107–136, 2006.
[57] S. Ramachandran and R. J. Mooney, “Theory re ﬁnement of bayesian
networks with hidden variables,” in Proceedings of the 14th Interna-
tional Conference on Machine Learning , Madison, Wisconson, USA,
July 1998, pp. 454–462.
[58] A. Arnold, R. Nallapati, and W. W. Cohen, “A comparative study of
methods for transductive transfer learning,” in Proceedings of the 7th
IEEE International Conference on Data Mining Workshops . Washing-
ton, DC, USA: IEEE Computer Society, 2007, pp. 77–82.
[59] T. Joachims, “Transductive inference for text classi ﬁcation using support
vector machines,” in Proceedings of the Sixteenth International Confer-
ence on Machine Learning , San Francisco, CA, USA, 1999, pp. 200–
209.
[60] V . N. Vapnik, Statistical Learning Theory . New York: Wiley-
Interscience, September 1998.
[61] S. Ben-David, J. Blitzer, K. Crammer, and F. Pereira, “Analysis of rep-
resentations for domain adaptation,” in Proceedings of the 20th Annual
Conference on Neural Information Processing Systems . Cambridge,
MA: MIT Press, 2007, pp. 137–144.
[62] J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. Wortman, “Learn-
ing bounds for domain adaptation,” in Proceedings of the 21st Annual
Conference on Neural Information Processing Systems . Cambridge,
MA: MIT Press, 2008, pp. 129–136.
[63] D. Xing, W. Dai, G.-R. Xue, and Y . Yu, “Bridged re ﬁnement for transfer
learning,” in 11th European Conference on Principles and Practice of
Knowledge Discovery in Databases , ser. Lecture Notes in Computer
Science. Warsaw, Poland: Springer, September 2007, pp. 324–335.
[64] X. Ling, W. Dai, G.-R. Xue, Q. Yang, and Y . Yu, “Spectral domain-
transfer learning,” in Proceedings of the 14th ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and Data Mining . Las
Vegas, Nevada: ACM, August 2008, pp. 488–496.
[65] G.-R. Xue, W. Dai, Q. Yang, and Y . Yu, “Topic-bridged plsa for
cross-domain text classi ﬁcation,” in Proceedings of the 31st Annual
International ACM SIGIR Conference on Research and Developmentin Information Retrieval . Singapore: ACM, July 2008, pp. 627–634.
[66] S. J. Pan, J. T. Kwok, and Q. Yang, “Transfer learning via dimensionality
reduction,” in Proceedings of the 23rd AAAI Conference on Arti ﬁcial
Intelligence , Chicago, Illinois, USA, July 2008, pp. 677–682.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.
15
[67] S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, “Domain adaptation via
transfer component analysis,” in Proceedings of the 21st International
Joint Conference on Arti ﬁcial Intelligence , Pasadena, California, 2009.
[68] M. M. H. Mahmud and S. R. Ray, “Transfer learning using kolmogorov
complexity: Basic theory and empirical evaluations,” in Proceedings of
the 20th Annual Conference on Neural Information Processing Systems .
Cambridge, MA: MIT Press, 2008, pp. 985–992.
[69] E. Eaton, M. desJardins, and T. Lane, “Modeling transfer relationships
between learning tasks for improved inductive transfer,” in Machine
Learning and Knowledge Discovery in Databases, European Confer-
ence, ECML/PKDD 2008 , ser. Lecture Notes in Computer Science.
Antwerp, Belgium: Springer, September 2008, pp. 317–332.
[70] M. T. Rosenstein, Z. Marx, and L. P. Kaelbling, “To transfer or not to
transfer,” in a NIPS-05 Workshop on Inductive Transfer: 10 Years Later ,
December 2005.
[71] S. Ben-David and R. Schuller, “Exploiting task relatedness for multiple
task learning,” in Proceedings of the Sixteenth Annual Conference on
Learning Theory . San Francisco: Morgan Kaufmann, 2003, pp. 825–
830.
[72] B. Bakker and T. Heskes, “Task clustering and gating for bayesian
multitask learning,” Journal of Machine Learning Reserch ,v o l .4 ,p p .
83–99, 2003.
[73] A. Argyriou, A. Maurer, and M. Pontil, “An algorithm for transfer learn-
ing in a heterogeneous environment,” in Machine Learning and Knowl-
edge Discovery in Databases, European Conference, ECML/PKDD
2008 , ser. Lecture Notes in Computer Science. Antwerp, Belgium:
Springer, September 2008, pp. 71–85.
[74] R. Raina, A. Y . Ng, and D. Koller, “Constructing informative priors using
transfer learning.” in Proceedings of the 23rd International Conference
on Machine Learning , Pittsburgh, Pennsylvania, USA, June 2006, pp.
713–720.
[75] J. Yin, Q. Yang, and L. M. Ni, “Adaptive temporal radio maps for indoor
location estimation,” in Proceedings of the 3rd IEEE International
Conference on Pervasive Computing and Communications , Kauai Island,
Hawaii, USA, March 2005.
[76] S. J. Pan, J. T. Kwok, Q. Yang, and J. J. Pan, “Adaptive localization in
a dynamic WiFi environment through multi-view learning,” in Proceed-
ings of the 22nd AAAI Conference on Arti ﬁcial Intelligence , Vancouver,
British Columbia, Canada, July 2007, pp. 1108–1113.
[77] V . W. Zheng, Q. Yang, W. Xiang, and D. Shen, “Transferring localization
models over time,” in Proceedings of the 23rd AAAI Conference on
Artiﬁcial Intelligence , Chicago, Illinois, USA, July 2008, pp. 1421–1426.
[78] S. J. Pan, D. Shen, Q. Yang, and J. T. Kwok, “Transferring localization
models across space,” in Proceedings of the 23rd AAAI Conference on
Artiﬁcial Intelligence , Chicago, Illinois, USA, July 2008, pp. 1383–1388.
[79] V . W. Zheng, S. J. Pan, Q. Yang, and J. J. Pan, “Transferring multi-device
localization models using latent multi-task learning,” in Proceedings of
the 23rd AAAI Conference on Arti ﬁcial Intelligence , Chicago, Illinois,
USA, July 2008, pp. 1427–1432.
[80] H. Zhuo, Q. Yang, D. H. Hu, and L. Li, “Transferring knowledge
from another domain for learning action models,” in Proceedings of
10th Paci ﬁc Rim International Conference on Arti ﬁcial Intelligence ,
December 2008.
[81] V . C. Raykar, B. Krishnapuram, J. Bi, M. Dundar, and R. B. Rao,
“Bayesian multiple instance learning: automatic feature selection andinductive transfer,” in Proceedings of the 25th International Conference
on Machine learning . Helsinki, Finland: ACM, July 2008, pp. 808–815.
[82] X. Ling, G.-R. Xue, W. Dai, Y . Jiang, Q. Yang, and Y . Yu, “Can chinese
web pages be classi ﬁed with english data source?” in Proceedings of
the 17th International Conference on World Wide Web . Beijing, China:
ACM, April 2008, pp. 969–978.
[83] Q. Yang, S. J. Pan, and V . W. Zheng, “Estimating location using Wi-Fi,”
IEEE Intelligent Systems , vol. 23, no. 1, pp. 8–13, 2008.
[84] X. Shi, W. Fan, and J. Ren, “Actively transfer domain knowledge,” in
Machine Learning and Knowledge Discovery in Databases, European
Conference, ECML/PKDD 2008 , ser. Lecture Notes in Computer Sci-
ence. Antwerp, Belgium: Springer, September 2008, pp. 342–357.
[85] G. Kuhlmann and P. Stone, “Graph-based domain mapping for transfer
learning in general games,” in 18th European Conference on Machine
Learning , ser. Lecture Notes in Computer Science. Warsaw, Poland:
Springer, September 2007, pp. 188–200.
[86] W. Dai, Y . Chen, G.-R. Xue, Q. Yang, and Y . Yu, “Translated learning,”
inProceedings of 21st Annual Conference on Neural Information
Processing Systems , 2008.
[87] B. Li, Q. Yang, and X. Xue, “Transfer learning for collaborative
ﬁltering via a rating-matrix generative model,” in Proceedings of the26th International Conference on Machine Learning , Montreal, Quebec,
Canada, June 2009.
[88] B. Li, Q. Yang, and X. Xue, “Can movies and books collaborate? - cross-
domain collaborative ﬁltering for sparsity reduction,” in Proceedings
of the 21st International Joint Conference on Arti ﬁcial Intelligence ,
Pasadena, California, USA, July 2009.
Sinno Jialin Pan is a PhD candidate in the
Department of Computer Science and Engineer-
ing, the Hong Kong University of Science andTechnology. He received the MS and BS de-
grees from Applied Mathematics Department,
Sun Y at-sen University, China, in 2003 and
2005, respectively. His research interests in-
clude transfer learning, semi-supervised learn-ing, and their applications in pervasive comput-
ing and Web mining. He is a member of AAAI.
Contact him at the Department of Computer
Science and Engineering, Hong Kong Univ. of Science and Tech-
nology, Clearwater Bay, Kowloon, Hong Kong; sinnopan@cse.ust.hk;
http://www.cse.ust.hk/ ∼sinnopan.
Qiang Yang is a faculty member in the Hong
Kong University of Science and Technology’sDepartment of Computer Science and Engineer-
ing. His research interests are data mining and
machine learning, AI planning and sensor based
activity recognition. He received his PhD de-
gree in Computer Science from the University ofMaryland, College Park, and Bachelor’s degree
from Peking University in Astrophysics. He is a
fellow of IEEE, member of AAAI and ACM, aformer associate editor for the IEEE TKDE, and
a current associate editor for IEEE Intelligent Systems. Contact him
at the Department of Computer Science and Engineering, Hong Kong
Univ. of Science and Technology, Clearwater Bay, Kowloon, Hong Kong;
qyang@cse.ust.hk; http://www.cse.ust.hk/ ∼qyang.IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERINGThis article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may ch ange prior to final publication.


Learning with catastrophes

A *catastrophe* is an event so bad that we are not willing to let it happen even a single time. For example, we would be unhappy if our self-driving car *ever* accelerates to 65 mph in a residential area and hits a pedestrian.

Catastrophes present a theoretical challenge for traditional machine learning — typically there is no way to reliably avoid catastrophic behavior without strong statistical assumptions.

In this post, I’ll lay out a very general model for catastrophes in which they areavoidable under much weaker statistical assumptions*.* I think this framework applies to the most important kinds of catastrophe, and will be especially relevant to AI alignment.

Designing practical algorithms that work in this model is an open problem. In a [subsequent post](/techniques-for-optimizing-worst-case-performance-39eafec74b99) I describe what I currently see as the most promising angles of attack.

Modeling catastrophes
---------------------

We consider an agent A interacting with the environment over a sequence of episodes. Each episode produces a transcript τ, consisting of the agent’s observations and actions, along with a reward *r* ∈ [0, 1]. Our primary goal is to quickly learn an agent which receives high reward. (Supervised learning is the special case where each transcripts consist of a single input and a label for that input.)

While training, we assume that we have an oracle which can determine whether a transcript τ is “catastrophic.” For example, we might show a transcript to a QA analyst and ask them if it looks catastrophic. This oracle can be applied to arbitrary sequences of observations and actions, including those that don’t arise from an actual episode*.* So training can begin before the very first interaction with nature, using only calls to the oracle.

Intuitively, a transcript should only be marked catastrophic if it satisfies two conditions:

1. The agent made a catastrophically bad decision.
2. The agent’s observations are plausible: we have a right to expect the agent to be able to handle those observations.

While actually interacting with the environment, the agent cannot query the oracle — there is no time to wait for a QA engineer to review a proposed action to check if it would be catastrophic.

Moreover, if interaction with nature ever produces a catastrophic transcript, we immediately fail. The performance of an algorithm is characterized by two parameters: the probability of catastrophic failure, and the total reward assuming no catastrophic failure.

We assume that there are some policies such that no matter what nature does, the resulting transcript is *never* catastrophic.

Traditionally in RL the goal is to get as much reward as the best policy from some class C. We’ slightly weaken that goal, and instead aim to do as well as the best policy from C that never makes a catastrophic decision.

Batch learning
--------------

I’ve described an online version of learning with catastrophes. We can also consider the batch version, where the learner is first given a large number of “training” episodes.

In the batch version, there is no penalty for catastrophes at training time, and we don’t care about training error. The two performance parameters are test-time performance and test-time catastrophe probability.

The oracle
----------

This definition depends on an oracle who determines which transcripts are catastrophic.

For weak AI systems, the oracle may be a human. But a powerful AI system might take actions which are catastrophic but which look inoffensive to a human judge, so this approach doesn’t cut it.

In general, the judge should be a human+AI team which is *more* competent than the system being trained, armed with an adequate solution to the [informed oversight problem](https://medium.com/ai-control/the-informed-oversight-problem-1b51b4f66b35).

Approach
========

Learning with catastrophes is straightforward *given an unlimited number of queries to the catastrophe oracle*. Given any online learning algorithm A, we can “harden” it by running the following process before the beginning of each episode (a similar process can be applied to a batch learner):

1. Search over all possible environments, running A on each one to obtain a transcript.
2. If we find a catastrophic transcript τ, then add τ to A’s training data with a reward of −1, and go back to step 1.

It’s easy to prove that this process converges, if A is competent: the number of times we invoke step 2 is at most the time required to learn an optimal catastrophe-free policy (plus the number of episodes).

The big problem with this algorithm is the exponential search in step 1.

Adversarial training
--------------------

The most obvious approach to *tractable* learning with catastrophes is to replace step 1 with a tractable algorithm that approximates the search. This process is an example of [adversarial training](https://arxiv.org/abs/1412.6572).

We can formalize this as a simple game between the agent A and an adversary B. The two agents alternate taking moves, with A playing the role of the agent and B playing the role of the environment. B wins if the resulting transcript is catastrophic, A wins if it is not catastrophic.

Before each episode, we play this game N times, where N > 0 is a hyperparameter. If B is able to win *any* of the games, then we add the resulting transcript to A’s training data and start again. Once B is not able to win any games, then we move on to a “real” episode. We hope that B is good enough at the game that if it can’t produce a catastrophic transcript, then the real environment can’t produce a catastrophic transcript either.

More precisely, before each episode we perform the following process:

1. Set *i* = 0.
2. A and B alternate taking moves, producing a transcript τ.
3. If τ is catastrophic, we add τ to A’s training data with a reward of −1, and add τ to B’s training data with a reward of +1. Then we go back to step 1.
4. If τ is not catastrophic, we add τ to B’s training data with a reward of −1.
5. If *i* < N, we increment *i* and go back to step 2.

I discuss this idea in more detail in my [post on red teams](https://medium.com/@paulfchristiano/b5b6de33dc76). There are serious problems with this approach and I don’t think it can work on its own, but fortunately it seems combinable with [other techniques](https://arxiv.org/abs/1412.6572).

Conclusion
==========

Learning with catastrophes is a very general model of catastrophic failures which avoids being obviously impossible. I think that designing competent algorithms for learning with catastrophes may be an important ingredient in a successful approach to AI alignment.