what to transfer
“What to transfer” asks which part of knowledge can be transferred across domains or tasks. Some knowledge is specific for individual domains or tasks, and some knowledge may be common between different domains such that they may help improve performance for the target domain or task.
how to transfer
After discovering which knowledge can be transferred, learning algorithms need to be developed to transfer the knowledge, which corresponds to the “how to transfer” issue.
when to transfer
Most current work on transfer learning focuses on “What to transfer” and “How to transfer”, by implicitly assuming that the source and target domains be related to each other. However, how to avoid negative transfer is an important open issue that is attracting more and more attention in the future
The fundamental motivation for Transfer learning in the field of machine learning was
discussed in a NIPS-95 workshop on “Learning to Learn” 1, which focused on the need for lifelong machine-learning methods that retain and reuse previously learned knowledge
- inductive transfer learning (推导迁移学习)
- transductive transfer learning (转导迁移学习)
- unsupervised transfer learning (无监督迁移学习)
Transfer learning is classified to three different settings: inductive transfer learning, transductive transfer learning and unsupervised transfer learning. Most previous works focused on the former two settings. Unsupervised transfer learning may attract more and more attention in the future.
the target task is different from the source task, no matter when the source and target domains are the same or not.
In this case, some labeled data in the target domain are required to induce an objective predictive model fT(·) for use in the target domain. In addition, according to different situations of labeled and unlabeled data in the source domain, we can further categorize the inductive transfer learning setting into two cases.
- A lot of labeled data in the source domain are available
In this case, the inductive transfer learning setting is similar to the multi-task learning setting. However, the inductive transfer learning setting only aims at achieving high performance in the target task by transferring knowledge from the source task while multi-task learning tries to learn the target and source task simultaneously
- No labeled data in the source domain are available.
In this case, the inductive transfer learning setting is similar to the self-taught learning setting, which is first proposed by Raina et al. . In the self-taught learning setting, the label spaces between the source and target domains may be different, which implies the side information of the source domain cannot be used directly. Thus, it’s similar to the inductive transfer learning setting where the labeled data in the source domain are unavailable.
- The feature spaces between the source and target domains are different,
- The feature spaces between domains are the same but the marginal probability distributions of the input data are different.
The latter case of the transductive transfer learning setting is related to domain adaptation for knowledge transfer in text classification  and sample selection bias  or co-variate shift , whose assumptions are similar.
In this situation, no labeled data in the target domain are available while a lot of labeled data in the source domain are available. In addition, according to different situations between the source and target domains, we can further categorize the transductive transfer learning setting into two cases
different task but related to the source task, unsupervised transfer learning focuses on solving unsupervised learning tasks in the target domain. Thus, no labeled data available in both source and target domains in training)
- instance-transfer approach
- feature-representation-transfer approach
- parameter-transfer approach
- relational-knowledge-transfer approach
The former three contexts have an i.i.d assumption on the data while the last context deals with transfer learning on relational data. Most of these approaches assume that the selected source domain is related to the target domain.
- how to avoid negative transfer
- most existing transfer learning algorithms so far focused on improving generalization across different distributions between source and target domains or tasks.
- 目前机器学习算法和深度学习算法的一个前提条件：the training and test data are drawn from the same feature space and the same distribution.