Why curriculum learning & self-paced learning work in big/noisy data: A theoretical perspective
Open Access
- 1 September 2015
- journal article
- Published by American Institute of Mathematical Sciences (AIMS) in Big Data and Information Analytics
- Vol. 1 (1) , 111-127
Abstract
Since being recently raised, curriculum learning (CL) and self-paced learning (SPL) have attracted increasing attention due to its multiple successful applications. While currently the rationality of this learning regime is heuristically inspired by the cognitive principle of humans, there still isn't a sound theory to explain the intrinsic mechanism leading to its effectiveness, especially on some successful attempts on big/noise data. To address this issue, this paper presents some theoretical results for revealing the insights under this learning scheme. Specifically, we first formulate a new learning problem aiming to learn a proper classifier from samples generated from the training distribution which is deviated from the target distribution. Furthermore, we find that the CL/SPL regime provides a feasible solving strategy for this learning problem. Especially, by first introducing high-confidence/easy samples and gradually involving low-confidence/complex ones into learning, the CL/SPL process latently minimizes an upper bound of the expected risk under target distribution, purely using the data from the deviated training distribution. We further construct a new SPL learning algorithm based on random sampling, which better complies with our theory, and substantiate its effectiveness by experiments implemented on synthetic and real data.Keywords
This publication has 9 references indexed in Scilit:
- Easy Samples First2014
- NEIL: Extracting Visual Knowledge from Web Data2013
- Self-paced dictionary learning for image classification2012
- Learning the easy things first: Self-paced visual category discovery2011
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011
- Supervised Learning with Minimal EffortLecture Notes in Computer Science, 2010
- Curriculum learning2009
- Learning Theory2007
- On the mathematical foundations of learningBulletin of the American Mathematical Society, 2001