Cliff-Learning
Authors:
- Tony T. WangTWTony T. Wang
- Igor ZablotchiIZIgor Zablotchi
- Nir ShavitNSNir Shavit
- Jonathan S. RosenfeldJRJonathan S. Rosenfeld
Preprint
Abstract
We study the data-scaling of transfer learning from foundation models in the low-downstream-data regime. We observe an intriguing phenomenon which we call cliff-learning. Cliff-learning refers to regions of data-scaling laws where performance improves at a faster than power law rate (i.e. regions of concavity on a log-log scaling plot). We conduct an in-depth investigation of foundation-model cliff-learning and study toy models of the phenomenon. We observe that the degree of cliff-learning reflects the degree of compatibility between the priors of a learning algorithm and the task being learned.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Explaining neural scaling lawsProceedings of the National Academy of Sciences of the United States of America, 2024
- The Shape of Learning Curves: A ReviewIEEE Transactions on Pattern Analysis and Machine Intelligence, 2022
- Neural representational geometry underlies few-shot concept learningProceedings of the National Academy of Sciences, 2022
- A Distribution-Free Theory of Nonparametric Regression2002