Polynomial Regression as a Task for Understanding In-context Learning Through Finetuning and Alignment
Authors:
- Max WilcoxsonMWMax Wilcoxson
- Morten SvendgårdMSMorten Svendgård
- Ria DoshiRDRia Doshi
- Dylan DavisDDDylan Davis
- Reya VirRVReya Vir
- Anant SahaiASAnant Sahai
Preprint
Abstract
Simple function classes have emerged as toy problems to better understand in-context-learning in transformer-based architectures used for large language models. But previously proposed simple function classes like linear regression or multi-layer-perceptrons lack the structure required to explore things like prompting and alignment within models capable of in-context-learning. We propose univariate polynomial regression as a function class that is just rich enough to study prompting and alignment, while allowing us to visualize and understand what is going on clearly.Keywords
All Related Versions
This publication has 2 references indexed in Scilit:
- Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and AssessmentIEEE Transactions on Pattern Analysis and Machine Intelligence, 2026
- The Power of Scale for Parameter-Efficient Prompt Tuning2021