Pretraining and the lasso

Authors:
Publisher website
Google Scholar
Abstract
Pre-training is a powerful paradigm in machine learning to pass information across models. For example, suppose one has a modest-sized dataset of images of cats and dogs and plans to fit a deep neural network to classify them. With pre-training, we start with a neural network trained on a large corpus of images of not just cats and dogs but hundreds of classes. We fix all network weights except the top layer(s) and fine tune on our dataset. This often results in dramatically better performance than training solely on our dataset. Here, we ask: ‘Can pre-training help the lasso?’. We propose a framework where the lasso is fit on a large dataset and then fine-tuned on a smaller dataset. The latter can be a subset of the original, or have a different but related outcome. This framework has a wide variety of applications, including stratified and multi-response models. In the stratified model setting, lasso pre-training first estimates coefficients common to all groups, then estimates group-specific coefficients during fine-tuning. Under appropriate assumptions, support recovery of the common coefficients is superior to the usual lasso trained on individual groups. This separate identification of common and individual coefficients also aids scientific understanding.
Funding Information
  • National Center for Advancing Translational Sciences of the National Institutes of Health
  • NHGRI
  • NIMH
  • NIH (5R01EB001988-16)
  • NSF (19DMS1208164)
  • NSF (DMS-2134248)
  • NSF CAREER (CCF-2236829)
  • U.S. Army Research Office Early Career (W911NF-21-1-0242)
  • Stanford Precourt Institute
  • Stanford Data Science Scholars Program
  • Stanford Graduate Fellowship
  • NIGMS
  • Stanford University Discovery Innovation Award
  • Chan Zuckerberg Data Insights
  • Western Norway Regional Health Authority

This publication has 14 references indexed in Scilit: