An Introduction to Bilevel Optimization: Foundations and applications in signal processing and machine learning
Top Cited Papers
- 1 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Magazine
- Vol. 41 (1) , 38-59
Abstract
Recently, bilevel optimization (BLO) has taken center stage in some very exciting developments in the area of signal processing (SP) and machine learning (ML). Roughly speaking, BLO is a classical optimization problem that involves two levels of hierarchy (i.e., upper and lower levels), wherein obtaining the solution to the upper-level problem requires solving the lower-level one. BLO has become popular largely because it is powerful in modeling problems in SP and ML, among others, that involve optimizing nested objective functions. Prominent applications of BLO range from resource allocation for wireless systems to adversarial ML. In this work, we focus on a class of tractable BLO problems that often appear in SP and ML applications. We provide an overview of some basic concepts of this class of BLO problems, such as their optimality conditions, standard algorithms (including their optimization principles and practical implementations) as well as how they can be leveraged to obtain state-of-the-art results for several key SP and ML applications. Further, we discuss some recent advances in BLO theory and its implications for applications, and we point out some limitations of the state of the art that require significant future research efforts. We hope that this article, together with the associated open source BLO toolbox we developed for algorithm benchmarking, can serve to accelerate the adoption of BLO as a generic tool to model, analyze, and innovate on a wide array of emerging SP and ML applications.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- On a Computationally Ill-Behaved Bilevel Problem with a Continuous and Nonconvex Lower LevelJournal of Optimization Theory and Applications, 2023
- A survey on bilevel optimization under uncertaintyEuropean Journal of Operational Research, 2023
- Bilevel Optimization: Theory, Algorithms, Applications and a Bibliography2020
- Continual lifelong learning with neural networks: A reviewNeural Networks, 2019
- A Very Brief Introduction to Machine Learning With Applications to Communication SystemsIEEE Transactions on Cognitive Communications and Networking, 2018
- A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and ApplicationsIEEE Transactions on Evolutionary Computation, 2017
- The Implicit Function Theorem2013
- Conjugate gradient methodWIREs Computational Statistics, 2009
- An overview of bilevel optimizationAnnals of Operations Research, 2007
- Channel Estimation for MIMO-OFDM Systems by Modal Analysis/FilteringIEEE Transactions on Communications, 2006