First-Order Federated Bilevel Learning
Open Access
- 11 April 2025
- journal article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (21) , 22029-22037
Abstract
Federated bilevel optimization (FBO) has garnered significant attention lately, driven by its promising applications in meta-learning and hyperparameter optimization. Existing algorithms generally aim to approximate the gradient of the upper-level objective function (hypergradient) in the federated setting. However, because of the nonlinearity of the hypergradient and client drift, they often involve complicated computations. These computations, like multiple optimization sub-loops and second-order derivative evaluations, end up with significant memory consumption and high computational costs. In this paper, we propose a computationally and memory-efficient FBO algorithm named MemFBO. MemFBO features a fully single-loop structure with all involved variables updated simultaneously, and uses only first-order gradient information for all local updates. We show that MemFBO exhibits a linear convergence speedup with milder assumptions in both partial and full client participation scenarios. We further implement MemFBO in a novel FBO application for federated data cleaning. Our experiments, conducted on this application and federated hyper-representation, demonstrate the effectiveness of the proposed algorithm.Keywords
This publication has 16 references indexed in Scilit:
- On penalty-based bilevel gradient descent methodMathematical Programming, 2025
- Network Utility Maximization with Unknown Utility Functions: A Distributed, Data-Driven Bilevel Optimization Approach2023
- A Two-Timescale Stochastic Algorithm Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-CriticSIAM Journal on Optimization, 2023
- BiG-Fed: Bilevel Optimization Enhanced Graph-Aided Federated LearningIEEE Transactions on Big Data, 2022
- A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and ApplicationsIEEE Transactions on Evolutionary Computation, 2017
- A First Order Method for Solving Convex Bilevel Optimization ProblemsSIAM Journal on Optimization, 2017
- On solving simple bilevel programs with a nonconvex lower level programMathematical Programming, 2013
- Cubic regularization of Newton method and its global performanceMathematical Programming, 2006
- An extended Kuhn–Tucker approach for linear bilevel programmingApplied Mathematics and Computation, 2005
- New Branch-and-Bound Rules for Linear Bilevel ProgrammingSIAM Journal on Scientific and Statistical Computing, 1992