Nicola Bastianello

This paper studies distributed composite optimization in open multi-agent systems, where agents may join and leave the network over time. We propose a variant of Open ADMM in which the composite proximal step is approximated through a finite number of proximal-gradient iterations, enabling agents to handle composite objectives even when the exact proximal operator is not available in closed form. Since the inexact computation of the proximal step induces additive errors, we establish a convergence result for paracontractive open multi-agent systems subject to bounded additive perturbations that capture both inexact local updates and noisy communications. Under mild assumptions on the network topology, the proposed algorithm converges linearly to a neighborhood of the optimal consensus set, with an explicit bound on the steady-state tracking error. The theoretical guarantees are corroborated by numerical experiments on distributed learning applications, namely linear regression (supervised learning) and principal component analysis (unsupervised learning).

We address distributed learning problems, both nonconvex and convex, over undirected networks. In particular, we design a novel algorithm based on the distributed Alternating Direction Method of Multipliers (ADMM) to tackle the challenges of high communication costs, and large datasets. Our design deals with these challenges i) by enabling the agents to perform multiple local training steps between each round of communications; and ii) by allowing the agents to employ stochastic gradients while carrying out local computations. We show that the proposed algorithm converges to a neighborhood of a stationary point, for nonconvex problems, and of an optimal point, for convex problems.We also propose a variant of the algorithm to incorporate variance reduction thus achieving exact convergence. We show that the resulting algorithm indeed converges to a stationary (or optimal) point, and moreover that local training accelerates convergence. We thoroughly compare the proposed algorithms with the state of the art, both theoretically and through numerical results.

Modern artificial intelligence relies on networks of agents that collect data, process information, and exchange it with neighbors to collaboratively solve optimization and learning problems. This article introduces a novel distributed algorithm to address a broad class of these problems in open networks, where the number of participating agents may vary due to several factors, such as autonomous decisions, heterogeneous resource availability, or denial of service (DoS) attacks. Extending the current literature, the convergence analysis of the proposed algorithm is based on the newly developed Theory of Open Operators, which characterizes an operator as open when the set of components to be updated changes over time, yielding time-varying operators acting on sequences of points of different dimensions and compositions. The mathematical tools and convergence results developed here provide a general framework for evaluating distributed algorithms in open networks, enabling characterization of their performance in terms of the punctual distance to the optimal solution, in contrast with regret-based metrics that assess cumulative performance over a finite-time horizon. As illustrative examples, the proposed algorithm is used to solve dynamic consensus and tracking problems on different metrics, such as average, median, and min/max, as well as classification problems with logistic loss functions.

In this paper, we address two practical challenges of distributed learning in multi-agent network systems, namely personalization and resilience. Personalization is the need of heterogeneous agents to learn local models tailored to their own data and tasks, while still generalizing well; on the other hand, the learning process must be resilient to cyberattacks or anomalous training data to avoid disruption. Motivated by a conceptual affinity between these two requirements, we devise a distributed learning algorithm that combines distributed gradient descent and the Friedkin-Johnsen model of opinion dynamics to fulfill both of them. We quantify its convergence speed and the neighborhood that contains the final learned models, which can be easily controlled by tuning the algorithm parameters to enforce a more personalized/resilient behavior. We numerically showcase the effectiveness of our algorithm on synthetic and real-world distributed learning tasks, where it achieves high global accuracy both for personalized models and with malicious agents compared to standard strategies.

In this paper, we design a novel distributed learning algorithm using stochastic compressed communications. In detail, we pursue a modular approach, merging ADMM and a gradient-based approach, benefiting from the robustness of the former and the computational efficiency of the latter. Additionally, we integrate a stochastic integral action (error feedback) enabling almost sure rejection of the compression error. We analyze the resulting method in nonconvex scenarios and guarantee almost sure asymptotic convergence to the set of stationary points of the problem. This result is obtained using system-theoretic tools based on stochastic timescale separation. We corroborate our findings with numerical simulations in nonconvex classification.

In this article, we propose a novel distributed algorithm for consensus optimization over networks and a robust extension tailored to deal with asynchronous agents and packet losses. Indeed, to robustly achieve dynamic consensus on the solution estimates and the global descent direction, we embed in our algorithms a distributed implementation of the alternating direction method of multipliers. Such a mechanism is suitably interlaced with a local proportional action steering each agent estimate to the solution of the original consensus optimization problem. First, in the case of ideal networks, by using tools from system theory, we prove the linear convergence of the scheme with strongly convex costs. Then, by exploiting the averaging theory, we extend such a first result to prove that the robust extension of our method preserves linear convergence in the case of asynchronous agents and packet losses. Further, by using the notion of input-to-state stability, we also guarantee the robustness of the schemes with respect to additional, generic errors affecting the agents' updates. Finally, some numerical simulations confirm our theoretical findings and compare our algorithms with other distributed schemes in terms of speed and robustness.

In this paper, we depart from the widely-used gradient descent-based hierarchical federated learning (FL) algorithms to develop a novel hierarchical FL framework based on the alternating direction method of multipliers (ADMM), leveraging a network architecture consisting of a single cloud server and multiple edge servers, where each edge server is dedicated to a specific client set. Within this framework, we propose two novel FL algorithms, which both use ADMM in the top layer: one that employs ADMM in the lower layer and another that uses the conventional gradient descent-based approach. The proposed framework enhances privacy, and experiments demonstrate the superiority of the proposed algorithms compared to the conventional algorithms in terms of learning convergence and accuracy. Additionally, gradient descent on the lower layer performs well even if the number of local steps is very limited, while ADMM on both layers lead to better performance otherwise.

The recent deployment of multiagent networks has enabled the distributed solution of learning problems, where agents cooperate to train a global model without sharing their local, private data. This work specifically targets some prevalent challenges inherent to distributed learning: 1) online training, i.e., the local data change over time; 2) asynchronous agent computations; 3) unreliable and limited communications; and 4) inexact local computations. To tackle these challenges, we apply the distributed operator theoretical (DOT) version of the alternating direction method of multipliers (ADMM), which we call “DOT-ADMM.” We prove that if the DOT-ADMM operator is metric subregular, then it converges with a linear rate for a large class of (not necessarily strongly) convex learning problems toward a bounded neighborhood of the optimal time-varying solution, and characterize how such neighborhood depends on 1)–4). We first derive an easy-to-verify condition for ensuring the metric subregularity of an operator, followed by tutorial examples on linear and logistic regression problems. We corroborate the theoretical analysis with numerical simulations comparing DOT-ADMM with other state-of-the-art algorithms, showing that only the proposed algorithm exhibits robustness to 1)–4).

This paper proposes a framework to study the convergence of stochastic optimization and learning algorithms. The framework is modeled over the different challenges that these algorithms pose, such as (i) the presence of random additive errors ( e.g. due to stochastic gradients), and (ii) random coordinate updates ( e.g. due to asynchrony in distributed set-ups). The paper covers both convex and strongly convex problems, and it also analyzes online scenarios, involving changes in the data and costs. The paper relies on interpreting stochastic algorithms as the iterated application of stochastic operators, thus allowing us to use the powerful tools of operator theory. In particular, we consider operators characterized by additive errors with sub-Weibull distribution (which parameterize a broad class of errors by their tail probability), and random updates. In this framework we derive convergence results in mean and in high probability, by providing bounds to the distance of the current iteration from a solution of the optimization or learning problem. The contributions are discussed in light of federated learning applications.

This manuscript proposes novel distributed algorithms for solving the dynamic consensus problem in discrete-time multi-agent systems on three different objective functions: the average, the maximum, and the median. In this problem, each agent has access to an external time-varying scalar signal and aims to estimate and track a function of all the signals by exploiting only local communications with other agents. By recasting the problem as an online distributed optimization problem, the proposed algorithms are derived based on the distributed implementation of the alternating direction method of multipliers (ADMM) and are thus amenable to a unified analysis technique. A major contribution is that of proving linear convergence of these ADMM-based algorithms for the specific dynamic consensus problems of interest, for which current results could only guarantee sub-linear convergence. In particular, the tracking error is shown to converge within a bound, whereas the steady-state error is zero. Numerical simulations corroborate the theoretical findings, empirically show the robustness of the proposed algorithms to re-initialization errors, and compare their performance with that of state-of-the-art algorithms.