Nirupam Gupta
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Nirupam Gupta has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Nitin H. Vaidya
Georgetown University
6 shared publications - Nikhil Chopra
Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA
5 shared publications - Kushal Chakrabarti
Division of Data and Decision Sciences, Tata Consultancy Services Research, Mumbai, India
3 shared publications - Rachid Guerraoui
EPFL
3 shared publications - Rafael Pinot
Sorbonne University, Paris, France
2 shared publications - Shuo Liu
School of Energy and Chemical Engineering, Tianjin Renai College, Tianjin 301636, China
2 shared publications
Latest Publications
- Open Access
- 17 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 131-134
Brief Announcement: A Case for Byzantine Machine Learning
The success of machine learning (ML) has been intimately linked with the availability of large amounts of data, typically collected from heterogeneous sources and processed on vast networks of computing devices (also called workers). Beyond accuracy, the use of ML in critical domains such as healthcare and autonomous driving calls for robustness against data poisoning and faulty workers. The problem of Byzantine ML formalizes these robustness issues by considering a distributed ML environment in which workers (storing a portion of the global dataset) can deviate arbitrarily from the prescribed algorithm. Although the problem has attracted a lot of attention from a theoretical point of view, its practical importance for addressing realistic faults (where the behavior of any worker is locally constrained) remains unclear. It has been argued that the seemingly weaker threat model where only workers' local datasets get poisoned is more reasonable. We highlight here some important results on the efficacy of Byzantine robustness for tackling data poisoning. In particular, we discuss cases where, while tolerating a wider range of faulty behaviors, Byzantine ML yields solutions that are optimal even under the weaker threat model of data poisoning.
- 17 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 131-134
Open AccessBrief Announcement: A Case for Byzantine Machine Learning
The success of machine learning (ML) has been intimately linked with the availability of large amounts of data, typically collected from heterogeneous sources and processed on vast networks of computing devices (also called workers). Beyond accuracy, the use of ML in critical domains such as healthcare and autonomous driving calls for robustness against data poisoning and faulty workers. The problem of Byzantine ML formalizes these robustness issues by considering a distributed ML environment in which workers (storing a portion of the global dataset) can deviate arbitrarily from the prescribed algorithm. Although the problem has attracted a lot of attention from a theoretical point of view, its practical importance for addressing realistic faults (where the behavior of any worker is locally constrained) remains unclear. It has been argued that the seemingly weaker threat model where only workers' local datasets get poisoned is more reasonable. We highlight here some important results on the efficacy of Byzantine robustness for tackling data poisoning. In particular, we discuss cases where, while tolerating a wider range of faulty behaviors, Byzantine ML yields solutions that are optimal even under the weaker threat model of data poisoning.
- Open Access
- 12 April 2024
- Published by Springer Nature
- p. 189-196
Robust and Private Federated Learning on LLMs
Large Language Models (LLMs) have gained significant attention in recent years due to their potential to revolutionize various industries and sectors. However, scaling LLMs further requires access to substantial linguistic resources that are being rapidly depleted. Moreover, the available text sources such as emails, social media interactions, or internal documents may contain private information, making them susceptible to misuse. On-premises Federated Learning (FL) with privacy-preserving model updates is an alternative avenue for LLMs’ development that ensures data sovereignty and enables peers to collaborate while ensuring that the sensitive parts of their private data cannot be reconstructed. However, in the case of large-scale FL, there is also a risk of malicious users attempting to poison LLMs for their benefit. The problem of protecting the learning procedure against such users is known as Byzantine-robustness, and it is crucial to develop models that perform accurately despite faulty machines and poisonous data. Designing FL methods that are simultaneously privacy-preserving and Byzantine-robust is challenging. However, ongoing research suggests ways to incorporate the differentially-private Gaussian mechanism for privacy preservation and spectral robust-averaging for robustness. However, whether this approach applies to LLMs or whether a major player in the domain would emerge and capture all private information sources through network effects remains to be seen.
- 12 April 2024
- Published by Springer Nature
- p. 189-196
Open AccessRobust and Private Federated Learning on LLMs
Large Language Models (LLMs) have gained significant attention in recent years due to their potential to revolutionize various industries and sectors. However, scaling LLMs further requires access to substantial linguistic resources that are being rapidly depleted. Moreover, the available text sources such as emails, social media interactions, or internal documents may contain private information, making them susceptible to misuse. On-premises Federated Learning (FL) with privacy-preserving model updates is an alternative avenue for LLMs’ development that ensures data sovereignty and enables peers to collaborate while ensuring that the sensitive parts of their private data cannot be reconstructed. However, in the case of large-scale FL, there is also a risk of malicious users attempting to poison LLMs for their benefit. The problem of protecting the learning procedure against such users is known as Byzantine-robustness, and it is crucial to develop models that perform accurately despite faulty machines and poisonous data. Designing FL methods that are simultaneously privacy-preserving and Byzantine-robust is challenging. However, ongoing research suggests ways to incorporate the differentially-private Gaussian mechanism for privacy preservation and spectral robust-averaging for robustness. However, whether this approach applies to LLMs or whether a major player in the domain would emerge and capture all private information sources through network effects remains to be seen.
- Open Access
- 9 April 2024
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 56 (7) , 1-39
Byzantine Machine Learning: A Primer
The problem of Byzantine resilience in distributed machine learning, a.k.a. Byzantine machine learning , consists of designing distributed algorithms that can train an accurate model despite the presence of Byzantine nodes—that is, nodes with corrupt data or machines that can misbehave arbitrarily. By now, many solutions to this important problem have been proposed, most of which build upon the classical stochastic gradient descent scheme. Yet, the literature lacks a unified structure of this emerging field. Consequently, the general understanding on the principles of Byzantine machine learning remains poor. This article addresses this issue by presenting a primer on Byzantine machine learning. In particular, we introduce three pillars of Byzantine machine learning, namely the concepts of breakdown point , robustness , and gradient complexity , to curate the efficacy of a solution. The introduced systematization enables us to (i) bring forth the merits and limitations of the state-of-the-art solutions, and (ii) pave a clear path for future advancements in this field.
- 9 April 2024
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 56 (7) , 1-39
Open AccessByzantine Machine Learning: A Primer
The problem of Byzantine resilience in distributed machine learning, a.k.a. Byzantine machine learning , consists of designing distributed algorithms that can train an accurate model despite the presence of Byzantine nodes—that is, nodes with corrupt data or machines that can misbehave arbitrarily. By now, many solutions to this important problem have been proposed, most of which build upon the classical stochastic gradient descent scheme. Yet, the literature lacks a unified structure of this emerging field. Consequently, the general understanding on the principles of Byzantine machine learning remains poor. This article addresses this issue by presenting a primer on Byzantine machine learning. In particular, we introduce three pillars of Byzantine machine learning, namely the concepts of breakdown point , robustness , and gradient complexity , to curate the efficacy of a solution. The introduced systematization enables us to (i) bring forth the merits and limitations of the state-of-the-art solutions, and (ii) pave a clear path for future advancements in this field.
- Open Access
Impact of Redundancy on Resilience in Distributed Optimization and Learning
This report considers the problem of resilient distributed optimization and stochastic learning in a server-based architecture. The system comprises a server and multiple agents, where each agent has its own local cost function. The agents collaborate with the server to find a minimum of the aggregate of the local cost functions. In the context of stochastic learning, the local cost of an agent is the loss function computed over the data at that agent. In this report, we consider this problem in a system wherein some of the agents may be Byzantine faulty and some of the agents may be slow (also called stragglers). In this setting, we investigate the conditions under which it is possible to obtain an "approximate" solution to the above problem. In particular, we introduce the notion of $(f, r; \epsilon)$-resilience to characterize how well the true solution is approximated in the presence of up to $f$ Byzantine faulty agents, and up to $r$ slow agents (or stragglers) -- smaller $\epsilon$ represents a better approximation. We also introduce a measure named $(f, r; \epsilon)$-redundancy to characterize the redundancy in the cost functions of the agents. Greater redundancy allows for a better approximation when solving the problem of aggregate cost minimization. In this report, we constructively show (both theoretically and empirically) that $(f, r; \mathcal{O}(\epsilon))$-resilience can indeed be achieved in practice, given that the local cost functions are sufficiently redundant.Open AccessImpact of Redundancy on Resilience in Distributed Optimization and Learning
This report considers the problem of resilient distributed optimization and stochastic learning in a server-based architecture. The system comprises a server and multiple agents, where each agent has its own local cost function. The agents collaborate with the server to find a minimum of the aggregate of the local cost functions. In the context of stochastic learning, the local cost of an agent is the loss function computed over the data at that agent. In this report, we consider this problem in a system wherein some of the agents may be Byzantine faulty and some of the agents may be slow (also called stragglers). In this setting, we investigate the conditions under which it is possible to obtain an "approximate" solution to the above problem. In particular, we introduce the notion of $(f, r; \epsilon)$-resilience to characterize how well the true solution is approximated in the presence of up to $f$ Byzantine faulty agents, and up to $r$ slow agents (or stragglers) -- smaller $\epsilon$ represents a better approximation. We also introduce a measure named $(f, r; \epsilon)$-redundancy to characterize the redundancy in the cost functions of the agents. Greater redundancy allows for a better approximation when solving the problem of aggregate cost minimization. In this report, we constructively show (both theoretically and empirically) that $(f, r; \mathcal{O}(\epsilon))$-resilience can indeed be achieved in practice, given that the local cost functions are sufficiently redundant. - 16 January 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Control of Network Systems
- Vol. 10 (4) , 1669-1681
Byzantine Fault-Tolerance in Federated Local SGD Under $2f$-Redundancy
We consider the problem of Byzantine fault-tolerance in federated machine learning. In this problem, the system comprises multiple agents each with local data, and a trusted centralized coordinator. In fault-free setting, the agents collaborate with the coordinator to find a minimizer of the aggregate of their local cost functions defined over their local data. We consider a scenario where some agents ($f$ out of$N$ ) are Byzantine faulty. Such agents need not follow a prescribed algorithm correctly, and may communicate arbitrary incorrect information to the coordinator. In the presence of Byzantine agents, a more reasonable goal for the non-faulty agents is to find a minimizer of the aggregate cost function of only the non-faulty agents. This particular goal is commonly referred as exact fault-tolerance . Recent work has shown that exact fault-tolerance is achievable if only if the non-faulty agents satisfy the property of$2f$ -redundancy . Now, under this property, techniques are known to impart exact fault-tolerance to the distributed implementation of the classical stochastic gradient-descent (SGD) algorithm. However, we do not know of any such techniques for the federated local SGD algorithm - a more commonly used method for federated machine learning. To address this issue, we propose a novel technique named comparative elimination (CE). We show that, under$2f$ -redundancy, the federated local SGD algorithm with CE can indeed obtain exact fault-tolerance in the deterministic setting when the non-faulty agents can accurately compute gradients of their local cost functions. In the general stochastic case, when agents can only compute unbiased noisy estimates of their local gradients, our algorithm achieves approximate fault-tolerance with approximation error proportional to the variance of stochastic gradients and the fraction of Byzantine agents.- 16 January 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Control of Network Systems
- Vol. 10 (4) , 1669-1681
Byzantine Fault-Tolerance in Federated Local SGD Under $2f$-Redundancy
We consider the problem of Byzantine fault-tolerance in federated machine learning. In this problem, the system comprises multiple agents each with local data, and a trusted centralized coordinator. In fault-free setting, the agents collaborate with the coordinator to find a minimizer of the aggregate of their local cost functions defined over their local data. We consider a scenario where some agents ($f$ out of$N$ ) are Byzantine faulty. Such agents need not follow a prescribed algorithm correctly, and may communicate arbitrary incorrect information to the coordinator. In the presence of Byzantine agents, a more reasonable goal for the non-faulty agents is to find a minimizer of the aggregate cost function of only the non-faulty agents. This particular goal is commonly referred as exact fault-tolerance . Recent work has shown that exact fault-tolerance is achievable if only if the non-faulty agents satisfy the property of$2f$ -redundancy . Now, under this property, techniques are known to impart exact fault-tolerance to the distributed implementation of the classical stochastic gradient-descent (SGD) algorithm. However, we do not know of any such techniques for the federated local SGD algorithm - a more commonly used method for federated machine learning. To address this issue, we propose a novel technique named comparative elimination (CE). We show that, under$2f$ -redundancy, the federated local SGD algorithm with CE can indeed obtain exact fault-tolerance in the deterministic setting when the non-faulty agents can accurately compute gradients of their local cost functions. In the general stochastic case, when agents can only compute unbiased noisy estimates of their local gradients, our algorithm achieves approximate fault-tolerance with approximation error proportional to the variance of stochastic gradients and the fraction of Byzantine agents.- Open Access
- 4 January 2023
- Published by Association for Computing Machinery (ACM)
- p. 80-89
Impact of Redundancy on Resilience in Distributed Optimization and Learning
This paper considers the problem of resilient distributed optimization and stochastic learning in a server-based architecture. The system comprises a server and multiple agents, where each agent has its own local cost function. The agents collaborate with the server to find a minimum of the aggregate of the local cost functions. In the context of stochastic learning, the local cost of an agent is the loss function computed over the data at that agent. In this paper, we consider this problem in a system wherein some of the agents may be Byzantine faulty and some of the agents may be slow (also called stragglers). In this setting, we investigate the conditions under which it is possible to obtain an “approximate” solution to the above problem. In particular, we introduce the notion of (f, r;ϵ)-resilience to characterize how well the true solution is approximated in the presence of up to f Byzantine faulty agents, and up to r slow agents (or stragglers) – smaller ϵ represents a better approximation. We also introduce a measure named (f, r;ϵ)-redundancy to characterize the redundancy in the cost functions of the agents. Greater redundancy allows for a better approximation when solving the problem of aggregate cost minimization. In this paper, we constructively show (both theoretically and empirically) that -resilience can indeed be achieved in practice, given that the local cost functions are sufficiently redundant. Our empirical evaluation considers a distributed gradient descent (DGD)-based solution; for distributed learning in the presence of Byzantine and asynchronous agents, we also evaluate a distributed stochastic gradient descent (D-SGD)-based algorithm.
- 4 January 2023
- Published by Association for Computing Machinery (ACM)
- p. 80-89
Open AccessImpact of Redundancy on Resilience in Distributed Optimization and Learning
This paper considers the problem of resilient distributed optimization and stochastic learning in a server-based architecture. The system comprises a server and multiple agents, where each agent has its own local cost function. The agents collaborate with the server to find a minimum of the aggregate of the local cost functions. In the context of stochastic learning, the local cost of an agent is the loss function computed over the data at that agent. In this paper, we consider this problem in a system wherein some of the agents may be Byzantine faulty and some of the agents may be slow (also called stragglers). In this setting, we investigate the conditions under which it is possible to obtain an “approximate” solution to the above problem. In particular, we introduce the notion of (f, r;ϵ)-resilience to characterize how well the true solution is approximated in the presence of up to f Byzantine faulty agents, and up to r slow agents (or stragglers) – smaller ϵ represents a better approximation. We also introduce a measure named (f, r;ϵ)-redundancy to characterize the redundancy in the cost functions of the agents. Greater redundancy allows for a better approximation when solving the problem of aggregate cost minimization. In this paper, we constructively show (both theoretically and empirically) that -resilience can indeed be achieved in practice, given that the local cost functions are sufficiently redundant. Our empirical evaluation considers a distributed gradient descent (DGD)-based solution; for distributed learning in the presence of Byzantine and asynchronous agents, we also evaluate a distributed stochastic gradient descent (D-SGD)-based algorithm.
- 5 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Control of Network Systems
- Vol. 9 (2) , 811-822
On Preconditioning of Decentralized Gradient-Descent When Solving a System of Linear Equations
This paper considers solving an overdetermined system of linear equations in peer-to-peer multi-agent networks. The network is assumed to be synchronous and (strongly) connected. Each agent has a set of local data points, and their goal is to compute a linear model that fits the collective data points. In principle, the agents can apply the decentralized gradient-descent method (DGD). However, when the data matrix is ill-conditioned, DGD requires many iterations to converge and is unstable against system noise. We propose a decentralized pre-conditioning technique to mitigate the deleterious effects of the data points' conditioning on the convergence rate of DGD. The proposed algorithm converges linearly, with an improved convergence rate than DGD. Considering the practical scenario where the computations performed by the agents are corrupted, we study the robustness guarantee of the proposed algorithm. Additionally, we apply the proposed algorithm for solving decentralized state estimation problems. The empirical results show our proposed state predictor's favorable convergence rate and robustness against system noise compared to prominent decentralized algorithms.
- 5 April 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Control of Network Systems
- Vol. 9 (2) , 811-822
On Preconditioning of Decentralized Gradient-Descent When Solving a System of Linear Equations
This paper considers solving an overdetermined system of linear equations in peer-to-peer multi-agent networks. The network is assumed to be synchronous and (strongly) connected. Each agent has a set of local data points, and their goal is to compute a linear model that fits the collective data points. In principle, the agents can apply the decentralized gradient-descent method (DGD). However, when the data matrix is ill-conditioned, DGD requires many iterations to converge and is unstable against system noise. We propose a decentralized pre-conditioning technique to mitigate the deleterious effects of the data points' conditioning on the convergence rate of DGD. The proposed algorithm converges linearly, with an improved convergence rate than DGD. Considering the practical scenario where the computations performed by the agents are corrupted, we study the robustness guarantee of the proposed algorithm. Additionally, we apply the proposed algorithm for solving decentralized state estimation problems. The empirical results show our proposed state predictor's favorable convergence rate and robustness against system noise compared to prominent decentralized algorithms.
- Open Access
- 21 July 2021
- Published by Association for Computing Machinery (ACM)
- Vol. 31, 379-389
Approximate Byzantine Fault-Tolerance in Distributed Optimization
This paper considers the problem of Byzantine fault-tolerance in distributed multi-agent optimization. In this problem, each agent has a local cost function, and in the fault-free case, the goal is to design a distributed algorithm that allows all the agents to find a minimum point of all the agents' aggregate cost function. We consider a scenario where some agents might be Byzantine faulty that renders the original goal of computing a minimum point of all the agents' aggregate cost vacuous. A more reasonable objective for an algorithm in this scenario is to allow all the non-faulty agents to compute the minimum point of only the non-faulty agents' aggregate cost. Prior work shows that if there are up to f (out of n) Byzantine agents then a minimum point of the non-faulty agents' aggregate cost can be computed exactly if and only if the non-faulty agents' costs satisfy a certain redundancy property called 2f-redundancy. However, 2f-redundancy is an ideal property that can be satisfied only in systems free from noise or uncertainties, which can make the goal of exact fault-tolerance unachievable in some applications. Thus, we introduce the notion of (f,ε)-resilience, a generalization of exact fault-tolerance wherein the objective is to find an approximate minimum point of the non-faulty aggregate cost, with ε accuracy. This approximate fault-tolerance can be achieved under a weaker condition that is easier to satisfy in practice, compared to 2f-redundancy. We obtain necessary and sufficient conditions for achieving (f, ε)-resilience characterizing the correlation between relaxation in redundancy and approximation in resilience. In case when the agents' cost functions are differentiable, we obtain conditions for (f, ε)-resilience of the distributed gradient-descent method when equipped with robust gradient aggregation; such as comparative gradient elimination or coordinate-wise trimmed mean.
- 21 July 2021
- Published by Association for Computing Machinery (ACM)
- Vol. 31, 379-389
Open AccessApproximate Byzantine Fault-Tolerance in Distributed Optimization
This paper considers the problem of Byzantine fault-tolerance in distributed multi-agent optimization. In this problem, each agent has a local cost function, and in the fault-free case, the goal is to design a distributed algorithm that allows all the agents to find a minimum point of all the agents' aggregate cost function. We consider a scenario where some agents might be Byzantine faulty that renders the original goal of computing a minimum point of all the agents' aggregate cost vacuous. A more reasonable objective for an algorithm in this scenario is to allow all the non-faulty agents to compute the minimum point of only the non-faulty agents' aggregate cost. Prior work shows that if there are up to f (out of n) Byzantine agents then a minimum point of the non-faulty agents' aggregate cost can be computed exactly if and only if the non-faulty agents' costs satisfy a certain redundancy property called 2f-redundancy. However, 2f-redundancy is an ideal property that can be satisfied only in systems free from noise or uncertainties, which can make the goal of exact fault-tolerance unachievable in some applications. Thus, we introduce the notion of (f,ε)-resilience, a generalization of exact fault-tolerance wherein the objective is to find an approximate minimum point of the non-faulty aggregate cost, with ε accuracy. This approximate fault-tolerance can be achieved under a weaker condition that is easier to satisfy in practice, compared to 2f-redundancy. We obtain necessary and sufficient conditions for achieving (f, ε)-resilience characterizing the correlation between relaxation in redundancy and approximation in resilience. In case when the agents' cost functions are differentiable, we obtain conditions for (f, ε)-resilience of the distributed gradient-descent method when equipped with robust gradient aggregation; such as comparative gradient elimination or coordinate-wise trimmed mean.
- Open Access
Approximate Byzantine Fault-Tolerance in Distributed Optimization
This paper considers the problem of Byzantine fault-tolerance in distributed multi-agent optimization. In this problem, each agent has a local cost function, and in the fault-free case, the goal is to design a distributed algorithm that allows all the agents to find a minimum point of all the agents' aggregate cost function. We consider a scenario where some agents might be Byzantine faulty that renders the original goal of computing a minimum point of all the agents' aggregate cost vacuous. A more reasonable objective for an algorithm in this scenario is to allow all the non-faulty agents to compute the minimum point of only the non-faulty agents' aggregate cost. Prior work shows that if there are up to $f$ (out of $n$) Byzantine agents then a minimum point of the non-faulty agents' aggregate cost can be computed exactly if and only if the non-faulty agents' costs satisfy a certain redundancy property called $2f$-redundancy. However, $2f$-redundancy is an ideal property that can be satisfied only in systems free from noise or uncertainties, which can make the goal of exact fault-tolerance unachievable in some applications. Thus, we introduce the notion of $(f,\epsilon)$-resilience, a generalization of exact fault-tolerance wherein the objective is to find an approximate minimum point of the non-faulty aggregate cost, with $\epsilon$ accuracy. This approximate fault-tolerance can be achieved under a weaker condition that is easier to satisfy in practice, compared to $2f$-redundancy. We obtain necessary and sufficient conditions for achieving $(f,\epsilon)$-resilience characterizing the correlation between relaxation in redundancy and approximation in resilience. In case when the agents' cost functions are differentiable, we obtain conditions for $(f,\epsilon)$-resilience of the distributed gradient-descent method when equipped with robust gradient aggregation.Open AccessApproximate Byzantine Fault-Tolerance in Distributed Optimization
This paper considers the problem of Byzantine fault-tolerance in distributed multi-agent optimization. In this problem, each agent has a local cost function, and in the fault-free case, the goal is to design a distributed algorithm that allows all the agents to find a minimum point of all the agents' aggregate cost function. We consider a scenario where some agents might be Byzantine faulty that renders the original goal of computing a minimum point of all the agents' aggregate cost vacuous. A more reasonable objective for an algorithm in this scenario is to allow all the non-faulty agents to compute the minimum point of only the non-faulty agents' aggregate cost. Prior work shows that if there are up to $f$ (out of $n$) Byzantine agents then a minimum point of the non-faulty agents' aggregate cost can be computed exactly if and only if the non-faulty agents' costs satisfy a certain redundancy property called $2f$-redundancy. However, $2f$-redundancy is an ideal property that can be satisfied only in systems free from noise or uncertainties, which can make the goal of exact fault-tolerance unachievable in some applications. Thus, we introduce the notion of $(f,\epsilon)$-resilience, a generalization of exact fault-tolerance wherein the objective is to find an approximate minimum point of the non-faulty aggregate cost, with $\epsilon$ accuracy. This approximate fault-tolerance can be achieved under a weaker condition that is easier to satisfy in practice, compared to $2f$-redundancy. We obtain necessary and sufficient conditions for achieving $(f,\epsilon)$-resilience characterizing the correlation between relaxation in redundancy and approximation in resilience. In case when the agents' cost functions are differentiable, we obtain conditions for $(f,\epsilon)$-resilience of the distributed gradient-descent method when equipped with robust gradient aggregation. - Open Access
- 25 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2248-2253
Robustness of Iteratively Pre-Conditioned Gradient-Descent Method: The Case of Distributed Linear Regression Problem
This paper considers the problem of multi-agent distributed linear regression in the presence of system noises. In this problem, the system comprises multiple agents wherein each agent locally observes a set of data points, and the agents' goal is to compute a linear model that best fits the collective data points observed by all the agents. We consider a server-based distributed architecture where the agents interact with a common server to solve the problem; however, the server cannot access the agents' data points. We consider a practical scenario wherein the system either has observation noise, i.e., the data points observed by the agents are corrupted, or has process noise, i.e., the computations performed by the server and the agents are corrupted. In noise-free systems, the recently proposed distributed linear regression algorithm, named the Iteratively Pre-conditioned Gradient-descent (IPG) method, has been claimed to converge faster than related methods. In this paper, we study the robustness of the IPG method, against both the observation noise and the process noise. We empirically show that the robustness of the IPG method compares favorably to the state-of-the-art algorithms.
- 25 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2248-2253
Open AccessRobustness of Iteratively Pre-Conditioned Gradient-Descent Method: The Case of Distributed Linear Regression Problem
This paper considers the problem of multi-agent distributed linear regression in the presence of system noises. In this problem, the system comprises multiple agents wherein each agent locally observes a set of data points, and the agents' goal is to compute a linear model that best fits the collective data points observed by all the agents. We consider a server-based distributed architecture where the agents interact with a common server to solve the problem; however, the server cannot access the agents' data points. We consider a practical scenario wherein the system either has observation noise, i.e., the data points observed by the agents are corrupted, or has process noise, i.e., the computations performed by the server and the agents are corrupted. In noise-free systems, the recently proposed distributed linear regression algorithm, named the Iteratively Pre-conditioned Gradient-descent (IPG) method, has been claimed to converge faster than related methods. In this paper, we study the robustness of the IPG method, against both the observation noise and the process noise. We empirically show that the robustness of the IPG method compares favorably to the state-of-the-art algorithms.