BLBen Liang

Ben Liang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Ben Liang has a total of 74 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Harnessing Teacher’s Explanation for Improved Knowledge Distillation

    by Sayantan Chowdhury, Ben Liang, Ali Tizghadam, Ilijc Albanese

    Knowledge distillation (KD) improves the performance of a low-complexity student model with the help of a more powerful teacher. The teacher in KD is a black-box model’ imparting knowledge to the student only through its predictions. This limits the amount of transferred knowledge. In this work, we introduce a novel Knowledge Explaining Distillation (KED) framework, which allows the student to learn not only from the teacher’s predictions but also from the teacher’s explanations. We propose a class of superfeature-explaining teachers that provide explanation over groups of features, along with the corresponding student model. We also present methods for constructing the superfeatures in structured and unstructured data. We then extend KED to allow augmentation with hidden-representation distillation methods and to work with a limited amount of training data using chimeric sets. Our experiments over a variety of datasets show that KED students can substantially outperform KD students of similar complexity.

  • Power-Efficient Over-the-Air Aggregation With Receive Beamforming for Federated Learning

    by Faeze Moradi Kalarde, Min Dong, Ben Liang, Yahia A. Eldemerdash Ahmed, Ho Ting Cheng

    This paper studies power-efficient uplink transmission design for federated learning (FL) that employs over-the-air analog aggregation and multi-antenna beamforming at the server. We jointly optimize device transmit weights and receive beamforming at each FL communication round to minimize the total device transmit power while ensuring convergence in FL training. Through our convergence analysis, we establish sufficient conditions on the aggregation error to guarantee FL training convergence. Utilizing these conditions, we reformulate the power minimization problem into a unique bi-convex structure that contains a transmit beamforming optimization subproblem and a receive beamforming feasibility subproblem. Despite this unconventional structure, we propose a novel alternating optimization (AO) approach that guarantees monotonic decrease of the objective value, to allow convergence to a partial optimum. We further consider imperfect channel state information (CSI), which requires accounting for the channel estimation errors in the power minimization problem and FL convergence analysis. We propose a CSI-error-aware joint beamforming algorithm, which can substantially outperform one that does not account for channel estimation errors. Simulation with canonical classification datasets demonstrates that our proposed methods achieve significant power reduction compared to existing benchmarks across a wide range of parameter settings, while attaining the same target accuracy under the same convergence rate.


  • Open Access

    Adaptive Sparsification for Communication-Efficient Distributed Learning

    by Faeze Moradi Kalarde, Ben Liang, Min Dong, Yahia A. Eldemerdash Ahmed, Ho Ting Cheng


  • Regularized Top-$ k $: A Bayesian Framework for Gradient Sparsification

    by Ali Bereyhi, Ben Liang, Gary Boudreau, Ali Afana

    Error accumulation is effective for gradient sparsification in distributed settings: initially-unselected gradient entries are eventually selected as their accumulated error exceeds a certain level. The accumulation essentially behaves as a scaling of the learning rate for the selected entries. Although this property prevents the slow-down of lateral movements in distributed gradient descent, it can deteriorate convergence in some settings. This work proposes a novel sparsification scheme that controls the learning rate scaling of error accumulation. The development of this scheme follows two major steps: first, gradient sparsification is formulated as an inverse probability (inference) problem, and the Bayesian optimal sparsification mask is derived as a maximum-a-posteriori estimator. Using the prior distribution inherited from TOP-k, we derive a new sparsification algorithm which can be interpreted as a regularized form of TOP-k. We call this algorithm regularized TOP-k (REGTOP-k). It utilizes past aggregated gradients to evaluate posterior statistics of the next aggregation. It then prioritizes the local accumulated gradient entries based on these posterior statistics. We validate our derivation through various numerical experiments. In distributed linear regression, it is observed that while TOP-k remains at a fixed distance from the global optimum, REGTOP-k converges to the global optimum at significantly higher compression ratios. We further demonstrate the generalization of this observation by employing REGTOP-k in distributed training of ResNet-18 on CIFAR-10, as well as fine-tuning of multiple computer vision models on the ImageNette dataset. Our numerical results confirm that as the compression ratio increases, REGTOP-k sparsification noticeably outperforms TOP-k.


  • Improving Wireless Federated Learning via Joint Downlink-Uplink Beamforming Over Analog Transmission

    by Chong Zhang, Min Dong, Ben Liang, Ali Afana, Yahia A. Eldemerdash Ahmed

    Federated learning (FL) over wireless networks using analog transmission can efficiently utilize the communication resource but is susceptible to errors caused by noisy wireless links. In this paper, assuming a multi-antenna base station, we jointly design downlink-uplink beamforming to maximize FL training convergence over time-varying wireless channels. We derive the round-trip model updating equation and use it to analyze the FL training convergence to capture the effects of downlink and uplink beamforming and the local model training on the global model update. Aiming to maximize the FL training convergence rate, we propose a low-complexity joint downlink-uplink beamforming (JDUBF) algorithm, which adopts a greedy approach to decompose the multi-round joint optimization and convert it into per-round online joint optimization problems. The per-round problem is further decomposed into three subproblems over a block coordinate descent framework, where we show that each subproblem can be efficiently solved by projected gradient descent with fast closed-form updates. An efficient initialization method that leads to a closed-form initial point is also proposed to accelerate the convergence of JDUBF. Simulation demonstrates that JDUBF substantially outperforms the conventional separate-link beamforming design.


  • Open Access

    Age-of-Information Minimization With Weight Limits for Semi-Asynchronous Online Distributed Optimization

    by Juncheng Wang, Ben Liang, Min Dong, Gary Boudreau, Ali Afana

    We consider online distributed optimization where a server and multiple devices collaborate to minimize a sequence of time-varying global loss functions. To accommodate slow devices that may require multiple time slots to compute their local decisions, the server uses semi-asynchronous aggregation of the local decisions, which complicates device scheduling and performance optimization. In this work, we first analyze the convergence of semi-asynchronous aggregation in the presence of time-varying local update delays and loss-function weights. Our analysis leads to an online scheduling problem to minimize the accumulated age of information on the local decision updates, subject to individual long-term constraints on the total weights of the scheduled devices. We then design an efficient scheduling policy, termed Age-of-Information Minimization with Weight Limits (AIMWeL), through a modified Lyapunov optimization approach that uses the weighted sum of linear age-of-information values and quadratic virtual queues as a new Lyapunov function. We show that AIMWeL has bounded optimality ratio, via a novel double relaxation approach to handle the unique scheduling-dependent communication indicator with time-varying probabilities of completing local decision update caused by semi-asynchronous aggregation. When AIMWeL is applied to semi-asynchronous federated learning, our simulation results based on standard image classification datasets demonstrate that AIMWeL uses significantly less time to reach the same classification accuracy achieved by the current best alternatives for both convex logistic regression and non-convex convolutional neural networks.


  • Open Access

    Clipper : Online Joint Client Sampling and Power Allocation for Wireless Federated Learning

    by Wen Xu, Ben Liang, Gary Boudreau, Hamza Sokun

    Communication overhead is a main bottleneck in federated learning (FL) especially in the wireless environment due to the limited data rate and unstable radio channels. The communication challenge necessitates holistic selection of participating clients that accounts for both the computation needs and communication cost, as well as judicious allocation of the limited transmission resource. Meanwhile, the random unpredictable nature of both the training data samples and the communication channels requires an online optimization approach that adapts to the changing system state over time. In this work, we consider a general framework of online joint client sampling and power allocation for wireless FL under time-varying communication channels. We formulate it as a stochastic network optimization problem that admits a Lyapunov-typed solution approach. This leads to per-training-round subproblems with a special bi-convex structure, which we leverage to propose globally optimal solutions, culminating in a meta algorithm that provides strong performance guarantees. We further study three specific FL problems covering multiple scenarios, namely, with IID or non-IID data, whether robustness against data drift is required, and with unbiased or biased client sampling. We derive detailed algorithms for each of these problems. Simulation with standard classification tasks demonstrate that the proposed communication-aware algorithms outperform their counterparts under a wide range of learning and communication scenarios.