Meng Wang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Meng Wang has a total of 76 co-authors. Here are the top co-authors by number of shared publications:
- Joe H. Chow
Rensselaer Polytechnic Institute
11 shared publications Shuai Zhang
New Jersey Institute of Technology University of Pennsylvania University of Louisiana at Lafayette Rensselaer Polytechnic Insitute New Jersey Institute of Technology
6 shared publications- Sijia Liu
Michigan State University
6 shared publications - Jinjun Xiong
College of AI, Cyber and Computing, The University of Texas at San Antonio, San Antonio, TX, USA
4 shared publications Pengzhi Gao
Baidu Research
4 shared publications- Pin-Yu Chen
IBM Research, Yorktown Heights, New York, USA
4 shared publications
Latest Publications
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Power Systems
- Vol. 41  (4) , 2866-2878
Mid-Term Load Forecasting With Minimal Data: An In-Context-Learning-Aware Approach Using Large Language Models
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Power Systems
- Vol. 41  (4) , 2866-2878
Mid-Term Load Forecasting With Minimal Data: An In-Context-Learning-Aware Approach Using Large Language Models
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 17  (4) , 3564-3576
Behind-the-Meter PV Disaggregation With Copula-Constrained Bayesian Variational Autoencoder and Feature Clustering
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Smart Grid
- Vol. 17  (4) , 3564-3576
Behind-the-Meter PV Disaggregation With Copula-Constrained Bayesian Variational Autoencoder and Feature Clustering
- Open Access
A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases. Furthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations.
Open AccessA Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
The recent empirical success of Mamba and other selective state space models (SSMs) has renewed interest in non-attention architectures for sequence modeling, yet their theoretical foundations remain underexplored. We present a first-step analysis of generalization and learning dynamics for a simplified but representative Mamba block: a single-layer, single-head selective SSM with input-dependent gating, followed by a two-layer MLP trained via gradient descent (GD). Our study adopts a structured data model with tokens that include both class-relevant and class-irrelevant patterns under token-level noise and examines two canonical regimes: majority-voting and locality-structured data sequences. We prove that the model achieves guaranteed generalization by establishing non-asymptotic sample complexity and convergence rate bounds, which improve as the effective signal increases and the noise decreases. Furthermore, we show that the gating vector aligns with class-relevant features while ignoring irrelevant ones, thereby formalizing a feature-selection role similar to attention but realized through selective recurrence. Numerical experiments on synthetic data justify our theoretical results. Overall, our results provide principled insight into when and why Mamba-style selective SSMs learn efficiently, offering a theoretical counterpoint to Transformer-centric explanations.
- Open Access
Theoretical Analysis of Contrastive Learning under Imbalanced Data: From Training Dynamics to a Pruning Solution
Contrastive learning has emerged as a powerful framework for learning generalizable representations, yet its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world applications. Such an imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve through three distinct stages of training, with different dynamics for majority features, minority features, and noise. We further show that minority features reduce representational capacity, increase the need for more complex architectures, and hinder the separation of ground-truth features from noise. Inspired by these neuron-level behaviors, we show that pruning restores performance degraded by imbalance and enhances feature separation, offering both conceptual insights and practical guidance. Major theoretical findings are validated through numerical experiments.
Open AccessTheoretical Analysis of Contrastive Learning under Imbalanced Data: From Training Dynamics to a Pruning Solution
Contrastive learning has emerged as a powerful framework for learning generalizable representations, yet its theoretical understanding remains limited, particularly under imbalanced data distributions that are prevalent in real-world applications. Such an imbalance can degrade representation quality and induce biased model behavior, yet a rigorous characterization of these effects is lacking. In this work, we develop a theoretical framework to analyze the training dynamics of contrastive learning with Transformer-based encoders under imbalanced data. Our results reveal that neuron weights evolve through three distinct stages of training, with different dynamics for majority features, minority features, and noise. We further show that minority features reduce representational capacity, increase the need for more complex architectures, and hinder the separation of ground-truth features from noise. Inspired by these neuron-level behaviors, we show that pruning restores performance degraded by imbalance and enhances feature separation, offering both conceptual insights and practical guidance. Major theoretical findings are validated through numerical experiments.
- 17 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24  (2) , 337-340
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.
- 17 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24  (2) , 337-340
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.
- Open Access
Can Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis
The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e., making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning. Despite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments.
Open AccessCan Mamba Learn In Context with Outliers? A Theoretical Generalization Analysis
The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e., making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning. Despite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments.
- Open Access
Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
Jumping connections enable Graph Convolutional Networks (GCNs) to overcome over-smoothing, while graph sparsification reduces computational demands by selecting a sub-matrix of the graph adjacency matrix during neighborhood aggregation. Learning GCNs with graph sparsification has shown empirical success across various applications, but a theoretical understanding of the generalization guarantees remains limited, with existing analyses ignoring either graph sparsification or jumping connections. This paper presents the first learning dynamics and generalization analysis of GCNs with jumping connections using graph sparsification. Our analysis demonstrates that the generalization accuracy of the learned model closely approximates the highest achievable accuracy within a broad class of target functions dependent on the proposed sparse effective adjacency matrix $A^*$. Thus, graph sparsification maintains generalization performance when $A^*$ preserves the essential edges that support meaningful message propagation. We reveal that jumping connections lead to different sparsification requirements across layers. In a two-hidden-layer GCN, the generalization is more affected by the sparsified matrix deviations from $A^*$ of the first layer than the second layer. To the best of our knowledge, this marks the first theoretical characterization of jumping connections' role in sparsification requirements. We validate our theoretical results on benchmark datasets in deep GCNs.Open AccessTheoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsification
Jumping connections enable Graph Convolutional Networks (GCNs) to overcome over-smoothing, while graph sparsification reduces computational demands by selecting a sub-matrix of the graph adjacency matrix during neighborhood aggregation. Learning GCNs with graph sparsification has shown empirical success across various applications, but a theoretical understanding of the generalization guarantees remains limited, with existing analyses ignoring either graph sparsification or jumping connections. This paper presents the first learning dynamics and generalization analysis of GCNs with jumping connections using graph sparsification. Our analysis demonstrates that the generalization accuracy of the learned model closely approximates the highest achievable accuracy within a broad class of target functions dependent on the proposed sparse effective adjacency matrix $A^*$. Thus, graph sparsification maintains generalization performance when $A^*$ preserves the essential edges that support meaningful message propagation. We reveal that jumping connections lead to different sparsification requirements across layers. In a two-hidden-layer GCN, the generalization is more affected by the sparsified matrix deviations from $A^*$ of the first layer than the second layer. To the best of our knowledge, this marks the first theoretical characterization of jumping connections' role in sparsification requirements. We validate our theoretical results on benchmark datasets in deep GCNs. - Top Cited Papers
- 7 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Power Systems
- Vol. 40  (5) , 4220-4232
Empower Pre-Trained Large Language Models for Building-Level Load Forecasting
Short-term building-level load forecasting is significant for enhancing the stability and efficiency of power grids. Despite the superior forecasting performance, machine learning methods heavily rely on sufficient historical load data for training. This paper addresses the challenge of limited or unavailable historical data, which often occurs in new communities or due to data storage issues. This paper proposes BlackInter, a novel black-box tuning inductive adapter based on pre-trained large language models (LLMs), specifically tailored for building-level load forecasting. Our method leverages the inherent generalization capabilities of LLMs with no need for pre-training on similar domains and fine-tuning by target data. Furthermore, BlackInter exploits spatial correlations among nearby buildings to improve forecast accuracy. It directly adapts to different building sizes, a feature lacking in existing approaches. Our approach does not require prior knowledge of the LLM's structure or parameters. We validate the effectiveness of the LLM-based BlackInter using real-world datasets and compare it with four existing methods. Under the settings of limited and no historical data, the prediction error by our method can be only 42.88% and 69.44% of the error by the best alternative methods, respectively.
- 7 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Power Systems
- Vol. 40  (5) , 4220-4232
Top Cited PapersEmpower Pre-Trained Large Language Models for Building-Level Load Forecasting
Short-term building-level load forecasting is significant for enhancing the stability and efficiency of power grids. Despite the superior forecasting performance, machine learning methods heavily rely on sufficient historical load data for training. This paper addresses the challenge of limited or unavailable historical data, which often occurs in new communities or due to data storage issues. This paper proposes BlackInter, a novel black-box tuning inductive adapter based on pre-trained large language models (LLMs), specifically tailored for building-level load forecasting. Our method leverages the inherent generalization capabilities of LLMs with no need for pre-training on similar domains and fine-tuning by target data. Furthermore, BlackInter exploits spatial correlations among nearby buildings to improve forecast accuracy. It directly adapts to different building sizes, a feature lacking in existing approaches. Our approach does not require prior knowledge of the LLM's structure or parameters. We validate the effectiveness of the LLM-based BlackInter using real-world datasets and compare it with four existing methods. Under the settings of limited and no historical data, the prediction error by our method can be only 42.88% and 69.44% of the error by the best alternative methods, respectively.
- 22 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Power Systems
- Vol. 40  (2) , 1335-1347
Unifying Load Disaggregation and Prediction for Buildings With Behind-the-Meter Solar
Real-time building-level load forecasting is important for demand response and power system planning. Behind-the-meter (BTM) solar generation in buildings is not directly measured, resulting in a lack of native load measurements, even in recorded historical data. This invisibility of native load data makes load forecasting challenging for BTM buildings. Our idea is to learn the unknown and time-varying spatial correlations of nearby buildings to enhance the overall load forecasting accuracy. To the best of our knowledge, this paper, for the first time, integrates load disaggregation and load forecasting without requiring historical native load measurements on BTM consumers. The proposed method, ULoFo, has a computationally efficient load disaggregation component and a state-of-the-art forecasting component. ULoFo also has two interaction strategies, graph sparsification, and input refurbishment, to leverage the intermediate forecasting result to enhance disaggregation accuracy, which in turn further promotes native load forecasting accuracy. ULoFo is demonstrated to outperform existing methods in practical datasets.
- 22 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Power Systems
- Vol. 40  (2) , 1335-1347
Unifying Load Disaggregation and Prediction for Buildings With Behind-the-Meter Solar
Real-time building-level load forecasting is important for demand response and power system planning. Behind-the-meter (BTM) solar generation in buildings is not directly measured, resulting in a lack of native load measurements, even in recorded historical data. This invisibility of native load data makes load forecasting challenging for BTM buildings. Our idea is to learn the unknown and time-varying spatial correlations of nearby buildings to enhance the overall load forecasting accuracy. To the best of our knowledge, this paper, for the first time, integrates load disaggregation and load forecasting without requiring historical native load measurements on BTM consumers. The proposed method, ULoFo, has a computationally efficient load disaggregation component and a state-of-the-art forecasting component. ULoFo also has two interaction strategies, graph sparsification, and input refurbishment, to leverage the intermediate forecasting result to enhance disaggregation accuracy, which in turn further promotes native load forecasting accuracy. ULoFo is demonstrated to outperform existing methods in practical datasets.
- 7 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 18  (2) , 216-231
How Does Promoting the Minority Fraction Affect Generalization? A Theoretical Study of One-Hidden-Layer Neural Network on Group Imbalance
Group imbalance has been a known problem in empirical risk minimization (ERM), where the achieved high average accuracy is accompanied by low accuracy in a minority group. Despite algorithmic efforts to improve the minority group accuracy, a theoretical generalization analysis of ERM on individual groups remains elusive. By formulating the group imbalance problem with the Gaussian Mixture Model, this paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance. Although our theoretical framework is centered on binary classification using a one-hidden-layer neural network, to the best of our knowledge, we provide the first theoretical analysis of the group-level generalization of ERM in addition to the commonly studied average generalization performance. Sample insights of our theoretical results include that when all group-level co-variance is in the medium regime and all mean are close to zero, the learning performance is most desirable in the sense of a small sample complexity, a fast training rate, and a high average and group-level testing accuracy. Moreover, we show that increasing the fraction of the minority group in the training data does not necessarily improve the generalization performance of the minority group. Our theoretical results are validated on both synthetic and empirical datasets, such as CelebA and CIFAR-10 in image classification.
- 7 March 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Journal of Selected Topics in Signal Processing
- Vol. 18  (2) , 216-231
How Does Promoting the Minority Fraction Affect Generalization? A Theoretical Study of One-Hidden-Layer Neural Network on Group Imbalance
Group imbalance has been a known problem in empirical risk minimization (ERM), where the achieved high average accuracy is accompanied by low accuracy in a minority group. Despite algorithmic efforts to improve the minority group accuracy, a theoretical generalization analysis of ERM on individual groups remains elusive. By formulating the group imbalance problem with the Gaussian Mixture Model, this paper quantifies the impact of individual groups on the sample complexity, the convergence rate, and the average and group-level testing performance. Although our theoretical framework is centered on binary classification using a one-hidden-layer neural network, to the best of our knowledge, we provide the first theoretical analysis of the group-level generalization of ERM in addition to the commonly studied average generalization performance. Sample insights of our theoretical results include that when all group-level co-variance is in the medium regime and all mean are close to zero, the learning performance is most desirable in the sense of a small sample complexity, a fast training rate, and a high average and group-level testing accuracy. Moreover, we show that increasing the fraction of the minority group in the training data does not necessarily improve the generalization performance of the minority group. Our theoretical results are validated on both synthetic and empirical datasets, such as CelebA and CIFAR-10 in image classification.