Linsen Ma
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Linsen Ma has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
- Rui Xie
Rensselaer Polytechnic Institute, Troy, NY, USA
6 shared publications - Tong Zhang
Rensselaer Polytechnic Institute, Troy, NY, USA
6 shared publications - Asad Ul Haq
Rensselaer Polytechnic Institute, Troy, NY, USA
4 shared publications - Liu Liu
Rensselaer Polytechnic Institute
4 shared publications - Yunhua Fang
Rensselaer Polytechnic Institute, Troy, NY, USA
4 shared publications - Feng Chen
Indiana University Bloomington
2 shared publications
Latest Publications
- Open Access
- 11 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Towards Encrypted Data Compression with Computational Storage Drives
- 11 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessTowards Encrypted Data Compression with Computational Storage Drives
- 20 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computers
- Vol. 75 (4) , 1349-1362
TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
LLM inference is increasingly limited by memory bandwidth, and the bottleneck worsens at long context as the KV cache grows. CXL memory adds capacity to offload weights and KV, but its link and device-side DDR bandwidth are far below HBM, so decoding stalls once traffic shifts to the CXL tier. Many CXL controllers are starting to add generic lossless compression, yet applying commodity codecs directly to standard word-major LLM tensors is largely ineffective, especially for token-major KV streams. We propose TRACE (Traffic-Reduced Architecture for Compression and Elasticity), which preserves the unmodified CXL.mem interface but changes the device-internal representation. It stores tensors in a channel-major, disaggregated bit-plane layout, and applies a KV-specific transform before compression, converting mixed-field words into low-entropy plane streams that commodity codecs can compress. The same substrate enables precision-proportional fetch by reading only the required bit-planes. Across public LLMs, TRACE reduces BF16 weight footprint by 25.2% and BF16 KV footprint by 46.9% losslessly, with per-layer KV ratios peaking at 2.69$\times$ . In trace-driven system modeling, once KV spills to CXL, GPT-OSS-120B-MXFP4 improves throughput at 128k tokens from 16.28 to 68.99 tok/s (4.24$\times$ ). DRAMSim3 shows up to 40.3% lower DRAM access energy under plane-aligned fetch. A 7 nm SystemVerilog implementation sustains 256 GB/s device bandwidth. Relative to a CXL controller with generic inline lossless compression, TRACE only adds 7.2% area, 4.7% power, and 6.0% load-to-use latency at 2 GHz and 0.7 V.- 20 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computers
- Vol. 75 (4) , 1349-1362
TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
LLM inference is increasingly limited by memory bandwidth, and the bottleneck worsens at long context as the KV cache grows. CXL memory adds capacity to offload weights and KV, but its link and device-side DDR bandwidth are far below HBM, so decoding stalls once traffic shifts to the CXL tier. Many CXL controllers are starting to add generic lossless compression, yet applying commodity codecs directly to standard word-major LLM tensors is largely ineffective, especially for token-major KV streams. We propose TRACE (Traffic-Reduced Architecture for Compression and Elasticity), which preserves the unmodified CXL.mem interface but changes the device-internal representation. It stores tensors in a channel-major, disaggregated bit-plane layout, and applies a KV-specific transform before compression, converting mixed-field words into low-entropy plane streams that commodity codecs can compress. The same substrate enables precision-proportional fetch by reading only the required bit-planes. Across public LLMs, TRACE reduces BF16 weight footprint by 25.2% and BF16 KV footprint by 46.9% losslessly, with per-layer KV ratios peaking at 2.69$\times$ . In trace-driven system modeling, once KV spills to CXL, GPT-OSS-120B-MXFP4 improves throughput at 128k tokens from 16.28 to 68.99 tok/s (4.24$\times$ ). DRAMSim3 shows up to 40.3% lower DRAM access energy under plane-aligned fetch. A 7 nm SystemVerilog implementation sustains 256 GB/s device bandwidth. Relative to a CXL controller with generic inline lossless compression, TRACE only adds 7.2% area, 4.7% power, and 6.0% load-to-use latency at 2 GHz and 0.7 V.- 17 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (2) , 337-340
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.
- 17 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (2) , 337-340
Accelerating LLM Inference via Dynamic KV Cache Placement in Heterogeneous Memory System
Large Language Model (LLM) inference is increasingly constrained by memory bandwidth, with frequent access to the key-value (KV) cache dominating data movement. While attention sparsity reduces some memory traffic, the relevance of past tokens varies over time, requiring the full KV cache to remain accessible and sustaining pressure on both bandwidth and capacity. With advances in interconnects such as NVLink and LPDDR5X, modern AI hardware now integrates high-bandwidth memory (HBM) with high-speed off-package DRAM, making heterogeneous memory systems a practical solution. This work investigates dynamic KV cache placement across such systems to maximize aggregated bandwidth utilization under capacity constraints. Rather than proposing a specific scheduling policy, we formulate the placement problem mathematically and derive a theoretical upper bound, revealing substantial headroom for runtime optimization. To our knowledge, this is the first formal treatment of dynamic KV cache scheduling in heterogeneous memory systems for LLM inference.
- Open Access
- 6 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 15-27
ZipCXL: CXL-based Main Memory Compression at Low Performance Penalty
- 6 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 15-27
Open AccessZipCXL: CXL-based Main Memory Compression at Low Performance Penalty
- 1 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (2) , 313-316
Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure
High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability requirements, poses a growing barrier to scalable deployment. This work explores a system-level approach to cost reduction by eliminating on-die ECC and shifting all fault management to the memory controller. We introduce a domain-specific ECC framework combining large-codeword Reed–Solomon (RS) correction with lightweight fine-grained CRC detection, differential parity updates to mitigate write amplification, and tunable protection based on data importance. Our evaluation using LLM inference workloads shows that, even under raw HBM bit error rates up to $10^{-3}$, the system retains 78% of throughput while maintaining at least 97% PIQA accuracy and 94% MMLU accuracy relative to error-free HBM. By treating reliability as a tunable system parameter rather than a fixed hardware constraint, our design opens a new path toward low-cost, high-performance HBM deployment in AI infrastructure.
- 1 October 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 24 (2) , 313-316
Breaking the HBM Bit Cost Barrier: Domain-Specific ECC for AI Inference Infrastructure
High-Bandwidth Memory (HBM) delivers exceptional bandwidth and energy efficiency for AI workloads, but its high cost per bit, driven in part by stringent on-die reliability requirements, poses a growing barrier to scalable deployment. This work explores a system-level approach to cost reduction by eliminating on-die ECC and shifting all fault management to the memory controller. We introduce a domain-specific ECC framework combining large-codeword Reed–Solomon (RS) correction with lightweight fine-grained CRC detection, differential parity updates to mitigate write amplification, and tunable protection based on data importance. Our evaluation using LLM inference workloads shows that, even under raw HBM bit error rates up to $10^{-3}$, the system retains 78% of throughput while maintaining at least 97% PIQA accuracy and 94% MMLU accuracy relative to error-free HBM. By treating reliability as a tunable system parameter rather than a fixed hardware constraint, our design opens a new path toward low-cost, high-performance HBM deployment in AI infrastructure.
- Open Access
- 30 September 2024
- Published by Association for Computing Machinery (ACM)
- p. 116-128
ZipCache: A DRAM/SSD Cache with Built-in Transparent Compression
- 30 September 2024
- Published by Association for Computing Machinery (ACM)
- p. 116-128
Open AccessZipCache: A DRAM/SSD Cache with Built-in Transparent Compression