Efficient Nonlinear Equalization in IM/DD Systems via Clustering-Based Non-Uniform Quantization
- 6 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Journal of Lightwave Technology
- Vol. 44 (9) , 3383-3392
Abstract
The feedforward equalizer (FFE), the decision feedback equalizer (DFE), and Volterra-based equalizer are widely adopted and shown effective in mitigating linear or nonlinear channel impairments in intensity-modulation direct-detection (IM/DD) systems. However, in practical hardware implementations, these equalizers often rely on floating-point (FP) representations for tap coefficients, resulting in high memory consumption and significant energy overhead. In this paper, we propose two novel equalization solutions with clustering-based non-uniform quantization schemes: automatic grouping-enabled non-uniform quantization (AGNQ) and $k$-means clustering-based non-uniform quantization (KCNQ), to address this challenge. In a 20 km C-band pulse amplitude modulation (PAM)-8 IM/DD transmission system employing a directly modulated laser (DML), experiments are conducted to evaluate the performance of the proposed methods under various equalization schemes. At a transmission rate of 36 GBaud, KCNQ achieves a bit error rate (BER) below the 25% soft-decision forward error correction (SD-FEC) threshold in FFE using only 2-bit quantization, and meets the 7% hard-decision FEC (HD-FEC) threshold with 4-bit and 6-bit quantization in Volterra equalizer and Volterra-DFE (VDFE), respectively. Compared to the 32-bit FP baselines, KCNQ reduces coefficient storage by 83.0%, 81.3% and 76.4% for FFE, Volterra, and VDFE equalizers. In contrast, AGNQ achieves reductions of 80.2%, 74.5%, and 67.5%. Compared with AGNQ, KCNQ can further improve memory efficiency by 13.9%, 26.6%, and 27.4%. The proposed approach not only reduces storage requirements but also maintains reliable communication quality compared with original equalization approaches, offering a balanced trade-off between complexity efficiency and performance.Keywords
Funding Information
- National Key Research and Development Program of China (2024YFB29NL00100)
- National Natural Science Foundation of China (62501328)
- Fund of State Key Laboratory of Photonics and Communications, China (2025QZKF07)
This publication has 44 references indexed in Scilit:
- Impact of receiver bandwidth on the performance of nonlinear volterra equalizer in IM/DD systemsOptics Communications, 2024
- Mixed-precision integer-arithmetic-only neural network-based equalizers for DML-based short-reach IM/DD systemsIET Conference Proceedings, 2023
- Long Short Term Memory Neural Network (LSTMNN) and inter-symbol feature extraction for 160 Gbit/s PAM4 from silicon micro-ring transmitterOptics Communications, 2022
- Experimental implementation of a neural network optical channel equalizer in restricted hardware using pruning and quantizationScientific Reports, 2022
- Zero-Multiplier Sparse DNN Equalization for Fiber-Optic QAM Systems with Probabilistic Amplitude Shaping2021
- High-capacity direct-detection systems2020
- 200 Gbps/Lane IM/DD Technologies for Short Reach Optical InterconnectsJournal of Lightwave Technology, 2019
- Low-bit Quantization of Neural Networks for Efficient Inference2019
- Digital Signal Processing for Short-Reach Optical Communications: A Review of Current Technologies and Future TrendsJournal of Lightwave Technology, 2018
- Electronic Data Processing for Error and Dispersion CompensationJournal of Lightwave Technology, 2006