An Optimized Deep Learning Approach for Multiclass Anomaly Detection
Open Access
- 11 February 2026
- journal article
- research article
- Published by MDPI AG in Information
- Vol. 17 (2) , 183
Abstract
The increasing scale and imbalance of modern network traffic pose significant challenges for multi-class intrusion detection systems (IDSs), particularly in identifying rare attack types. Traditional intrusion detection approaches based on supervised classification or unsupervised anomaly detection often suffer from limited generalization under severe class imbalance, high-dimensional feature spaces, and noisy traffic, resulting in poor detection of minority attack classes. To address these limitations, this study presents a hybrid intrusion detection framework that integrates unsupervised feature learning, anomaly scoring, and supervised classification within a unified pipeline. A denoising autoencoder trained exclusively on normal traffic is employed to learn compact and noise-resistant feature representations, while an isolation forest independently generates statistical anomaly scores. These complementary features are then fused and classified using a Light Gradient Boosting Machine (LightGBM). The main contribution of this work lies in the effective integration of these components, combined with a balanced training strategy based on the Synthetic Minority Over-sampling Technique with Edited Nearest Neighbors (SMOTE-ENN), as well as robust validation procedures. The framework is evaluated on the Network Security Laboratory Knowledge Discovery and Data Mining dataset (NSL-KDD) and the UNSW-NB15 intrusion detection dataset using stratified cross-validation and multiple independent runs. Experimental results demonstrate consistently high classification accuracy (~99%) and strong macro-F1 performance (>97%) across all attack categories on both NSL-KDD and UNSW-NB15 datasets. The framework achieves exceptional detection of rare classes (R2L: 99% F1, U2R: 100% F1), significantly outperforming prior approaches (AE-SAC: 83.97% F1, RL-NIDS: poor U2R recall), while maintaining low inference latency (~2–3 ms per sample, 415 samples/second) suitable for real-time network security deployment.Keywords
This publication has 41 references indexed in Scilit:
- IoT-based intrusion detection system using explainable multi-class deep learning approachesComputers and Electrical Engineering, 2025
- Real-Time Anomaly Detection in Network Traffic Using Graph Neural Networks and Random Forest2024
- Feature Extraction and Anomaly Detection Using Different Autoencoders for Modeling Intrusion Detection SystemsArabian Journal for Science and Engineering, 2024
- Time series forecasting and anomaly detection using deep learningComputers & Chemical Engineering, 2023
- A soft actor-critic reinforcement learning algorithm for network intrusion detectionComputers & Security, 2023
- AD-CGAN: Contrastive Generative Adversarial Network for Anomaly Detection2022
- A two-stage intrusion detection system with auto-encoder and LSTMsApplied Soft Computing, 2022
- Chameleon: Optimized feature selection using particle swarm optimization and ensemble methods for network anomaly detectionComputers & Security, 2022
- Representation learning-based network intrusion detection system by capturing explicit and implicit feature interactionsComputers & Security, 2022
- Generative and Autoencoder Models for Large-Scale Mutivariate Unsupervised Anomaly Detection2021