SepSpace: payload-free open-set network traffic classification via supervised contrastive representation learning and prototype-radius rejection
Open Access
- 11 August 2026
- journal article
- research article
- Published by Springer Nature in Cybersecurity
- Vol. 9 (1) , 204
Abstract
Network traffic classification (NTC) increasingly needs to go beyond predefined categories and identify traffic from previously unseen classes, namely the open-set recognition problem in NTC. This problem remains challenging because encrypted traffic limits the practicality of payload-dependent features, while existing methods still lack representation learning tailored to open-set discrimination. This study aims to develop a payload-free open-set NTC framework that constructs an embedding space suitable for both known-class classification and unseen-class rejection. To this end, we propose SepSpace, which models network flows using meta-feature sequences. SepSpace introduces supervised contrastive learning to construct a discriminative embedding space with compact intra-class structures and separated inter-class boundaries. Combined with a prototype-radius rejection mechanism, SepSpace achieves effective unseen-class rejection while maintaining known-class classification performance. Meanwhile, to comprehensively evaluate open-set performance, we introduce a class-holdout evaluation protocol and conduct experiments on four benchmark datasets together with the conventional cross-dataset protocol. Experimental results demonstrate that, under the class-holdout protocol, SepSpace achieves the highest average area under the receiver operating characteristic curve (AUROC) of 0.9015 and open-set classification rate (OSCR) of 0.6351 among the evaluated methods; under the cross-dataset protocol, it obtains a higher average AUROC of 0.9396. The implementation and datasets of SepSpace are available at https://github.com/BITbla/SepSpace .Keywords
Funding Information
- the Major Science and Technology Special Project of Henan Province (241110210200)
This publication has 27 references indexed in Scilit:
- Plug-in enhancement framework: Breaking through performance bottleneck of pre-trained models for encrypted traffic classificationComputer Networks, 2026
- CTDMF-SC: cybersecurity threat detection and mitigation framework for smart citiesInternational Journal of Information Security, 2026
- A malware traffic detection method based on Victim-Attacker interaction patternsComputers & Security, 2025
- nNFST: A single-model approach for multiclass novelty detection in network intrusion detection systemsJournal of Network and Computer Applications, 2025
- Generalized Out-of-Distribution Detection: A SurveyInternational Journal of Computer Vision, 2024
- Feature mining for encrypted malicious traffic detection with deep learning and other machine learning algorithmsComputers & Security, 2023
- A review of Machine Learning-based zero-day attack detection: Challenges and future directionsComputer Communications, 2022
- Data stream classification with novel class detection: a review, comparison and challengesKnowledge and Information Systems, 2021
- Deep packet: a novel approach for encrypted traffic classification using deep learningSoft Computing, 2019
- An empirical comparison of botnet detection methodsComputers & Security, 2014