Dual-Threshold Conformal Deferral for Trustworthy Security Alert Triage
Open Access
- 9 September 2026
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 15 (18) , 4084
Abstract
Automated alert triage can reduce Security Operations Center (SOC) workload, yet the validation-tuned thresholds deployed systems rely on carry no finite-sample control of their operational error rates and degrade unpredictably under distribution shift. We present a model-agnostic dual-threshold conformal deferral architecture: high-score alerts are auto-escalated under finite-sample marginal class-conditional control of the benign-escalation probability (budget α), low-score alerts are auto-closed under matching control of the threat-miss probability (budget β), and the rest are deferred to an analyst. It needs no retraining and closes an automatic zone rather than certifying what the calibration data cannot support. We evaluate it on a reinforcement-learning investigation agent in a simulated SOC and on four classifiers trained on CIC-IDS2017 and tested on CSE-CIC-IDS2018, using stratified 25,000-flow calibration and evaluation samples, with attack-type recall computed over the full 16.2-million-flow corpus. Pooling episodes from ten trained policies across two evaluation datasets, the architecture automated 73.7% of decisions at α = β = 0.01—a figure for that predefined pooled mixture rather than a per-policy or per-dataset guarantee—realizing benign auto-escalation and threat auto-close rates of 0.0099 and 0.0101 and deferring the hardest ~26% of alerts. After recalibration on labeled target-domain data, severe cross-dataset degradation appears not as a silent error but as sharply reduced certifiable automation, with deferral rising to 79–99% for the most affected classifiers. This visibility is a property of the recalibrated layer: thresholds left un-recalibrated after a shift continue to certify nothing while still deciding, so the architecture requires periodic recalibration on labelled target-domain alerts to deliver it. Substituting open-weight language models for the analyst inside the band failed a pre-specified criterion at every scale tested from 7B to 32B across two model families, with the discriminative signal flat in model size and far below the first-stage policy’s own.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Machine Learning in Network Intrusion Detection: A Cross-Dataset Generalization StudyIEEE Access, 2024
- Enhancing trustworthiness in ML-based network intrusion detection with uncertainty quantificationJournal of Reliable Intelligent Environments, 2024
- Conformal Triage for Medical Imaging AI DeploymentmedRxiv, 2024
- DEEPCASE: Semi-Supervised Contextual Analysis of Security Events2022
- Algorithmic Learning in a Random World2022
- Security Operations Center: A Systematic Study and Open ChallengesIEEE Access, 2020
- Inter-dataset generalization strength of supervised machine learning methods for intrusion detectionJournal of Information Security and Applications, 2020
- Distribution-Free Predictive Inference for RegressionJournal of the American Statistical Association, 2018
- UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set)2015
- Inductive Confidence Machines for Regression2002