Andrea Ceccarelli

The ever-evolving landscape of attacks and the growing complexity of ICT systems make crafting anomaly-based intrusion and error detectors difficult: they must accurately detect attacks and promptly perform detections. Although improving and comparing the detection capability is the focus of most research works, the timeliness of the detection is less considered and often insufficiently evaluated or discussed. In this paper, we argue the relevance of measuring the temporal latency of attacks, and we propose an evaluation approach for detectors to ensure a trade-off between correct and in-time detection. Briefly, the approach relates the false positive rate to the temporal latency of attacks, ultimately leading to guidelines for configuring a detector. We discuss and apply the strategy to compose datasets for intrusion detection that can support the computation of our metrics. We exercise our approach by evaluating different intrusion and error detectors in three industrial cases: i) an embedded railway on-board system that optimizes public mobility, ii) an edge device for the Industrial Internet of Things, and iii) an IoT network that monitors an industrial facility. Results show that considering latency in addition to traditional metrics like the false positive rate, precision, and recall gives an additional fundamental perspective on the actual performance of the detector and should be considered when assessing and configuring intrusion detectors.

Traffic Signs Recognition (TSR) is a fundamental task for the implementation of automated driving assistance systems. Major technical advancements have been achieved thanks to the adoption of machine learning classifiers, and especially Deep Neural Networks (DNNs), which were successfully used to correctly classify most of the traffic signs. A small number of misclassifications persist, that are due to various reasons ranging from a flawed learning process of classifiers to low-quality images, adverse environmental conditions, or adversarial attacks to the TSR. Misclassifications are unlikely, but still, they are a severe safety hazard. This paper discusses five strategies that can be adopted for safe, robust, and reliable TSR, that comprise data augmentation, noise removals, corruption, and classifier misclassification detection. We compare these strategies in an experimental campaign, where we inject camera failures into images of traffic signs from two public datasets. Results show that corruption and misclassification detectors can identify and discard in advance a huge fraction of the images that are going to be misclassified but may as well discard images that would have been correctly classified. We conclude that corruption and misclassification detectors can be used with DNNs to reduce misclassifications of TSR components and provide a more reliable functionality.

Machine Learning algorithms that perform classification are increasingly been adopted in Information and Communication Technology (ICT) systems and infrastructures due to their capability to profile their expected behavior and detect anomalies due to ongoing errors or intrusions. Deploying a classifier for a given system requires conducting comparison and sensitivity analyses that are time-consuming, require domain expertise, and may even not achieve satisfactory classification performance, resulting in a waste of money and time for practitioners and stakeholders. This paper predicts the expected performance of classifiers without needing to select, craft, exercise, or compare them, requiring minimal expertise and machinery. Should classification performance be predicted worse than expectations, the users could focus on improving data quality and monitoring systems instead of wasting time in exercising classifiers, saving key time and money. The prediction strategy uses scores of feature rankers, which are processed by regressors to predict metrics such as Matthews Correlation Coefficient (MCC) and Area Under ROC-Curve (AUC) for quantifying classification performance. We validate our prediction strategy through a massive experimental analysis using up to 12 feature rankers that process features from 23 public datasets, creating additional variants in the process and exercising supervised and unsupervised classifiers. Our findings show that it is possible to predict the value of performance metrics for supervised or unsupervised classifiers with a mean average error (MAE) of residuals lower than 0.1 for many classification tasks. The predictors are publicly available in a Python library whose usage is straightforward and does not require domain-specific skill or expertise.

We argue that object detectors in the safety critical domain should prioritize detection of objects that are most likely to interfere with the actions of the autonomous actor. Especially, this applies to objects that can impact the actor's safety and reliability. To quantify the impact of object (mis)detection on safety and reliability in the context of autonomous driving, we propose new object detection measures that reward the correct identification of objects that are most dangerous and most likely to affect driving decisions. To achieve this, we build an object criticality model to reward the detection of the objects based on proximity, orientation, and relative velocity with respect to the subject vehicle. Then, we apply our model on the recent autonomous driving dataset nuScenes, and we compare nine object detectors. Results show that, in several settings, object detectors that perform best according to the nuScenes ranking are not the preferable ones when the focus is shifted on safety and reliability.

The railway domain is regulated by rigorous safety standards to ensure that specific safety goals are met. Often, safety-critical systems rely on custom hardware-software components that are built from scratch to achieve specific functional and non-functional requirements. Instead, the (partial) usage of Commercial Off-The-Shelf (COTS) components is very attractive as it potentially allows reducing cost and time to market. Unfortunately, COTS components do not individually offer enough guarantees in terms of safety and security to be used in critical systems as they are. In such a context, RFI (Rete Ferroviaria Italiana), a major player in Europe for railway infrastructure management, aims at equipping track-side workers with COTS devices to remotely and safely interact with the existing interlocking system, drastically improving the performance of maintenance operations. This paper describes the first effort to update existing (embedded) railway systems to a more recent cyber-physical system paradigm. Our Remote Worker Dashboard (RWD) pairs the existing safe interlocking machinery alongside COTS mobile components, making cyber and physical components cooperate to provide the user with responsive, safe, and secure service. Specifically, the RWD is a SIL4 cyber-physical system to support maintenance of actuators and railways in which COTS mobile devices are safely used by track-side workers. The concept, development, implementation, verification, and validation activities to build the RWD were carried out in compliance with the applicable CENELEC standards required by certification bodies to declare compliance with specific guidelines.