Thomas Laurent

Analysis of rugby match and training footage is particularly useful for coaches and players to understand and improve their tackling technique, and potentially lower the rate of injuries. Machine learning models (in particular for pose estimation) promise to streamline rugby analysis. However models trained for “general purpose” computer vision tasks, such as pose estimation and object detection, frequently fail as a result of the challenging conditions and significant domain shift that rugby footage presents: high-impact, close-contact play causes problems such as occlusions, motion blur, and unconventional body orientations. It is therefore crucial to understand the specific conditions which cause these systems to fail so they can be prioritised during pre-processing and expensive manual data collection. In this paper we leverage Met-Pose, a metamorphic testing system to understand the specific conditions that cause pose estimation systems to fail. Metamorphic testing is particularly advantageous as this approach side-steps the need for costly, manually labelled data. Our ongoing project on applying pose estimation for rugby analysis employs MediaPipe, a popular, widely used pose estimation system, on rugby broadcast footage. We show how applying metamorphic testing to a sport analytics application can reveal situations that challenge the model without the need for any manual data labelling. For example, our results show that in this context, MediaPipe is particularly sensitive to motion blur and colour loss, but less so to lighting and resolution changes. Furthermore, we show how this process can be adapted to focus on particular aspects of an application by proposing a new metamorphic rule exploring the effect of including or excluding context on MediaPipe’s results. Our results show where MediaPipe struggles in complex, real-world sporting scenarios and also offer concrete insights for improving data augmentation, data collection and system design in sports analytics.

Concerns about the cumulative effects of head acceleration events in rugby are growing, but how tackle events lead to direct head contact in women's rugby remains underexplored. This cross‐sectional study aimed to develop and evaluate a machine learning model to identify characteristics associated with direct head contact and incorrect tackler head placement in elite women's rugby. Match situational and precontact technical characteristics (n = 31) from 1500 randomly selected tackle events were coded visually and retrospectively analyzed from the 2022–23 Women's Six Nations Championship. A machine learning model was developed and evaluated using a grid search with 5‐fold cross‐validations and F₁ scores (i.e., a measure of predictive performance). The top modifiable characteristics associated with the target outcomes across 100 test sets were identified by mutual importance and decision tree modeling. The top modifiable characteristics linked to direct head contact to the tackler were incorrect head placement, coming to balance, and foot placement. Tackle direction, point of contact on the tackler, and multiplayer tackles were key characteristics for incorrect tackler head placement. Tackler drop height, front/oblique tackle direction, and multiplayer tackles were strongly associated with direct head contact to the ball‐carrier. Incorrect tackler head placement, the direction of tackle, tackler drop height, and multiplayer tackles are key characteristics in direct head contact events in elite women's rugby. Addressing these characteristics should be prioritized in contact training practices, education resources, and law enforcement to enhance player safety and direct head contact events in the women's game.

In AI-enabled CPSs, DNNs are used as controllers for the physical system. Despite their advantages, DNN controllers can produce wrong control decisions, which can lead to safety risks for the system. Once wrong behaviors are detected, the DNN controller should be fixed. DNN repair is a technique that allows to perform this fine-grained improvement. However, state-of-the-art DNN repair techniques require ground-truth labels to guide the repair. For AI-enabled CPSs, these are not available, as it is not possible to assess whether a specific control decision is correct. Nevertheless, it is possible to assess whether the DNN controller leads to wrong behaviors of the controlled system by considering system-level requirements. In this paper, following this observation, we propose a novel DNN repair approach that is guided by system-level specifications. The approach takes in input a system-level specification, some tests violating the specification, and some faulty DNN weights. The approach searches for alternative weight values with the goal of fixing the behavior on the failing tests without breaking the passing tests. We also propose a heuristic that allows us to accelerate the search by avoiding the execution of some tests. Experiments on real-world AI-enabled CPSs show that the approach effectively repairs their controllers.

As DNNs are embedded in more and more critical systems, it is essential to ensure that they perform well on specific inputs. DNN repair has shown good results in fixing specific misclassifications in already trained models using additional data, even surpassing additional training. In safety-critical applications, such as autonomous driving, collaboration between industrial actors would lead to more representative datasets for repair, that would enable to obtain more robust models and thus safer systems. However, these companies are reluctant to share their data, to both protect their intellectual property and the privacy of their users. Federated Learning is an approach that allows for collaborative, privacy-preserving training of DNNs. Inspired by this technique, this work proposes Federated Repair in order to collaboratively repair a DNN model without the need for sharing any raw data. We implemented Federated Repair based on a state-of-the-art DNN repair technique, and applied it to three DNN models, with federation size from 2 to 10. Results show that Federated Repair can achieve the same repair efficiency as non-federated DNN repair using the pooled data, despite the presence of rounding errors when aggregating clients' results.

DNN repair is an effective technique applied after training to enhance the class-specific accuracy of classifier models, where a low failure rate is required on specific classes. The repair methods introduced in recent studies assume that they are applied to fully trained models. In this paper, we argue that this could not always be the best choice. We analyse the performance of DNN models under various training times and repair combinations. Through meticulously designed experiments on two real-world datasets and a carefully curated assessment score, we show that applying DNN repair earlier in the training process, and not only at its end, can be beneficial. Thus, we encourage the research community to consider the idea of when to apply DNN repair in the model development.

Deep Neural Networks (DNNs) are finding a place at the heart of more and more critical systems, and it is necessary to ensure they perform in as correct a way as possible. Search-based repair methods, that search for new values for target neuron weights in the network to better process fault-inducing inputs, have shown promising results. These methods rely on fault localisation to determine what weights the search should target. However, as the search progresses and the network evolves, the weights responsible for the faults in the system will change, and the search will lose in effectiveness. In this work, we propose an adaptive search method for DNN repair that adaptively updates the target weights during the search by performing fault localisation on the current state of the model. We propose and implement two methods to decide when to update the target weights, based on the progress of the search's fitness value or on the evolution of fault localisation results. We apply our technique to two image classification DNN architectures against a dataset of autonomous driving images, and compare it with a state-of-the art search-based DNN repair approach.

Autonomous Driving Systems (ADSs) are promising, but must show they are secure and trustworthy before adoption. Simulation-based testing is a widely adopted approach, where the ADS is run in a simulated environment over specific scenarios. Coverage criteria specify what needs to be covered to consider the ADS sufficiently tested. However, existing criteria do not guarantee to exercise the different decisions that the ADS can make, which is essential to assess its correctness. ADSs usually compute their decisions using parameterised rule-based systems and cost functions, such as cost components or decision thresholds. In this article, we argue that the parameters characterise the decision process, as their values affect the ADS’s final decisions. Therefore, we propose parameter coverage, a criterion requiring to cover the ADS’s parameters. A scenario covers a parameter if changing its value leads to different simulation results, meaning it is relevant for the driving decisions made in the scenario. Since ADS simulators are slightly uncertain, we employ statistical methods to assess multiple simulation runs for execution difference and coverage. Experiments using the Autonomoose ADS show that the criterion discriminates between different scenarios and that the cost of computing coverage can be managed with suitable heuristics.

Understanding the performance characteristics of software systems is particular relevant when looking at design alternatives. However, it is a very challenging problem, due to the complexity of interpreting the role and incidence of the different system elements on performance metrics of interest, such as system response time or resources utilisation. This work introduces JSIMutate, a tool that makes use of queueing network performance models and enables the analysis of mutations of a model reflecting possible design changes to support designers in identifying the model elements that contribute to improving or worsening the system's performance.