Bruno Legeard
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Bruno Legeard has a total of 77 co-authors. Here are the top co-authors by number of shared publications:
- Fabrice Bouquet
University of Franche-Comte
16 shared publications Fabien Peureux
12 shared publications- Mark Utting
University of Queensland
11 shared publications - Alexandre Vernotte
Institut FEMTO-ST, UMR CNRS 6174
8 shared publications Elizabeta Fourneret
Smartesting Solutions & Services, Besançon, France
5 shared publicationsFrédéric Dadeau
Institut FEMTO-ST
5 shared publications
Latest Publications
- Open Access
- 22 June 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (ISSTA) , 206-228
Are Autonomous Web Agents Good Testers?
Despite advances in automated testing, manual testing remains prevalent due to the high maintenance demands associated with test script fragility—scripts often break with minor changes in application structure. Recent developments in Large Language Models (LLMs) offer a potential alternative by powering Autonomous Web Agents (AWAs) that can autonomously interact with applications. These agents may serve as Autonomous Test Agents (ATAs), potentially reducing the need for maintenance-heavy automated scripts by utilising natural language instructions similar to those used by human testers. This paper investigates the feasibility of adapting AWAs for natural language test case execution and how to evaluate them. We contribute with (1) a benchmark of three offline web applications, and a suite of 113 manual test cases, split between passing and failing cases, to evaluate and compare ATAs performance, (2) SeeAct-ATA and pinATA, two open-source ATA implementations capable of executing test steps, verifying assertions and giving verdicts, and (3) comparative experiments using our benchmark that quantifies our ATAs effectiveness. Finally we also proceed to a qualitative evaluation to identify the limitations of PinATA, our best performing implementation. Our findings reveal that our simple implementation, SeeAct-ATA, does not perform well compared to our more advanced PinATA implementation when executing test cases (50% performance improvement). However, while PinATA obtains around 60% of correct verdict and up to a promising 94% specificity, we identify several limitations that need to be addressed to develop more resilient and reliable ATAs, paving the way for robust, low maintenance test automation.
- 22 June 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (ISSTA) , 206-228
Open AccessAre Autonomous Web Agents Good Testers?
Despite advances in automated testing, manual testing remains prevalent due to the high maintenance demands associated with test script fragility—scripts often break with minor changes in application structure. Recent developments in Large Language Models (LLMs) offer a potential alternative by powering Autonomous Web Agents (AWAs) that can autonomously interact with applications. These agents may serve as Autonomous Test Agents (ATAs), potentially reducing the need for maintenance-heavy automated scripts by utilising natural language instructions similar to those used by human testers. This paper investigates the feasibility of adapting AWAs for natural language test case execution and how to evaluate them. We contribute with (1) a benchmark of three offline web applications, and a suite of 113 manual test cases, split between passing and failing cases, to evaluate and compare ATAs performance, (2) SeeAct-ATA and pinATA, two open-source ATA implementations capable of executing test steps, verifying assertions and giving verdicts, and (3) comparative experiments using our benchmark that quantifies our ATAs effectiveness. Finally we also proceed to a qualitative evaluation to identify the limitations of PinATA, our best performing implementation. Our findings reveal that our simple implementation, SeeAct-ATA, does not perform well compared to our more advanced PinATA implementation when executing test cases (50% performance improvement). However, while PinATA obtains around 60% of correct verdict and up to a promising 94% specificity, we identify several limitations that need to be addressed to develop more resilient and reliable ATAs, paving the way for robust, low maintenance test automation.
- 28 November 2023
- Published by Springer Nature
- p. 317-337
Trusted IoT and Testing
- Open Access
Detecting Intentional AIS Shutdown in Open Sea Maritime Surveillance Using Self-Supervised Deep Learning
In maritime traffic surveillance, detecting illegal activities, such as illegal fishing or transshipment of illicit products is a crucial task of the coastal administration. In the open sea, one has to rely on Automatic Identification System (AIS) message transmitted by on-board transponders, which are captured by surveillance satellites. However, insincere vessels often intentionally shut down their AIS transponders to hide illegal activities. In the open sea, it is very challenging to differentiate intentional AIS shutdowns from missing reception due to protocol limitations, bad weather conditions or restricting satellite positions. This paper presents a novel approach for the detection of abnormal AIS missing reception based on self-supervised deep learning techniques and transformer models. Using historical data, the trained model predicts if a message should be received in the upcoming minute or not. Afterwards, the model reports on detected anomalies by comparing the prediction with what actually happens. Our method can process AIS messages in real-time, in particular, more than 500 Millions AIS messages per month, corresponding to the trajectories of more than 60 000 ships. The method is evaluated on 1-year of real-world data coming from four Norwegian surveillance satellites. Using related research results, we validated our method by rediscovering already detected intentional AIS shutdowns.
Open AccessDetecting Intentional AIS Shutdown in Open Sea Maritime Surveillance Using Self-Supervised Deep Learning
In maritime traffic surveillance, detecting illegal activities, such as illegal fishing or transshipment of illicit products is a crucial task of the coastal administration. In the open sea, one has to rely on Automatic Identification System (AIS) message transmitted by on-board transponders, which are captured by surveillance satellites. However, insincere vessels often intentionally shut down their AIS transponders to hide illegal activities. In the open sea, it is very challenging to differentiate intentional AIS shutdowns from missing reception due to protocol limitations, bad weather conditions or restricting satellite positions. This paper presents a novel approach for the detection of abnormal AIS missing reception based on self-supervised deep learning techniques and transformer models. Using historical data, the trained model predicts if a message should be received in the upcoming minute or not. Afterwards, the model reports on detected anomalies by comparing the prediction with what actually happens. Our method can process AIS messages in real-time, in particular, more than 500 Millions AIS messages per month, corresponding to the trajectories of more than 60 000 ships. The method is evaluated on 1-year of real-world data coming from four Norwegian surveillance satellites. Using related research results, we validated our method by rediscovering already detected intentional AIS shutdowns.
- 10 February 2022
- Published by Elsevier in Computers & Security
- Vol. 116
CAE: Contextual auto-encoder for multivariate time-series anomaly detection in air transportation
- 10 February 2022
- Published by Elsevier in Computers & Security
- Vol. 116
CAE: Contextual auto-encoder for multivariate time-series anomaly detection in air transportation
- 22 February 2021
- Published by Springer Nature in International Journal on Software Tools for Technology Transfer
- Vol. 24 (2) , 127-158
A domain-specific language to design false data injection tests for air traffic control systems
- 22 February 2021
- Published by Springer Nature in International Journal on Software Tools for Technology Transfer
- Vol. 24 (2) , 127-158
A domain-specific language to design false data injection tests for air traffic control systems
- Open Access
- 1 December 2020
- Published by MDPI AG in Proceedings
- Vol. 59 (1) , 9
Improved Testing of AI-Based Anomaly Detection Systems Using Synthetic Surveillance Data
Major transportation surveillance protocols have not been specified with cyber security in mind and therefore provide no encryption nor identification. These issues expose air and sea transport to false data injection attacks (FDIAs), in which an attacker modifies, blocks or emits fake surveillance messages to dupe controllers and surveillance systems. There has been growing interest in conducting research on machine learning-based anomaly detection systems that address these new threats. However, significant amounts of data are needed to achieve meaningful results with this type of model. Raw, genuine data can be obtained from existing databases but need to be preprocessed before being fed to a model. Acquiring anomalous data is another challenge: such data is much too scarce for both the Automatic Dependent Surveillance–Broadcast (ADS-B) and the Automatic Identification System (AIS). Crafting anomalous data by hand, which has been the sole method applied to date, is hardly suitable for broad detection model testing. This paper proposes an approach built upon existing libraries and ideas that offers ML researchers the necessary tools to facilitate the access and processing of genuine data as well as to automatically generate synthetic anomalous surveillance data to constitute broad, elaborated test datasets. We demonstrate the usability of the approach by discussing work in progress that includes the reproduction of related work, creation of relevant datasets and design of advanced anomaly detection models for both domains of application.
- 1 December 2020
- Published by MDPI AG in Proceedings
- Vol. 59 (1) , 9
Open AccessImproved Testing of AI-Based Anomaly Detection Systems Using Synthetic Surveillance Data
Major transportation surveillance protocols have not been specified with cyber security in mind and therefore provide no encryption nor identification. These issues expose air and sea transport to false data injection attacks (FDIAs), in which an attacker modifies, blocks or emits fake surveillance messages to dupe controllers and surveillance systems. There has been growing interest in conducting research on machine learning-based anomaly detection systems that address these new threats. However, significant amounts of data are needed to achieve meaningful results with this type of model. Raw, genuine data can be obtained from existing databases but need to be preprocessed before being fed to a model. Acquiring anomalous data is another challenge: such data is much too scarce for both the Automatic Dependent Surveillance–Broadcast (ADS-B) and the Automatic Identification System (AIS). Crafting anomalous data by hand, which has been the sole method applied to date, is hardly suitable for broad detection model testing. This paper proposes an approach built upon existing libraries and ideas that offers ML researchers the necessary tools to facilitate the access and processing of genuine data as well as to automatically generate synthetic anomalous surveillance data to constitute broad, elaborated test datasets. We demonstrate the usability of the approach by discussing work in progress that includes the reproduction of related work, creation of relevant datasets and design of advanced anomaly detection models for both domains of application.
- Open Access
- 1 October 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 84-94
Supporting efficient test automation using lightweight MBT
The Agile and DevOps transformation of software development practices enhances the need for increased automation of functional testing, especially for regression testing. This poses challenges both in the effort that needs to be devoted to the creation and maintenance of automated test scripts, and in their relevance (i.e. their alignment with business needs). Test automation is still difficult to implement and maintain and the return on investment comes late while projects tend to be short. In this context, we have experimented a lightweight model-based test automation approach to address both productivity and relevance challenges. It integrates test automation through a simple process and tool-chain experimented on large IT projects.
- 1 October 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 84-94
Open AccessSupporting efficient test automation using lightweight MBT
The Agile and DevOps transformation of software development practices enhances the need for increased automation of functional testing, especially for regression testing. This poses challenges both in the effort that needs to be devoted to the creation and maintenance of automated test scripts, and in their relevance (i.e. their alignment with business needs). Test automation is still difficult to implement and maintain and the return on investment comes late while projects tend to be short. In this context, we have experimented a lightweight model-based test automation approach to address both productivity and relevance challenges. It integrates test automation through a simple process and tool-chain experimented on large IT projects.
- 1 June 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Improving Internet of Things device certification with policy-based management
The fast growing rate of the IoT systems with strong pressure to put devices on the market as soon as possible makes these systems vulnerable targets for cyber criminals, as recently seen in the Mirai botnet Distributed Denial-of-Service (DDoS) attack. A way to mitigate these threats is to enforce a comprehensive security certification process of IoT devices based on common standards. In this paper, we present an approach to improve certification of IoT devices using a combination of model-based testing and policy-based management in order to detect post certification vulnerabilities and act on them by introducing runtime policy enforcement capabilities. More precisely, we address these attacks using policy enforcement in order to correct vulnerable IoT device behavior and protect users even if security and privacy were not properly addressed by the device manufactures. We describe the details of our approach and, focusing on authorization vulnerabilities, we present a case study for the oneM2M standard showing how our solution can be applied in practice.
- 1 June 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-6
Improving Internet of Things device certification with policy-based management
The fast growing rate of the IoT systems with strong pressure to put devices on the market as soon as possible makes these systems vulnerable targets for cyber criminals, as recently seen in the Mirai botnet Distributed Denial-of-Service (DDoS) attack. A way to mitigate these threats is to enforce a comprehensive security certification process of IoT devices based on common standards. In this paper, we present an approach to improve certification of IoT devices using a combination of model-based testing and policy-based management in order to detect post certification vulnerabilities and act on them by introducing runtime policy enforcement capabilities. More precisely, we address these attacks using policy enforcement in order to correct vulnerable IoT device behavior and protect users even if security and privacy were not properly addressed by the device manufactures. We describe the details of our approach and, focusing on authorization vulnerabilities, we present a case study for the oneM2M standard showing how our solution can be applied in practice.
- 9 February 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 627-632
Security certification and labelling in Internet of Things
In recent years, security and privacy aspects of IoT have received considerable attention from the industry and research communities. Because IoT will be more pervasive in the everyday life of the citizens, and it may be used in safety related applications (e.g., road transportation), its security threats may be more damaging than conventional Internet threats. Due to processing and memory constraints, the provision of security functions could be quite challenging in IoT. In addition, IoT devices must operate in a dynamic environment in terms of communication interfaces and fast upgrade cycle (e.g., patching), which imposes severe security requirements to designer and developers. Privacy aspects are also relevant because of the large amount of data collected by IoT sensors. In this context, the security certification of IoT devices is an important element to support the development and deployment of trusted IoT systems and applications. The objective of this paper is to investigate IoT security certification taking into consideration the current security certification frameworks, standards, and their related limitations identified by the industry and research communities. This paper proposes a new approach for security certification in IoT, which addresses the identified limitations and links formal models to testing and certification.
- 9 February 2017
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 627-632
Security certification and labelling in Internet of Things
In recent years, security and privacy aspects of IoT have received considerable attention from the industry and research communities. Because IoT will be more pervasive in the everyday life of the citizens, and it may be used in safety related applications (e.g., road transportation), its security threats may be more damaging than conventional Internet threats. Due to processing and memory constraints, the provision of security functions could be quite challenging in IoT. In addition, IoT devices must operate in a dynamic environment in terms of communication interfaces and fast upgrade cycle (e.g., patching), which imposes severe security requirements to designer and developers. Privacy aspects are also relevant because of the large amount of data collected by IoT sensors. In this context, the security certification of IoT devices is an important element to support the development and deployment of trusted IoT systems and applications. The objective of this paper is to investigate IoT security certification taking into consideration the current security certification frameworks, standards, and their related limitations identified by the industry and research communities. This paper proposes a new approach for security certification in IoT, which addresses the identified limitations and links formal models to testing and certification.
- 5 October 2016
- Published by Springer Nature
- p. 727-742
Model-Based Testing as a Service for IoT Platforms
- 5 October 2016
- Published by Springer Nature
- p. 727-742
Model-Based Testing as a Service for IoT Platforms