Examining the Potential of Generative Language Models for Aviation Safety Analysis: Case Study and Insights Using the Aviation Safety Reporting System (ASRS)
Open Access
Abstract
This research investigates the potential application of generative language models, especially ChatGPT, in aviation safety analysis as a means to enhance the efficiency of safety analyses and accelerate the time it takes to process incident reports. In particular, ChatGPT was leveraged to generate incident synopses from narratives, which were subsequently compared with ground-truth synopses from the Aviation Safety Reporting System (ASRS) dataset. The comparison was facilitated by using embeddings from Large Language Models (LLMs), with aeroBERT demonstrating the highest similarity due to its aerospace-specific fine-tuning. A positive correlation was observed between the synopsis length and its cosine similarity. In a subsequent phase, human factors issues involved in incidents, as identified by ChatGPT, were compared to human factors issues identified by safety analysts. The precision was found to be 0.61, with ChatGPT demonstrating a cautious approach toward attributing human factors issues. Finally, the model was utilized to execute an evaluation of accountability. As no dedicated ground-truth column existed for this task, a manual evaluation was conducted to compare the quality of outputs provided by ChatGPT to the ground truths provided by safety analysts. This study discusses the advantages and pitfalls of generative language models in the context of aviation safety analysis and proposes a human-in-the-loop system to ensure responsible and effective utilization of such models, leading to continuous improvement and fostering a collaborative approach in the aviation safety domain.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Machine Learning and Natural Language Processing for Prediction of Human Factors in Aviation Incident ReportsAerospace, 2021
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models2020
- Stress, Pressure and Fatigue on Aircraft Maintenance PersonalInternational Review of Aerospace Engineering (IREASE), 2019
- Maintenance and inspection as risk factors in helicopter accidents: Analysis and recommendationsPLOS ONE, 2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- Natural language processing for aviation safety reports: From classification to interactive analysisComputers in Industry, 2016
- Human Factors in Aviation2010
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Human Factors: The Last Frontier of Aviation Safety?The International Journal of Aviation Psychology, 2004
- Human factors and aviation safety: what the industry has, what the industry needsErgonomics, 2000