Examining the Potential of Generative Language Models for Aviation Safety Analysis: Case Study and Insights using the Aviation Safety Reporting System (ASRS)
Preprint
- 11 July 2023
- preprint
- Published by MDPI AG in Preprints.org
Abstract
This research investigates the potential application of generative language models, especially ChatGPT, in aviation safety analysis as a means to enhance the efficiency of safety analyses and accelerate the time it takes to process incident reports. In particular, ChatGPT was leveraged to generate incident synopses from narratives, which were subsequently compared with ground truth synopses from the Aviation Safety Reporting System (ASRS) dataset. The comparison was facilitated by using embeddings from Large Language Models (LLMs), with aeroBERT demonstrating the highest similarity due to its aerospace-specific fine-tuning. A positive correlation was observed between synopsis length and their cosine similarity. In a subsequent phase, human factor issues involved in incidents as identified by ChatGPT were compared to human factor issues identified by safety analysts. A concurrence rate of 61% was found, with ChatGPT demonstrating a cautious approach towards attributing human factor issues. Finally, the model was used to attribute incidents to relevant parties. As no dedicated ground truth column existed for this task, a manual evaluation was conducted. ChatGPT attributed the majority of incidents to the Flight Crew, ATC, Ground Personnel, and Maintenance. This study opens new avenues for leveraging AI in aviation safety analysis.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- SafeAeroBERT: Towards a Safety-Informed Aerospace-Specific Language Model2023
- aeroBERT-NER: Named-Entity Recognition for Aerospace Requirements Engineering using BERT2023
- MLCM: Multi-Label Confusion MatrixIEEE Access, 2022
- RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models2020
- Stress, Pressure and Fatigue on Aircraft Maintenance PersonalInternational Review of Aerospace Engineering (IREASE), 2019
- Maintenance and inspection as risk factors in helicopter accidents: Analysis and recommendationsPLOS ONE, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Human Factors: The Last Frontier of Aviation Safety?The International Journal of Aviation Psychology, 2004