Matthias Kleinert

Automatic speech recognition for air traffic control (ATC) faces severe training data scarcity due to operational recording restrictions and expensive domain-expert transcription requirements. We address this limitation by developing an automated pipeline that extracts large-scale, high-quality training data from publicly available YouTube streams of virtual ATC simulator sessions from networks such as VATSIM and IVAO. Our approach systematically processes over 2,000 hours of content spanning 709 videos from virtual airports and airspaces in 17 countries across multiple continents, operational domains (ground, tower, approach, en-route), and diverse speaker accents. The pipeline employs speaker diarization for utterance segmentation, parallel transcription using three complementary automatic speech recognition (ASR) architectures with distinct error characteristics, and Large Language Model-based transcript fusion that synthesizes improved pseudo-labels while filtering non-ATC content. Manual verification on a stratified 120-minute evaluation set demonstrates 10.2% word error rate for controller speech and 18.3% for pilot speech—representing 37% relative improvement over the best individual model and establishing pseudo-label quality sufficient for downstream model training. We show the feasibility of this approach by training a compact 115M-parameter ASR model exclusively on automatically generated transcripts without any manually annotated operational data. Evaluation on the operational ATCO2 benchmark reveals 21.1% word error rate compared to 35.6% for published baselines trained on smaller manually-transcribed datasets, despite the domain gap between virtual and operational ATC, while achieving approximately five times faster inference. These results demonstrate that large-scale geographically and acoustically diverse, pseudo-labeled data can effectively compensate for moderate label noise when training specialized-domain speech recognition systems. We openly release the complete processing pipeline, curated video collection, and our trained model to enable reproducible research.

While traffic levels are predicted to rise, nearly all European air navigation service providers suffer from staff shortages. In most cases, two air traffic controllers are deployed to control one airspace sector. Enabling the deployment of one controller per sector could be a solution to staff shortage problems. For this Single-Controller Operation (SCO) concept, a demonstrator with integrated support tools based on advanced information technology was developed. These partially automate some controller tasks to allow one controller to work off the same traffic amount as a controller team. The system was tested in a human-in-the-loop real-time simulation under varying traffic loads using a 2 × 2 within-subjects design. The variables assessed include separation minima infringements, exit flight level deviations, instantaneous self-assessment, voice communication, flight distance, and fuel burn. The results show no negative influence on safety, workload, situational awareness, operational efficiency, and environment, with 80% of maximum allowed declared capacity. Thus, SCO has the potential to mitigate staff shortages and raise cost efficiency by 40%. These results showcase the feasibility of the SCO concept under nominal conditions. Assessments with different traffic levels, non-nominal conditions, and an interdependent multi-sector SCO layout are recommended for further investigations.

Air traffic controllers (ATCos) quantified the benefits of automatic speech recognition and understanding (ASRU) on workload and flight safety. As a baseline procedure, ATCos manually enter all verbal clearances into the aircraft radar labels by mouse. In our proposed solution, ATCos are supported by ASRU, which is capable of delivering the required radar label updates automatically. ATCos need to visually review the ASRU-based label updates and only have to make corrections in case of misinterpretations. Overall, the amount of time required for manually inserting clearances, i.e., by selecting the correct input in the radar labels, was reduced from 12,700 s during 14 hours of simulation time down to 405 s when ATCos were supported by ASRU. Considering the additional time of mental workload for verifying ASRU output, there is still a saving of more than one-third of the time for radar label updates. This paper also considers safety aspects, i.e., how often incorrect inputs into aircraft radar labels occur with ASRU. The number of wrong or missing inputs is less than without ASRU support. This paper advances the use case that ASRU could potentially improve safety and efficiency for ATCo operations for arrivals.

The information air traffic controllers (ATCos) communicate via radio telephony is valuable for digital assistants to provide additional safety. Yet, ATCos have to enter this information manually. Assistant-based speech recognition (ABSR) has proven to be a lightweight technology that automatically extracts and successfully feeds the content of ATC communication into digital systems without additional human effort. This article explains how ABSR can be integrated into an advanced surface movement guidance and control system (A-SMGCS). The described validations were performed in the complex apron simulation training environment of Frankfurt Airport with 14 apron controllers in a human-in-the-loop simulation in summer 2022. The integration significantly reduces the workload of controllers and increases safety as well as overall performance. Based on a word error rate of 3.1%, the command recognition rate was 91.8% with a callsign recognition rate of 97.4%. This performance was enabled by the integration of A-SMGCS and ABSR: the command recognition rate improves by more than 15% absolute by considering A-SMGCS data in ABSR.

Assistant Based Speech Recognition (ABSR) systems for air traffic control radiotelephony communication have shown their potential to reduce air traffic controllers’ (ATCos) workload. Related research activities mainly focused on utterances for approach and en-route traffic. This is one of the first investigations of how ABSR could support ATCos in a tower environment. Ten ATCos from Lithuania and Austria participated in a human-in-the-loop simulation to validate ABSR support within a prototypic multiple remote tower controller working position. The ABSR supports ATCos by (1) highlighting recognized callsigns, (2) inputting recognized commands from ATCo utterances in electronic flight strips, (3) offering correction of ABSR output, (4) automatically accepting ABSR output, and (5) feeding the digital air traffic control system. This paper assesses human factors such as workload, situation awareness, and usability when ATCos are supported by ABSR. Those assessments result from a system with a relevant command recognition rate of 82.9% and a callsign recognition rate of 94.2%. Workload reductions and usability improvement with p-values below 0.25 are obtained for the case when the ABSR system is compared to the baseline situation without ABSR support. This motivates the technology to be brought to a higher technology readiness level, which is also confirmed by subjective feedback from questionnaires and objective measurement of workload reduction based on a performed secondary task.

Automatic speech recognition and understanding (ASRU) for air traffic control (ATC) has been investigated in different ATC environments and applications. The objective of this study was to quantify the effect of ASRU support for air traffic controllers (ATCos) radar label maintenance in terms of safety and human performance. Therefore, an implemented ASRU system was validated within a human-in-the-loop environment by ATCos in different traffic-density scenarios. In the baseline condition, ATCos performed radar label maintenance by entering verbally instructed ATC commands with a mouse and keyboard. In the proposed solution, ATCos were supported by ASRU, which achieved a command recognition rate of 92.5% with a command error rate of 2.4%. ASRU support reduced the number of wrong or missing inputs from ATCos into the radar label by a factor of two, which contemporaneously improved their situational awareness. Furthermore, ATCos where able to perform more successful secondary tasks when using ASRU support, indicating a greater capacity to handle unexpected events. The results from NASA TLX showed that the perceived workload decreased with a statistical significance of 4.3% across all scenarios. In conclusion, this study provides evidence that using ASRU for radar label maintenance can significantly reduce workload and improve flight safety.

As researchers around the globe develop applications for the use of Automatic Speech Recognition and Understanding (ASRU) in the Air Traffic Management (ATM) domain, Air Traffic Control (ATC) language ontologies will play a critical role in enabling research collaboration. The MITRE Corporation (MITRE) and the German Aerospace Center (DLR), having independently developed ATC language ontologies for specific applications, recently compared these ontologies to identify opportunities for improvement and harmonization. This paper extends the topic in two ways. First, this paper describes the specific ways in which ontologies facilitate the sharing of and collaboration on data, models, algorithms, metrics, and applications in the ATM domain. Second, this paper provides comparative analysis of word frequencies in ATC speech in the United States and Europe to illustrate that, whereas methods and tools for evaluating ASRU applications can be shared across researchers, the specific models would not work well between regions due to differences in the underlying corpus data.

In this paper we propose a novel virtual simulation-pilot engine for speeding up air traffic controller (ATCo) training by integrating different state-of-the-art artificial intelligence (AI)-based tools. The virtual simulation-pilot engine receives spoken communications from ATCo trainees, and it performs automatic speech recognition and understanding. Thus, it goes beyond only transcribing the communication and can also understand its meaning. The output is subsequently sent to a response generator system, which resembles the spoken read-back that pilots give to the ATCo trainees. The overall pipeline is composed of the following submodules: (i) an automatic speech recognition (ASR) system that transforms audio into a sequence of words; (ii) a high-level air traffic control (ATC)-related entity parser that understands the transcribed voice communication; and (iii) a text-to-speech submodule that generates a spoken utterance that resembles a pilot based on the situation of the dialogue. Our system employs state-of-the-art AI-based tools such as Wav2Vec 2.0, Conformer, BERT and Tacotron models. To the best of our knowledge, this is the first work fully based on open-source ATC resources and AI tools. In addition, we develop a robust and modular system with optional submodules that can enhance the system’s performance by incorporating real-time surveillance data, metadata related to exercises (such as sectors or runways), or even a deliberate read-back error to train ATCo trainees to identify them. Our ASR system can reach as low as 5.5% and 15.9% absolute word error rates (WER) on high- and low-quality ATC audio. We also demonstrate that adding surveillance data into the ASR can yield a callsign detection accuracy of more than 96%.

In the H2020 project “Satellite-borne and INsitu Observations to Predict The Initiation of Convection for ATM” (SINOPTICA), an air traffic controller support system was extended to organize approaching traffic even under severe weather conditions. During project runtime, traffic days with extreme weather events in the Po Valley were analyzed, an arrival manager was extended with a module for 4D diversion trajectory calculation, two display variants for severe weather conditions in an air traffic controller primary display were developed, and the airport Milano Malpensa was modelled for an air traffic simulation. On the meteorological side, three new forecasting techniques were developed to better nowcast weather events affecting tactical air traffic operations and used to automatically organize arrival traffic. Additionally, short-range weather forecasts with high spatial resolution were elaborated using radar-based nowcasting and a numerical weather prediction model with data assimilation. This nowcast information was integrated into the extended arrival manager for the sequencing and guiding of approaching aircraft even in adverse weather situations. The combination of fast and reliable weather nowcasts with a guidance support system enables severe weather diversion coordination in combination with a visualization of its dynamics on traffic situation displays.