Constantin Orasan

Research on Automatic Speech Recognition (ASR)-assisted interpreting has concentrated on simultaneous conference interpreting, leaving remote public service settings underexplored. This article reviews methodological approaches to ASR-assisted interpreting research and presents two pilot studies on remote legal and medical interpreting. In the legal pilot, two professional interpreters completed ASR and non-ASR tasks in simultaneous and consecutive modes; ASR was associated with higher NTR accuracy and lower weighted error scores, especially in consecutive tasks, while interviews showed selective transcript use alongside style- and attention-related trade-offs. In the medical pilot, four trainee interpreters completed four ASR display conditions; full transcripts and ASR-fed summaries were associated with lower perceived workload, while the eye-tracking component exposed design issues requiring refinement. Together, the pilots show how domain-specific conditions shape research design, user responses, and the methodological challenges of studying AI-assisted interpreting in Public Service Interpreting.

Reference-less evaluation of machine translation, or Quality Estimation (QE), is vital for low-resource language pairs where high-quality references are often unavailable. In this study, we investigate segment-level QE methods comparing encoder-based models such as MonoTransQuest, CometKiwi, and xCOMET with various decoder-based methods (Tower+, ALOPE, and other instruction-fine-tuned language models). Our work primarily focused on utilizing eight low-resource language pairs, involving both English on the source side and the target side of the translation. Results indicate that while fine-tuned encoder-based models remain strong performers across most low-resource language pairs, decoder-based Large Language Models (LLMs) show clear improvements when adapted through instruction tuning. Importantly, the ALOPE framework further enhances LLM performance beyond standard fine-tuning, demonstrating its effectiveness in narrowing the gap with encoder-based approaches and highlighting its potential as a viable strategy for low-resource QE. In addition, our experiments demonstrates that with adaptation techniques such as LoRA (Low Rank Adapters) and quantization, decoder-based QE models can be trained with competitive GPU memory efficiency, though they generally require substantially more disk space than encoder-based models. Our findings highlight the effectiveness of encoder-based models for low-resource QE and suggest that advances in cross-lingual modeling will be key to improving LLM-based QE in the future.

The increasing adoption of artificial intelligence across domains presents new opportunities to enhance access to justice. In this paper, we introduce a human-centric AI tool that utilises advances in Automatic Speech Recognition (ASR) and Large Language Models (LLMs) to facilitate semantic linking between written UK Supreme Court (SC) judgements and their corresponding hearing videos. The motivation stems from the critical role UK SC hearings play in shaping landmark legal decisions, which often span several hours and remain difficult to navigate manually. Our approach involves two key components: (1) a customised ASR system fine-tuned on 139 h of manually edited SC hearing transcripts and legal documents and (2) a semantic linking module powered by GPT-based text embeddings adapted to the legal domain. The ASR system addresses domain-specific transcription challenges by incorporating a custom language model and legal phrase extraction techniques. The semantic linking module uses fine-tuned embeddings to match judgement paragraphs with relevant spans in the hearing transcripts. Quantitative evaluation shows that our customised ASR system improves transcription accuracy by 9% compared to generic ASR baselines. Furthermore, our adapted GPT embeddings achieve an F1 score of 0.85 in classifying relevant links between judgement text and hearing transcript segments. These results demonstrate the effectiveness of our system in streamlining access to critical legal information and supporting legal professionals in interpreting complex judicial decisions.

Large Language Models (LLMs) have shown remarkable performance across a wide range of natural language processing tasks. Quality Estimation (QE) for Machine Translation (MT), which assesses the quality of a source-target pair without relying on reference translations, remains a challenging cross-lingual task for LLMs. The challenges stem from the inherent limitations of existing LLM-based QE systems, which are pre-trained for causal language modelling rather than regression-specific tasks, further elevated by the presence of low-resource languages given pre-training data distribution. This paper introduces ALOPE, an adaptive layer-optimization framework designed to enhance LLM-based QE by restructuring Transformer representations through layer-wise adaptation for improved regression-based prediction. Our framework integrates low-rank adapters (LoRA) with regression task heads, leveraging selected pre-trained Transformer layers for improved cross-lingual alignment. In addition to the layer-specific adaptation, ALOPE introduces two strategies-dynamic weighting, which adaptively combines representations from multiple layers, and multi-head regression, which aggregates regression losses from multiple heads for QE. Our framework shows improvements over various existing LLM-based QE approaches. Empirical evidence suggests that intermediate Transformer layers in LLMs provide contextual representations that are more aligned with the cross-lingual nature of the QE task. We make resultant models and framework code publicly available for further research, also allowing existing LLM-based MT frameworks to be scaled with QE capabilities.

This chapter reviews quality as a multifaceted and highly debated yet central concept within interpreting studies. It highlights the crucial relevance of quality with regards to deepening our understanding of the influence of different technology-related modalities of interpreting on current and future practice. The chapter begins by revisiting foundational dimensions of quality in interpreting and examines the key challenges posed by the intersection of technology and interpreting. It then synthesises how quality can be evaluated and measured and considers the added complexities introduced by technology. To this end, this chapter provides a comprehensive review of the methods and perspectives used to assess quality in various technologised workflows, grouping them into three categories, namely, top-down, bottom-up, and automated methods. The chapter also demonstrates how technology integration into different interpreting practices makes the process of assessing quality even more intricate, yet increasingly needed, while also outlining the benefits and drawbacks of technology-driven quality assessment. The chapter concludes by discussing future developments, particularly the need for a hybrid approach to quality evaluation that balances scalability with deeper, more nuanced insights to address the complex, nuanced interplay between interpreting and technology.

Evaluating machine translation (MT) of user-generated content (UGC) involves unique challenges such as checking whether the nuance of emotions from the source are preserved in the target text. Recent studies have proposed emotion-related datasets, frameworks and models to automatically evaluate MT quality of Chinese UGC, without relying on reference translations. However, whether these models are robust to the challenge of preserving emotional nuances has been left largely unexplored. To address this gap, we introduce a novel method inspired by information theory which generates challenging Chinese homophone words related to emotions, by leveraging the concept of self-information. Our approach generates homophones that were observed to cause translation errors in emotion preservation, and exposes vulnerabilities in MT systems and their evaluation methods when tackling emotional UGC. We evaluate the efficacy of our method using human evaluation for the quality of these generated homophones, and compare it with an existing one, showing that our method achieves higher correlation with human judgments. The generated Chinese homophones, along with their manual translations, are utilized to generate perturbations and to probe the robustness of existing quality evaluation models, including models trained using multi-task learning, fine-tuned variants of multilingual language models, as well as large language models (LLMs). Our results indicate that LLMs with larger size exhibit higher stability and robustness to such perturbations. We release our data and code for reproducibility and further research.

This paper reports on the results from a pilot study investigating the impact of automatic speech recognition (ASR) technology on interpreting quality in remote healthcare interpreting settings. Employing a within-subjects experiment design with four randomised conditions, this study utilises scripted medical consultations to simulate dialogue interpreting tasks. It involves four trainee interpreters with a language combination of Chinese and English. It also gathers participants' experience and perceptions of ASR support through cued retrospective reports and semi-structured interviews. Preliminary data suggest that the availability of ASR, specifically the access to full ASR transcripts and to ChatGPT-generated summaries based on ASR, effectively improved interpreting quality. Varying types of ASR output had different impacts on the distribution of interpreting error types. Participants reported similar interactive experiences with the technology, expressing their preference for full ASR transcripts. This pilot study shows encouraging results of applying ASR to dialogue-based healthcare interpreting and offers insights into the optimal ways to present ASR output to enhance interpreter experience and performance. However, it should be emphasised that the main purpose of this study was to validate the methodology and that further research with a larger sample size is necessary to confirm these findings.

This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0-100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.

Daria Sokova, Anastasiia Bezobrazova, Constantin Orasan. Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 2025). 2025.