Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Sami Ul Haq has a total of 7 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Visual Semantics in MT Evaluation: Do Image Descriptions Help with Assessment of Multimodal MT Quality?

    by Sami Ul Haq, Sheila Castilho, Yvette Graham

    Multimodal machine translation (MMT) aims to integrate visual context with textual data to improve the translation of ambiguous source text, such as the inclusion of an image as additional context. However, the evaluation of systems largely still relies on automatic metrics designed to evaluate text alone, and do not account for additional modalities during evaluation. The lack of dedicated MMT evaluation methods often results in inconsistent findings and creates uncertainty regarding the actual contribution of visual context in translation. In this work, we examine the performance of state-of-the-art trained and untrained evaluation metrics, particularly when comparing multimodal and text-only systems. Our evaluation focuses on the degree to which existing metrics are sensitive enough to distinguish between multimodal and text-only machine translation systems. We further investigate the potential for automatically generated image descriptions to serve as effective contextual signals for improving metric sensitivity to multimodal tasks. Our results show that incorporating such visual information into supervised metrics yields better alignment with human judgment. While all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, both n-gram-based and embedding-based metrics struggled with respect to contrastive evaluation designed to capture context-dependent errors. Furthermore, we discuss how the presence of visual context may influence human evaluator judgment, observing that given the opportunity, human ratings are often substantially revised, further emphasizing the critical role of context in the evaluation of MMT.

  • Open Access

    Visual Semantics in MT Evaluation: Do Image Descriptions Help with Assessment of Multimodal MT Quality?

    by Sami Ul Haq, Sheila Castilho, Yvette Graham

    Multimodal machine translation aims to integrate visual context, such as images, with textual data to improve the translation of ambiguous source text. However, the evaluation of these systems still largely relies on traditional automatic metrics, which do not account for additional modalities during evaluation. The lack of dedicated evaluation methods often results in inconsistent findings and creates uncertainty regarding the actual contribution of visual context in translation. In this work, we examine the performance of state-of-the-art trained and untrained automatic evaluation metrics, particularly when comparing multimodal and text-only systems. Our evaluation focuses on whether existing metrics are sensitive enough to distinguish between multimodal and text-only machine translation systems. We further investigate whether automatically generated image descriptions can serve as effective contextual signals for improving metric sensitivity to multimodal tasks. Our results show that incorporating such visual information into supervised metrics yields better alignment with human judgments. While all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, both n-gram–based and embedding-based metrics struggled on a contrastive evaluation set designed to capture context-dependent errors. Furthermore, we discuss how the presence of visual context influences human evaluators' judgments, as ratings were often substantially revised, further emphasising the critical role of context in the evaluation of multimodal machine translation systems.

  • Open Access

    Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality

    by Sami Ul Haq, Sheila Castilho, Yvette Graham

    Machine Translation (MT) has achieved remarkable performance, with growing interest in speech translation and multimodal approaches. However, despite these advancements, MT quality assessment remains largely text centric, typically relying on human experts who read and compare texts. Since many real-world MT applications (e.g Google Translate Voice Mode, iFLYTEK Translator) involve translation being spoken rather printed or read, a more natural way to assess translation quality would be through speech as opposed text-only evaluations. This study compares text-only and audio-based evaluations of 10 MT systems from the WMT General MT Shared Task, using crowd-sourced judgments collected via Amazon Mechanical Turk. We additionally, performed statistical significance testing and self-replication experiments to test reliability and consistency of audio-based approach. Crowd-sourced assessments based on audio yield rankings largely consistent with text only evaluations but, in some cases, identify significant differences between translation systems. We attribute this to speech richer, more natural modality and propose incorporating speech-based assessments into future MT evaluation frameworks.

  • Open Access

    Audio-Based Crowd-Sourced Evaluation of Machine Translation Quality

    by Sami Haq, Sheila Castilho, Yvette Graham

    Sami Haq, Sheila Castilho, Yvette Graham. Proceedings of the Tenth Conference on Machine Translation. 2025.

  • Context-Aware Neural Machine Translation using Selected Context

    by Sami Ul Haq, Sadaf Abdul Rauf, Arslan Shaukat, Muhammad Hassan Arif

    Context-aware neural machine translation has attracted much attention recently by promising sophisticated contextual information integration into conventional neural machine translation. However, context-aware NMT is challenged with effective context aggregation and increased training time due integration of extra information. In this work, we study the effect of encoding selective contextual information using pre-trained models for effective contextual integration and performance optimization. We conduct experiments on different context selection methods and quantify that encoding selected context significantly reduces the training time while maintaining superiority over sentence-level NMT models. Specifically, we experimented on IWSLT English↔German translation task and show encoding selected keywords as context is sufficient and obtains best translation results.