Yvette Graham
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Yvette Graham has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Cathal Gurrin
Dublin City University, Dublin, Ireland
3 shared publications - Chenyang Lyu
College of Computer Science and Technology, Zhejiang University, Hangzhou, Zhejiang, China
3 shared publications - Jennifer Foster
Dublin City University
3 shared publications - Tianbo Ji
School of Transportation and Civil Engineering, Nantong University, Nantong 226019, China
3 shared publications - Naushad Alam
SFI Insight Centre for Data Analytics, Dublin City University, Dublin, Ireland
2 shared publications - Chenyang Lyu
Alibaba International Digital Commerce, Hangzhou, China
1 shared publication
Latest Publications
- Open Access
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 3776-3780
Towards Emotional Intelligence in Conversational AI: How Well Can LLMs Recognise Emotion in Conversations?
- 19 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 3776-3780
Open AccessTowards Emotional Intelligence in Conversational AI: How Well Can LLMs Recognise Emotion in Conversations?
- Open Access
- 15 July 2026
- Published by MDPI AG in Information
- Vol. 17 (7) , 689
Visual Semantics in MT Evaluation: Do Image Descriptions Help with Assessment of Multimodal MT Quality?
Multimodal machine translation (MMT) aims to integrate visual context with textual data to improve the translation of ambiguous source text, such as the inclusion of an image as additional context. However, the evaluation of systems largely still relies on automatic metrics designed to evaluate text alone, and do not account for additional modalities during evaluation. The lack of dedicated MMT evaluation methods often results in inconsistent findings and creates uncertainty regarding the actual contribution of visual context in translation. In this work, we examine the performance of state-of-the-art trained and untrained evaluation metrics, particularly when comparing multimodal and text-only systems. Our evaluation focuses on the degree to which existing metrics are sensitive enough to distinguish between multimodal and text-only machine translation systems. We further investigate the potential for automatically generated image descriptions to serve as effective contextual signals for improving metric sensitivity to multimodal tasks. Our results show that incorporating such visual information into supervised metrics yields better alignment with human judgment. While all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, both n-gram-based and embedding-based metrics struggled with respect to contrastive evaluation designed to capture context-dependent errors. Furthermore, we discuss how the presence of visual context may influence human evaluator judgment, observing that given the opportunity, human ratings are often substantially revised, further emphasizing the critical role of context in the evaluation of MMT.
- 15 July 2026
- Published by MDPI AG in Information
- Vol. 17 (7) , 689
Open AccessVisual Semantics in MT Evaluation: Do Image Descriptions Help with Assessment of Multimodal MT Quality?
Multimodal machine translation (MMT) aims to integrate visual context with textual data to improve the translation of ambiguous source text, such as the inclusion of an image as additional context. However, the evaluation of systems largely still relies on automatic metrics designed to evaluate text alone, and do not account for additional modalities during evaluation. The lack of dedicated MMT evaluation methods often results in inconsistent findings and creates uncertainty regarding the actual contribution of visual context in translation. In this work, we examine the performance of state-of-the-art trained and untrained evaluation metrics, particularly when comparing multimodal and text-only systems. Our evaluation focuses on the degree to which existing metrics are sensitive enough to distinguish between multimodal and text-only machine translation systems. We further investigate the potential for automatically generated image descriptions to serve as effective contextual signals for improving metric sensitivity to multimodal tasks. Our results show that incorporating such visual information into supervised metrics yields better alignment with human judgment. While all metrics successfully distinguished image-aware from image-agnostic systems on general test sets, both n-gram-based and embedding-based metrics struggled with respect to contrastive evaluation designed to capture context-dependent errors. Furthermore, we discuss how the presence of visual context may influence human evaluator judgment, observing that given the opportunity, human ratings are often substantially revised, further emphasizing the critical role of context in the evaluation of MMT.
- Open Access
- 10 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 82-87
Memento 4.0: A Prototype Conversational Search System for LSC'24
The practice of lifelogging, capturing one's daily experiences through wearable devices, has evolved significantly over the last decade, presenting both challenges and opportunities in information retrieval. This paper presents an early prototype of a conversational lifelog retrieval system designed to address the open challenges in this domain. Our system integrates a hierarchical event segmentation approach to automatically organize lifelog data into meaningful events, facilitating event-based retrieval over traditional image retrieval. Moreover, we incorporate a question-answering pipeline, leveraging large language models such as GPT-3.5 Turbo and Mistral7B, to enable free-form natural language interaction with the lifelog dataset. Moreover, we enhance our system's user interface by building on previous versions to streamline event-based retrieval and question-answering functionalities.
- 10 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 82-87
Open AccessMemento 4.0: A Prototype Conversational Search System for LSC'24
The practice of lifelogging, capturing one's daily experiences through wearable devices, has evolved significantly over the last decade, presenting both challenges and opportunities in information retrieval. This paper presents an early prototype of a conversational lifelog retrieval system designed to address the open challenges in this domain. Our system integrates a hierarchical event segmentation approach to automatically organize lifelog data into meaningful events, facilitating event-based retrieval over traditional image retrieval. Moreover, we incorporate a question-answering pipeline, leveraging large language models such as GPT-3.5 Turbo and Mistral7B, to enable free-form natural language interaction with the lifelog dataset. Moreover, we enhance our system's user interface by building on previous versions to streamline event-based retrieval and question-answering functionalities.
- Open Access
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 3975-3984
Graph-Based Video-Language Learning with Multi-Grained Audio-Visual Alignment
Video-language learning has attracted significant attention in the fields of multimedia, computer vision and natural language processing in recent years. One of the key challenges in this area is how to effectively integrate visual and linguistic information to enable machines to understand video content and query information. In this work, we leverage graph-based representations and multi-grained audio-visual alignment to address this challenge. First, our approach starts by transforming video and query inputs into visual-scene graphs and semantic role graphs using a visual-scene parser and semantic role labeler respectively. These graphs are then encoded using graph neural networks to obtain enriched representations and combined to obtain a video-query joint representation that enhances the semantic expressivity of the inputs. Second, to achieve accurate matching of relevant parts of audio and visual features, we propose a multi-grained alignment module that aligns the audio and visual features at multiple scales. This enables us to effectively fuse the audio and visual information in a way that is consistent with the semantic-level information captured by the graph-based representations. Experiments on five representative datasets collected for Video Retrieval and Video Question Answering tasks show that our approach outperforms the literature on several metrics. Our extensive ablation studies demonstrate the effectiveness of graph-based representation and multi-grained audio-visual alignment.
- 26 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 3975-3984
Open AccessGraph-Based Video-Language Learning with Multi-Grained Audio-Visual Alignment
Video-language learning has attracted significant attention in the fields of multimedia, computer vision and natural language processing in recent years. One of the key challenges in this area is how to effectively integrate visual and linguistic information to enable machines to understand video content and query information. In this work, we leverage graph-based representations and multi-grained audio-visual alignment to address this challenge. First, our approach starts by transforming video and query inputs into visual-scene graphs and semantic role graphs using a visual-scene parser and semantic role labeler respectively. These graphs are then encoded using graph neural networks to obtain enriched representations and combined to obtain a video-query joint representation that enhances the semantic expressivity of the inputs. Second, to achieve accurate matching of relevant parts of audio and visual features, we propose a multi-grained alignment module that aligns the audio and visual features at multiple scales. This enables us to effectively fuse the audio and visual information in a way that is consistent with the semantic-level information captured by the graph-based representations. Experiments on five representative datasets collected for Video Retrieval and Video Question Answering tasks show that our approach outperforms the literature on several metrics. Our extensive ablation studies demonstrate the effectiveness of graph-based representation and multi-grained audio-visual alignment.
- Open Access
- 12 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 41-46
Memento 3.0: An Enhanced Lifelog Search Engine for LSC’23
In this work, we present our system Memento 3.0 for participation in the Lifelog Search Challenge 2023, which is a successor to the previous 2 iterations of our system called Memento 1.0 [1] and Memento 2.0 [2]. Memento 3.0 employs image-text embeddings derived from OpenAI CLIP models as well as larger OpenCLIP models trained on ∼ 5x more data. Our system also significantly reduces the query processing time by almost 75% when compared to its predecessor systems by employing a cluster-based search technique. We additionally make important updates to the system’s user interface to offer more flexibility to the user and at the same time be better suited to efficiently handle new query types introduced in the Lifelog Search Challenge.
- 12 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 41-46
Open AccessMemento 3.0: An Enhanced Lifelog Search Engine for LSC’23
In this work, we present our system Memento 3.0 for participation in the Lifelog Search Challenge 2023, which is a successor to the previous 2 iterations of our system called Memento 1.0 [1] and Memento 2.0 [2]. Memento 3.0 employs image-text embeddings derived from OpenAI CLIP models as well as larger OpenCLIP models trained on ∼ 5x more data. Our system also significantly reduces the query processing time by almost 75% when compared to its predecessor systems by employing a cluster-based search technique. We additionally make important updates to the system’s user interface to offer more flexibility to the user and at the same time be better suited to efficiently handle new query types introduced in the Lifelog Search Challenge.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 183-189
Is a Video worth n n Images? A Highly Efficient Approach to Transformer-based Video Question Answering
Chenyang Lyu, Tianbo Ji, Yvette Graham, Jennifer Foster. Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 183-189
Open AccessIs a Video worth n n Images? A Highly Efficient Approach to Transformer-based Video Question Answering
Chenyang Lyu, Tianbo Ji, Yvette Graham, Jennifer Foster. Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). 2023.
- Open Access
QAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers found within the passage. In recent years, the introduction of neural generation models has resulted in substantial improvements of automatically generated questions in terms of quality, especially compared to traditional approaches that employ manually crafted heuristics. However, the metrics commonly applied in QG evaluations have been criticized for their low agreement with human judgement. We therefore propose a new reference-free evaluation metric that has the potential to provide a better mechanism for evaluating QG systems, called QAScore. Instead of fine-tuning a language model to maximize its correlation with human judgements, QAScore evaluates a question by computing the cross entropy according to the probability that the language model can correctly generate the masked words in the answer to that question. Furthermore, we conduct a new crowd-sourcing human evaluation experiment for the QG evaluation to investigate how QAScore and other metrics can correlate with human judgements. Experiments show that QAScore obtains a stronger correlation with the results of our proposed human evaluation method compared to existing traditional word-overlap-based metrics such as BLEU and ROUGE, as well as the existing pretrained-model-based metric BERTScore.
Open AccessQAScore -- An Unsupervised Unreferenced Metric for the Question Generation Evaluation
Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers found within the passage. In recent years, the introduction of neural generation models has resulted in substantial improvements of automatically generated questions in terms of quality, especially compared to traditional approaches that employ manually crafted heuristics. However, the metrics commonly applied in QG evaluations have been criticized for their low agreement with human judgement. We therefore propose a new reference-free evaluation metric that has the potential to provide a better mechanism for evaluating QG systems, called QAScore. Instead of fine-tuning a language model to maximize its correlation with human judgements, QAScore evaluates a question by computing the cross entropy according to the probability that the language model can correctly generate the masked words in the answer to that question. Furthermore, we conduct a new crowd-sourcing human evaluation experiment for the QG evaluation to investigate how QAScore and other metrics can correlate with human judgements. Experiments show that QAScore obtains a stronger correlation with the results of our proposed human evaluation method compared to existing traditional word-overlap-based metrics such as BLEU and ROUGE, as well as the existing pretrained-model-based metric BERTScore.
- Open Access
Extending the Scope of Out-of-Domain: Examining QA models in multiple subdomains
Past works that investigate out-of-domain performance of QA systems have mainly focused on general domains (e.g. news domain, wikipedia domain), underestimating the importance of subdomains defined by the internal characteristics of QA datasets. In this paper, we extend the scope of "out-of-domain" by splitting QA examples into different subdomains according to their several internal characteristics including question type, text length, answer position. We then examine the performance of QA systems trained on the data from different subdomains. Experimental results show that the performance of QA systems can be significantly reduced when the train data and test data come from different subdomains. These results question the generalizability of current QA systems in multiple subdomains, suggesting the need to combat the bias introduced by the internal characteristics of QA datasets.
Open AccessExtending the Scope of Out-of-Domain: Examining QA models in multiple subdomains
Past works that investigate out-of-domain performance of QA systems have mainly focused on general domains (e.g. news domain, wikipedia domain), underestimating the importance of subdomains defined by the internal characteristics of QA datasets. In this paper, we extend the scope of "out-of-domain" by splitting QA examples into different subdomains according to their several internal characteristics including question type, text length, answer position. We then examine the performance of QA systems trained on the data from different subdomains. Experimental results show that the performance of QA systems can be significantly reduced when the train data and test data come from different subdomains. These results question the generalizability of current QA systems in multiple subdomains, suggesting the need to combat the bias introduced by the internal characteristics of QA datasets.