PMIScore: An Unsupervised Approach to Quantify Dialogue Engagement
Open Access
- 12 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 99-110
Abstract
High dialogue engagement is a crucial indicator of an effective conversation. A reliable measure of engagement could help benchmark large language models, enhance the effectiveness of human-computer interactions, or improve personal communication skills. However, quantifying engagement is challenging, since it is subjective and lacks a ''gold standard''. This paper proposes PMIScore, an efficient unsupervised approach to quantify dialogue engagement. It uses pointwise mutual information (PMI), which is the probability of generating a response conditioning on the conversation history. Thus, PMIScore offers a clear interpretation of engagement. As directly computing PMI is intractable due to the complexity of dialogues, PMIScore learned it through a dual form of divergence. The algorithm includes generating positive and negative dialogue pairs, extracting embeddings by large language models (LLMs), and training a small neural network using a mutual information loss function. We validated PMIScore on both synthetic and real-world datasets. Our results demonstrate the effectiveness of PMIScore in PMI estimation and the reasonableness of the PMI metric itself.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Joint Dual Learning With Mutual Information Maximization for Natural Language Understanding and Generation in DialoguesIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
- C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation2023
- Representation Learning for Conversational Data using Discourse Mutual Information Maximization2022
- A Large-Scale Chinese Short-Text Conversation Dataset2020
- KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation2020
- An Information Theoretic Framework For Designing Information Elicitation Mechanisms That Reward Truth-tellingACM Transactions on Economics and Computation, 2019
- Water from Two Rocks2018
- Estimating Divergence Functionals and the Likelihood Ratio by Convex Risk MinimizationIEEE Transactions on Information Theory, 2010
- On surrogate loss functions and f-divergencesThe Annals of Statistics, 2009
- A General Class of Coefficients of Divergence of One Distribution from AnotherJournal of the Royal Statistical Society Series B: Statistical Methodology, 1966