Jackie Chi Kit Cheung
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Jackie Chi Kit Cheung has a total of 35 co-authors. Here are the top co-authors by number of shared publications:
- Ian Porada
Mila AI Institute McGill University AI Institute McGill University Canada
5 shared publications - Alexandra Olteanu
Microsoft Research Microsoft Research Microsoft Research Stanford University Carnegie Mellon University Harvard University Drextel University Oxford University Microsoft Research
4 shared publications - Adam Trischler
Microsoft Research Montréal Canada
2 shared publications Ines Arous
McGill University
2 shared publications- Maxime Darrin
International Laboratory on Learning Systems
2 shared publications Meng Cao
Mila -Quebec Artificial Intelligence Institute
2 shared publications
Latest Publications
- Open Access
PreSumm: Predicting Summarization Performance Without Summarizing
Despite recent advancements in automatic summarization, state-of-the-art models do not summarize all documents equally well, raising the question: why? While prior research has extensively analyzed summarization models, little attention has been given to the role of document characteristics in influencing summarization performance. In this work, we explore two key research questions. First, do documents exhibit consistent summarization quality across multiple systems? If so, can we predict a document's summarization performance without generating a summary? We answer both questions affirmatively and introduce PreSumm, a novel task in which a system predicts summarization performance based solely on the source document. Our analysis sheds light on common properties of documents with low PreSumm scores, revealing that they often suffer from coherence issues, complex content, or a lack of a clear main theme. In addition, we demonstrate PreSumm's practical utility in two key applications: improving hybrid summarization workflows by identifying documents that require manual summarization and enhancing dataset quality by filtering outliers and noisy documents. Overall, our findings highlight the critical role of document properties in summarization performance and offer insights into the limitations of current systems that could serve as the basis for future improvements.
Open AccessPreSumm: Predicting Summarization Performance Without Summarizing
Despite recent advancements in automatic summarization, state-of-the-art models do not summarize all documents equally well, raising the question: why? While prior research has extensively analyzed summarization models, little attention has been given to the role of document characteristics in influencing summarization performance. In this work, we explore two key research questions. First, do documents exhibit consistent summarization quality across multiple systems? If so, can we predict a document's summarization performance without generating a summary? We answer both questions affirmatively and introduce PreSumm, a novel task in which a system predicts summarization performance based solely on the source document. Our analysis sheds light on common properties of documents with low PreSumm scores, revealing that they often suffer from coherence issues, complex content, or a lack of a clear main theme. In addition, we demonstrate PreSumm's practical utility in two key applications: improving hybrid summarization workflows by identifying documents that require manual summarization and enhancing dataset quality by filtering outliers and noisy documents. Overall, our findings highlight the critical role of document properties in summarization performance and offer insights into the limitations of current systems that could serve as the basis for future improvements.
- Open Access
Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution
Challenge sets such as the Winograd Schema Challenge (WSC) are used to benchmark systems' ability to resolve ambiguities in natural language. If one assumes as in existing work that solving a given challenge set is at least as difficult as solving some more general task, then high performance on the challenge set should indicate high performance on the general task overall. However, we show empirically that this assumption of difficulty does not always hold. In particular, we demonstrate that despite the strong performance of prompted language models (LMs) on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier. Motivated by these findings, we propose a method for ensembling a prompted LM with a supervised, task-specific system that is overall more accurate at resolving pronominal coreference across datasets. Finally, we emphasize that datasets involving the same linguistic phenomenon draw on distinct, but overlapping, capabilities, and evaluating on any one dataset alone does not provide a complete picture of a system's overall capability.
Open AccessSolving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution
Challenge sets such as the Winograd Schema Challenge (WSC) are used to benchmark systems' ability to resolve ambiguities in natural language. If one assumes as in existing work that solving a given challenge set is at least as difficult as solving some more general task, then high performance on the challenge set should indicate high performance on the general task overall. However, we show empirically that this assumption of difficulty does not always hold. In particular, we demonstrate that despite the strong performance of prompted language models (LMs) on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier. Motivated by these findings, we propose a method for ensembling a prompted LM with a supervised, task-specific system that is overall more accurate at resolving pronominal coreference across datasets. Finally, we emphasize that datasets involving the same linguistic phenomenon draw on distinct, but overlapping, capabilities, and evaluating on any one dataset alone does not provide a complete picture of a system's overall capability.
- Open Access
A Controlled Reevaluation of Coreference Resolution Models
All state-of-the-art coreference resolution (CR) models involve finetuning a pretrained language model. Whether the superior performance of one CR model over another is due to the choice of language model or other factors, such as the task-specific architecture, is difficult or impossible to determine due to lack of a standardized experimental setup. To resolve this ambiguity, we systematically evaluate five CR models and control for certain design decisions including the pretrained language model used by each. When controlling for language model size, encoder-based CR models outperform more recent decoder-based models in terms of both accuracy and inference speed. Surprisingly, among encoder-based CR models, more recent models are not always more accurate, and the oldest CR model that we test generalizes the best to out-of-domain textual genres. We conclude that controlling for the choice of language model reduces most, but not all, of the increase in F1 score reported in the past five years.
Open AccessA Controlled Reevaluation of Coreference Resolution Models
All state-of-the-art coreference resolution (CR) models involve finetuning a pretrained language model. Whether the superior performance of one CR model over another is due to the choice of language model or other factors, such as the task-specific architecture, is difficult or impossible to determine due to lack of a standardized experimental setup. To resolve this ambiguity, we systematically evaluate five CR models and control for certain design decisions including the pretrained language model used by each. When controlling for language model size, encoder-based CR models outperform more recent decoder-based models in terms of both accuracy and inference speed. Surprisingly, among encoder-based CR models, more recent models are not always more accurate, and the oldest CR model that we test generalizes the best to out-of-domain textual genres. We conclude that controlling for the choice of language model reduces most, but not all, of the increase in F1 score reported in the past five years.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 489-506
Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution
Ian Porada, Jackie CK Cheung. Proceedings of the 28th Conference on Computational Natural Language Learning. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 489-506
Open AccessSolving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution
Ian Porada, Jackie CK Cheung. Proceedings of the 28th Conference on Computational Natural Language Learning. 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 39-43
Human-Centered Evaluation of Language Technologies
Su Lin Blodgett, Jackie Chi Kit Cheung, Vera Liao, Ziang Xiao. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 39-43
Open AccessHuman-Centered Evaluation of Language Technologies
Su Lin Blodgett, Jackie Chi Kit Cheung, Vera Liao, Ziang Xiao. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts. 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 16349-16365
ECBD: Evidence-Centered Benchmark Design for NLP
Yu Lu Liu, Su Lin Blodgett, Jackie Cheung, Q. Vera Liao, Alexandra Olteanu, Ziang Xiao. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 16349-16365
Open AccessECBD: Evidence-Centered Benchmark Design for NLP
Yu Lu Liu, Su Lin Blodgett, Jackie Cheung, Q. Vera Liao, Alexandra Olteanu, Ziang Xiao. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 12696-12717
COSMIC: Mutual Information for Task-Agnostic Summarization Evaluation
Maxime Darrin, Philippe Formont, Jackie Cheung, Pablo Piantanida. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 12696-12717
Open AccessCOSMIC: Mutual Information for Task-Agnostic Summarization Evaluation
Maxime Darrin, Philippe Formont, Jackie Cheung, Pablo Piantanida. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 12737-12752
GLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews
Maxime Darrin, Ines Arous, Pablo Piantanida, Jackie Cheung. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 12737-12752
Open AccessGLIMPSE: Pragmatically Informative Multi-Document Summarization for Scholarly Reviews
Maxime Darrin, Ines Arous, Pablo Piantanida, Jackie Cheung. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 15380-15395
Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective
Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Cheung. Findings of the Association for Computational Linguistics ACL 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 15380-15395
Open AccessChallenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective
Ian Porada, Alexandra Olteanu, Kaheer Suleman, Adam Trischler, Jackie Cheung. Findings of the Association for Computational Linguistics ACL 2024. 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 212-219
Separately Parameterizing Singleton Detection Improves End-to-end Neural Coreference Resolution
Xiyuan Zou, Yiran Li, Ian Porada, Jackie Cheung. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 212-219
Open AccessSeparately Parameterizing Singleton Detection Improves End-to-end Neural Coreference Resolution
Xiyuan Zou, Yiran Li, Ian Porada, Jackie Cheung. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 2024.