Matt Post
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Matt Post has a total of 21 co-authors. Here are the top co-authors by number of shared publications:
- Vikas Raunak
Microsoft Azure AI
4 shared publications - Rachel Wicks
Human Language Technology Center of Excellence Johns Hopkins University
3 shared publications - Tom Kocmi
Microsoft Redmond Washington USA
3 shared publications Elijah Rippeth
University of Maryland
2 shared publications- Marcin Junczys-Dowmunt
Microsoft Azure AI
2 shared publications Roman Grundkiewicz
Sapienza University of Rome
2 shared publications
Latest Publications
- Open Access
PEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation
We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised Quality Estimation (QE) metric family that reframes reference-free Machine Translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for Minimum Bayes Risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.
Open AccessPEAR: Pairwise Evaluation for Automatic Relative Scoring in Machine Translation
We present PEAR (Pairwise Evaluation for Automatic Relative Scoring), a supervised Quality Estimation (QE) metric family that reframes reference-free Machine Translation (MT) evaluation as a graded pairwise comparison. Given a source segment and two candidate translations, PEAR predicts the direction and magnitude of their quality difference. The metrics are trained using pairwise supervision derived from differences in human judgments, with an additional regularization term that encourages sign inversion under candidate order reversal. On the WMT24 meta-evaluation benchmark, PEAR outperforms strictly matched single-candidate QE baselines trained with the same data and backbones, isolating the benefit of the proposed pairwise formulation. Despite using substantially fewer parameters than recent large metrics, PEAR surpasses far larger QE models and reference-based metrics. Our analysis further indicates that PEAR yields a less redundant evaluation signal relative to other top metrics. Finally, we show that PEAR is an effective utility function for Minimum Bayes Risk (MBR) decoding, reducing pairwise scoring cost at negligible impact.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 84-92
Effects of automatic alignment on speech translation metrics
Matt Post, Hieu Hoang. Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025). 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 84-92
Open AccessEffects of automatic alignment on speech translation metrics
Matt Post, Hieu Hoang. Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025). 2025.
- Open Access
SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
Reference-based metrics that operate at the sentence-level typically outperform quality estimation metrics, which have access only to the source and system output. This is unsurprising, since references resolve ambiguities that may be present in the source. In this paper, we investigate whether additional source context can effectively substitute for a reference. We present a metric named SLIDE (SLIding Document Evaluator), which operates on blocks of sentences. SLIDE leverages a moving window that slides over each document in the test set, feeding each chunk of sentences into an unmodified, off-the-shelf quality estimation model. We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline, in some cases even eliminating the gap with reference-base metrics. This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities. This finding is especially pertinent for reference-free document-level evaluation, wherein SLIDE could provide higher-quality pairwise system assessments while only requiring document boundary annotations.
Open AccessSLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
Reference-based metrics that operate at the sentence-level typically outperform quality estimation metrics, which have access only to the source and system output. This is unsurprising, since references resolve ambiguities that may be present in the source. In this paper, we investigate whether additional source context can effectively substitute for a reference. We present a metric named SLIDE (SLIding Document Evaluator), which operates on blocks of sentences. SLIDE leverages a moving window that slides over each document in the test set, feeding each chunk of sentences into an unmodified, off-the-shelf quality estimation model. We find that SLIDE obtains significantly higher pairwise system accuracy than its sentence-level baseline, in some cases even eliminating the gap with reference-base metrics. This suggests that source context may provide the same information as a human reference in disambiguating source ambiguities. This finding is especially pertinent for reference-free document-level evaluation, wherein SLIDE could provide higher-quality pairwise system assessments while only requiring document boundary annotations.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 328-335
PyMarian: Fast Neural Machine Translation and Evaluation in Python
Thamme Gowda, Roman Grundkiewicz, Elijah Rippeth, Matt Post, Marcin Junczys-Dowmunt. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 328-335
Open AccessPyMarian: Fast Neural Machine Translation and Evaluation in Python
Thamme Gowda, Roman Grundkiewicz, Elijah Rippeth, Matt Post, Marcin Junczys-Dowmunt. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1125-1139
Evaluation and Large-scale Training for Contextual Machine Translation
Matt Post, Marcin Junczys-Dowmunt. Proceedings of the Ninth Conference on Machine Translation. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1125-1139
Open AccessEvaluation and Large-scale Training for Contextual Machine Translation
Matt Post, Marcin Junczys-Dowmunt. Proceedings of the Ninth Conference on Machine Translation. 2024.
- Top Cited PapersOpen Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1999-2014
Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
Tom Kocmi, Vilém Zouhar, Christian Federmann, Matt Post. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1999-2014
Top Cited PapersOpen AccessNavigating the Metrics Maze: Reconciling Score Magnitudes and Accuracies
Tom Kocmi, Vilém Zouhar, Christian Federmann, Matt Post. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 9876-9890
Recovering document annotations for sentence-level bitext
Rachel Wicks, Matt Post, Philipp Koehn. Findings of the Association for Computational Linguistics ACL 2024. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 9876-9890
Open AccessRecovering document annotations for sentence-level bitext
Rachel Wicks, Matt Post, Philipp Koehn. Findings of the Association for Computational Linguistics ACL 2024. 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 205-211
SLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
Vikas Raunak, Tom Kocmi, Matt Post. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 205-211
Open AccessSLIDE: Reference-free Evaluation for Machine Translation using a Sliding Document Window
Vikas Raunak, Tom Kocmi, Matt Post. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers). 2024.
- Open Access
Identifying Context-Dependent Translations for Evaluation Set Production
A major impediment to the transition to context-aware machine translation is the absence of good evaluation metrics and test sets. Sentences that require context to be translated correctly are rare in test sets, reducing the utility of standard corpus-level metrics such as COMET or BLEU. On the other hand, datasets that annotate such sentences are also rare, small in scale, and available for only a few languages. To address this, we modernize, generalize, and extend previous annotation pipelines to produce CTXPRO, a tool that identifies subsets of parallel documents containing sentences that require context to correctly translate five phenomena: gender, formality, and animacy for pronouns, verb phrase ellipsis, and ambiguous noun inflections. The input to the pipeline is a set of hand-crafted, per-language, linguistically-informed rules that select contextual sentence pairs using coreference, part-of-speech, and morphological features provided by state-of-the-art tools. We apply this pipeline to seven languages pairs (EN into and out-of DE, ES, FR, IT, PL, PT, and RU) and two datasets (OpenSubtitles and WMT test sets), and validate its performance using both overlap with previous work and its ability to discriminate a contextual MT system from a sentence-based one. We release the CTXPRO pipeline and data as open source.
Open AccessIdentifying Context-Dependent Translations for Evaluation Set Production
A major impediment to the transition to context-aware machine translation is the absence of good evaluation metrics and test sets. Sentences that require context to be translated correctly are rare in test sets, reducing the utility of standard corpus-level metrics such as COMET or BLEU. On the other hand, datasets that annotate such sentences are also rare, small in scale, and available for only a few languages. To address this, we modernize, generalize, and extend previous annotation pipelines to produce CTXPRO, a tool that identifies subsets of parallel documents containing sentences that require context to correctly translate five phenomena: gender, formality, and animacy for pronouns, verb phrase ellipsis, and ambiguous noun inflections. The input to the pipeline is a set of hand-crafted, per-language, linguistically-informed rules that select contextual sentence pairs using coreference, part-of-speech, and morphological features provided by state-of-the-art tools. We apply this pipeline to seven languages pairs (EN into and out-of DE, ES, FR, IT, PL, PT, and RU) and two datasets (OpenSubtitles and WMT test sets), and validate its performance using both overlap with previous work and its ability to discriminate a contextual MT system from a sentence-based one. We release the CTXPRO pipeline and data as open source.
- Top Cited PapersOpen Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 812-814
Evaluating Metrics for Document-context Evaluation in Machine Translation
Vikas Raunak, Tom Kocmi, Matt Post. Proceedings of the Eighth Conference on Machine Translation. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 812-814
Top Cited PapersOpen AccessEvaluating Metrics for Document-context Evaluation in Machine Translation
Vikas Raunak, Tom Kocmi, Matt Post. Proceedings of the Eighth Conference on Machine Translation. 2023.