SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
Authors:
- Iñigo Lopez-GazpioILIñigo Lopez-Gazpio
Preprint
Abstract
Semantic Textual Similarity (STS) measures the meaning similarity of sentences. Applications include machine translation (MT), summarization, generation, question answering (QA), short answer grading, semantic search, dialog and conversational systems. The STS shared task is a venue for assessing the current state-of-the-art. The 2017 task focuses on multilingual and cross-lingual pairs with one sub-track exploring MT quality estimation (MTQE) data. The task obtained strong participation from 31 teams, with 17 participating in all language tracks. We summarize performance and review a selection of well performing methods. Analysis highlights common errors, providing insight into the limitations of existing models. To support ongoing work on semantic representations, the STS Benchmark is introduced as a new shared training and evaluation set carefully selected from the corpus of English STS shared task data (2012-2017).Keywords
All Related Versions
- Version 1, 2017-07-31, ArXiv
- Published version: .
This publication has 18 references indexed in Scilit:
- Bag of Tricks for Efficient Text Classification2017
- Representing Sentences as Low-Rank Subspaces2017
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data2017
- SICK through the SemEval glasses. Lesson learned from the evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailmentLanguage Resources and Evaluation, 2016
- UMD-TTIC-UW at SemEval-2016 Task 1: Attention-Based Multi-Perspective Convolutional Neural Networks for Textual Similarity Measurement2016
- DBnary: Wiktionary as a Lemon-based multilingual lexical resource in RDFSemantic Web: – Interoperability, Usability, Applicability, 2015
- cocor: A Comprehensive Solution for the Statistical Comparison of CorrelationsPLOS ONE, 2015
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014
- Recognizing textual entailment: Rational, evaluation and approaches – ErratumNatural Language Engineering, 2010
- Efficient Convolution Kernels for Dependency and Constituent Syntactic Trees2006