Mathias Creutz
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Mathias Creutz has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Mikko Kurimo
Department of Information and Communications Engineering Aalto University Espoo Finland
5 shared publications - Anssi Moisio
Department of Information and Communications Engineering Aalto University Finland
4 shared publications - Sami Virpioja
Department of Digital Humanities, University of Helsinki, Helsinki, Finland
4 shared publications - Jörg Tiedemann
University of Edinburgh University of Helsinki
3 shared publications - Krista Lagus
University of Helsinki
3 shared publications Raúl Vázquez
University of Helsinki, Department of Digital Humanities
3 shared publications
Latest Publications
- Open Access
LLMs' morphological analyses of complex FST-generated Finnish words
Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.
Open AccessLLMs' morphological analyses of complex FST-generated Finnish words
Rule-based language processing systems have been overshadowed by neural systems in terms of utility, but it remains unclear whether neural NLP systems, in practice, learn the grammar rules that humans use. This work aims to shed light on the issue by evaluating state-of-the-art LLMs in a task of morphological analysis of complex Finnish noun forms. We generate the forms using an FST tool, and they are unlikely to have occurred in the training sets of the LLMs, therefore requiring morphological generalisation capacity. We find that GPT-4-turbo has some difficulties in the task while GPT-3.5-turbo struggles and smaller models Llama2-70B and Poro-34B fail nearly completely.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1-10
Correcting Challenging Finnish Learner Texts With Claude, GPT-3.5 and GPT-4 Large Language Models
Mathias Creutz. Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1-10
Open AccessCorrecting Challenging Finnish Learner Texts With Claude, GPT-3.5 and GPT-4 Large Language Models
Mathias Creutz. Proceedings of the Ninth Workshop on Noisy and User-generated Text (W-NUT 2024). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1-6
Toward the Modular Training of Controlled Paraphrase Adapters
Teemu Vahtola, Mathias Creutz. Proceedings of the 1st Workshop on Modular and Open Multilingual NLP (MOOMIN 2024). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 1-6
Open AccessToward the Modular Training of Controlled Paraphrase Adapters
Teemu Vahtola, Mathias Creutz. Proceedings of the 1st Workshop on Modular and Open Multilingual NLP (MOOMIN 2024). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 242-254
LLMs’ morphological analyses of complex FST-generated Finnish words
Anssi Moisio, Mathias Creutz, Mikko Kurimo. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 242-254
Open AccessLLMs’ morphological analyses of complex FST-generated Finnish words
Anssi Moisio, Mathias Creutz, Mikko Kurimo. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. 2024.
- Open Access
On Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation
Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language tasks in order to understand the abilities and limitations of systems deployed in the wild. To this end, our GenBench Collaborative Benchmarking Task submission utilises the distribution-based compositionality assessment (DBCA) framework to split the Europarl translation corpus into a training and a test set in such a way that the test set requires compositional generalisation capacity. Specifically, the training and test sets have divergent distributions of dependency relations, testing NMT systems' capability of translating dependencies that they have not been trained on. This is a fully-automated procedure to create natural language compositionality benchmarks, making it simple and inexpensive to apply it further to other datasets and languages. The code and data for the experiments is available at https://github.com/aalto-speech/dbca.
Open AccessOn Using Distribution-Based Compositionality Assessment to Evaluate Compositional Generalisation in Machine Translation
Compositional generalisation (CG), in NLP and in machine learning more generally, has been assessed mostly using artificial datasets. It is important to develop benchmarks to assess CG also in real-world natural language tasks in order to understand the abilities and limitations of systems deployed in the wild. To this end, our GenBench Collaborative Benchmarking Task submission utilises the distribution-based compositionality assessment (DBCA) framework to split the Europarl translation corpus into a training and a test set in such a way that the test set requires compositional generalisation capacity. Specifically, the training and test sets have divergent distributions of dependency relations, testing NMT systems' capability of translating dependencies that they have not been trained on. This is a fully-automated procedure to create natural language compositionality benchmarks, making it simple and inexpensive to apply it further to other datasets and languages. The code and data for the experiments is available at https://github.com/aalto-speech/dbca.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 204-213
On using distribution-based compositionality assessment to evaluate compositional generalisation in machine translation
Anssi Moisio, Mathias Creutz, Mikko Kurimo. Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 204-213
Open AccessOn using distribution-based compositionality assessment to evaluate compositional generalisation in machine translation
Anssi Moisio, Mathias Creutz, Mikko Kurimo. Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. 2023.
- Open Access
- 2 June 2020
- Published by MIT Press in Computational Linguistics
- Vol. 46 (2) , 387-424
A Systematic Study of Inner-Attention-Based Sentence Representations in Multilingual Neural Machine Translation
Neural machine translation has considerably improved the quality of automatic translations by learning good representations of input sentences. In this paper, we explore a multilingual translation ...
- 2 June 2020
- Published by MIT Press in Computational Linguistics
- Vol. 46 (2) , 387-424
Open AccessA Systematic Study of Inner-Attention-Based Sentence Representations in Multilingual Neural Machine Translation
Neural machine translation has considerably improved the quality of automatic translations by learning good representations of input sentences. In this paper, we explore a multilingual translation ...
- Open Access
- 17 May 2019
- Published by University of Oslo Library in Digital Humanities in the Nordic and Baltic Countries Publications
- Vol. 2 (1) , 33-48
Annotation of subtitle paraphrases using a new web tool
This paper analyzes the manual annotation effort carried out to produce Opusparcus, the Open Subtitles Paraphrase Corpus for six European languages. Within the scope of the project, a new web-based annotation tool was created. We discuss the design choices behind the tool as well as the setup of the annotation task. We also evaluate the annotations obtained. Two independent annotators needed to decide to what extent two sentences approximately meant the same thing. The sentences originate from subtitles from movies and TV shows, which constitutes an interesting genre of mostly colloquial language. Inter-annotator agreement was found to be on par with a well-known previous paraphrase resource from the news domain, the Microsoft Research Paraphrase Corpus (MSRPC). Our annotation tool is open source. The tool can be used for closed projects with restricted access and controlled user authentification as well as open crowdsourced projects, in which anyone can participate and user identification takes place based on IP addresses.
- 17 May 2019
- Published by University of Oslo Library in Digital Humanities in the Nordic and Baltic Countries Publications
- Vol. 2 (1) , 33-48
Open AccessAnnotation of subtitle paraphrases using a new web tool
This paper analyzes the manual annotation effort carried out to produce Opusparcus, the Open Subtitles Paraphrase Corpus for six European languages. Within the scope of the project, a new web-based annotation tool was created. We discuss the design choices behind the tool as well as the setup of the annotation task. We also evaluate the annotations obtained. Two independent annotators needed to decide to what extent two sentences approximately meant the same thing. The sentences originate from subtitles from movies and TV shows, which constitutes an interesting genre of mostly colloquial language. Inter-annotator agreement was found to be on par with a well-known previous paraphrase resource from the news domain, the Microsoft Research Paraphrase Corpus (MSRPC). Our annotation tool is open source. The tool can be used for closed projects with restricted access and controlled user authentification as well as open crowdsourced projects, in which anyone can participate and user identification takes place based on IP addresses.
- Open Access
- 1 January 2019
- Published by Association for Computational Linguistics (ACL)
- p. 27-32
An Evaluation of Language-Agnostic Inner-Attention-Based Representations in Machine Translation
- 1 January 2019
- Published by Association for Computational Linguistics (ACL)
- p. 27-32
Open AccessAn Evaluation of Language-Agnostic Inner-Attention-Based Representations in Machine Translation
- Open Access
- 1 January 2019
- Published by Association for Computational Linguistics (ACL)
Multilingual NMT with a Language-Independent Attention Bridge
- 1 January 2019
- Published by Association for Computational Linguistics (ACL)
Open AccessMultilingual NMT with a Language-Independent Attention Bridge