David Mareček
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
David Mareček has a total of 39 co-authors. Here are the top co-authors by number of shared publications:
- Rudolf Rosa
Faculty of Mathematics and Physics Institute of Formal and Applied Linguistics Charles University
8 shared publications - Martin Popel
Faculty of Mathematics and Physics Charles University Institute of Formal and Applied Linguistics Prague Czechia
4 shared publications Tomasz Limisiewicz
School of Computer Science Faculty of Mathematics and Physics The Hebrew University of Jerusalem Charles University in Prague
4 shared publications- Zdeněk Žabokrtský
Charles University
4 shared publications - Tomasz Limisiewicz
Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics Charles University Prague Czech Republic
3 shared publications - Tomáš Musil
Institute of Formal and Applied Linguistics Faculty of Mathematics and Physics Charles University Prague Czech Republic
3 shared publications
Latest Publications
- Open Access
Transforming Hidden States into Binary Semantic Features
Large language models follow a lineage of many NLP applications that were directly inspired by distributional semantics, but do not seem to be closely related to it anymore. In this paper, we propose to employ the distributional theory of meaning once again. Using Independent Component Analysis to overcome some of its challenging aspects, we show that large language models represent semantic features in their hidden states.
Open AccessTransforming Hidden States into Binary Semantic Features
Large language models follow a lineage of many NLP applications that were directly inspired by distributional semantics, but do not seem to be closely related to it anymore. In this paper, we propose to employ the distributional theory of meaning once again. Using Independent Component Analysis to overcome some of its challenging aspects, we show that large language models represent semantic features in their hidden states.
- Open Access
Exploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation
We study the effect of tokenization on gender bias in machine translation, an aspect that has been largely overlooked in previous works. Specifically, we focus on the interactions between the frequency of gendered profession names in training data, their representation in the subword tokenizer's vocabulary, and gender bias. We observe that female and non-stereotypical gender inflections of profession names (e.g., Spanish "doctora" for "female doctor") tend to be split into multiple subword tokens. Our results indicate that the imbalance of gender forms in the model's training corpus is a major factor contributing to gender bias and has a greater impact than subword splitting. We show that analyzing subword splits provides good estimates of gender-form imbalance in the training data and can be used even when the corpus is not publicly available. We also demonstrate that fine-tuning just the token embedding layer can decrease the gap in gender prediction accuracy between female and male forms without impairing the translation quality.
Open AccessExploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation
We study the effect of tokenization on gender bias in machine translation, an aspect that has been largely overlooked in previous works. Specifically, we focus on the interactions between the frequency of gendered profession names in training data, their representation in the subword tokenizer's vocabulary, and gender bias. We observe that female and non-stereotypical gender inflections of profession names (e.g., Spanish "doctora" for "female doctor") tend to be split into multiple subword tokens. Our results indicate that the imbalance of gender forms in the model's training corpus is a major factor contributing to gender bias and has a greater impact than subword splitting. We show that analyzing subword splits provides good estimates of gender-form imbalance in the training data and can be used even when the corpus is not publicly available. We also demonstrate that fine-tuning just the token embedding layer can decrease the gap in gender prediction accuracy between female and male forms without impairing the translation quality.
- Open Access
Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training
Open AccessTokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the word-level tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 885-896
Exploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation
Bar Iluz, Tomasz Limisiewicz, Gabriel Stanovsky, David Mareček. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 885-896
Open AccessExploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation
Bar Iluz, Tomasz Limisiewicz, Gabriel Stanovsky, David Mareček. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 835-848
The Functional Relevance of Probed Information: A Case Study
Michael Hanna, Roberto Zamparelli, David Mareček. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 835-848
Open AccessThe Functional Relevance of Probed Information: A Case Study
Michael Hanna, Roberto Zamparelli, David Mareček. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.
- Open Access
Introducing Orthogonal Constraint in Structural Probes
With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of word embeddings is performed in order to approximate the topology of dependency structures. In this work, we introduce a new type of structural probing, where the linear projection is decomposed into 1. isomorphic space rotation; 2. linear scaling that identifies and scales the most relevant dimensions. In addition to syntactic dependency, we evaluate our method on novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memorization.
Open AccessIntroducing Orthogonal Constraint in Structural Probes
With the recent success of pre-trained models in NLP, a significant focus was put on interpreting their representations. One of the most prominent approaches is structural probing (Hewitt and Manning, 2019), where a linear projection of word embeddings is performed in order to approximate the topology of dependency structures. In this work, we introduce a new type of structural probing, where the linear projection is decomposed into 1. isomorphic space rotation; 2. linear scaling that identifies and scales the most relevant dimensions. In addition to syntactic dependency, we evaluate our method on novel tasks (lexical hypernymy and position in a sentence). We jointly train the probes for multiple tasks and experimentally show that lexical and syntactic information is separated in the representations. Moreover, the orthogonal constraint makes the Structural Probes less vulnerable to memorization.
- Open Access
- 1 January 2021
- Published by Association for Computational Linguistics (ACL)
- p. 4589-4598
Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes
Tomasz Limisiewicz, David Mareček. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
- 1 January 2021
- Published by Association for Computational Linguistics (ACL)
- p. 4589-4598
Open AccessExamining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes
Tomasz Limisiewicz, David Mareček. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.
- Open Access
Universal Dependencies according to BERT: both more specific and more general
This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.
Open AccessUniversal Dependencies according to BERT: both more specific and more general
This work focuses on analyzing the form and extent of syntactic abstraction captured by BERT by extracting labeled dependency trees from self-attentions. Previous work showed that individual BERT heads tend to encode particular dependency relation types. We extend these findings by explicitly comparing BERT relations to Universal Dependencies (UD) annotations, showing that they often do not match one-to-one. We suggest a method for relation identification and syntactic tree construction. Our approach produces significantly more consistent dependency trees than previous work, showing that it better explains the syntactic abstractions in BERT. At the same time, it can be successfully applied with only a minimal amount of supervision and generalizes well across languages.
- Open Access
Syntax Representation in Word Embeddings and Neural Networks -- A Survey
Neural networks trained on natural language processing tasks capture syntax even though it is not provided as a supervision signal. This indicates that syntactic analysis is essential to the understating of language in artificial intelligence systems. This overview paper covers approaches of evaluating the amount of syntactic information included in the representations of words for different neural network architectures. We mainly summarize re-search on English monolingual data on language modeling tasks and multilingual data for neural machine translation systems and multilingual language models. We describe which pre-trained models and representations of language are best suited for transfer to syntactic tasks.
Open AccessSyntax Representation in Word Embeddings and Neural Networks -- A Survey
Neural networks trained on natural language processing tasks capture syntax even though it is not provided as a supervision signal. This indicates that syntactic analysis is essential to the understating of language in artificial intelligence systems. This overview paper covers approaches of evaluating the amount of syntactic information included in the representations of words for different neural network architectures. We mainly summarize re-search on English monolingual data on language modeling tasks and multilingual data for neural machine translation systems and multilingual language models. We describe which pre-trained models and representations of language are best suited for transfer to syntactic tasks.
- 1 September 2020
- Published by Springer Nature
- p. 180-188
Measuring Memorization Effect in Word-Level Neural Networks Probing
- 1 September 2020
- Published by Springer Nature
- p. 180-188
Measuring Memorization Effect in Word-Level Neural Networks Probing