Ekaterina Shutova
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Ekaterina Shutova has a total of 63 co-authors. Here are the top co-authors by number of shared publications:
- Helen Yannakoudakis
King's College London
12 shared publications - Pushkar Mishra
Dept. of Informatics ILLC ILLC University of Amsterdam King's College London University of Amsterdam
9 shared publications - Rochelle Choenni
University of Amsterdam University of Edinburgh
8 shared publications - Dan Garrette
Google DeepMind University of Amsterdam University of Amsterdam
4 shared publications - Daniel Hershcovich
University of Copenhagen
3 shared publications Helen Yannakoudakis
The ALTA Institute
3 shared publications
Latest Publications
- Open Access
NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning
Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation. The former, such as LoRA, introduce additional modules to adapt the model to downstream tasks, offering strong memory efficiency. However, their representational capacity is often limited, making them less suitable for fine-grained adaptation. In contrast, the latter directly fine-tunes a carefully chosen subset of the original model parameters, allowing for more precise and effective adaptation, but at the cost of significantly increased memory consumption. To reconcile this trade-off, we propose NeuroAda, a novel PEFT method that enables fine-grained model finetuning while maintaining high memory efficiency. Our approach first identifies important parameters (i.e., connections within the network) as in selective adaptation, and then introduces bypass connections for these selected parameters. During finetuning, only the bypass connections are updated, leaving the original model parameters frozen. Empirical results on 23+ tasks spanning both natural language generation and understanding demonstrate that NeuroAda achieves state-of-the-art performance with as little as $\leq \textbf{0.02}\%$ trainable parameters, while reducing CUDA memory usage by up to 60%. We release our code here: https://github.com/FightingFighting/NeuroAda.git.Open AccessNeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning
Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation. The former, such as LoRA, introduce additional modules to adapt the model to downstream tasks, offering strong memory efficiency. However, their representational capacity is often limited, making them less suitable for fine-grained adaptation. In contrast, the latter directly fine-tunes a carefully chosen subset of the original model parameters, allowing for more precise and effective adaptation, but at the cost of significantly increased memory consumption. To reconcile this trade-off, we propose NeuroAda, a novel PEFT method that enables fine-grained model finetuning while maintaining high memory efficiency. Our approach first identifies important parameters (i.e., connections within the network) as in selective adaptation, and then introduces bypass connections for these selected parameters. During finetuning, only the bypass connections are updated, leaving the original model parameters frozen. Empirical results on 23+ tasks spanning both natural language generation and understanding demonstrate that NeuroAda achieves state-of-the-art performance with as little as $\leq \textbf{0.02}\%$ trainable parameters, while reducing CUDA memory usage by up to 60%. We release our code here: https://github.com/FightingFighting/NeuroAda.git. - Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 10960-10977
NeuroAda: Activating Each Neuron’s Potential for Parameter-Efficient Fine-Tuning
Zhi Zhang, Yixian Shen, Congfeng Cao, Ekaterina Shutova. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 10960-10977
Open AccessNeuroAda: Activating Each Neuron’s Potential for Parameter-Efficient Fine-Tuning
Zhi Zhang, Yixian Shen, Congfeng Cao, Ekaterina Shutova. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- Top Cited PapersOpen Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 5034-5096
Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning
Joy Crosbie, Ekaterina Shutova. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 5034-5096
Top Cited PapersOpen AccessInduction Heads as an Essential Mechanism for Pattern Matching in In-context Learning
Joy Crosbie, Ekaterina Shutova. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 7592-7608
Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models
Srishti Yadav, Zhi Zhang, Daniel Hershcovich, Ekaterina Shutova. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 7592-7608
Open AccessBeyond Words: Exploring Cultural Value Sensitivity in Multimodal Models
Srishti Yadav, Zhi Zhang, Daniel Hershcovich, Ekaterina Shutova. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.
- Open Access
- 10 December 2024
- Published by Qeios Ltd in Qeios Preprints
Proactive Gradient Conflict Mitigation in Multi-Task Learning:A Sparse Training Perspective
Advancing towards generalist agents necessitates the concurrent processing of multiple tasks using a unified model, thereby underscoring the growing significance of simultaneous model training on multiple downstream tasks. A common issue in multi-task learning is the occurrence of gradient conflict, which leads to potential competition among different tasks during joint training. This competition often results in improvements in one task at the expense of deterioration in another. Although several optimization methods have been developed to address this issue by manipulating task gradients for better task balancing, they cannot decrease the incidence of gradient conflict. In this paper, we systematically investigate the occurrence of gradient conflict across different methods and propose a strategy to reduce such conflicts through sparse training (ST), wherein only a portion of the model’s parameters are updated during training while keeping the rest unchanged. Our extensive experiments demonstrate that ST effectively mitigates conflicting gradients and leads to superior performance. Furthermore, ST can be easily integrated with gradient manipulation techniques, thus enhancing their effectiveness.
- 10 December 2024
- Published by Qeios Ltd in Qeios Preprints
Open AccessProactive Gradient Conflict Mitigation in Multi-Task Learning:A Sparse Training Perspective
Advancing towards generalist agents necessitates the concurrent processing of multiple tasks using a unified model, thereby underscoring the growing significance of simultaneous model training on multiple downstream tasks. A common issue in multi-task learning is the occurrence of gradient conflict, which leads to potential competition among different tasks during joint training. This competition often results in improvements in one task at the expense of deterioration in another. Although several optimization methods have been developed to address this issue by manipulating task gradients for better task balancing, they cannot decrease the incidence of gradient conflict. In this paper, we systematically investigate the occurrence of gradient conflict across different methods and propose a strategy to reduce such conflicts through sparse training (ST), wherein only a portion of the model’s parameters are updated during training while keeping the rest unchanged. Our extensive experiments demonstrate that ST effectively mitigates conflicting gradients and leads to superior performance. Furthermore, ST can be easily integrated with gradient manipulation techniques, thus enhancing their effectiveness.
- Open Access
Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective
Advancing towards generalist agents necessitates the concurrent processing of multiple tasks using a unified model, thereby underscoring the growing significance of simultaneous model training on multiple downstream tasks. A common issue in multi-task learning is the occurrence of gradient conflict, which leads to potential competition among different tasks during joint training. This competition often results in improvements in one task at the expense of deterioration in another. Although several optimization methods have been developed to address this issue by manipulating task gradients for better task balancing, they cannot decrease the incidence of gradient conflict. In this paper, we systematically investigate the occurrence of gradient conflict across different methods and propose a strategy to reduce such conflicts through sparse training (ST), wherein only a portion of the model's parameters are updated during training while keeping the rest unchanged. Our extensive experiments demonstrate that ST effectively mitigates conflicting gradients and leads to superior performance. Furthermore, ST can be easily integrated with gradient manipulation techniques, thus enhancing their effectiveness.
Open AccessProactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective
Advancing towards generalist agents necessitates the concurrent processing of multiple tasks using a unified model, thereby underscoring the growing significance of simultaneous model training on multiple downstream tasks. A common issue in multi-task learning is the occurrence of gradient conflict, which leads to potential competition among different tasks during joint training. This competition often results in improvements in one task at the expense of deterioration in another. Although several optimization methods have been developed to address this issue by manipulating task gradients for better task balancing, they cannot decrease the incidence of gradient conflict. In this paper, we systematically investigate the occurrence of gradient conflict across different methods and propose a strategy to reduce such conflicts through sparse training (ST), wherein only a portion of the model's parameters are updated during training while keeping the rest unchanged. Our extensive experiments demonstrate that ST effectively mitigates conflicting gradients and leads to superior performance. Furthermore, ST can be easily integrated with gradient manipulation techniques, thus enhancing their effectiveness.
- Open Access
- 12 August 2024
- Published by Open Publishing Association in Electronic Proceedings in Theoretical Computer Science
- Vol. 406, 197-215
Density Matrices for Metaphor Understanding
In physics, density matrices are used to represent mixed states, i.e. probabilistic mixtures of pure states. This concept was used to model lexical ambiguity in [31]. In this paper, we consider metaphor as a type of lexical ambiguity, and examine whether metaphorical meaning can be effectively modelled using mixtures of word senses. We find that modelling metaphor is significantly more difficult than other kinds of lexical ambiguity, but that our best-performing density matrix method outperforms simple baselines as well as some neural language models.
- 12 August 2024
- Published by Open Publishing Association in Electronic Proceedings in Theoretical Computer Science
- Vol. 406, 197-215
Open AccessDensity Matrices for Metaphor Understanding
In physics, density matrices are used to represent mixed states, i.e. probabilistic mixtures of pure states. This concept was used to model lexical ambiguity in [31]. In this paper, we consider metaphor as a type of lexical ambiguity, and examine whether metaphorical meaning can be effectively modelled using mixtures of word senses. We find that modelling metaphor is significantly more difficult than other kinds of lexical ambiguity, but that our best-performing density matrix method outperforms simple baselines as well as some neural language models.
- Open Access
The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning
Texts written in different languages reflect different culturally-dependent beliefs of their writers. Thus, we expect multilingual LMs (MLMs), that are jointly trained on a concatenation of text in multiple languages, to encode different cultural values for each language. Yet, as the 'multilinguality' of these LMs is driven by cross-lingual sharing, we also have reason to belief that cultural values bleed over from one language into another. This limits the use of MLMs in practice, as apart from being proficient in generating text in multiple languages, creating language technology that can serve a community also requires the output of LMs to be sensitive to their biases (Naous et al., 2023). Yet, little is known about how cultural values emerge and evolve in MLMs (Hershcovich et al., 2022a). We are the first to study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning. Focusing on the fine-tuning stage allows us to study the interplay between value shifts when exposed to new linguistic experience from different data sources and languages. Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.
Open AccessThe Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning
Texts written in different languages reflect different culturally-dependent beliefs of their writers. Thus, we expect multilingual LMs (MLMs), that are jointly trained on a concatenation of text in multiple languages, to encode different cultural values for each language. Yet, as the 'multilinguality' of these LMs is driven by cross-lingual sharing, we also have reason to belief that cultural values bleed over from one language into another. This limits the use of MLMs in practice, as apart from being proficient in generating text in multiple languages, creating language technology that can serve a community also requires the output of LMs to be sensitive to their biases (Naous et al., 2023). Yet, little is known about how cultural values emerge and evolve in MLMs (Hershcovich et al., 2022a). We are the first to study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning. Focusing on the fine-tuning stage allows us to study the interplay between value shifts when exposed to new linguistic experience from different data sources and languages. Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 558-573
Are LLMs classical or nonmonotonic reasoners? Lessons from generics
Alina Leidinger, Robert Van Rooij, Ekaterina Shutova. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 558-573
Open AccessAre LLMs classical or nonmonotonic reasoners? Lessons from generics
Alina Leidinger, Robert Van Rooij, Ekaterina Shutova. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 15042-15058
The Echoes of Multilinguality: Tracing Cultural Value Shifts during Language Model Fine-tuning
Rochelle Choenni, Anne Lauscher, Ekaterina Shutova. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 15042-15058
Open AccessThe Echoes of Multilinguality: Tracing Cultural Value Shifts during Language Model Fine-tuning
Rochelle Choenni, Anne Lauscher, Ekaterina Shutova. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.