ESEkaterina Shutova

Ekaterina Shutova

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Ekaterina Shutova has a total of 63 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning

    by Zhi Zhang, Yixian Shen, Congfeng Cao, Ekaterina Shutova

    Existing parameter-efficient fine-tuning (PEFT) methods primarily fall into two categories: addition-based and selective in-situ adaptation. The former, such as LoRA, introduce additional modules to adapt the model to downstream tasks, offering strong memory efficiency. However, their representational capacity is often limited, making them less suitable for fine-grained adaptation. In contrast, the latter directly fine-tunes a carefully chosen subset of the original model parameters, allowing for more precise and effective adaptation, but at the cost of significantly increased memory consumption. To reconcile this trade-off, we propose NeuroAda, a novel PEFT method that enables fine-grained model finetuning while maintaining high memory efficiency. Our approach first identifies important parameters (i.e., connections within the network) as in selective adaptation, and then introduces bypass connections for these selected parameters. During finetuning, only the bypass connections are updated, leaving the original model parameters frozen. Empirical results on 23+ tasks spanning both natural language generation and understanding demonstrate that NeuroAda achieves state-of-the-art performance with as little as $\leq \textbf{0.02}\%$ trainable parameters, while reducing CUDA memory usage by up to 60%. We release our code here: https://github.com/FightingFighting/NeuroAda.git.

  • Open Access

    NeuroAda: Activating Each Neuron’s Potential for Parameter-Efficient Fine-Tuning

    by Zhi Zhang, Yixian Shen, Congfeng Cao, Ekaterina Shutova

    Zhi Zhang, Yixian Shen, Congfeng Cao, Ekaterina Shutova. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

  • Top Cited Papers
    Open Access

    Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

    by Joy Crosbie, Ekaterina Shutova

    Joy Crosbie, Ekaterina Shutova. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.


  • Open Access

    Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models

    by Srishti Yadav, Zhi Zhang, Daniel Hershcovich, Ekaterina Shutova

    Srishti Yadav, Zhi Zhang, Daniel Hershcovich, Ekaterina Shutova. Findings of the Association for Computational Linguistics: NAACL 2025. 2025.


  • Open Access

    Proactive Gradient Conflict Mitigation in Multi-Task Learning:A Sparse Training Perspective

    by Zhi Zhang, Jiayi Shen, Shanghang Zhang, Ekaterina Shutova

    Advancing towards generalist agents necessitates the concurrent processing of multiple tasks using a unified model, thereby underscoring the growing significance of simultaneous model training on multiple downstream tasks. A common issue in multi-task learning is the occurrence of gradient conflict, which leads to potential competition among different tasks during joint training. This competition often results in improvements in one task at the expense of deterioration in another. Although several optimization methods have been developed to address this issue by manipulating task gradients for better task balancing, they cannot decrease the incidence of gradient conflict. In this paper, we systematically investigate the occurrence of gradient conflict across different methods and propose a strategy to reduce such conflicts through sparse training (ST), wherein only a portion of the model’s parameters are updated during training while keeping the rest unchanged. Our extensive experiments demonstrate that ST effectively mitigates conflicting gradients and leads to superior performance. Furthermore, ST can be easily integrated with gradient manipulation techniques, thus enhancing their effectiveness.


  • Open Access

    Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective

    by Zhi Zhang, Jiayi Shen, Congfeng Cao, Gaole Dai, Shiji Zhou, Qizhe Zhang, Shanghang Zhang, Ekaterina Shutova

    Advancing towards generalist agents necessitates the concurrent processing of multiple tasks using a unified model, thereby underscoring the growing significance of simultaneous model training on multiple downstream tasks. A common issue in multi-task learning is the occurrence of gradient conflict, which leads to potential competition among different tasks during joint training. This competition often results in improvements in one task at the expense of deterioration in another. Although several optimization methods have been developed to address this issue by manipulating task gradients for better task balancing, they cannot decrease the incidence of gradient conflict. In this paper, we systematically investigate the occurrence of gradient conflict across different methods and propose a strategy to reduce such conflicts through sparse training (ST), wherein only a portion of the model's parameters are updated during training while keeping the rest unchanged. Our extensive experiments demonstrate that ST effectively mitigates conflicting gradients and leads to superior performance. Furthermore, ST can be easily integrated with gradient manipulation techniques, thus enhancing their effectiveness.


  • Open Access

    Density Matrices for Metaphor Understanding

    by Jay Owers, Ekaterina Shutova, Martha Lewis

    In physics, density matrices are used to represent mixed states, i.e. probabilistic mixtures of pure states. This concept was used to model lexical ambiguity in [31]. In this paper, we consider metaphor as a type of lexical ambiguity, and examine whether metaphorical meaning can be effectively modelled using mixtures of word senses. We find that modelling metaphor is significantly more difficult than other kinds of lexical ambiguity, but that our best-performing density matrix method outperforms simple baselines as well as some neural language models.

  • Open Access

    The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning

    by Rochelle Choenni, Anne Lauscher, Ekaterina Shutova

    Texts written in different languages reflect different culturally-dependent beliefs of their writers. Thus, we expect multilingual LMs (MLMs), that are jointly trained on a concatenation of text in multiple languages, to encode different cultural values for each language. Yet, as the 'multilinguality' of these LMs is driven by cross-lingual sharing, we also have reason to belief that cultural values bleed over from one language into another. This limits the use of MLMs in practice, as apart from being proficient in generating text in multiple languages, creating language technology that can serve a community also requires the output of LMs to be sensitive to their biases (Naous et al., 2023). Yet, little is known about how cultural values emerge and evolve in MLMs (Hershcovich et al., 2022a). We are the first to study how languages can exert influence on the cultural values encoded for different test languages, by studying how such values are revised during fine-tuning. Focusing on the fine-tuning stage allows us to study the interplay between value shifts when exposed to new linguistic experience from different data sources and languages. Lastly, we use a training data attribution method to find patterns in the fine-tuning examples, and the languages that they come from, that tend to instigate value shifts.


  • Open Access

    Are LLMs classical or nonmonotonic reasoners? Lessons from generics

    by Alina Leidinger, Robert Van Rooij, Ekaterina Shutova

    Alina Leidinger, Robert Van Rooij, Ekaterina Shutova. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2024.


  • Open Access

    The Echoes of Multilinguality: Tracing Cultural Value Shifts during Language Model Fine-tuning

    by Rochelle Choenni, Anne Lauscher, Ekaterina Shutova

    Rochelle Choenni, Anne Lauscher, Ekaterina Shutova. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.