Syntactic Analysis in Transformers through Attention Heads
Open Access
- 29 September 2025
- conference paper
- Published by Sociedade Brasileira de Computação
- p. 295-306
Abstract
The advances in Natural Language Processing (NLP) have led to the development of Transformer architectures, such as BERT. One of the most prominent features of this architecture is the attention mechanism. However, the application of attention mechanisms to languages other than English, like Brazilian Portuguese, remains underexplored. This work analyzes the attention heads in a Transformer architecture, considering the syntactic relations in a sentence. We analyze how attention patterns align with syntactic dependencies involving phenomena such as transitive verbs, reflexive pronouns, and subordinate clauses, as realized in Brazilian Portuguese. Results described the existence of specialized heads within arcs, such as subject–verb and verb–object. These findings open up new research opportunities to evaluate syntactic sensitivity in Transformer models and to contribute to the development of more linguistically informed models for Brazilian Portuguese.Keywords
This publication has 9 references indexed in Scilit:
- Scaling and Adapting Large Language Models for Portuguese Open Information Extraction: A Comparative Study of Fine-Tuning and LoRA2025
- BERTimbau: Pretrained BERT Models for Brazilian Portuguese2020
- Are All Languages Created Equal in Multilingual BERT?2020
- BERT Rediscovers the Classical NLP Pipeline2019
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned2019
- How Multilingual is Multilingual BERT?2019
- Open Sesame: Getting inside BERT’s Linguistic Knowledge2019
- What Does BERT Look at? An Analysis of BERT’s Attention2019
- An Analysis of Encoder Representations in Transformer-Based Machine Translation2018