Alessio Miaschi

Prompting Large Language Models (LLMs) offers new opportunities for extracting technical knowledge from industrial documents with limited programming expertise and annotated data. Yet, in knowledge-intensive domains, practitioners face a key implementation question: should general-purpose LLMs be adapted to the target task through prompting or supervised fine-tuning? This study compares fine-tuning and prompting, implemented through representative encoder-only and decoder-only LLMs, for extracting Functional Requirements and Design Parameters from patents. We formulate the task at two levels: sentence classification and Named Entity Recognition (NER), supported by a manually annotated dataset of 6,000 patent sentences. Our results indicate that fine-tuning encoder-only models outperform decoder-only models using prompting, with performance gaps up to 0.1 F1-score for sentence classification and 0.4 F1-score for NER. Using around 1,000 annotated examples for fine-tuning is enough to outperform decoder-only models, showing that fine-tuning is practical even in data-scarce domains. The study provides methodological and practical guidance for applying LLMs in patent analysis, new product development, and requirement engineering. The annotated dataset is made openly available to support future research: https://anonymous.4open.science/r/axiomatic-design-dataset-1298/README.md

Large Language Models (LLMs) enable researchers and practitioners to extract knowledge from text through prompting, without extensive programming expertise, large annotated datasets, or fine-tuning. This accessibility has accelerated LLM adoption but raises methodological concerns, particularly in knowledge-intensive technical domains, where annotated data are scarce and LLM evaluations remain limited. Hence, guidance on key methodological decisions, including LLM architecture selection and data requirements for evaluation or fine-tuning, is still limited. This study compares encoder-only and decoder-only LLMs for extracting technical knowledge from patents, focusing on Functional Requirements and Design Parameters. We formulate the task at two levels: sentence classification and Named Entity Recognition (NER), supported by a manually annotated dataset of 6,000 patent sentences. Our results indicate that fine-tuned encoder-only models outperform decoder-only models using prompting, with performance gaps up to 0.1 F1-score for sentence classification and 0.4 F1-score for NER. Using around 1,000 annotated examples for fine-tuning is enough to outperform decoder-only models, showing that fine-tuning is practical even in data-scarce domains. Our findings provide methodological and practical guidance for applying LLMs in technical contexts, and we make the annotated dataset openly available (https://anonymous.4open.science/r/axiomatic-design-dataset-1298/README.md) to support future research.

The paper introduces Parallel Trees, a novel multilingual treebank collection that includes 20 treebanks for 10 languages. The distinguishing property of this resource is that the sentences of each language are annotated using two syntactic representation paradigms (SRPs), respectively based on the notions of dependency and constituency. By aligning the annotations of existing resources, Parallel Trees represents an example of exploiting pre-existing treebanks to adapt them to novel applications. To illustrate its potential, we present a case study where the resource is employed as a benchmark to investigate whether and how BERT, one of the first prominent neural language models (NLMs), is sensitive to the dependency- and constituency-based approaches for representing the syntactic structure of a sentence. The case study results indicate that the model’s sensitivity fluctuates across languages and experimental settings. The unique nature of the Parallel Trees resource creates the prerequisites for innovative studies comparing dependency and phrase-structure trees, allowing for more focused investigations without the interference of lexical variation.

Davide Testa, Giovanni Bonetta, Raffaella Bernardi, Alessandro Bondielli, Alessandro Lenci, Alessio Miaschi, Lucia Passaro, Bernardo Magnini. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.

Mobile app reviews are a large-scale data source for software-related knowledge generation activities, including software maintenance, evolution and feedback analysis. Effective extraction of features (i.e., functionalities or characteristics) from these reviews is key to support analysis on the acceptance of these features, identification of relevant new feature requests and prioritization of feature development, among others. Traditional methods focus on syntactic pattern-based approaches, typically context-agnostic, evaluated on a closed set of apps, difficult to replicate and limited to a reduced set and domain of apps. Meanwhile, the pervasiveness of Large Language Models (LLMs) based on the Transformer architecture in software engineering tasks lays the groundwork for empirical evaluation of the performance of these models to support feature extraction. In this study, we present T-FREX, a Transformer-based, fully automatic approach for mobile app review feature extraction. First, we collect a set of ground truth features from users in a real crowdsourced software recommendation platform and transfer them automatically into a dataset of app reviews. Then, we use this newly created dataset to fine-tune multiple LLMs on a named entity recognition task under different data configurations. We assess the performance of T-FREX with respect to this ground truth, and we complement our analysis by comparing T-FREX with a baseline method from the field. Finally, we assess the quality of new features predicted by T-FREX through an external human evaluation. Results show that T-FREX outperforms on average the traditional syntactic-based method, especially when discovering new features from a domain for which the model has been fine-tuned.

Purpose: The authors’ goal is to investigate variations in the writing style of book reviews published on different social reading platforms and referring to books of different genres, which enables acquiring insights into communication strategies adopted by readers to share their reading experiences.Design/methodology/approach: The authors propose a corpus-based study focused on the analysis of A Good Review, a novel corpus of online book reviews written in Italian, posted on Amazon and Goodreads, and covering six literary fiction genres. The authors rely on stylometric analysis to explore the linguistic properties and lexicon of reviews and the authors conducted automatic classification experiments using multiple approaches and feature configurations to predict either the review's platform or the literary genre.Findings: The analysis of user-generated reviews demonstrates that language is a quite variable dimension across reading platforms, but not as much across book genres. The classification experiments revealed that features modelling the syntactic structure of the sentence are reliable proxies for discerning Amazon and Goodreads reviews, whereas lexical information showed a higher predictive role for automatically discriminating the genre.Originality/value: The high availability of cultural products makes information services necessary to help users navigate these resources and acquire information from unstructured data. This study contributes to a better understanding of the linguistic characteristics of user-generated book reviews, which can support the development of linguistically-informed recommendation services. Additionally, the authors release a novel corpus of online book reviews meant to support the reproducibility and advancements of the research.

The outstanding performance recently reached by neural language models (NLMs) across many natural language processing (NLP) tasks has steered the debate towards understanding whether NLMs implicitly learn linguistic competence. Probes, i.e., supervised models trained using NLM representations to predict linguistic properties, are frequently adopted to investigate this issue. However, it is still questioned if probing classification tasks really enable such investigation or if they simply hint at surface patterns in the data. This work contributes to this debate by presenting an approach to assessing the effectiveness of a suite of probing tasks aimed at testing the linguistic knowledge implicitly encoded by one of the most prominent NLMs, BERT. To this aim, we compared the performance of probes when predicting gold and automatically altered values of a set of linguistic features. Our experiments were performed on Italian and were evaluated across BERT’s layers and for sentences with different lengths. As a general result, we observed higher performance in the prediction of gold values, thus suggesting that the probing model is sensitive to the distortion of feature values. However, our experiments also showed that the length of a sentence is a highly influential factor that is able to confound the probing model’s predictions.

In this paper, we propose a comprehensive linguistic study aimed at assessing the implicit behaviour of one of the most prominent Neural Language Model (NLM) based on Transformer architectures, BERT [15], when dealing with a particular source of noisy data, namely essays written by L1 Italian learners containing a variety of errors targeting grammar, orthography and lexicon. Differently from previous works, we focus on the pre-training stage and we devise two evaluation tasks aimed at assessing the impact of errors on sentence-level inner representations from two complementary perspectives, i.e. robustness and sensitivity. Our experiments show that BERT's ability to compute sentence similarity and to correctly encode a set of raw and morpho-syntactic properties of a sentence are differently modulated by the category of errors and that the error hierarchies in terms of robustness and sensitivity change across layer-wise representations.