Pietro Bernardelle
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Pietro Bernardelle has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Gianluca Demartini
The University of Queensland, Brisbane, Australia
10 shared publications - Stefano Civelli
The University of Queensland, Brisbane, Australia
8 shared publications - Kevin Roitero
University of Udine, udine, Italy
3 shared publications - Riccardo Lunardi
University of Udine, udine, Italy
3 shared publications - Joel Mackenzie
The University of Queensland, Brisbane, Australia
2 shared publications - Leon Fröhling
GESIS - Leibniz Institute for the Social Sciences, Cologne, Germany
2 shared publications
Latest Publications
- Open Access
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 3689-3700
Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 3689-3700
Open AccessPersona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluation
- Open Access
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 307-318
LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal
Large language models (LLMs) are increasingly used in information retrieval (IR) pipelines as relevance judges and re-rankers. Yet most analyses remain output-centric, evaluating generated labels or scores while offering limited insight into how relevance is represented inside the model. In this work, we study whether query-document (q-d) relevance is linearly decodable from residual-stream activations in instruction-tuned LLMs, how this internal signal compares with generated relevance judgments, and whether it transfers across languages. Using the TREC DL20 and MIRACL test collections, we guide medium-scale LLMs (4-9B parameters) with UMBRELA-style relevance judgment prompts, extract last-token activations from every transformer layer, and train linear probes to predict relevance labels. We compare probe predictions with generated judgments and use TREC DL20 to test whether probe-derived pseudo-labels preserve system rankings when compared with human judgments. Our results suggest that q-d relevance is encoded as a depth-dependent internal signal: probe performance is weak in early layers and strongest in middle-to-late layers, indicating that relevance becomes more linearly accessible after contextual integration. Most importantly, in several models, validation-selected probes match or outperform generated judgments and better preserve system rankings, revealing a separation between internal relevance representation and external expression. Multilingual experiments further suggest partial cross-language portability, although transfer remains weaker than within-language decoding. Overall, this work provides a representation-level perspective on LLM-based relevance assessment. Layer-wise probing can help diagnose where relevance emerges, when generated judgments fail to reflect internally available evidence, and how relevance representations vary across languages, datasets, and model families.
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 307-318
Open AccessLLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal
Large language models (LLMs) are increasingly used in information retrieval (IR) pipelines as relevance judges and re-rankers. Yet most analyses remain output-centric, evaluating generated labels or scores while offering limited insight into how relevance is represented inside the model. In this work, we study whether query-document (q-d) relevance is linearly decodable from residual-stream activations in instruction-tuned LLMs, how this internal signal compares with generated relevance judgments, and whether it transfers across languages. Using the TREC DL20 and MIRACL test collections, we guide medium-scale LLMs (4-9B parameters) with UMBRELA-style relevance judgment prompts, extract last-token activations from every transformer layer, and train linear probes to predict relevance labels. We compare probe predictions with generated judgments and use TREC DL20 to test whether probe-derived pseudo-labels preserve system rankings when compared with human judgments. Our results suggest that q-d relevance is encoded as a depth-dependent internal signal: probe performance is weak in early layers and strongest in middle-to-late layers, indicating that relevance becomes more linearly accessible after contextual integration. Most importantly, in several models, validation-selected probes match or outperform generated judgments and better preserve system rankings, revealing a separation between internal relevance representation and external expression. Multilingual experiments further suggest partial cross-language portability, although transfer remains weaker than within-language decoding. Overall, this work provides a representation-level perspective on LLM-based relevance assessment. Layer-wise probing can help diagnose where relevance emerges, when generated judgments fail to reflect internally available evidence, and how relevance representations vary across languages, datasets, and model families.
- Open Access
- 9 September 2026
- Published by Elsevier in Information Processing & Management
- Vol. 64 (2)
Political Ideology Shifts in Large Language Models
- 9 September 2026
- Published by Elsevier in Information Processing & Management
- Vol. 64 (2)
Open AccessPolitical Ideology Shifts in Large Language Models
- Open Access
- 18 April 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
Ideology-Based LLMs for Content Moderation
Large language models (LLMs) are increasingly used in content moderation systems, where ensuring fairness and objectivity is essential. In this study, we examine how persona adoption influences the consistency and fairness of harmful content classification across different LLM architectures, model sizes, and content modalities (i.e., language vs. vision). At first glance, headline performance metrics suggest that personas have little impact on overall classification accuracy. However, a closer analysis reveals important behavioral shifts. Personas with different ideological leanings display distinct propensities to label content as harmful, showing that the lens through which a model “views” input can subtly shape its judgments. Agreement analyses highlight that models, particularly larger ones, tend to align more closely with personas from the same political ideology, strengthening within-ideology consistency while widening divergence across ideological groups. To show this effect more directly, we conducted a study on a politically targeted task, which confirmed that personas not only behave more coherently within their own ideology but also exhibit a tendency to defend their perspective while downplaying harmfulness in opposing views. Together, these findings highlight how persona conditioning can introduce subtle ideological biases into LLM outputs, raising concerns about the use of AI systems that may reinforce partisan perspectives under the guise of neutrality.
- 18 April 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
Open AccessIdeology-Based LLMs for Content Moderation
Large language models (LLMs) are increasingly used in content moderation systems, where ensuring fairness and objectivity is essential. In this study, we examine how persona adoption influences the consistency and fairness of harmful content classification across different LLM architectures, model sizes, and content modalities (i.e., language vs. vision). At first glance, headline performance metrics suggest that personas have little impact on overall classification accuracy. However, a closer analysis reveals important behavioral shifts. Personas with different ideological leanings display distinct propensities to label content as harmful, showing that the lens through which a model “views” input can subtly shape its judgments. Agreement analyses highlight that models, particularly larger ones, tend to align more closely with personas from the same political ideology, strengthening within-ideology consistency while widening divergence across ideological groups. To show this effect more directly, we conducted a study on a politically targeted task, which confirmed that personas not only behave more coherently within their own ideology but also exhibit a tendency to defend their perspective while downplaying harmfulness in opposing views. Together, these findings highlight how persona conditioning can introduce subtle ideological biases into LLM outputs, raising concerns about the use of AI systems that may reinforce partisan perspectives under the guise of neutrality.
- Open Access
A Shared Geometry of Difficulty in Multilingual Language Models
Predicting problem-difficulty in large language models (LLMs) refers to estimating how difficult a task is according to the model itself, typically by training linear probes on its internal representations. In this work, we study the multilingual geometry of problem-difficulty in LLMs by training linear probes using the AMC subset of the Easy2Hard benchmark, translated into 21 languages. We found that difficulty-related signals emerge at two distinct stages of the model internals, corresponding to shallow (early-layers) and deep (later-layers) internal representations, that exhibit functionally different behaviors. Probes trained on deep representations achieve high accuracy when evaluated on the same language but exhibit poor cross-lingual generalization. In contrast, probes trained on shallow representations generalize substantially better across languages, despite achieving lower within-language performance. Together, these results suggest that LLMs first form a language-agnostic representation of problem difficulty, which subsequently becomes language-specific. This closely aligns with existing findings in LLM interpretability showing that models tend to operate in an abstract conceptual space before producing language-specific outputs. We demonstrate that this two-stage representational process extends beyond semantic content to high-level meta-cognitive properties such as problem-difficulty estimation.
Open AccessA Shared Geometry of Difficulty in Multilingual Language Models
Predicting problem-difficulty in large language models (LLMs) refers to estimating how difficult a task is according to the model itself, typically by training linear probes on its internal representations. In this work, we study the multilingual geometry of problem-difficulty in LLMs by training linear probes using the AMC subset of the Easy2Hard benchmark, translated into 21 languages. We found that difficulty-related signals emerge at two distinct stages of the model internals, corresponding to shallow (early-layers) and deep (later-layers) internal representations, that exhibit functionally different behaviors. Probes trained on deep representations achieve high accuracy when evaluated on the same language but exhibit poor cross-lingual generalization. In contrast, probes trained on shallow representations generalize substantially better across languages, despite achieving lower within-language performance. Together, these results suggest that LLMs first form a language-agnostic representation of problem difficulty, which subsequently becomes language-specific. This closely aligns with existing findings in LLM interpretability showing that models tend to operate in an abstract conceptual space before producing language-specific outputs. We demonstrate that this two-stage representational process extends beyond semantic content to high-level meta-cognitive properties such as problem-difficulty estimation.
- Open Access
Political Advertising on Facebook During the 2022 Australian Federal Election: A Social Identity Perspective
The spread of targeted advertising on social media platforms has revolutionized political marketing strategies. Monitoring these digital campaigns is essential for maintaining transparency and accountability in democratic processes. Leveraging Meta's Ad Library, we analyze political advertising on Facebook and Instagram during the 2022 Australian federal election campaign. We investigate temporal, demographic, and geographical patterns in the advertising strategies of major Australian political actors to establish an empirical evidence base, and interpret these findings through the lens of Social Identity Theory (SIT). Our findings not only reveal significant disparities in spending and reach among parties, but also in persuasion strategies being deployed in targeted online campaigns. We observe a marked increase in advertising activity as the election approached, peaking just before the mandated media blackout period. Demographic analysis shows distinct targeting strategies, with parties focusing more on younger demographics and exhibiting gender-based differences in ad impressions. Regional distribution of ads largely mirrored population densities, with some parties employing more targeted approaches in specific states. Moreover, we found that parties emphasized different themes aligned with their ideologies-major parties focused on party names and opponents, while smaller parties emphasized issue-specific messages. Drawing on SIT, we interpret these findings within Australia's compulsory voting context, suggesting that parties employed distinct persuasion strategies. With turnout guaranteed, major parties focused on reinforcing partisan identities to prevent voter defection, while smaller parties cultivated issue-based identities to capture the support of disaffected voters who are obligated to participate.
Open AccessPolitical Advertising on Facebook During the 2022 Australian Federal Election: A Social Identity Perspective
The spread of targeted advertising on social media platforms has revolutionized political marketing strategies. Monitoring these digital campaigns is essential for maintaining transparency and accountability in democratic processes. Leveraging Meta's Ad Library, we analyze political advertising on Facebook and Instagram during the 2022 Australian federal election campaign. We investigate temporal, demographic, and geographical patterns in the advertising strategies of major Australian political actors to establish an empirical evidence base, and interpret these findings through the lens of Social Identity Theory (SIT). Our findings not only reveal significant disparities in spending and reach among parties, but also in persuasion strategies being deployed in targeted online campaigns. We observe a marked increase in advertising activity as the election approached, peaking just before the mandated media blackout period. Demographic analysis shows distinct targeting strategies, with parties focusing more on younger demographics and exhibiting gender-based differences in ad impressions. Regional distribution of ads largely mirrored population densities, with some parties employing more targeted approaches in specific states. Moreover, we found that parties emphasized different themes aligned with their ideologies-major parties focused on party names and opponents, while smaller parties emphasized issue-specific messages. Drawing on SIT, we interpret these findings within Australia's compulsory voting context, suggesting that parties employed distinct persuasion strategies. With turnout guaranteed, major parties focused on reinforcing partisan identities to prevent voter defection, while smaller parties cultivated issue-based identities to capture the support of disaffected voters who are obligated to participate.
- Open Access
Political Ideology Shifts in Large Language Models
Large language models (LLMs) are increasingly deployed in politically sensitive settings, raising concerns about their potential to encode, amplify, or be steered toward specific ideologies. We investigate how adopting synthetic personas influences ideological expression in LLMs across seven models (7B-70B+ parameters) from multiple families, using the Political Compass Test as a standardized probe. Our analysis reveals four consistent patterns: (i) larger models display broader and more polarized implicit ideological coverage; (ii) susceptibility to explicit ideological cues grows with scale; (iii) models respond more strongly to right-authoritarian than to left-libertarian priming; and (iv) thematic content in persona descriptions induces systematic and predictable ideological shifts, which amplify with size. These findings indicate that both scale and persona content shape LLM political behavior. As such systems enter decision-making, educational, and policy contexts, their latent ideological malleability demands attention to safeguard fairness, transparency, and safety.
Open AccessPolitical Ideology Shifts in Large Language Models
Large language models (LLMs) are increasingly deployed in politically sensitive settings, raising concerns about their potential to encode, amplify, or be steered toward specific ideologies. We investigate how adopting synthetic personas influences ideological expression in LLMs across seven models (7B-70B+ parameters) from multiple families, using the Political Compass Test as a standardized probe. Our analysis reveals four consistent patterns: (i) larger models display broader and more polarized implicit ideological coverage; (ii) susceptibility to explicit ideological cues grows with scale; (iii) models respond more strongly to right-authoritarian than to left-libertarian priming; and (iv) thematic content in persona descriptions induces systematic and predictable ideological shifts, which amplify with size. These findings indicate that both scale and persona content shape LLM political behavior. As such systems enter decision-making, educational, and policy contexts, their latent ideological malleability demands attention to safeguard fairness, transparency, and safety.
- Open Access
- 8 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 864-867
Mapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas
- 8 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 864-867
Open AccessMapping and Influencing the Political Ideology of Large Language Models using Synthetic Personas
- Open Access
- 8 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 1963-1968
The Impact of Persona-based Political Perspectives on Hateful Content Detection
- 8 May 2025
- Published by Association for Computing Machinery (ACM)
- p. 1963-1968
Open AccessThe Impact of Persona-based Political Perspectives on Hateful Content Detection
- Open Access
- 8 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 236-240
Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
Aligning the output of Large Language Models (LLMs) with human preferences (e.g., by means of reinforcement learning with human feedback, or RLHF) is essential for ensuring their effectiveness in real-world scenarios. Despite significant advancements in LLM alignment techniques, the impact of different type of preference data on model performance has yet to be systematically explored. In this study, we investigate the scalability, data efficiency, and effectiveness of Direct Preference Optimization (DPO) in fine-tuning pre-trained LLMs, aiming to reduce their dependency on extensive amounts of preference data, which is expensive to collect. We (1) systematically compare the performance of models fine-tuned with varying percentages of a combined preference judgement dataset to define the improvement curve of DPO and assess its effectiveness in data-constrained environments; and (2) provide insights for the development of an optimal approach for selective preference data usage. Our study reveals that increasing the amount of data used for training generally enhances and stabilizes model performance. Moreover, the use of a combination of diverse datasets significantly improves model effectiveness. Furthermore, when models are trained separately using different types of prompts, models trained with conversational prompts outperformed those trained with question answering prompts.
- 8 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 236-240
Open AccessOptimizing LLMs with Direct Preferences: A Data Efficiency Perspective
Aligning the output of Large Language Models (LLMs) with human preferences (e.g., by means of reinforcement learning with human feedback, or RLHF) is essential for ensuring their effectiveness in real-world scenarios. Despite significant advancements in LLM alignment techniques, the impact of different type of preference data on model performance has yet to be systematically explored. In this study, we investigate the scalability, data efficiency, and effectiveness of Direct Preference Optimization (DPO) in fine-tuning pre-trained LLMs, aiming to reduce their dependency on extensive amounts of preference data, which is expensive to collect. We (1) systematically compare the performance of models fine-tuned with varying percentages of a combined preference judgement dataset to define the improvement curve of DPO and assess its effectiveness in data-constrained environments; and (2) provide insights for the development of an optimal approach for selective preference data usage. Our study reveals that increasing the amount of data used for training generally enhances and stabilizes model performance. Moreover, the use of a combination of diverse datasets significantly improves model effectiveness. Furthermore, when models are trained separately using different types of prompts, models trained with conversational prompts outperformed those trained with question answering prompts.