Francesco Pierri
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Francesco Pierri has a total of 215 co-authors. Here are the top co-authors by number of shared publications:
- Matthew R. DeVerna
Stanford University, Stanford, USA
7 shared publications - Stefano Ceri
Politecnico di Milano, Milan, Italy
7 shared publications Filippo Menczer
Observatory on Social Media, Indiana University, Bloomington, USA
6 shared publications- Giovanni Bonaccorsi
Department of Management, Economics and Industrial Engineering, Politecnico di Milano, Milan, Italy
6 shared publications - Kai-Cheng Yang
Binghamton University, Binghamton, USA
6 shared publications Stefano Ceri
Politecnico di Milano
6 shared publications
Latest Publications
- 6 October 2026
- Published by Springer Nature
- p. 550-564
From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs
- 6 October 2026
- Published by Springer Nature
- p. 550-564
From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs
- Open Access
- 1 September 2026
- Published by Association for Computing Machinery (ACM) in Communications of the ACM
- Vol. 69 (9) , 40-43
Research Opportunities and Challenges of the EU’s Digital Services Act
Proposing practical measures to ensure the Digital Services Act fulfills its transparency and accountability goals.
- 1 September 2026
- Published by Association for Computing Machinery (ACM) in Communications of the ACM
- Vol. 69 (9) , 40-43
Open AccessResearch Opportunities and Challenges of the EU’s Digital Services Act
Proposing practical measures to ensure the Digital Services Act fulfills its transparency and accountability goals.
- Top Cited Papers
- 9 June 2026
- Published by Springer Nature in Nature Human Behaviour
- Vol. 10 (7) , 1182-1186
A reporting checklist for large language models in behavioural science
- 9 June 2026
- Published by Springer Nature in Nature Human Behaviour
- Vol. 10 (7) , 1182-1186
Top Cited PapersA reporting checklist for large language models in behavioural science
- Open Access
- 30 May 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 17 (5) , 1-33
Evaluating Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards
Recent research has highlighted that assigning specific personas to large language models (LLMs) can significantly increase harmful content generation. However, limited attention has been given to persona-driven toxicity in non-Western contexts, particularly in Chinese-based LLMs. In this paper, we perform a large-scale, cross-model analysis of refusal behavior and persona-driven toxicity amplification across four Chinese LLMs, leveraging a comprehensive dataset of over 1 400 000 generated texts. We identify significant disparities in persona-driven refusal behavior, including systematic gender differences in refusal triggering across the evaluated Chinese LLMs. Furthermore, we provide quantitative evidence of persona-driven toxicity amplification with respect to model default baselines. We show that this amplification—whose magnitude varies substantially across models—is driven by interactions across several factors, involving persona conditioning, prompting strategy, target social group, and model-specific safety mechanisms. Leveraging model-specific regression analyses, we systematically characterize how persona categories, target social groups, and prompt templates independently and jointly shape both refusal behavior and output toxicity. As a complementary case study, we further explore an iterative, evaluator-guided mitigation strategy based on model feedback with an external LLM evaluator, demonstrating that highly toxic outputs can be substantially reduced without costly model retraining. Overall, our findings highlight the importance of culturally contextualized safety evaluations for Chinese-language LLMs and provide a structured framework for assessing persona-induced risks and exploratory mitigation strategies in LLM-generated content.
- 30 May 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 17 (5) , 1-33
Open AccessEvaluating Chinese Large Language Models: The Influence of Persona Assignment on Stereotypes and Safeguards
Recent research has highlighted that assigning specific personas to large language models (LLMs) can significantly increase harmful content generation. However, limited attention has been given to persona-driven toxicity in non-Western contexts, particularly in Chinese-based LLMs. In this paper, we perform a large-scale, cross-model analysis of refusal behavior and persona-driven toxicity amplification across four Chinese LLMs, leveraging a comprehensive dataset of over 1 400 000 generated texts. We identify significant disparities in persona-driven refusal behavior, including systematic gender differences in refusal triggering across the evaluated Chinese LLMs. Furthermore, we provide quantitative evidence of persona-driven toxicity amplification with respect to model default baselines. We show that this amplification—whose magnitude varies substantially across models—is driven by interactions across several factors, involving persona conditioning, prompting strategy, target social group, and model-specific safety mechanisms. Leveraging model-specific regression analyses, we systematically characterize how persona categories, target social groups, and prompt templates independently and jointly shape both refusal behavior and output toxicity. As a complementary case study, we further explore an iterative, evaluator-guided mitigation strategy based on model feedback with an external LLM evaluator, demonstrating that highly toxic outputs can be substantially reduced without costly model retraining. Overall, our findings highlight the importance of culturally contextualized safety evaluations for Chinese-language LLMs and provide a structured framework for assessing persona-induced risks and exploratory mitigation strategies in LLM-generated content.
- Open Access
From Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs
Subtitles are essential for video accessibility and audience engagement. Modern Automatic Speech Recognition (ASR) systems, built upon Encoder-Decoder neural network architectures and trained on massive amounts of data, have progressively reduced transcription errors on standard benchmark datasets. However, their performance in real-world production environments, particularly for non-English content like long-form Italian videos, remains largely unexplored. This paper presents a case study on developing a professional subtitling system for an Italian media company. To inform our system design, we evaluated four state-of-the-art ASR models (Whisper Large v2, AssemblyAI Universal, Parakeet TDT v3 0.6b, and WhisperX) on a 50-hour dataset of Italian television programs. The study highlights their strengths and limitations, benchmarking their performance against the work of professional human subtitlers. The findings indicate that, while current models cannot meet the media industry's accuracy needs for full autonomy, they can serve as highly effective tools for enhancing human productivity. We conclude that a human-in-the-loop (HITL) approach is crucial and present the production-grade, cloud-based infrastructure we designed to support this workflow.
Open AccessFrom Speech to Subtitles: Evaluating ASR Models in Subtitling Italian Television Programs
Subtitles are essential for video accessibility and audience engagement. Modern Automatic Speech Recognition (ASR) systems, built upon Encoder-Decoder neural network architectures and trained on massive amounts of data, have progressively reduced transcription errors on standard benchmark datasets. However, their performance in real-world production environments, particularly for non-English content like long-form Italian videos, remains largely unexplored. This paper presents a case study on developing a professional subtitling system for an Italian media company. To inform our system design, we evaluated four state-of-the-art ASR models (Whisper Large v2, AssemblyAI Universal, Parakeet TDT v3 0.6b, and WhisperX) on a 50-hour dataset of Italian television programs. The study highlights their strengths and limitations, benchmarking their performance against the work of professional human subtitlers. The findings indicate that, while current models cannot meet the media industry's accuracy needs for full autonomy, they can serve as highly effective tools for enhancing human productivity. We conclude that a human-in-the-loop (HITL) approach is crucial and present the production-grade, cloud-based infrastructure we designed to support this workflow.
- Open Access
- 5 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 190735-190744
A Conceptual Map for Exploring the Landscape of Large Language Models
Selecting and evaluating the most suitable Large Language Model (LLM) for a given task remains a significant challenge, particularly in domain-specific applications such as healthcare and legal research, which require models to provide transparency regarding training data, task specialization, and compliance with ethical standards. Despite the availability of open-source platforms for sharing models and datasets, challenges persist in accessing critical information. Incomplete metadata and inconsistent documentation hinder efficient model discovery, comparison, and adoption. In this paper, we introduce a simple conceptual map for LLMs, designed to assist researchers and practitioners in understanding the complex landscape of generative models. We provide a rationale for our modeling choices and a comprehensive description of the map in terms of four interconnected entities. Our primary objective is to provide all practitioners in the field — not just developers, but managers, testers, and other people who are part of producing and using technology — a clear terminology for expressing how to address the LLM landscape. A secondary objective is a call on industry stakeholders — including collaborative platforms and model providers—to enhance transparency and reproducibility in LLM research and deployment.
- 5 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 190735-190744
Open AccessA Conceptual Map for Exploring the Landscape of Large Language Models
Selecting and evaluating the most suitable Large Language Model (LLM) for a given task remains a significant challenge, particularly in domain-specific applications such as healthcare and legal research, which require models to provide transparency regarding training data, task specialization, and compliance with ethical standards. Despite the availability of open-source platforms for sharing models and datasets, challenges persist in accessing critical information. Incomplete metadata and inconsistent documentation hinder efficient model discovery, comparison, and adoption. In this paper, we introduce a simple conceptual map for LLMs, designed to assist researchers and practitioners in understanding the complex landscape of generative models. We provide a rationale for our modeling choices and a comprehensive description of the map in terms of four interconnected entities. Our primary objective is to provide all practitioners in the field — not just developers, but managers, testers, and other people who are part of producing and using technology — a clear terminology for expressing how to address the LLM landscape. A secondary objective is a call on industry stakeholders — including collaborative platforms and model providers—to enhance transparency and reproducibility in LLM research and deployment.
- Open Access
Probing Social Identity Bias in Chinese LLMs with Gendered Pronouns and Social Groups
Large language models (LLMs) are increasingly deployed in user-facing applications, raising concerns about their potential to reflect and amplify social biases. We investigate social identity framing in Chinese LLMs using Mandarin-specific prompts across ten representative Chinese LLMs, evaluating responses to ingroup ("We") and outgroup ("They") framings, and extending the setting to 240 social groups salient in the Chinese context. To complement controlled experiments, we further analyze Chinese-language conversations from a corpus of real interactions between users and chatbots. Across models, we observe systematic ingroup-positive and outgroup-negative tendencies, which are not confined to synthetic prompts but also appear in naturalistic dialogue, indicating that bias dynamics might strengthen in real interactions. Our study provides a language-aware evaluation framework for Chinese LLMs, demonstrating that social identity biases documented in English generalize cross-linguistically and intensify in user-facing contexts.
Open AccessProbing Social Identity Bias in Chinese LLMs with Gendered Pronouns and Social Groups
Large language models (LLMs) are increasingly deployed in user-facing applications, raising concerns about their potential to reflect and amplify social biases. We investigate social identity framing in Chinese LLMs using Mandarin-specific prompts across ten representative Chinese LLMs, evaluating responses to ingroup ("We") and outgroup ("They") framings, and extending the setting to 240 social groups salient in the Chinese context. To complement controlled experiments, we further analyze Chinese-language conversations from a corpus of real interactions between users and chatbots. Across models, we observe systematic ingroup-positive and outgroup-negative tendencies, which are not confined to synthetic prompts but also appear in naturalistic dialogue, indicating that bias dynamics might strengthen in real interactions. Our study provides a language-aware evaluation framework for Chinese LLMs, demonstrating that social identity biases documented in English generalize cross-linguistically and intensify in user-facing contexts.
- Open Access
Towards an Automated Framework to Audit Youth Safety on TikTok
This paper investigates the effectiveness of TikTok's enforcement mechanisms for limiting the exposure of harmful content to youth accounts. We collect over 7000 videos, classify them as harmful vs not-harmful, and then simulate interactions using age-specific sockpuppet accounts through both passive and active engagement strategies. We also evaluate the performance of large language (LLMs) and vision-language models (VLMs) in detecting harmful content, identifying key challenges in precision and scalability. Preliminary results show minimal differences in content exposure between adult and youth accounts, raising concerns about the platform's age-based moderation. These findings suggest that the platform needs to strengthen youth safety measures and improve transparency in content moderation.
Open AccessTowards an Automated Framework to Audit Youth Safety on TikTok
This paper investigates the effectiveness of TikTok's enforcement mechanisms for limiting the exposure of harmful content to youth accounts. We collect over 7000 videos, classify them as harmful vs not-harmful, and then simulate interactions using age-specific sockpuppet accounts through both passive and active engagement strategies. We also evaluate the performance of large language (LLMs) and vision-language models (VLMs) in detecting harmful content, identifying key challenges in precision and scalability. Preliminary results show minimal differences in content exposure between adult and youth accounts, raising concerns about the platform's age-based moderation. These findings suggest that the platform needs to strengthen youth safety measures and improve transparency in content moderation.
- Top Cited PapersOpen Access
- 7 June 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the International AAAI Conference on Web and Social Media
- Vol. 19, 522-535
Nicer than Humans: How Do Large Language Models Behave in the Prisoner's Dilemma?
The behavior of Large Language Models (LLMs) as artificial social agents is largely unexplored, and we still lack extensive evidence of how these agents react to simple social stimuli. Testing the behavior of AI agents in classic Game Theory experiments provides a promising theoretical framework for evaluating the norms and values of these agents in archetypal social situations. In this work, we investigate the cooperative behavior of three LLMs (Llama2, Llama3, and GPT3.5) when playing the Iterated Prisoner's Dilemma against random adversaries displaying various levels of hostility. We introduce a systematic methodology to evaluate an LLM's comprehension of the game rules and its capability to parse historical gameplay logs for decision-making. We conducted simulations of games lasting for 100 rounds and analyzed the LLMs' decisions in terms of dimensions defined in the behavioral economics literature. We find that all models tend not to initiate defection but act cautiously, favoring cooperation over defection only when the opponent's defection rate is low. Overall, LLMs behave at least as cooperatively as the typical human player, although our results indicate some substantial differences among models. In particular, Llama2 and GPT3.5 are more cooperative than humans, and especially forgiving and non-retaliatory for opponent defection rates below 30%. More similar to humans, Llama3 exhibits consistently uncooperative and exploitative behavior unless the opponent always cooperates. Our systematic approach to the study of LLMs in game theoretical scenarios is a step towards using these simulations to inform practices of LLM auditing and alignment.
- 7 June 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the International AAAI Conference on Web and Social Media
- Vol. 19, 522-535
Top Cited PapersOpen AccessNicer than Humans: How Do Large Language Models Behave in the Prisoner's Dilemma?
The behavior of Large Language Models (LLMs) as artificial social agents is largely unexplored, and we still lack extensive evidence of how these agents react to simple social stimuli. Testing the behavior of AI agents in classic Game Theory experiments provides a promising theoretical framework for evaluating the norms and values of these agents in archetypal social situations. In this work, we investigate the cooperative behavior of three LLMs (Llama2, Llama3, and GPT3.5) when playing the Iterated Prisoner's Dilemma against random adversaries displaying various levels of hostility. We introduce a systematic methodology to evaluate an LLM's comprehension of the game rules and its capability to parse historical gameplay logs for decision-making. We conducted simulations of games lasting for 100 rounds and analyzed the LLMs' decisions in terms of dimensions defined in the behavioral economics literature. We find that all models tend not to initiate defection but act cautiously, favoring cooperation over defection only when the opponent's defection rate is low. Overall, LLMs behave at least as cooperatively as the typical human player, although our results indicate some substantial differences among models. In particular, Llama2 and GPT3.5 are more cooperative than humans, and especially forgiving and non-retaliatory for opponent defection rates below 30%. More similar to humans, Llama3 exhibits consistently uncooperative and exploitative behavior unless the opponent always cooperates. Our systematic approach to the study of LLMs in game theoretical scenarios is a step towards using these simulations to inform practices of LLM auditing and alignment.
- Top Cited PapersOpen Access
- 7 June 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the International AAAI Conference on Web and Social Media
- Vol. 19, 1346-1357
Toxic Bias: Perspective API Misreads German as More Toxic
Proprietary public APIs play a crucial and growing role as research tools among social scientists. Among such APIs, Google's machine learning-based Perspective API is extensively utilized for assessing the toxicity of social media messages, providing both an important resource for researchers and automatic content moderation. However, this paper exposes an important bias in Perspective API concerning German language text. Through an in-depth examination of several datasets, we uncover intrinsic language biases within the multilingual model of Perspective API. We find that the toxicity assessment of German content produces significantly higher toxicity levels than other languages. This finding is robust across various translations, topics, and data sources, and has significant consequences for both research and moderation strategies that rely on Perspective API. For instance, we show that, on average, four times more tweets and users would be moderated when using the German language compared to their English translation. Our findings point to broader risks associated with the widespread use of proprietary APIs within the computational social sciences.
- 7 June 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the International AAAI Conference on Web and Social Media
- Vol. 19, 1346-1357
Top Cited PapersOpen AccessToxic Bias: Perspective API Misreads German as More Toxic
Proprietary public APIs play a crucial and growing role as research tools among social scientists. Among such APIs, Google's machine learning-based Perspective API is extensively utilized for assessing the toxicity of social media messages, providing both an important resource for researchers and automatic content moderation. However, this paper exposes an important bias in Perspective API concerning German language text. Through an in-depth examination of several datasets, we uncover intrinsic language biases within the multilingual model of Perspective API. We find that the toxicity assessment of German content produces significantly higher toxicity levels than other languages. This finding is robust across various translations, topics, and data sources, and has significant consequences for both research and moderation strategies that rely on Perspective API. For instance, we show that, on average, four times more tweets and users would be moderated when using the German language compared to their English translation. Our findings point to broader risks associated with the widespread use of proprietary APIs within the computational social sciences.