Xinyue Shen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Xinyue Shen has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Yang Zhang
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
7 shared publications - Michael Backes
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
6 shared publications - Zeyuan Chen
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
3 shared publications - Savvas Zannettou
Delft University of Technology, Delft, Netherlands
2 shared publications - Xinlei He
Wuhan University, Wuhan, China
2 shared publications - Yage Zhang
CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
2 shared publications
Latest Publications
- Open Access
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 287-301
'Shut Up and Let Me Enjoy My Otome': Understanding and Measuring the Toxicity in Otome Game Communities
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 287-301
Open Access'Shut Up and Let Me Enjoy My Otome': Understanding and Measuring the Toxicity in Otome Game Communities
- Open Access
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 5896-5910
Does AI Amplify Online Hate- A Multi-Modal Measurement Study on 4chan's /pol/
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 5896-5910
Open AccessDoes AI Amplify Online Hate- A Multi-Modal Measurement Study on 4chan's /pol/
- Open Access
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 4432-4446
Real Money, Fake Models: Deceptive Model Claims in Shadow APIs
- 14 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 4432-4446
Open AccessReal Money, Fake Models: Deceptive Model Claims in Shadow APIs
- Top Cited PapersOpen Access
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 3221-3235
UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images
- 19 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 3221-3235
Top Cited PapersOpen AccessUnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images
- Top Cited PapersOpen Access
- 2 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 2251-2265
MGTBench: Benchmarking Machine-Generated Text Detection
Nowadays, powerful large language models (LLMs) such as ChatGPT have demonstrated revolutionary power in a variety of natural language processing (NLP) tasks such as text classification, sentiment analysis, language translation, and question-answering. Consequently, the detection of machine-generated texts (MGTs) is becoming increasingly crucial as LLMs become more advanced and prevalent. These models have the ability to generate human-like language, making it challenging to discern whether a text is authored by a human or a machine. This raises concerns regarding authenticity, accountability, and potential bias. However, existing methods for detecting MGTs are evaluated using different model architectures, datasets, and experimental settings, resulting in a lack of a comprehensive evaluation framework that encompasses various methodologies. Furthermore, it remains unclear how existing detection methods would perform against powerful LLMs. In this paper, we fill this gap by proposing the first benchmark framework for MGT detection against powerful LLMs, named MGTBench. Extensive evaluations on public datasets with curated texts generated by various powerful LLMs such as ChatGPT-turbo and Claude demonstrate the effectiveness of different detection methods. Our ablation study shows that a larger number of words in general leads to better performance and most detection methods can achieve similar performance with much fewer training samples. Additionally, our findings reveal that metric-based/model-based detection methods exhibit better transferability across different LLMs/datasets. Furthermore, we delve into a more challenging task: text attribution, where the goal is to identify the originating model of a given text, i.e., whether it is a specific LLM or authored by a human. Our findings indicate that the model-based detection methods still perform well in the text attribution task. To investigate the robustness of different detection methods, we consider three adversarial attacks, namely paraphrasing, random spacing, and adversarial perturbations. We discover that these attacks can significantly diminish detection effectiveness, underscoring the critical need for the development of more robust detection methods. We envision that MGTBench will serve as a benchmark tool to accelerate future investigations involving the evaluation of powerful MGT detection methods on their respective datasets and the development of more advanced MGT detection methods.
- 2 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 2251-2265
Top Cited PapersOpen AccessMGTBench: Benchmarking Machine-Generated Text Detection
Nowadays, powerful large language models (LLMs) such as ChatGPT have demonstrated revolutionary power in a variety of natural language processing (NLP) tasks such as text classification, sentiment analysis, language translation, and question-answering. Consequently, the detection of machine-generated texts (MGTs) is becoming increasingly crucial as LLMs become more advanced and prevalent. These models have the ability to generate human-like language, making it challenging to discern whether a text is authored by a human or a machine. This raises concerns regarding authenticity, accountability, and potential bias. However, existing methods for detecting MGTs are evaluated using different model architectures, datasets, and experimental settings, resulting in a lack of a comprehensive evaluation framework that encompasses various methodologies. Furthermore, it remains unclear how existing detection methods would perform against powerful LLMs. In this paper, we fill this gap by proposing the first benchmark framework for MGT detection against powerful LLMs, named MGTBench. Extensive evaluations on public datasets with curated texts generated by various powerful LLMs such as ChatGPT-turbo and Claude demonstrate the effectiveness of different detection methods. Our ablation study shows that a larger number of words in general leads to better performance and most detection methods can achieve similar performance with much fewer training samples. Additionally, our findings reveal that metric-based/model-based detection methods exhibit better transferability across different LLMs/datasets. Furthermore, we delve into a more challenging task: text attribution, where the goal is to identify the originating model of a given text, i.e., whether it is a specific LLM or authored by a human. Our findings indicate that the model-based detection methods still perform well in the text attribution task. To investigate the robustness of different detection methods, we consider three adversarial attacks, namely paraphrasing, random spacing, and adversarial perturbations. We discover that these attacks can significantly diminish detection effectiveness, underscoring the critical need for the development of more robust detection methods. We envision that MGTBench will serve as a benchmark tool to accelerate future investigations involving the evaluation of powerful MGT detection methods on their respective datasets and the development of more advanced MGT detection methods.
- Top Cited PapersOpen Access
- 2 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 1671-1685
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.
- 2 December 2024
- Published by Association for Computing Machinery (ACM)
- p. 1671-1685
Top Cited PapersOpen Access"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.
- Open Access
- 15 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 3403-3417
Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models
State-of-the-art Text-to-Image models like Stable Diffusion and DALLE\cdot2 are revolutionizing how people generate visual content. At the same time, society has serious concerns about how adversaries can exploit such models to generate problematic or unsafe images. In this work, we focus on demystifying the generation of unsafe images and hateful memes from Text-to-Image models. We first construct a typology of unsafe images consisting of five categories (sexually explicit, violent, disturbing, hateful, and political). Then, we assess the proportion of unsafe images generated by four advanced Text-to-Image models using four prompt datasets. We find that Text-to-Image models can generate a substantial percentage of unsafe images; across four models and four prompt datasets, 14.56% of all generated images are unsafe. When comparing the four Text-to-Image models, we find different risk levels, with Stable Diffusion being the most prone to generating unsafe content (18.92% of all generated images are unsafe). Given Stable Diffusion's tendency to generate more unsafe content, we evaluate its potential to generate hateful meme variants if exploited by an adversary to attack a specific individual or community. We employ three image editing methods, DreamBooth, Textual Inversion, and SDEdit, which are supported by Stable Diffusion to generate variants. Our evaluation result shows that 24% of the generated images using DreamBooth are hateful meme variants that present the features of the original hateful meme and the target individual/community; these generated images are comparable to hateful meme variants collected from the real world. Overall, our results demonstrate that the danger of large-scale generation of unsafe images is imminent. We discuss several mitigating measures, such as curating training data, regulating prompts, and implementing safety filters, and encourage better safeguard tools to be developed to prevent unsafe generation.1 Our code is available at https://github.com/YitingQu/unsafe-diffusion.
- 15 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 3403-3417
Open AccessUnsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models
State-of-the-art Text-to-Image models like Stable Diffusion and DALLE\cdot2 are revolutionizing how people generate visual content. At the same time, society has serious concerns about how adversaries can exploit such models to generate problematic or unsafe images. In this work, we focus on demystifying the generation of unsafe images and hateful memes from Text-to-Image models. We first construct a typology of unsafe images consisting of five categories (sexually explicit, violent, disturbing, hateful, and political). Then, we assess the proportion of unsafe images generated by four advanced Text-to-Image models using four prompt datasets. We find that Text-to-Image models can generate a substantial percentage of unsafe images; across four models and four prompt datasets, 14.56% of all generated images are unsafe. When comparing the four Text-to-Image models, we find different risk levels, with Stable Diffusion being the most prone to generating unsafe content (18.92% of all generated images are unsafe). Given Stable Diffusion's tendency to generate more unsafe content, we evaluate its potential to generate hateful meme variants if exploited by an adversary to attack a specific individual or community. We employ three image editing methods, DreamBooth, Textual Inversion, and SDEdit, which are supported by Stable Diffusion to generate variants. Our evaluation result shows that 24% of the generated images using DreamBooth are hateful meme variants that present the features of the original hateful meme and the target individual/community; these generated images are comparable to hateful meme variants collected from the real world. Overall, our results demonstrate that the danger of large-scale generation of unsafe images is imminent. We discuss several mitigating measures, such as curating training data, regulating prompts, and implementing safety filters, and encourage better safeguard tools to be developed to prevent unsafe generation.1 Our code is available at https://github.com/YitingQu/unsafe-diffusion.
- 1 March 2013
- Published by American Association of Avian Pathologists (AAAP) in Avian Diseases
- Vol. 57 (1) , 8-14
Infectivity and Pathogenicity of Newcastle Disease Virus Strains of Different Avian Origin and Different Virulence for Mallard Ducklings
SUMMARY. Experimental infections of Newcastle disease virus (NDV) strains of different avian origin and different virulence in mallard (Anas platyrhynchos) ducklings were undertaken to evaluate infectivity and pathogenicity of NDV for ducks and the potential role of ducks in the epidemiology of Newcastle disease (ND). Ducklings were experimentally infected with seven NDV strains, and their clinical sign, weight gain, antibody response, virus shedding, and virus distribution in tissues were investigated. The duck origin virulent strain duck/Jiangsu/JSD0812/2008 (JSD0812) and the Chinese standard virulent strain F48E8 were highly pathogenic for ducklings. They caused high morbidity and mortality, and they distributed extensively in various tissues of infected ducklings. Other strains, including pigeon origin virulent strain pigeon/Jiangsu/JSP0204/2002 (JSP0204), chicken origin virulent strain chicken/Jiangsu/JSC0804/2008 (JSC0804), goose origin virulent goose/Jiangsu/JSG0210/2002 (JSG0210), and vaccine stra... RESUMEN. Infectividad y patogenicidad de cepas del virus de la enfermedad de Newcastle de diferente origen aviar y su virulencia para patitos de collar. Se llevaron a cabo infecciones experimentales del virus de la enfermedad de Newcastle con cepas de diferente origen aviar y con diferencias en su virulencia para patitos de collar (Anas platyrhynchos). Se evaluó la infectividad y la patogenicidad del virus de Newcastle en patos y el posible papel de estas aves en la epidemiología de la enfermedad de Newcastle (ND). Se infectaron experimentalmente patitos con siete cepas del virus de Newcastle y se observaron los signos clínicos, el aumento de peso, la respuesta de anticuerpos, la eliminación del virus, y la distribución del virus en los tejidos. La cepa virulenta originada en patos duck/Jiangsu/JSD0812/2008 (JSD0812) y la cepa virulenta estándar de China fueron altamente patógenas para los patitos. Estas cepas causaron morbilidad y mortalidad altas, y se distribuyeron ampliamente en varios tejidos de los ...
- 1 March 2013
- Published by American Association of Avian Pathologists (AAAP) in Avian Diseases
- Vol. 57 (1) , 8-14
Infectivity and Pathogenicity of Newcastle Disease Virus Strains of Different Avian Origin and Different Virulence for Mallard Ducklings
SUMMARY. Experimental infections of Newcastle disease virus (NDV) strains of different avian origin and different virulence in mallard (Anas platyrhynchos) ducklings were undertaken to evaluate infectivity and pathogenicity of NDV for ducks and the potential role of ducks in the epidemiology of Newcastle disease (ND). Ducklings were experimentally infected with seven NDV strains, and their clinical sign, weight gain, antibody response, virus shedding, and virus distribution in tissues were investigated. The duck origin virulent strain duck/Jiangsu/JSD0812/2008 (JSD0812) and the Chinese standard virulent strain F48E8 were highly pathogenic for ducklings. They caused high morbidity and mortality, and they distributed extensively in various tissues of infected ducklings. Other strains, including pigeon origin virulent strain pigeon/Jiangsu/JSP0204/2002 (JSP0204), chicken origin virulent strain chicken/Jiangsu/JSC0804/2008 (JSC0804), goose origin virulent goose/Jiangsu/JSG0210/2002 (JSG0210), and vaccine stra... RESUMEN. Infectividad y patogenicidad de cepas del virus de la enfermedad de Newcastle de diferente origen aviar y su virulencia para patitos de collar. Se llevaron a cabo infecciones experimentales del virus de la enfermedad de Newcastle con cepas de diferente origen aviar y con diferencias en su virulencia para patitos de collar (Anas platyrhynchos). Se evaluó la infectividad y la patogenicidad del virus de Newcastle en patos y el posible papel de estas aves en la epidemiología de la enfermedad de Newcastle (ND). Se infectaron experimentalmente patitos con siete cepas del virus de Newcastle y se observaron los signos clínicos, el aumento de peso, la respuesta de anticuerpos, la eliminación del virus, y la distribución del virus en los tejidos. La cepa virulenta originada en patos duck/Jiangsu/JSD0812/2008 (JSD0812) y la cepa virulenta estándar de China fueron altamente patógenas para los patitos. Estas cepas causaron morbilidad y mortalidad altas, y se distribuyeron ampliamente en varios tejidos de los ...