QFQixiang Fang

Qixiang Fang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Qixiang Fang has a total of 70 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    How Much Labelled Data Is Enough? A Simulation Study of Prompting Strategies for Unbiased and Reliable LLM-Based Annotation Workflows in Social Sciences

    by Qixiang Fang

    Large language models (LLMs) are increasingly used in (computational) social science to annotate text at scale. Typically, this approach requires researchers to provide (enough) gold labels, develop and evaluate candidate prompts, and decide on the most promising prompt. Meanwhile, many questions arise and remain unanswered: How much labelled data is enough? How should prompt evaluation be conducted? In particular, how can researchers prevent overfitting, an important consideration many social scientists are unfamiliar with. To address these questions, this paper reports a simulation study of prompt evaluation workflows for LLM-based text annotation. I compare intentionally biased yet often employed all-data reuse and validation-only baseline strategies with two defensible workflows: a two-way development/evaluation split and a three-way Explore--Select--Evaluate (ESE) workflow. Across 360 simulation scenarios and 360,000 workflow-level simulation results, all-data reuse and validation-only workflows produced positive reporting bias, especially under adaptive prompt development. In contrast, both two-way and ESE workflows had near-zero mean reporting bias and close-to-95% coverage. However, ESE did not generally select stronger prompts. In fixed-candidate settings, two-way selection outperformed ESE because ESE retained less data for formal prompt comparison without gaining any benefit from exploration. In adaptive settings, ESE reduced regret only modestly under some split ratios, while two-way workflows usually selected prompts with higher absolute true performance. The results suggest that a held-out evaluation set is essential, but that a separate Select stage should be used selectively rather than treated as universally optimal.

  • Open Access

    A Methodological Guide on Using Large Language Models for Text Annotation in the Social Sciences and Humanities with Python and R

    by Qixiang Fang, Javier Garcia-Bernardo, Erik-Jan van Kesteren

    Large language models (LLMs) have become an essential tool for social science and humanities (SSH) researchers who work with text. One particularly valuable application is automating text annotation, a traditionally time-consuming step in preparing data for empirical analysis. Yet many SSH researchers face two challenges: getting started with LLMs and understanding how to address their limitations. Practically, the rapid pace of model development can make LLMs seem inaccessible or intimidating, while even experienced users may overlook how annotation errors can bias downstream statistical analyses (e.g., regression estimates and $p$-values), even when annotation accuracy appears high.This paper provides a comprehensive, step-by-step methodological guide for using LLMs for text annotation in SSH research, with clear Python and R code snippets. We cover (1) how LLMs work and what they can and cannot do; (2) how to identify an LLM-suitable research project and establish minimum data and computational requirements; (3) how to design prompts and run annotation tasks; (4) how to evaluate annotation quality and iteratively refine prompts without overfitting; (5) how to integrate LLM annotations into downstream statistical analyses while accounting for annotation error; and (6) how to manage cost, efficiency, and reproducibility when scaling up annotation. Throughout, we provide clear and intuitive methodological reasoning, concrete examples, code snippets, and best-practice guidance to help researchers confidently and transparently incorporate LLM-based annotation into their scientific workflows.


  • Open Access

    Screenathon 2.0: human–AI collaborative screening applied to patient-generated health data

    by Jonas Bergmann, Tiago Azzi, Rutger Neeleman, Kianush Monschau, Berke Yazan, Elena Jalsovec, Emily Westerbeek, Felix Weijdema, Jonathan de Bruin, Qixiang Fang, et al.

    Systematic reviews are essential for evidence-based research, yet the traditional screening process is time-consuming and difficult to scale. Human-only screening can introduce inconsistency, while fully automated approaches employing Large Language Models often lack the contextual judgement required for complex decisions. To address this, we introduce a crowd-based screening methodology that integrates human expertise with adaptive machine learning. The methods have been applied in the context of a large EU project where experts from 27 collaborating partners jointly screened 5842 papers across eleven disease topics related to patient-generated health data in a span of 2 days. Post-processing played a central role in ensuring data quality, including topic reallocation, targeted full-text verification, and noisy-label filtering. This Screenathon resulted in 487 records being labeled as relevant and 6,463 records as irrelevant. The number of records screened per participant ranged from 3 to 2496, with a mean of 216.4 records per screener ( SE = 95.19). Exploratory analyses using survey results indicated increased trust in AI-assisted systematic reviewing after the event, along with generally positive evaluations of usability. The current Screenathon demonstrates that crowdsourced human–AI collaboration requires thoughtful training and calibration, together with strong post-processing safeguards.


  • Open Access

    Screenathon 2.0: Human–AI Collaborative Screening Applied to Patient-Generated Health Data

    by Jonas Bergmann, Tiago Azzi, Rutger Chris Neeleman, Kianush Monschau, Elena Jalsovec, Emily Westerbeek, Felix Weijdema, Jonathan de Bruin, Qixiang Fang, Rens van de Schoot, et al.

    Systematic reviews are essential for evidence-based research, yet the traditional screening process is time-consuming and difficult to scale. Human-only screening can introduce inconsistency, while fully automated approaches employing Large Language Models often lack the contextual judgement required for complex decisions. To address this, we introduce a crowd-based screening methodology that integrates human expertise with adaptive machine learning. The methods have been applied in the context of a large EU project where experts from 27 collaborating partners jointly screened 5,842 papers across eleven disease topics related to patient-generated health data in a span of only 2 days. Post-processing played a central role in ensuring data quality, including topic reallocation, targeted full-text verification, and noisy-label filtering. This Screenathon resulted in 487 records being labeled as relevant and 6,463 records as irrelevant. The number of records screened per participant ranged from 3 to 2,496, with a mean of 216.4 records per screener (SE = 95.19). Survey results indicated increased trust in AI-assisted systematic reviewing after the event, along with generally positive evaluations of usability. The current Screenathon demonstrates that human–AI collaboration can increase efficiency while maintaining rigor, provided that workflows include thoughtful training and calibration together with strong post-processing safeguards.


  • Open Access

    Data from the IMPROVE Project: Labelling Decisions and Metadata from an Umbrella Review on Patient-Generated Health Data in Digital Health

    by Kianush Monschau, Emily Westerbeek, Qixiang Fang, Jonas Bergmann, Giuseppe Fico, Manuel Ottaviano, Liss Hernandez, Hans Peeters, Ernst Hermens, Davide Guerri, et al.

    The EU-funded IMPROVE project seeks to build a platform that enables the smart use of patient-generated health data (PGHD), thereby enhancing digital healthcare efficiency. This paper introduces a dataset directly resulting from a large-scale systematic literature search carried out within the IMPROVE project, concerning the use of PGHD in treatments of five categories of chronic diseases: cardiovascular disease, chronic inflammation, neurology, oncology, and ophthalmology. Specifically, the dataset contains the metadata of 12.473 studies returned by databases, such as title, abstract and DOI, as well as inclusion decisions. The dataset is stored in both excel and csv format. It can be used to aid the understanding of PGHD use in healthcare and the development of AI-assisted systematic review tools.

  • Leveraging Measurement Theory for Natural Language Processing Research

    by Qixiang Fang

    This dissertation explores the intersection of natural language processing (NLP) and measurement theory. NLP is a field aimed at enabling machines to process and generate human languages, such as English, German, and Mandarin. These languages are complex, diverse, and full of irregularities, making them challenging for machines to handle compared to structured artificial languages, like programming languages. NLP research ranges from simple tasks, like word frequency analysis, to complex ones involving the understanding and generation of human language. Measurement theory, traditionally applied in social sciences, addresses how we measure various properties scientifically. Key concepts include construct validity, which examines whether a measure accurately represents what it intends to measure, and reliability, which focuses on the consistency of a measure across different conditions. This dissertation argues that many challenges in NLP relate to measurement issues and suggests that principles from measurement theory can help address these challenges, particularly by providing tools to evaluate and improve the quality of NLP models. The structure of the dissertation is as follows: Chapter 2 offers background on NLP and measurement theory, covering essential text representation techniques in NLP, the history of measurement theory, and recent discussions on its unification across fields. Chapters 3-5 apply measurement theory to evaluate NLP models: Chapter 3 explores the reliability of gender bias measures in NLP by using classical reliability estimators from social sciences. Chapter 4 adapts a construct validity testing framework to assess the quality of text representations for social science constructs. Chapter 5 introduces a psychometric-based benchmarking approach to evaluate large language models, demonstrated through a case study on eighth-grade math proficiency. Chapters 6-7 focus on using measurement theory to improve NLP model performance: Chapter 6 presents a framework for designing user models based on measurement principles, achieving better-quality user representations than current methods. Chapter 7 examines how integrating human values into model training can enhance models’ ability to recognize values in human arguments. Chapters 8 and 9 reflect on current NLP research challenges and propose future directions: Chapter 8 identifies challenges in text-based personality computing, offering potential solutions and avenues for research. Chapter 9 concludes with a summary of the dissertation’s findings and suggests future work at the intersection of measurement theory and NLP. This work underscores the potential of measurement theory to enhance NLP research by offering frameworks for evaluating and designing more reliable and valid models. By integrating these approaches, the dissertation aims to bridge NLP and measurement theory, advancing NLP's capability to address complex measurement challenges.

  • Open Access

    General-Purpose User Modeling with Behavioral Logs: A Snapchat Case Study

    by Qixiang Fang, Zhihan Zhou, Francesco Barbieri, Yozen Liu, Leonardo Neves, Dong Nguyen, Daniel Oberski, Maarten Bos, Ron Dotsch

    Learning general-purpose user representations based on user behavioral logs is an increasingly popular user modeling approach. It benefits from easily available, privacy-friendly yet expressive data, and does not require extensive re-tuning of the upstream user model for different downstream tasks. While this approach has shown promise in search engines and e-commerce applications, its fit for instant messaging platforms, a cornerstone of modern digital communication, remains largely uncharted. We explore this research gap using Snapchat data as a case study. Specifically, we implement a Transformer-based user model with customized training objectives and show that the model can produce high-quality user representations across a broad range of evaluation tasks, among which we introduce three new downstream tasks that concern pivotal topics in user research: user safety, engagement and churn. We also tackle the challenge of efficient extrapolation of long sequences at inference time, by applying a novel positional encoding method.


  • Open Access

    Epicurus at SemEval-2023 Task 4: Improving Prediction of Human Values behind Arguments by Leveraging Their Definitions

    by Christian Fang, Qixiang Fang, Dong Nguyen

    We describe our experiments for SemEval-2023 Task 4 on the identification of human values behind arguments (ValueEval). Because human values are subjective concepts which require precise definitions, we hypothesize that incorporating the definitions of human values (in the form of annotation instructions and validated survey items) during model training can yield better prediction performance. We explore this idea and show that our proposed models perform better than the challenge organizers' baselines, with improvements in macro F1 scores of up to 18%.

  • Open Access

    Modelling Stance Detection as Textual Entailment Recognition and Leveraging Measurement Knowledge from Social Sciences

    by Qixiang Fang, Anastasia Giachanou, Ayoub Bagheri

    Stance detection (SD) can be considered a special case of textual entailment recognition (TER), a generic natural language task. Modelling SD as TER may offer benefits like more training data and a more general learning scheme. In this paper, we present an initial empirical analysis of this approach. We apply it to a difficult but relevant test case where no existing labelled SD dataset is available, because this is where modelling SD as TER may be especially helpful. We also leverage measurement knowledge from social sciences to improve model performance. We discuss our findings and suggest future research directions.

  • Open Access

    Evaluating the construct validity of text embeddings with application to survey questions

    by Qixiang Fang, Dong Nguyen, Daniel L. Oberski

    Text embedding models from Natural Language Processing can map text data (e.g. words, sentences, documents) to meaningful numerical representations (a.k.a. text embeddings). While such models are increasingly applied in social science research, one important issue is often not addressed: the extent to which these embeddings are high-quality representations of the information needed to be encoded. We view this quality evaluation problem from a measurement validity perspective, and propose the use of the classic construct validity framework to evaluate the quality of text embeddings. First, we describe how this framework can be adapted to the opaque and high-dimensional nature of text embeddings. Second, we apply our adapted framework to an example where we compare the validity of survey question representation across text embedding models.