Mattia Samory

Content validation, the assessment of whether measurement items adequately capture their intended constructs, is essential to scale development in organizational research but remains resource-intensive when conducted through conventional human-rating procedures. Natural language processing (NLP) offers a scalable complement for estimating definitional correspondence and distinctiveness, yet no study has systematically compared the full range of available approaches against direct human validity judgments across both multi-item and single-item measurement formats. The present study evaluates 16 NLP models spanning five architectural families (statistical models, static word embeddings, contextual sentence embeddings, task-specific interaction models and large language models) using two independent human-rated benchmark datasets. Results indicated that no single approach dominated across validity components or measurement formats, and that computational complexity was neither necessary nor sufficient for predictive accuracy. For multi-item correspondence, contextual sentence embeddings achieved top-group accuracy alongside supervised content-validation models; the choice of similarity metric substantially moderated performance for static embeddings. For multi-item distinctiveness, a broad upper group spanning all major architectural families achieved statistically indistinguishable accuracy. Single-item correspondence revealed a task-imposed performance ceiling with no architectural advantage. Practical recommendations are derived for each use case, prioritizing reproducibility, interpretability, and generalizability alongside predictive accuracy. A freely available web application implementing the recommended pipeline is introduced.

Reddit, one of the largest global social media platforms, has undergone significant transformations since its inception in 2005. From a loosely structured, niche platform to a globally recognized company with a standardized and regulated governance system, Reddit’s evolution has been marked by a shift in the power dynamics between its owners, moderators, and users. 2015 and 2023 were marked by the occurrence of two prominent protests, termed “blackouts.” Moderators of numerous subreddits, though not all, disabled public access to their subreddits, thereby protesting the company’s policies and policy changes and challenging the company’s endeavors to exert further control over the platform. Drawing on Bourdieusian theory and relational methodology, we establish a computational social science approach to investigate the structural causes behind the two blackouts and contextualize the differences between them. We argue that these blackouts signify growing tensions within the socio-technical space of Reddit and an ongoing political, cultural, and economic reconfiguration of its power structure and political economy.

News coverage profoundly affects how countries and individuals behave in international relations. Yet, we have little empirical evidence of how news coverage varies across countries. To enable studies of global news coverage, we develop an efficient computational methodology that comprises three components: (i) a transformer model to estimate multilingual news similarity; (ii) a global event identification system that clusters news based on a similarity network of news articles; and (iii) measures of news synchrony across countries and news diversity within a country, based on country-specific distributions of news coverage of the global events. Each component achieves state-of-the art performance, scaling seamlessly to massive datasets of millions of news articles. We apply the methodology to 60 million news articles published globally between January 1 and June 30, 2020, across 124 countries and 10 languages, detecting 4357 news events. We identify the factors explaining diversity and synchrony of news coverage across countries. Our study reveals that news media tend to cover a more diverse set of events in countries with larger Internet penetration, more official languages, larger religious diversity, higher economic inequality, and larger populations. Coverage of news events is more synchronized between countries that not only actively participate in commercial and political relations---such as, pairs of countries with high bilateral trade volume, and countries that belong to the NATO military alliance or BRICS group of major emerging economies---but also countries that share certain traits: an official language, high GDP, and high democracy indices.

The disruptive offline mobilization of participants in online conspiracy theory (CT) discussions has highlighted the importance of understanding how online users may form radicalized conspiracy beliefs. While prior work researched the factors leading up to joining online CT discussions and provided theories of how conspiracy beliefs form, we have little understanding of how conspiracy radicalization evolves after users join CT discussion communities. In this paper, we provide the empirical modeling of various radicalization phases in online CT discussion participants. To unpack how conspiracy engagement is related to radicalization, we first characterize the users' journey through CT discussions via conspiracy engagement pathways. Specifically, by studying 36K Reddit users through their 169M contributions, we uncover four distinct pathways of conspiracy engagement: steady high, increasing, decreasing, and steady low. We further model three successive stages of radicalization guided by prior theoretical works. Specific sub-populations of users, namely those on steady high and increasing conspiracy engagement pathways, progress successively through various radicalization stages. In contrast, users on the decreasing engagement pathway show distinct behavior: they limit their CT discussions to specialized topics, participate in diverse discussion groups, and show reduced conformity with conspiracy subreddits. By examining users who disengage from online CT discussions, this paper provides promising insights about conspiracy recovery process.

Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney, Przemyslaw Grabowicz, Scott Hale, David Jurgens, Mattia Samory. Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022). 2022.