Louis Hickman
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Louis Hickman has a total of 353 co-authors. Here are the top co-authors by number of shared publications:
- Louis Tay
Department of Psychological Sciences, Purdue University, West Lafayette, Indiana, USA
11 shared publications Louis Tay
6 shared publications- Sang Eun Woo
Department of Psychological Sciences Purdue University West Lafayette Indiana USA
5 shared publications Vincent Ng
Department of Psychology, University of Houston, Houston, TX, United States
5 shared publications- Brandon Michael Booth
Department of Computer Science, University of Memphis, Memphis, Tennessee, USA
4 shared publications - Mesut Akdere
Purdue Polytechnic Institute Purdue University West Lafayette Indiana USA
3 shared publications
Latest Publications
- Open Access
- 26 May 2026
- Published by Center for Open Science in PsyArXiv
Assessing individual differences in adult theory of mind through mental state language: Human and artificial intelligence approaches
Theory of mind is central to human social behavior, yet empirical studies rely largely on biased measures with limited ecological validity. In the current manuscript, we propose and evaluate the use of mental state language as a spontaneous indicator of theory of mind expressed in social interaction. Among research participants (334 undergraduate students, 63.8% non-Hispanic White, 74.3% female) engaged in a dyadic cooperative task with an unfamiliar peer, mental state language demonstrated strong convergent validity with established measures of theory of mind and, when compared to established measures, superior predictive validity for interpersonal functioning. Using natural language processing, we demonstrated that automated models can reliably replicate human coding of mental state language, enabling scalable assessment. These findings suggest that mental state language is an empirically grounded and ecologically valid method for assessing theory of mind in dynamic everyday social interactions.
- 26 May 2026
- Published by Center for Open Science in PsyArXiv
Open AccessAssessing individual differences in adult theory of mind through mental state language: Human and artificial intelligence approaches
Theory of mind is central to human social behavior, yet empirical studies rely largely on biased measures with limited ecological validity. In the current manuscript, we propose and evaluate the use of mental state language as a spontaneous indicator of theory of mind expressed in social interaction. Among research participants (334 undergraduate students, 63.8% non-Hispanic White, 74.3% female) engaged in a dyadic cooperative task with an unfamiliar peer, mental state language demonstrated strong convergent validity with established measures of theory of mind and, when compared to established measures, superior predictive validity for interpersonal functioning. Using natural language processing, we demonstrated that automated models can reliably replicate human coding of mental state language, enabling scalable assessment. These findings suggest that mental state language is an empirically grounded and ecologically valid method for assessing theory of mind in dynamic everyday social interactions.
- Open Access
- 27 April 2026
- Published by SAGE Publications in Organizational Research Methods
Textual Similarity in Organizational Research: Review of Applications, Consistency of Methods, and Best Practice Recommendations
Organizational research increasingly uses natural language processing (NLP) to measure textual similarity. Despite common usage, the meaning and consistency of similarity measures (e.g., cosine similarity and Euclidean distance) across common NLP methods (e.g., n -grams and document embeddings) is unclear. This risks misalignment between theoretical constructs and textual measures, undermining the comparability of findings across studies. To address this gap, we review studies using textual similarity in organizational and psychological research, finding a jingle-jangle fallacy: identical labels are used for similarity estimates from different NLP methods, and different labels are used for the same method. Additionally, we examine the consistency of similarity measures across and within NLP methods. Different transformer-based embeddings’ similarity results are interchangeable. However, n -grams yield distinct, inconsistent results and are less appropriate for estimating similarity with distance measures. When applied to multi-word inputs, dictionaries and word embeddings return similar results reflecting linguistic style. We provide best practice recommendations and example code for operationalizing textual similarity, including clarifying which NLP methods correspond to content similarity, linguistic style similarity, and semantic similarity at the word, sentence, and document-levels of analysis.
- 27 April 2026
- Published by SAGE Publications in Organizational Research Methods
Open AccessTextual Similarity in Organizational Research: Review of Applications, Consistency of Methods, and Best Practice Recommendations
Organizational research increasingly uses natural language processing (NLP) to measure textual similarity. Despite common usage, the meaning and consistency of similarity measures (e.g., cosine similarity and Euclidean distance) across common NLP methods (e.g., n -grams and document embeddings) is unclear. This risks misalignment between theoretical constructs and textual measures, undermining the comparability of findings across studies. To address this gap, we review studies using textual similarity in organizational and psychological research, finding a jingle-jangle fallacy: identical labels are used for similarity estimates from different NLP methods, and different labels are used for the same method. Additionally, we examine the consistency of similarity measures across and within NLP methods. Different transformer-based embeddings’ similarity results are interchangeable. However, n -grams yield distinct, inconsistent results and are less appropriate for estimating similarity with distance measures. When applied to multi-word inputs, dictionaries and word embeddings return similar results reflecting linguistic style. We provide best practice recommendations and example code for operationalizing textual similarity, including clarifying which NLP methods correspond to content similarity, linguistic style similarity, and semantic similarity at the word, sentence, and document-levels of analysis.
- Open Access
- 1 April 2026
- Published by Center for Open Science in MetaArXiv
Investigating the replicability of the social and behavioural sciences
We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.
- 1 April 2026
- Published by Center for Open Science in MetaArXiv
Open AccessInvestigating the replicability of the social and behavioural sciences
We attempted replications of 274 claims of positive results from 164 quantitative papers published from 2009 to 2018 in 54 journals in the social and behavioral sciences. Replications were high-powered on average to detect the original effect size (Median = 99.6%), used original materials when relevant and available, and were peer-reviewed in advance through a standardized internal protocol. Replications showed statistically significant results in the original pattern for 151 of 274 claims (55.1% [95% CI 49.2 - 60.9%]) and for 80.8 of 164 papers (49.3% [95% CI 43.8 - 54.7%]) weighed for replicating multiple claims per paper. We observed modest variation in replication rates across disciplines (42.5% to 63.1%) though some estimates had high uncertainty. For claims with Pearson’s r effect sizes, the median was 0.25 [95% CI 0.21 - 0.27] for original studies and 0.10 [95% CI 0.09 - 0.13] for replication studies, a 82.4% [95% CI 67.8 - 88.2%] reduction in shared variance. Thirteen methods for evaluating replication success provided estimates ranging from 28.6% to 74.8% (median = 49.3%). Some decline in effect size and significance is expected based on power to detect original effects and regression to the mean due to replicating only positive results. The conditions that promote or inhibit replicability are worthy of additional investigation.
- 15 December 2025
- Published by American Psychological Association (APA) in Journal of Applied Psychology
When are item nuances useful for prediction in organizations? Comparing the validity of item-level, scale-level, and ensemble machine learning models.
Recent evidence suggests that prediction models using item scores, instead of the traditionally used scale scores, can more accurately predict outcomes of interest. However, little is known about the conditions under which item- or scale-level models are more suitable for prediction in organizational practice. To address this gap, we examined several real-world data sets for the presence of valid item nuances (i.e., criterion-valid item-specific variance that is lost when aggregating to scale scores). We then designed and conducted a Monte Carlo simulation based on empirical estimates to investigate the criterion-related validity of item- and scale-level models. In the simulation, we varied (a) the distribution of nuances among items, (b) the effect size of nuances, (c) the effect size of constructs, (d) scale internal consistency, and (e) training sample size. Results suggested that item-level models are recommended when relatively few items in a scale carry nuances, the nuance effect sizes are similar or larger in magnitude to the scale effect sizes, internal consistency is high, and training sample size is large. Our review of prior studies and analyses of real-world data suggest that the conditions favoring item-level models are not uncommon in organizational data. For conditions that do not substantiate the use of either item- or scale-level models, we also examined ensemble models and found that they can be an attractive alternative when the choice is unclear. We provide R code and recommendations for examining valid nuances in one's data and developing item-level, scale-level, and ensemble predictive models. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
- 15 December 2025
- Published by American Psychological Association (APA) in Journal of Applied Psychology
When are item nuances useful for prediction in organizations? Comparing the validity of item-level, scale-level, and ensemble machine learning models.
Recent evidence suggests that prediction models using item scores, instead of the traditionally used scale scores, can more accurately predict outcomes of interest. However, little is known about the conditions under which item- or scale-level models are more suitable for prediction in organizational practice. To address this gap, we examined several real-world data sets for the presence of valid item nuances (i.e., criterion-valid item-specific variance that is lost when aggregating to scale scores). We then designed and conducted a Monte Carlo simulation based on empirical estimates to investigate the criterion-related validity of item- and scale-level models. In the simulation, we varied (a) the distribution of nuances among items, (b) the effect size of nuances, (c) the effect size of constructs, (d) scale internal consistency, and (e) training sample size. Results suggested that item-level models are recommended when relatively few items in a scale carry nuances, the nuance effect sizes are similar or larger in magnitude to the scale effect sizes, internal consistency is high, and training sample size is large. Our review of prior studies and analyses of real-world data suggest that the conditions favoring item-level models are not uncommon in organizational data. For conditions that do not substantiate the use of either item- or scale-level models, we also examined ensemble models and found that they can be an attractive alternative when the choice is unclear. We provide R code and recommendations for examining valid nuances in one's data and developing item-level, scale-level, and ensemble predictive models. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
- Open Access
- 19 August 2025
- Published by Wiley in Applied Psychology
- Vol. 74 (4)
The role of intergroup threats for explaining political bias in screening decisions
In many places, the current political landscape is increasingly polarized. Unfortunately, such polarization may cause organizations to miss out on valuable human capital if their recruiters and hiring managers exhibit political bias. This research draws on integrated threat theory to investigate the implications of political polarization on applicant evaluations when information about their political views can be gleaned from cues in their application materials or social media. Through four lab studies (total N = 4223) involving Prolific participants with real‐world hiring experience, we explore how subtle cues about an applicant's inferred political beliefs influence screening decisions. Study 1 indicates that political bias manifests primarily as a bias against those with differing political views rather than favoritism toward those with similar views. Drawing from integrated threat theory, Studies 2–3 find that political dissimilarity between applicants and hiring managers affects evaluations through perceived intergroup threats, particularly symbolic threats arising from conflicting values and beliefs. Study 3 reveals that political bias persists even when candidates are clearly highly qualified for the position. Studies 1–3 used COVID‐related employment gaps, and Study 4 shows that these effects persist when evaluators are exposed to the applicant's views about abortion. We discuss how and when political bias may surface and offer suggestions for organizations to mitigate these biases to leverage the available human capital in their applicant pools.
- 19 August 2025
- Published by Wiley in Applied Psychology
- Vol. 74 (4)
Open AccessThe role of intergroup threats for explaining political bias in screening decisions
In many places, the current political landscape is increasingly polarized. Unfortunately, such polarization may cause organizations to miss out on valuable human capital if their recruiters and hiring managers exhibit political bias. This research draws on integrated threat theory to investigate the implications of political polarization on applicant evaluations when information about their political views can be gleaned from cues in their application materials or social media. Through four lab studies (total N = 4223) involving Prolific participants with real‐world hiring experience, we explore how subtle cues about an applicant's inferred political beliefs influence screening decisions. Study 1 indicates that political bias manifests primarily as a bias against those with differing political views rather than favoritism toward those with similar views. Drawing from integrated threat theory, Studies 2–3 find that political dissimilarity between applicants and hiring managers affects evaluations through perceived intergroup threats, particularly symbolic threats arising from conflicting values and beliefs. Study 3 reveals that political bias persists even when candidates are clearly highly qualified for the position. Studies 1–3 used COVID‐related employment gaps, and Study 4 shows that these effects persist when evaluators are exposed to the applicant's views about abortion. We discuss how and when political bias may surface and offer suggestions for organizations to mitigate these biases to leverage the available human capital in their applicant pools.
- Top Cited PapersOpen Access
- 1 July 2025
- Published by American Psychological Association (APA) in Journal of Applied Psychology
- Vol. 110 (7) , 979-1000
Whither bias goes, I will go: An integrative, systematic review of algorithmic bias mitigation.
- 1 July 2025
- Published by American Psychological Association (APA) in Journal of Applied Psychology
- Vol. 110 (7) , 979-1000
Top Cited PapersOpen AccessWhither bias goes, I will go: An integrative, systematic review of algorithmic bias mitigation.
- 1 June 2025
- Published by American Psychological Association (APA) in Journal of Applied Psychology
- Vol. 110 (6) , 846-858
Automated speech recognition bias in personnel selection: The case of automatically scored job interviews.
Organizations, researchers, and software increasingly use automatic speech recognition (ASR) to transcribe speech to text. However, ASR can be less accurate for (i.e., biased against) certain demographic subgroups. This is concerning, given that the machine-learning (ML) models used to automatically score video interviews use ASR transcriptions of interviewee responses as inputs. To address these concerns, we investigate the extent of ASR bias and its effects in automatically scored interviews. Specifically, we compare the accuracy of ASR transcription for English as a second language (ESL) versus non-ESL interviewees, people of color (and Black interviewees separately) versus White interviewees, and male versus female interviewees. Then, we test whether ASR bias causes bias in ML model scores-both in terms of differential convergent correlations (i.e., subgroup differences in correlations between observed and ML scores) and differential means (i.e., shifts in subgroup differences from observed to ML scores). To do so, we apply one human and four ASR transcription methods to two samples of mock video interviews (Ns = 1,014 and 414), and then we train and test models using these different transcripts to score multiple constructs. We observed significant bias in the commercial ASR services across nearly all comparisons, with the magnitude of bias differing across the ASR services. However, the transcription bias did not translate into meaningful measurement bias for the ML interview scores-whether in terms of differential convergent correlations or means. We discuss what these results mean for the nature of bias, fairness, and validity of ML models for scoring verbal open-ended responses. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
- 1 June 2025
- Published by American Psychological Association (APA) in Journal of Applied Psychology
- Vol. 110 (6) , 846-858
Automated speech recognition bias in personnel selection: The case of automatically scored job interviews.
Organizations, researchers, and software increasingly use automatic speech recognition (ASR) to transcribe speech to text. However, ASR can be less accurate for (i.e., biased against) certain demographic subgroups. This is concerning, given that the machine-learning (ML) models used to automatically score video interviews use ASR transcriptions of interviewee responses as inputs. To address these concerns, we investigate the extent of ASR bias and its effects in automatically scored interviews. Specifically, we compare the accuracy of ASR transcription for English as a second language (ESL) versus non-ESL interviewees, people of color (and Black interviewees separately) versus White interviewees, and male versus female interviewees. Then, we test whether ASR bias causes bias in ML model scores-both in terms of differential convergent correlations (i.e., subgroup differences in correlations between observed and ML scores) and differential means (i.e., shifts in subgroup differences from observed to ML scores). To do so, we apply one human and four ASR transcription methods to two samples of mock video interviews (Ns = 1,014 and 414), and then we train and test models using these different transcripts to score multiple constructs. We observed significant bias in the commercial ASR services across nearly all comparisons, with the magnitude of bias differing across the ASR services. However, the transcription bias did not translate into meaningful measurement bias for the ML interview scores-whether in terms of differential convergent correlations or means. We discuss what these results mean for the nature of bias, fairness, and validity of ML models for scoring verbal open-ended responses. (PsycInfo Database Record (c) 2025 APA, all rights reserved).
- Open Access
- 6 May 2025
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 33 (2)
Can Interviewees Fake Out AI? Comparing the Susceptibility and Mechanisms of Faking Across Self‐Reports, Human Interview Ratings, and AI Interview Ratings
Artificial intelligence (AI) is increasingly used to score employment interviews in the early stages of the hiring process, but AI algorithms may be particularly prone to interviewee faking. Our study compared the extent to which people can improve their scores on self‐report scales, structured and less structured human interview ratings, and AI interview ratings. Further, we replicate and extend prior research by examining how interviewee abilities and impression management tactics influence score inflation across scoring methods. Participants ( N = 152) completed simulated, asynchronous interviews in honest and applicant‐like conditions in a within‐subjects design. The AI algorithms in the study were trained to replicate question‐level structured interview ratings. Participants' scores increased most on self‐reports (overall Cohen's d = 0.62) and least on AI interview ratings (overall Cohen's d = 0.14), although AI score increases were similar to those observed for human interview ratings (overall Cohen's d = 0.22). On average, across conditions, AI interview ratings converged more strongly with structured human ratings based on behaviorally anchored rating scales than with less structured human ratings. Verbal ability only predicted score improvement on self‐reports, while increased use of honest defensive impression management tactics predicted improvement in AI and less structured human interview scores. Ability to identify criteria did not predict score improvement. Overall, these AI interview scores behaved similarly to structured human ratings. We discuss future possibilities for investigating faking in AI interviews, given that interviewees may try to “game” the system when aware that they are being evaluated by AI.
- 6 May 2025
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 33 (2)
Open AccessCan Interviewees Fake Out AI? Comparing the Susceptibility and Mechanisms of Faking Across Self‐Reports, Human Interview Ratings, and AI Interview Ratings
Artificial intelligence (AI) is increasingly used to score employment interviews in the early stages of the hiring process, but AI algorithms may be particularly prone to interviewee faking. Our study compared the extent to which people can improve their scores on self‐report scales, structured and less structured human interview ratings, and AI interview ratings. Further, we replicate and extend prior research by examining how interviewee abilities and impression management tactics influence score inflation across scoring methods. Participants ( N = 152) completed simulated, asynchronous interviews in honest and applicant‐like conditions in a within‐subjects design. The AI algorithms in the study were trained to replicate question‐level structured interview ratings. Participants' scores increased most on self‐reports (overall Cohen's d = 0.62) and least on AI interview ratings (overall Cohen's d = 0.14), although AI score increases were similar to those observed for human interview ratings (overall Cohen's d = 0.22). On average, across conditions, AI interview ratings converged more strongly with structured human ratings based on behaviorally anchored rating scales than with less structured human ratings. Verbal ability only predicted score improvement on self‐reports, while increased use of honest defensive impression management tactics predicted improvement in AI and less structured human interview scores. Ability to identify criteria did not predict score improvement. Overall, these AI interview scores behaved similarly to structured human ratings. We discuss future possibilities for investigating faking in AI interviews, given that interviewees may try to “game” the system when aware that they are being evaluated by AI.
- 25 March 2025
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 33 (2)
Asynchronous Video Interviews in Recruitment and Selection: Lights, Camera, Action!
- 25 March 2025
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 33 (2)
Asynchronous Video Interviews in Recruitment and Selection: Lights, Camera, Action!
- Open Access
- 4 March 2025
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 33 (2)
All Your Base Are Belong to Us: The Urgent Reality of Unproctored Testing in the Age of LLMs
The release of new generative artificial intelligence (AI) tools, including new large language models (LLMs), continues at a rapid pace. Upon the release of OpenAI's new o1 models, I reconducted Hickman et al.'s (2024) analyses examining how well LLMs perform on a quantitative ability (number series) test. GPT‐4 scored below the 20th percentile (compared to thousands of human test takers), but o1 scored at the 95th percentile. In response to these updated findings and Lievens and Dunlop's (2025) article about the effects of LLMs on the validity of pre‐employment assessments, I make an urgent call to action for selection and assessment researchers and practitioners. A recent survey suggests that a large proportion of applicants are already using generative AI tools to complete high‐stakes assessments, and it seems that no current assessments will be safe for long. Thus, I offer possibilities for the future of testing, detail their benefits and drawbacks, and provide recommendations. These possibilities are: increased use of proctoring, adding strict time limits, using LLM detection software, using think‐aloud (or similar) protocols, collecting and analyzing trace data, emphasizing samples over signs, and redesigning assessments to allow LLM use during completion. Several of these possibilities inspire future research to modernize assessment. Future research should seek to improve our understanding of how to design valid assessments that allow LLM use, how to effectively use trace test‐taker data, and whether think‐aloud protocols can help differentiate experts and novices.
- 4 March 2025
- Published by Wiley in International Journal of Selection and Assessment
- Vol. 33 (2)
Open AccessAll Your Base Are Belong to Us: The Urgent Reality of Unproctored Testing in the Age of LLMs
The release of new generative artificial intelligence (AI) tools, including new large language models (LLMs), continues at a rapid pace. Upon the release of OpenAI's new o1 models, I reconducted Hickman et al.'s (2024) analyses examining how well LLMs perform on a quantitative ability (number series) test. GPT‐4 scored below the 20th percentile (compared to thousands of human test takers), but o1 scored at the 95th percentile. In response to these updated findings and Lievens and Dunlop's (2025) article about the effects of LLMs on the validity of pre‐employment assessments, I make an urgent call to action for selection and assessment researchers and practitioners. A recent survey suggests that a large proportion of applicants are already using generative AI tools to complete high‐stakes assessments, and it seems that no current assessments will be safe for long. Thus, I offer possibilities for the future of testing, detail their benefits and drawbacks, and provide recommendations. These possibilities are: increased use of proctoring, adding strict time limits, using LLM detection software, using think‐aloud (or similar) protocols, collecting and analyzing trace data, emphasizing samples over signs, and redesigning assessments to allow LLM use during completion. Several of these possibilities inspire future research to modernize assessment. Future research should seek to improve our understanding of how to design valid assessments that allow LLM use, how to effectively use trace test‐taker data, and whether think‐aloud protocols can help differentiate experts and novices.