James M. LeBreton
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
James M. LeBreton has a total of 133 co-authors. Here are the top co-authors by number of shared publications:
Lawrence R. James
17 shared publications- Jesse S. Michel
Auburn University
6 shared publications Kelly T. Scherer
Purdue University
5 shared publications- Scott Tonidandel
Davidson College
5 shared publications Scott Tonidandel
Department of Psychology, Davidson College, Davidson, NC, USA
4 shared publicationsDace S. Svikis
Virginia Commonwealth University; Richmond; VA; USA
3 shared publications
Latest Publications
- Open Access
- 11 March 2026
- Published by Center for Open Science in PsyArXiv
AutoMetaCoder: An AI Agent for Meta-analysis Coding
Coding in Meta-analysis is a critical component in cumulative scientific research. However, it is labor-intensive and susceptible to human error. With recent advances in large language models (LLMs), this preprint introduces AutoMetaCoder, an open-source AI agent designed to support meta-analytic coding through a structured, rule-governed, and human-in-the-loop workflow. AutoMetaCoder decomposes coding into modular stages, including study screening, multi-level information extraction, and standardized output organization for downstream analyses, while explicitly logging interactions between researcher inputs and model outputs to enhance transparency and auditability. We provide a detailed walkthrough of the system and present initial validation results based on published psychological meta-analyses. AutoMetaCoder achieves high retrieval and accuracy rates (97% overall) when extracting paper, sample, variable, and effect size related information across 55 primary studies with over 5,000 coding entries. These findings suggest that agentic AI workflows can substantially reduce manual coding burden while preserving methodological rigor and transparency in meta-analysis.
- 11 March 2026
- Published by Center for Open Science in PsyArXiv
Open AccessAutoMetaCoder: An AI Agent for Meta-analysis Coding
Coding in Meta-analysis is a critical component in cumulative scientific research. However, it is labor-intensive and susceptible to human error. With recent advances in large language models (LLMs), this preprint introduces AutoMetaCoder, an open-source AI agent designed to support meta-analytic coding through a structured, rule-governed, and human-in-the-loop workflow. AutoMetaCoder decomposes coding into modular stages, including study screening, multi-level information extraction, and standardized output organization for downstream analyses, while explicitly logging interactions between researcher inputs and model outputs to enhance transparency and auditability. We provide a detailed walkthrough of the system and present initial validation results based on published psychological meta-analyses. AutoMetaCoder achieves high retrieval and accuracy rates (97% overall) when extracting paper, sample, variable, and effect size related information across 55 primary studies with over 5,000 coding entries. These findings suggest that agentic AI workflows can substantially reduce manual coding burden while preserving methodological rigor and transparency in meta-analysis.
- Open Access
- 20 July 2024
- Published by Elsevier in European Management Journal
- Vol. 42 (4) , 452-461
Multilevel empirical research: A call for more mixed-methods approaches
- 20 July 2024
- Published by Elsevier in European Management Journal
- Vol. 42 (4) , 452-461
Open AccessMultilevel empirical research: A call for more mixed-methods approaches
- 24 November 2022
- Published by Springer Nature in Journal of Business and Psychology
- Vol. 38 (2) , 239-258
Data Aggregation in Multilevel Research: Best Practice Recommendations and Tools for Moving Forward
- 24 November 2022
- Published by Springer Nature in Journal of Business and Psychology
- Vol. 38 (2) , 239-258
Data Aggregation in Multilevel Research: Best Practice Recommendations and Tools for Moving Forward
- Open Access
- 9 September 2022
- Published by Center for Open Science in PsyArXiv
Rejoinder to Commentaries on the Woo et al. (2022) Article
We respond to four commentaries (authored by Boykin; Gómez; Kuncel & Worrell; Nye & Ryan) on the Woo et al. (2022) article: Woo, S. E., LeBreton, J., Keith, M., & Tay, L. (in press). Bias, fairness, and validity in graduate admissions: A psychometric perspective. Perspectives on Psychological Science.
- 9 September 2022
- Published by Center for Open Science in PsyArXiv
Open AccessRejoinder to Commentaries on the Woo et al. (2022) Article
We respond to four commentaries (authored by Boykin; Gómez; Kuncel & Worrell; Nye & Ryan) on the Woo et al. (2022) article: Woo, S. E., LeBreton, J., Keith, M., & Tay, L. (in press). Bias, fairness, and validity in graduate admissions: A psychometric perspective. Perspectives on Psychological Science.
- 1 March 2021
- Published by American Psychological Association (APA) in Journal of Applied Psychology
- Vol. 106 (3) , 476-488
Assessing and interpreting interaction effects: A reply to Vancouver, Carlson, Dhanani, and Colton (2021).
Van Iddekinge et al. (2018)'s meta-analysis revealed that ability and motivation have mostly an additive rather than an interactive effect on performance. One of the methods they used to assess the ability × motivation interaction was moderated multiple regression (MMR). Vancouver et al. (2021) presented conceptual arguments that ability and motivation should interact to predict performance, as well as analytical and empirical arguments against the use of MMR to assess interaction effects. We describe problems with these arguments and show conceptually and empirically that MMR (and the ΔR and ΔR2 it yields) is an appropriate and effective method for assessing both the statistical significance and magnitude of interaction effects. Nevertheless, we also applied the alternative approach Vancouver et al. recommended to test for interactions to primary data sets (k = 69) from Van Iddekinge et al. These new results showed that the ability × motivation interaction was not significant in 90% of the analyses, which corroborated Van Iddekinge et al.'s original conclusion that the interaction rarely increments the prediction of performance beyond the additive effects of ability and motivation. In short, Van Iddekinge et al.'s conclusions remain unchanged and, given the conceptual and empirical problems we identified, we cannot endorse Vancouver et al.'s recommendation to change how researchers test interactions. We conclude by offering suggestions for how to assess and interpret interactions in future research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
- 1 March 2021
- Published by American Psychological Association (APA) in Journal of Applied Psychology
- Vol. 106 (3) , 476-488
Assessing and interpreting interaction effects: A reply to Vancouver, Carlson, Dhanani, and Colton (2021).
Van Iddekinge et al. (2018)'s meta-analysis revealed that ability and motivation have mostly an additive rather than an interactive effect on performance. One of the methods they used to assess the ability × motivation interaction was moderated multiple regression (MMR). Vancouver et al. (2021) presented conceptual arguments that ability and motivation should interact to predict performance, as well as analytical and empirical arguments against the use of MMR to assess interaction effects. We describe problems with these arguments and show conceptually and empirically that MMR (and the ΔR and ΔR2 it yields) is an appropriate and effective method for assessing both the statistical significance and magnitude of interaction effects. Nevertheless, we also applied the alternative approach Vancouver et al. recommended to test for interactions to primary data sets (k = 69) from Van Iddekinge et al. These new results showed that the ability × motivation interaction was not significant in 90% of the analyses, which corroborated Van Iddekinge et al.'s original conclusion that the interaction rarely increments the prediction of performance beyond the additive effects of ability and motivation. In short, Van Iddekinge et al.'s conclusions remain unchanged and, given the conceptual and empirical problems we identified, we cannot endorse Vancouver et al.'s recommendation to change how researchers test interactions. We conclude by offering suggestions for how to assess and interpret interactions in future research. (PsycInfo Database Record (c) 2021 APA, all rights reserved).
- 19 January 2021
- Published by Wiley in Applied Psychology
- Vol. 71 (2) , 710-739
Validity evidence and measurement equivalence for the Dutch translation of the conditional reasoning test for aggression
The Conditional Reasoning Test for Aggression (CRT‐A) indirectly measures the implicit motive to aggress by engaging respondents in inductive reasoning tasks. Most research involving the CRT‐A has been based on the original English version of the test with most data being collected in the United States. The purpose of the current paper is to evaluate the psychometric properties of a Dutch translation of this test and to examine if it could be used to predict measures of integrity. In the first of two studies, we evaluated the psychometric properties and measurement equivalence of the CRT‐A across US and Dutch samples. In the second study, we examined validity evidence for the Dutch version of the CRT‐A. Results from Study 1 indicated that the test was mostly equivalent across cultures (i.e. limited differential item functioning was detected). Results from Study 2 demonstrated that the Dutch version of the CRT‐A was correlated with measures of behavioral integrity and provided incremental prediction of integrity over and above traditional self‐report measures of explicit personality traits. We discuss the implications for using CRTs across different cultures and languages.
- 19 January 2021
- Published by Wiley in Applied Psychology
- Vol. 71 (2) , 710-739
Validity evidence and measurement equivalence for the Dutch translation of the conditional reasoning test for aggression
The Conditional Reasoning Test for Aggression (CRT‐A) indirectly measures the implicit motive to aggress by engaging respondents in inductive reasoning tasks. Most research involving the CRT‐A has been based on the original English version of the test with most data being collected in the United States. The purpose of the current paper is to evaluate the psychometric properties of a Dutch translation of this test and to examine if it could be used to predict measures of integrity. In the first of two studies, we evaluated the psychometric properties and measurement equivalence of the CRT‐A across US and Dutch samples. In the second study, we examined validity evidence for the Dutch version of the CRT‐A. Results from Study 1 indicated that the test was mostly equivalent across cultures (i.e. limited differential item functioning was detected). Results from Study 2 demonstrated that the Dutch version of the CRT‐A was correlated with measures of behavioral integrity and provided incremental prediction of integrity over and above traditional self‐report measures of explicit personality traits. We discuss the implications for using CRTs across different cultures and languages.
- Open Access
- 18 August 2020
- Published by Center for Open Science in PsyArXiv
Bias, Fairness, and Validity in Graduate Admissions: A Psychometric Perspective
As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement bias and fairness, in order to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used in graduate admission assessment: grade point average, personal statements, resumes/CVs, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures, and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions, as well as highlighting areas in which future research would be beneficial.
- 18 August 2020
- Published by Center for Open Science in PsyArXiv
Open AccessBias, Fairness, and Validity in Graduate Admissions: A Psychometric Perspective
As many schools and departments are considering the removal of the Graduate Record Examination (GRE) from their graduate admission processes to enhance equity and diversity in higher education, controversies arise. From a psychometric perspective, we see a critical need for clarifying the meanings of measurement bias and fairness, in order to create common ground for constructive discussions within the field of psychology, higher education, and beyond. We critically evaluate six major sources of information that are widely used in graduate admission assessment: grade point average, personal statements, resumes/CVs, letters of recommendation, interviews, and GRE. We review empirical research evidence available to date on the validity, bias, and fairness issues associated with each of these admission measures, and identify potential issues that have been overlooked in the literature. We conclude by suggesting several directions for practical steps to improve the current admissions decisions, as well as highlighting areas in which future research would be beneficial.
- 13 November 2019
- Published by Wiley in Personnel Psychology
- Vol. 73 (3) , 491-516
Maybe not so independent after all: The possibility, prevalence, and consequences of violating the independence assumptions in psychometric meta‐analysis
Psychometric meta‐analysis assumes that moderators are unrelated to study artifacts (e.g., criterion reliability), and that study artifacts are independent of true validities. Meeting these assumptions is important for researchers seeking to accurately partition the variance in effect sizes due to study artifacts from the variance due to meaningful moderators. Despite the critical role of these assumptions, we know very little about their tenability. To address this basic gap in the literature, we conducted three studies to determine if there are potential violations of the independence assumptions (Study 1), the prevalence of such violations (Study 2), and the consequences of violating the independence assumptions via a series of Monte Carlo simulations (Study 3). We found that violations of the independence assumptions are not only plausible but also routinely detected across a wide array of topics. Simulation results indicate that violating the independence assumptions can result in biases under certain circumstances, which are further accentuated due to the lack of stability in the estimators. We offer suggestions for the future use of psychometric meta‐analysis and discuss the implications for research focused on refining psychometric meta‐analysis.
- 13 November 2019
- Published by Wiley in Personnel Psychology
- Vol. 73 (3) , 491-516
Maybe not so independent after all: The possibility, prevalence, and consequences of violating the independence assumptions in psychometric meta‐analysis
Psychometric meta‐analysis assumes that moderators are unrelated to study artifacts (e.g., criterion reliability), and that study artifacts are independent of true validities. Meeting these assumptions is important for researchers seeking to accurately partition the variance in effect sizes due to study artifacts from the variance due to meaningful moderators. Despite the critical role of these assumptions, we know very little about their tenability. To address this basic gap in the literature, we conducted three studies to determine if there are potential violations of the independence assumptions (Study 1), the prevalence of such violations (Study 2), and the consequences of violating the independence assumptions via a series of Monte Carlo simulations (Study 3). We found that violations of the independence assumptions are not only plausible but also routinely detected across a wide array of topics. Simulation results indicate that violating the independence assumptions can result in biases under certain circumstances, which are further accentuated due to the lack of stability in the estimators. We offer suggestions for the future use of psychometric meta‐analysis and discuss the implications for research focused on refining psychometric meta‐analysis.
- Open Access
- 20 October 2019
- Published by SAGE Publications in Organizational Research Methods
- Vol. 23 (1) , 124-153
Conditional Reasoning: An Integrated Approach to Item Analysis
This article provides a review of the approach that James used when conducting item analyses on his conditional reasoning test items. That approach was anchored in classical test theory. Our article extends this work in two important ways. First, we offer a set of test development protocols that are tailored to the unique nature of conditional reasoning tests. Second, we further extend James’s approach by integrating his early test validation protocols (based on classical test theory) with more recent protocols (based on item response theory). We then apply our integrated item analytic framework to data collected on James’s first test, the conditional reasoning test for relative motive strength. We illustrate how this integrated approach furnishes additional diagnostic information that may allow researchers to make more informed and targeted revisions to an initial set of items.
- 20 October 2019
- Published by SAGE Publications in Organizational Research Methods
- Vol. 23 (1) , 124-153
Open AccessConditional Reasoning: An Integrated Approach to Item Analysis
This article provides a review of the approach that James used when conducting item analyses on his conditional reasoning test items. That approach was anchored in classical test theory. Our article extends this work in two important ways. First, we offer a set of test development protocols that are tailored to the unique nature of conditional reasoning tests. Second, we further extend James’s approach by integrating his early test validation protocols (based on classical test theory) with more recent protocols (based on item response theory). We then apply our integrated item analytic framework to data collected on James’s first test, the conditional reasoning test for relative motive strength. We illustrate how this integrated approach furnishes additional diagnostic information that may allow researchers to make more informed and targeted revisions to an initial set of items.