Detecting Careless Responding in Multidimensional Forced-Choice Questionnaires
Open Access
- 12 January 2024
- journal article
- research article
- Published by SAGE Publications in Educational and Psychological Measurement
- Vol. 84 (5) , 887-926
Abstract
The multidimensional forced-choice (MFC) format is an alternative to rating scales in which participants rank items according to how well the items describe them. Currently, little is known about how to detect careless responding in MFC data. The aim of this study was to adapt a number of indices used for rating scales to the MFC format and additionally develop several new indices that are unique to the MFC format. We applied these indices to a data set from an online survey ( N = 1,169) that included a series of personality questionnaires in the MFC format. The correlations among the careless responding indices were somewhat lower than those published for rating scales. Results from a latent profile analysis suggested that the majority of the sample (about 76–84%) did not respond carelessly, although the ones who did were characterized by different levels of careless responding. In a simulation study, we simulated different careless responding patterns and varied the overall proportion of carelessness in the samples. With one exception, the indices worked as intended conceptually. Taken together, the results suggest that careless responding also plays an important role in the MFC format. Recommendations on how it can be addressed are discussed.Keywords
All Related Versions
Funding Information
- Young Scholar Fund of the University of Konstanz awarded to Eunike Wetzel
- Deutsche Forschungsgemeinschaft (GRK 2277)
This publication has 62 references indexed in Scilit:
- Item Response Modeling of Paired Comparison and Ranking DataMultivariate Behavioral Research, 2010
- Random Responding as a Threat to the Validity of Effect Size Estimates in Correlational ResearchEducational and Psychological Measurement, 2010
- Psychometric Characteristics of a Public-Domain Self-Report Measure of Vocational Interests: The Oregon Vocational Interest ScalesJournal of Personality Assessment, 2010
- The HEXACO-60: A Short Measure of the Major Dimensions of PersonalityJournal of Personality Assessment, 2009
- Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web SurveyPublic Opinion Quarterly, 2009
- The international personality item pool and the future of public-domain personality measuresJournal of Research in Personality, 2006
- Reconsidering Forced-Choice Item Formats for Applicant Personality AssessmentHuman Performance, 2005
- Response Time Effort: A New Measure of Examinee Motivation in Computer-Based TestsApplied Measurement in Education, 2005
- Ascertaining the validity of individual protocols from Web-based personality inventoriesJournal of Research in Personality, 2005
- Factors Defined by Negatively Keyed Items: The Result of Careless Respondents?Applied Psychological Measurement, 1985