Fiona Patterson

Situational judgment tests (SJTs) are widely used to evaluate decision-making and problem-solving skills. These tests present respondents with situations (scenarios), and possible actions in response to each situation, which they must rate based on effectiveness or appropriateness. Despite having high face and criterion-related validities, these tests involve complex cognitive processes, and their construct validity has been notoriously difficult to establish. This is particularly challenging when several questions relate to the same scenario, thus creating local dependencies in item responses after controlling for the overall ability. Finding a suitable measurement model for SJTs would address the operational challenges of test development, maintenance, scoring, and equating the scores across multiple test forms and/or testing years. We propose to use a variant of the continuous response model (CRM; Samejima, 1973 ) to describe points gained at the level of scenario, regardless of the number of questions asked about the scenario. A simulation study testing the performance of this scenario-level model on data generated from categorical items nested within scenarios found that CRM fits well under a wide range of realistic conditions and resembles the true score better than a summated test score. We illustrate the proposed approach with an SJT presenting clinical dilemmas with several possible actions, each scored using four ordered categories. The SJT was completed by N = 2,740 examinees in operational assessment settings. The proposed CRM variant fitted the data well and was effective for assessing dimensionality, modeling and scoring responses, and evaluating test reliability. Implications for other operational uses such as form linking and pre-equating are discussed.

Specific social groups remain under-represented within dentistry. While the University Clinical Aptitude Test (UCAT) aims to widen participation in under-represented social groups, there is no evidence in dental education that this aim is being met. Data over two admission cycles (2012 and 2013), including 3246 applicants to 10 UK dental schools, were analysed. Applicant and selected pools were compared to the UK population. Multiple logistic regression was used to investigate the association between demographic variables and UCAT and receiving an offer of a place at dental school. Over-representation of Female, Asian, least deprived and grammar school groups were found in applicant and selected pools compared with the UK population. White ethnic applicants were significantly more selected than Black (OR 0.25), Asian (OR 0.57) and Mixed (OR 0.80) ethnicities, while least deprived applicants were significantly more selected than most deprived (OR 0.59). Grammar school education increased odds of selection by 1.8 when compared to state school. The addition of UCAT to the model for applicants reduced ethnic disparities but led to disparities between other groups. Current widening participation practices focus on attracting applicants from lower socio-economic groups. However, this study showed that ethnicity, sex and educational background biases also affect demographic diversity in dentistry. The UCAT shows promise in levelling the playing field; however, widening access measures will only succeed if selection committees radically change selection processes to address the systemic biases, enabling the dentists of tomorrow to represent the society they serve.

Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile. Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508). The SJT demonstrated good internal consistency (Cronbach’s alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity. This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

Objective. To design, implement, and psychometrically evaluate a situational judgement test (SJT) to use as a formative assessment of pharmacy students’ non-academic skills in an Australian-based university. Methods. An SJT was developed using a previously validated design process including involvement of subject matter experts. The first phase included design of a blueprint through stakeholder consultation and the development of bespoke attribute definitions and a tool specification. Following on from this, SJT items were developed through subject matter expert interviews and in-depth review process. Results. Students (702) from four different cohorts (first through fourth years) of a Bachelor of Pharmacy degree program completed the situational judgment test. Data from 648 students was eligible for inclusion in the analysis. The SJT demonstrated good reliability, appropriateness for use (difficulty and quality), fairness, and face validity. The variability in students’ scores suggested that the SJT may be a useful metric to identify students most in need of additional support. Conclusion. Evaluation of the SJT demonstrated that the tool was valid, reliable, fair and appropriate to use as a formative assessment. Through implementing an SJT such as this, pharmacy students are provided the opportunity to receive feedback on their non-academic skills and consider how to approach challenging or unfamiliar situations before entering the profession.

The addition of a consensus statement on the assessment of professionalism (Hodges et al. 2018) in this edition of the journal complements the two statements published in an earlier issue: one on A...

MedEdPublish (ISSN 2312–7996) is a highly visible, open access, specialist practitioner e-journal that enables academics, teachers, clinicians, researchers and students to publish their experiences, views and research findings relating to teaching, learning and assessment in medical and health professions education. An innovative and key feature of MedEdPublish is that the peer review process begins following publication.

This article was migrated. The article was marked as recommended. Over the last two decades, technological advancements internationally have meant that the Internet has become an important medium for recruitment and selection. Consequently, there is an increased need for research that examines the effectiveness of newer technology-mediated selection methods. This exploratory research study qualitatively explored applicant perceptions of fairness of asynchronous video interviews used in medical selection. Ten undergraduate medical students participated in a pilot asynchronous multiple-mini interview and were invited to share their experiences and perceptions in a follow-up interview. The data was transcribed verbatim and analysed using template analysis, with Gilliland's (1993) organisational justice theory guiding the original template. Many of the original themes from Gilliland's model were uncovered during analysis. Additionally, some significant themes were identified that did not form part of the original template and were therefore added to the final coding template - these were specifically relating to technology, including acceptability in a medical context; technical issues and adverse impact. Overall, results suggested that participants perceived asynchronous video interviews to be a fair method of selection. However, participants thought asynchronous interviews should only be used as part of an extensive selection process and furthermore, should not replace face-to-face interviews. Findings are discussed in line with existing research of fairness perceptions and justice theory in selection (Gilliland, 1993) and implications for research and practice are presented.