Valid questions: the development and evaluation of a new library learning analytics survey
- 21 July 2023
- journal article
- research article
- Published by Emerald Publishing in Performance Measurement and Metrics
- Vol. 24 (2) , 101-119
Abstract
Purpose: This article describes the development processes, sampling and analysis practices and the assessment of reliability and validity of a new 0survey that sought to evaluate undergraduate students' perceptions and expectations related to privacy and library participation in learning analytics studies. This article provides other researchers with information required to independently evaluate the survey's efficacy, as well as guidance for designing other surveys.Design/methodology/approach: Following question development, pre-survey validity assessments were made using subject matter expert panel review and cognitive interviews. Post-hoc analysis of survey construct reliability was evaluated using the Omega coefficient, while exploratory factor analysis was utilized to assess construct validity. Survey design limitations and potential bias effects are also examined.Findings: The survey exhibited a high level of reliability among research constructs, while the exploratory factor analysis results suggested that survey constructs contained multiple conceptual elements that should be measured separately for more nuanced analysis.Practical implications: This article provides a model for other researchers wishing to re-use the survey described or develop similar surveys.Social implications: As learning analytics interest continues to expand, engaging with the subjects, in this case students, of analysis is critical. Researchers need to ensure that captured measurements are appropriately valid in order to accurately represent the findings.Originality/value: This survey is one of very few addressing library learning analytics that has undergone extensive validity analysis of the conceptual constructs.Keywords
This publication has 21 references indexed in Scilit:
- The Student Expectations of Learning Analytics QuestionnaireJournal of Computer Assisted Learning, 2019
- The alpha and the omega of scale reliability and validity: why and how to abandon Cronbach’s alpha and the route towards more comprehensive assessment of scale qualityPsyArXiv, 2018
- Reliability, Population Classification and Weighting in Multidimensional Poverty Measurement: A Monte Carlo StudySocial Indicators Research, 2018
- An Institution Wide Approach to Learning Analytics2016
- From alpha to omega: A practical solution to the pervasive problem of internal consistency estimationBritish Journal of Psychology, 2013
- Recalculation of the Critical Values for Lawshe’s Content Validity RatioMeasurement and Evaluation in Counseling and Development, 2012
- Learning analytics2012
- On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s AlphaPsychometrika, 2009
- Methods for Testing and Evaluating Survey Questions2004
- A QUANTITATIVE APPROACH TO CONTENT VALIDITY1Personnel Psychology, 1975