What should we evaluate when we use technology in education?
- 23 January 2022
- journal article
- research article
- Published by Wiley in Journal of Computer Assisted Learning
- Vol. 38 (3) , 743-757
Abstract
Background: There is a lack of critical or empirical work interrogating the nature and purpose of evaluating technology use in education.Objectives: In this study, we examine the values underpinning the evaluation of technology use in education through field specialist perceptions. The study also poses critical reflections about the rigour of evaluation instruments development in the educational technology field.Method: A total of 48 domain specialists were surveyed to investigate the face and content validity of 39 items under eight constructs, with both qualitative and quantitative data from the survey analysed.Results: There was an alignment between the constructs that the specialists felt were important and the constructs that have typically been the focus of empirical studies, with field specialists indicating high relevance scores for technology (M = 3.24/4), learning outcomes (M = 3.20/4), affective elements (M = 3.19/4), behaviour (M = 3.15/4), presence/community (M = 3.07/4), teaching/pedagogy (M = 3.01/4), design (M = 2.96/4), and institutional environment (M = 2.86/4). Only a minority of other studies were found to perform face and content validity checks and even then only with small samples of respondents (usually n ≤ 5).Implications: Specialists in educational technology research confirm that all eight dimensions are important to consider when evaluating the use of technology in education. Thorough face and content validity processes should be adopted when developing educational technology evaluation instruments. Further work has validated an eight factor 28 item instrument for evaluating the use of technology in education using a large sample of students from a global open learning online course.Keywords
This publication has 101 references indexed in Scilit:
- Research trends in mobile and ubiquitous learning: a review of publications in selected journals from 2001 to 2010British Journal of Educational Technology, 2011
- Critical Issues and Common Pitfalls in Designing and Conducting Impact Studies in EducationEducational Evaluation and Policy Analysis, 2010
- The Bigger Picture: Institutional Perspectives on Interim Assessment TechnologiesPeabody Journal of Education, 2010
- Effects of ICT: Do we know what we should know?Education and Information Technologies, 2007
- Technological Pedagogical Content Knowledge: A Framework for Teacher KnowledgeTeachers College Record: the Voice of Scholarship in Education, 2006
- Using thematic analysis in psychologyQualitative Research in Psychology, 2006
- Confirmatory factor analysis of the cognitive failures questionnaire: evidence for dimensionality and construct validityPersonality and Individual Differences, 2004
- The use of expert judges in scale developmentJournal of Business Research, 2004
- Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach.Journal of Educational Psychology, 1992
- Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information TechnologyMIS Quarterly, 1989