Nathan D. Jones
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Nathan D. Jones has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Jing Liu
University of Maryland College Park, College Park, MD
2 shared publications - Julie Cohen
University of Virginia, Charlottesville, VA, USA
2 shared publications - Paiheng Xu
University of Maryland University of Maryland Boston University University of Virginia University of Maryland
2 shared publications - Wei Ai
University of Maryland College Park
2 shared publications - Allison F. Gilmour
American Institutes for Research, Arlington, VA, USA
1 shared publication - Benjamin Kelcey
University of Cincinnati, CECH Educational Studies, 2610 University Circle, Cincinnati, OH 45221, United States
1 shared publication
Latest Publications
- Open Access
The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education
Assessing instruction quality is a fundamental component of any improvement efforts in the education system. However, traditional manual assessments are expensive, subjective, and heavily dependent on observers' expertise and idiosyncratic factors, preventing teachers from getting timely and frequent feedback. Different from prior research that mostly focuses on low-inference instructional practices on a singular basis, this paper presents the first study that leverages Natural Language Processing (NLP) techniques to assess multiple high-inference instructional practices in two distinct educational settings: in-person K-12 classrooms and simulated performance tasks for pre-service teachers. This is also the first study that applies NLP to measure a teaching practice that is widely acknowledged to be particularly effective for students with special needs. We confront two challenges inherent in NLP-based instructional analysis, including noisy and long input data and highly skewed distributions of human ratings. Our results suggest that pretrained Language Models (PLMs) demonstrate performances comparable to the agreement level of human raters for variables that are more discrete and require lower inference, but their efficacy diminishes with more complex teaching practices. Interestingly, using only teachers' utterances as input yields strong results for student-centered variables, alleviating common concerns over the difficulty of collecting and transcribing high-quality student speech data in in-person teaching settings. Our findings highlight both the potential and the limitations of current NLP techniques in the education domain, opening avenues for further exploration.
Open AccessThe Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education
Assessing instruction quality is a fundamental component of any improvement efforts in the education system. However, traditional manual assessments are expensive, subjective, and heavily dependent on observers' expertise and idiosyncratic factors, preventing teachers from getting timely and frequent feedback. Different from prior research that mostly focuses on low-inference instructional practices on a singular basis, this paper presents the first study that leverages Natural Language Processing (NLP) techniques to assess multiple high-inference instructional practices in two distinct educational settings: in-person K-12 classrooms and simulated performance tasks for pre-service teachers. This is also the first study that applies NLP to measure a teaching practice that is widely acknowledged to be particularly effective for students with special needs. We confront two challenges inherent in NLP-based instructional analysis, including noisy and long input data and highly skewed distributions of human ratings. Our results suggest that pretrained Language Models (PLMs) demonstrate performances comparable to the agreement level of human raters for variables that are more discrete and require lower inference, but their efficacy diminishes with more complex teaching practices. Interestingly, using only teachers' utterances as input yields strong results for student-centered variables, alleviating common concerns over the difficulty of collecting and transcribing high-quality student speech data in in-person teaching settings. Our findings highlight both the potential and the limitations of current NLP techniques in the education domain, opening avenues for further exploration.
- Open Access
- 22 March 2024
- Published by Springer Nature in Prevention Science
- Vol. 27 (1) , 6-15
Expanding Education Researchers’ Access to Classroom Observation Data With a Remote and Cost-Effective Video Data Collection Protocol
The onset of the COVID-19 pandemic and associated long-term shifts to virtual instruction among most US schools presented notable challenges among education researchers. Ongoing projects conducted in school settings experienced sudden losses of access to teacher and student participants, in many cases leading to severe interruptions to data collection efforts. Perhaps most notably, upon returns to in-person instruction in the 2021/22 academic year most schools instigated strict policies limiting the number of non-school personnel who could enter school buildings, including researchers conducting in-person data collections. As such, many researchers had to find alternative means to gather data. In this paper, we offer a new protocol that we created in response to these challenges that allows for the secure and fully remote collection of video data in school settings. This new protocol not only addressed the immediate needs of the focal study but also addresses some of the most notable barriers to collecting classroom video data in the field of education research at large. In this paper, we describe the initial development and application of this protocol among a local study of elementary teachers, as well as the scaling of this protocol in a study of elementary teachers in multiple states. It is our hope that this protocol can expand education researchers’, practitioners’, and policymakers’ access to classroom video data.
- 22 March 2024
- Published by Springer Nature in Prevention Science
- Vol. 27 (1) , 6-15
Open AccessExpanding Education Researchers’ Access to Classroom Observation Data With a Remote and Cost-Effective Video Data Collection Protocol
The onset of the COVID-19 pandemic and associated long-term shifts to virtual instruction among most US schools presented notable challenges among education researchers. Ongoing projects conducted in school settings experienced sudden losses of access to teacher and student participants, in many cases leading to severe interruptions to data collection efforts. Perhaps most notably, upon returns to in-person instruction in the 2021/22 academic year most schools instigated strict policies limiting the number of non-school personnel who could enter school buildings, including researchers conducting in-person data collections. As such, many researchers had to find alternative means to gather data. In this paper, we offer a new protocol that we created in response to these challenges that allows for the secure and fully remote collection of video data in school settings. This new protocol not only addressed the immediate needs of the focal study but also addresses some of the most notable barriers to collecting classroom video data in the field of education research at large. In this paper, we describe the initial development and application of this protocol among a local study of elementary teachers, as well as the scaling of this protocol in a study of elementary teachers in multiple states. It is our hope that this protocol can expand education researchers’, practitioners’, and policymakers’ access to classroom video data.
- Open Access
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 4375-4389
The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education
Paiheng Xu, Jing Liu, Nathan Jones, Julie Cohen, Wei Ai. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
- 1 January 2024
- Published by Association for Computational Linguistics (ACL)
- p. 4375-4389
Open AccessThe Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education
Paiheng Xu, Jing Liu, Nathan Jones, Julie Cohen, Wei Ai. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.
- 16 June 2020
- Published by American Educational Research Association (AERA) in Educational Researcher
- Vol. 49 (9) , 645-655
Policies That Define Instruction: A Systematic Review of States’ and Districts’ Recommendations for Evaluating Special Educators
Educational policies addressing instruction may fail to acknowledge that effective instruction is not the same for all learners. We reviewed teacher evaluation systems across all states and the 25 largest districts to determine how states and districts approach the evaluation of special education teachers, a policy aimed at improving teaching effectiveness. We found that most states and districts did not provide guidance to schools for adapting evaluation systems for these teachers. Some states provided guidance on technical aspects of special education teacher evaluation, such as incorporating student achievement into special education teachers’ scores. Districts were more likely to focus on instructional considerations. We discuss the implications of these findings for policies that aim to promote the use of effective instructional practices.
- 16 June 2020
- Published by American Educational Research Association (AERA) in Educational Researcher
- Vol. 49 (9) , 645-655
Policies That Define Instruction: A Systematic Review of States’ and Districts’ Recommendations for Evaluating Special Educators
Educational policies addressing instruction may fail to acknowledge that effective instruction is not the same for all learners. We reviewed teacher evaluation systems across all states and the 25 largest districts to determine how states and districts approach the evaluation of special education teachers, a policy aimed at improving teaching effectiveness. We found that most states and districts did not provide guidance to schools for adapting evaluation systems for these teachers. Some states provided guidance on technical aspects of special education teacher evaluation, such as incorporating student achievement into special education teachers’ scores. Districts were more likely to focus on instructional considerations. We discuss the implications of these findings for policies that aim to promote the use of effective instructional practices.
- 1 March 2019
- Published by University of Chicago Press in The Elementary School Journal
- Vol. 119 (3) , 468-486
Competing Strands of Educational Reform Policy: Can Collaborative School Reform and Teacher Evaluation Reform Be Reconciled?
- 1 March 2019
- Published by University of Chicago Press in The Elementary School Journal
- Vol. 119 (3) , 468-486
Competing Strands of Educational Reform Policy: Can Collaborative School Reform and Teacher Evaluation Reform Be Reconciled?
- 6 August 2015
- Published by Wiley in Educational Measurement: Issues and Practice
- Vol. 34 (3) , 28-38
Using Test Scores From Students With Disabilities in Teacher Evaluation
- 6 August 2015
- Published by Wiley in Educational Measurement: Issues and Practice
- Vol. 34 (3) , 28-38
Using Test Scores From Students With Disabilities in Teacher Evaluation
- 19 December 2013
- Published by SAGE Publications in Assessment for Effective Intervention
- Vol. 39 (2) , 112-124
Examining the Use of Classroom Observations in the Evaluation of Special Education Teachers
A primary focus of recent educational policymaking has been on improving measures of teaching effectiveness, attributable in part to recent federal policies such as the Race to the Top initiative and Investing in Innovation. To date, neither researchers nor practitioners have arrived at a consensus on the best methods for evaluating special educators. While value-added scores are likely not suitable for the majority of special education teachers, observation systems appear to be more promising because they can be used across a variety of instructional settings and formats. To illustrate the steps that would be necessary to validate observation systems for use with special educators, we examine the observation system being most commonly used in school districts—Charlotte Danielson’s Framework for Teaching (FFT). Drawing on empirical research, we propose a definition of effective special education teaching, and we then use this definition to frame our investigation into FFT. We conclude with a series of recommendations for research that would establish the validity of using FFT to evaluate special educators. We also describe how the challenges specific to FFT would generalize to other observation systems.
- 19 December 2013
- Published by SAGE Publications in Assessment for Effective Intervention
- Vol. 39 (2) , 112-124
Examining the Use of Classroom Observations in the Evaluation of Special Education Teachers
A primary focus of recent educational policymaking has been on improving measures of teaching effectiveness, attributable in part to recent federal policies such as the Race to the Top initiative and Investing in Innovation. To date, neither researchers nor practitioners have arrived at a consensus on the best methods for evaluating special educators. While value-added scores are likely not suitable for the majority of special education teachers, observation systems appear to be more promising because they can be used across a variety of instructional settings and formats. To illustrate the steps that would be necessary to validate observation systems for use with special educators, we examine the observation system being most commonly used in school districts—Charlotte Danielson’s Framework for Teaching (FFT). Drawing on empirical research, we propose a definition of effective special education teaching, and we then use this definition to frame our investigation into FFT. We conclude with a series of recommendations for research that would establish the validity of using FFT to evaluate special educators. We also describe how the challenges specific to FFT would generalize to other observation systems.
- 1 January 2013
- Published by SAGE Publications in Exceptional Children
- Vol. 79 (3) , 365-383
The Role of School-Based Colleagues in Shaping the Commitment of Novice Special and General Education Teachers
We compare beginning special and general education teachers’ access to school-based colleagues. Our findings demonstrate that colleague relationships are critical for the experiences of beginning teachers, as are the school organizational norms that these beginning teachers experience. For special education teachers in particular, perception of colleague support was a strong predictor of retention plans. Similar results were seen with respect to their perception of the level of collective responsibility among the faculty. Taken together, these results suggest that schools and districts should make efforts to facilitate productive relationships between general and special education faculty, as well as to differentiate induction support for beginning special educators.
- 1 January 2013
- Published by SAGE Publications in Exceptional Children
- Vol. 79 (3) , 365-383
The Role of School-Based Colleagues in Shaping the Commitment of Novice Special and General Education Teachers
We compare beginning special and general education teachers’ access to school-based colleagues. Our findings demonstrate that colleague relationships are critical for the experiences of beginning teachers, as are the school organizational norms that these beginning teachers experience. For special education teachers in particular, perception of colleague support was a strong predictor of retention plans. Similar results were seen with respect to their perception of the level of collective responsibility among the faculty. Taken together, these results suggest that schools and districts should make efforts to facilitate productive relationships between general and special education faculty, as well as to differentiate induction support for beginning special educators.