James H. Martin
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
James H. Martin has a total of 157 co-authors. Here are the top co-authors by number of shared publications:
Wayne Ward
Department of Computer Science University of Colorado, Boulder, Colorado, USA
11 shared publications- Steven Bethard
University of Arizona
10 shared publications - Kirsten R. Butcher
University of Utah
7 shared publications Tamara Sumner
7 shared publicationsMartha Palmer
The Ronin Institute for Independent Scholarship Monclair NJ United States of America
6 shared publicationsRodney D. Nielsen
University of Colorado, Boulder, CO, USA
5 shared publications
Latest Publications
- Open Access
Towards Actionable Pedagogical Feedback: A Multi-Perspective Analysis of Mathematics Teaching and Tutoring Dialogue
Effective feedback is essential for refining instructional practices in mathematics education, and researchers often turn to advanced natural language processing (NLP) models to analyze classroom dialogues from multiple perspectives. However, utterance-level discourse analysis encounters two primary challenges: (1) multifunctionality, where a single utterance may serve multiple purposes that a single tag cannot capture, and (2) the exclusion of many utterances from domain-specific discourse move classifications, leading to their omission in feedback. To address these challenges, we proposed a multi-perspective discourse analysis that integrates domain-specific talk moves with dialogue act (using the flattened multi-functional SWBD-MASL schema with 43 tags) and discourse relation (applying Segmented Discourse Representation Theory with 16 relations). Our top-down analysis framework enables a comprehensive understanding of utterances that contain talk moves, as well as utterances that do not contain talk moves. This is applied to two mathematics education datasets: TalkMoves (teaching) and SAGA22 (tutoring). Through distributional unigram analysis, sequential talk move analysis, and multi-view deep dive, we discovered meaningful discourse patterns, and revealed the vital role of utterances without talk moves, demonstrating that these utterances, far from being mere fillers, serve crucial functions in guiding, acknowledging, and structuring classroom discourse. These insights underscore the importance of incorporating discourse relations and dialogue acts into AI-assisted education systems to enhance feedback and create more responsive learning environments. Our framework may prove helpful for providing human educator feedback, but also aiding in the development of AI agents that can effectively emulate the roles of both educators and students.
Open AccessTowards Actionable Pedagogical Feedback: A Multi-Perspective Analysis of Mathematics Teaching and Tutoring Dialogue
Effective feedback is essential for refining instructional practices in mathematics education, and researchers often turn to advanced natural language processing (NLP) models to analyze classroom dialogues from multiple perspectives. However, utterance-level discourse analysis encounters two primary challenges: (1) multifunctionality, where a single utterance may serve multiple purposes that a single tag cannot capture, and (2) the exclusion of many utterances from domain-specific discourse move classifications, leading to their omission in feedback. To address these challenges, we proposed a multi-perspective discourse analysis that integrates domain-specific talk moves with dialogue act (using the flattened multi-functional SWBD-MASL schema with 43 tags) and discourse relation (applying Segmented Discourse Representation Theory with 16 relations). Our top-down analysis framework enables a comprehensive understanding of utterances that contain talk moves, as well as utterances that do not contain talk moves. This is applied to two mathematics education datasets: TalkMoves (teaching) and SAGA22 (tutoring). Through distributional unigram analysis, sequential talk move analysis, and multi-view deep dive, we discovered meaningful discourse patterns, and revealed the vital role of utterances without talk moves, demonstrating that these utterances, far from being mere fillers, serve crucial functions in guiding, acknowledging, and structuring classroom discourse. These insights underscore the importance of incorporating discourse relations and dialogue acts into AI-assisted education systems to enhance feedback and create more responsive learning environments. Our framework may prove helpful for providing human educator feedback, but also aiding in the development of AI agents that can effectively emulate the roles of both educators and students.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 12027-12048
On the Role of Semantic Proto-roles in Semantic Analysis: What do LLMs know about agency?
Elizabeth Spaulding, Shafiuddin Rehan Ahmed, James Martin. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 12027-12048
Open AccessOn the Role of Semantic Proto-roles in Semantic Analysis: What do LLMs know about agency?
Elizabeth Spaulding, Shafiuddin Rehan Ahmed, James Martin. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- Top Cited PapersOpen Access
- 18 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 250-262
A Comparative Analysis of Automatic Speech Recognition Errors in Small Group Classroom Discourse
In collaborative learning environments, effective intelligent learning systems need to accurately analyze and understand the collaborative discourse between learners (i.e., group modeling) to provide adaptive support. We investigate how automatic speech recognition (ASR) errors influence discourse models of small group collaboration in noisy real-world classrooms. Our dataset consisted of 30 students recorded by consumer off-the-shelf microphones (Yeti Blue) while engaging in dyadic- and triadic- collaborative learning in a multi-day STEM curriculum unit. We found that two state-of-the-art ASR systems (Google Speech and OpenAI Whisper) yielded very high word error rates (0.822, 0.847) but very different profiles of error with Google being more conservative, rejecting 38% of utterances instead of 12% for Whisper. Next, we examined how these ASR errors influenced down-stream small group modeling based on pre-trained large language models for three tasks: Abstract Meaning Representation parsing (AMRParsing), on-task/off-task detection (OnTask), and Accountable Productive Talk prediction (TalkMove). As expected, models trained on clean human transcripts yielded degraded performance on all three tasks, measured by the transfer ratio (TR). However, the TR of the specific sentence-level AMRParsing task (.39 - .62) was much lower than that of the abstract discourse-level OnTask (.63- .94) and TalkMove tasks (.64-.72). Furthermore, different training strategies that incorporated ASR transcripts alone or as augmentations of human transcripts increased accuracy for the discourse-level tasks (OnTask and TalkMove) but not AMRParsing. Simulation experiments suggested that the models were tolerant of missing utterances in the dialog context, and that jointly improving ASR accuracy on important word classes (e.g., verbs and nouns) can improve performance across all tasks. Overall, our results provide insights into how different types of NLP-based tasks might be tolerant of ASR errors under extremely noisy conditions and provide suggestions for how to improve accuracy in small group modeling settings for a more equitable, engaging, and adaptive collaborative learning environment.
- 18 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 250-262
Top Cited PapersOpen AccessA Comparative Analysis of Automatic Speech Recognition Errors in Small Group Classroom Discourse
In collaborative learning environments, effective intelligent learning systems need to accurately analyze and understand the collaborative discourse between learners (i.e., group modeling) to provide adaptive support. We investigate how automatic speech recognition (ASR) errors influence discourse models of small group collaboration in noisy real-world classrooms. Our dataset consisted of 30 students recorded by consumer off-the-shelf microphones (Yeti Blue) while engaging in dyadic- and triadic- collaborative learning in a multi-day STEM curriculum unit. We found that two state-of-the-art ASR systems (Google Speech and OpenAI Whisper) yielded very high word error rates (0.822, 0.847) but very different profiles of error with Google being more conservative, rejecting 38% of utterances instead of 12% for Whisper. Next, we examined how these ASR errors influenced down-stream small group modeling based on pre-trained large language models for three tasks: Abstract Meaning Representation parsing (AMRParsing), on-task/off-task detection (OnTask), and Accountable Productive Talk prediction (TalkMove). As expected, models trained on clean human transcripts yielded degraded performance on all three tasks, measured by the transfer ratio (TR). However, the TR of the specific sentence-level AMRParsing task (.39 - .62) was much lower than that of the abstract discourse-level OnTask (.63- .94) and TalkMove tasks (.64-.72). Furthermore, different training strategies that incorporated ASR transcripts alone or as augmentations of human transcripts increased accuracy for the discourse-level tasks (OnTask and TalkMove) but not AMRParsing. Simulation experiments suggested that the models were tolerant of missing utterances in the dialog context, and that jointly improving ASR accuracy on important word classes (e.g., verbs and nouns) can improve performance across all tasks. Overall, our results provide insights into how different types of NLP-based tasks might be tolerant of ASR errors under extremely noisy conditions and provide suggestions for how to improve accuracy in small group modeling settings for a more equitable, engaging, and adaptive collaborative learning environment.
- Open Access
$2 * n$ is better than $n^2$: Decomposing Event Coreference Resolution into Two Tractable Problems
Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents. Most mention pairs are not coreferent, yet many that are coreferent can be identified through simple techniques such as lemma matching of the event triggers or the sentences in which they appear. Existing methods for training coreference systems sample from a largely skewed distribution, making it difficult for the algorithm to learn coreference beyond surface matching. Additionally, these methods are intractable because of the quadratic operations needed. To address these challenges, we break the problem of ECR into two parts: a) a heuristic to efficiently filter out a large number of non-coreferent pairs, and b) a training approach on a balanced set of coreferent and non-coreferent mention pairs. By following this approach, we show that we get comparable results to the state of the art on two popular ECR datasets while significantly reducing compute requirements. We also analyze the mention pairs that are "hard" to accurately classify as coreferent or non-coreferent. Code at https://github.com/ahmeshaf/lemma_ce_coref
Open Access$2 * n$ is better than $n^2$: Decomposing Event Coreference Resolution into Two Tractable Problems
Event Coreference Resolution (ECR) is the task of linking mentions of the same event either within or across documents. Most mention pairs are not coreferent, yet many that are coreferent can be identified through simple techniques such as lemma matching of the event triggers or the sentences in which they appear. Existing methods for training coreference systems sample from a largely skewed distribution, making it difficult for the algorithm to learn coreference beyond surface matching. Additionally, these methods are intractable because of the quadratic operations needed. To address these challenges, we break the problem of ECR into two parts: a) a heuristic to efficiently filter out a large number of non-coreferent pairs, and b) a training approach on a balanced set of coreferent and non-coreferent mention pairs. By following this approach, we show that we get comparable results to the state of the art on two popular ECR datasets while significantly reducing compute requirements. We also analyze the mention pairs that are "hard" to accurately classify as coreferent or non-coreferent. Code at https://github.com/ahmeshaf/lemma_ce_coref
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 136-145
How Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Shafiuddin Rehan Ahmed, Abhijnan Nath, Michael Regan, Adam Pollins, Nikhil Krishnaswamy, James H. Martin. Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII). 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 136-145
Open AccessHow Good Is the Model in Model-in-the-loop Event Coreference Resolution Annotation?
Shafiuddin Rehan Ahmed, Abhijnan Nath, Michael Regan, Adam Pollins, Nikhil Krishnaswamy, James H. Martin. Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII). 2023.
- 30 April 2021
- Published by Springer Nature in KI - Künstliche Intelligenz
- Vol. 35 (3-4) , 343-360
Designing a Uniform Meaning Representation for Natural Language Processing
- 30 April 2021
- Published by Springer Nature in KI - Künstliche Intelligenz
- Vol. 35 (3-4) , 343-360
Designing a Uniform Meaning Representation for Natural Language Processing
- 24 March 2021
- Published by Taylor & Francis
- p. 389-415
Representing and Acquiring Metaphor-Based Polysemy
Polysemy is a widespread systematic phenomena in English. Much of the polysemy underlying the most common English words is based on metaphor. An approach to the acquisition of polysemous word-senses based on metaphor is presented. Conventional conceptual metaphors are used to motivate and generate new word-senses. MIDAS (Metaphor, Interpretation, Denotation and Acquisition System) is a computer system that implements this approach.
- 24 March 2021
- Published by Taylor & Francis
- p. 389-415
Representing and Acquiring Metaphor-Based Polysemy
Polysemy is a widespread systematic phenomena in English. Much of the polysemy underlying the most common English words is based on metaphor. An approach to the acquisition of polysemous word-senses based on metaphor is presented. Conventional conceptual metaphors are used to motivate and generate new word-senses. MIDAS (Metaphor, Interpretation, Denotation and Acquisition System) is a computer system that implements this approach.
- 2 November 2020
- Published by American Institute of Aeronautics and Astronautics (AIAA)
(MBSE)2: Using MBSE to Architect, Implement, and Operate the MBSE System
Model-Based Systems Engineering (MBSE) is being applied in a new way at the enterprise level for the benefit of all its major programs and business units. For an enterprise to successfully implement MBSE, it must integrate models, datasets, tools, and infrastructure with appropriate processes, methods, and standards into a comprehensive, integrated MBSE “System”. This paper describes an approach being put to practice, which leverages MBSE principles and methods to architect this MBSE System, focusing on the descriptive models that serve as the foundation for MBSE. The approach is named (MBSE)2 since it uses MBSE to architect and implement the MBSE System. This (MBSE)2 approach is illustrated with example models of the MBSE System, showing how these models are used to inform the architecting process, monitor agile implementation of models, facilitate model-based execution of SE reviews and audits, and support other Government acquisition applications.
- 2 November 2020
- Published by American Institute of Aeronautics and Astronautics (AIAA)
(MBSE)2: Using MBSE to Architect, Implement, and Operate the MBSE System
Model-Based Systems Engineering (MBSE) is being applied in a new way at the enterprise level for the benefit of all its major programs and business units. For an enterprise to successfully implement MBSE, it must integrate models, datasets, tools, and infrastructure with appropriate processes, methods, and standards into a comprehensive, integrated MBSE “System”. This paper describes an approach being put to practice, which leverages MBSE principles and methods to architect this MBSE System, focusing on the descriptive models that serve as the foundation for MBSE. The approach is named (MBSE)2 since it uses MBSE to architect and implement the MBSE System. This (MBSE)2 approach is illustrated with example models of the MBSE System, showing how these models are used to inform the architecting process, monitor agile implementation of models, facilitate model-based execution of SE reviews and audits, and support other Government acquisition applications.
- Open Access
- 22 May 2018
- Published by Pensoft Publishers in Biodiversity Information Science and Standards
- Vol. 2, e26080
Automated Trait Extraction using ClearEarth, a Natural Language Processing System for Text Mining in Natural Sciences
The cTAKES package (using the ClearTK Natural Language Processing toolkit Bethard et al. 2014, http://cleartk.github.io/cleartk/) has been successfully used to automatically read clinical notes in the medical field (Albright et al. 2013, Styler et al. 2014). It is used on a daily basis to automatically process clinical notes and extract relevant information by dozens of medical institutions. ClearEarth is a collaborative project that brings together computational linguistics and domain scientists to port Natural Language Processing (NLP) modules trained on the same types of linguistic annotation to the fields of geology, cryology, and ecology. The goal for ClearEarth in the ecology domain is the extraction of ecologically-relevant terms, including eco-phenotypic traits from text and the assignment of those traits to taxa. Four annotators used Anafora (an annotation software; https://github.com/weitechen/anafora) to mark seven entity types (biotic, aggregate, abiotic, locality, quality, unit, value) and six reciprocal property types (synonym of/has synonym, part of/has part, subtype/supertype) in 133 documents from primarily Encyclopedia of Life (EOL) and Wikipedia according to project guidelines (https://github.com/ClearEarthProject/AnnotationGuidelines). Inter-annotator agreement ranged from 43% to 90%. Performance of ClearEarth on identifying named entities in biology text overall was good (precision: 85.56%; recall: 71.57%). The named entities with the best performance were organisms and their parts/products (biotic entities - precision: 72.09%; recall: 54.17%) and systems and environments (aggregate entities - precision: 79.23%; recall: 75.34%). Terms and their relationships extracted by ClearEarth can be embedded in the new ecocore ontology after vetting (http://www.obofoundry.org/ontology/ecocore.html). This project enables use of advanced industry and research software within natural sciences for downstream operations such as data discovery, assessment, and analysis. In addition, ClearEarth uses the NLP results to generate domain-specific ontologies and other semantic resources.
- 22 May 2018
- Published by Pensoft Publishers in Biodiversity Information Science and Standards
- Vol. 2, e26080
Open AccessAutomated Trait Extraction using ClearEarth, a Natural Language Processing System for Text Mining in Natural Sciences
The cTAKES package (using the ClearTK Natural Language Processing toolkit Bethard et al. 2014, http://cleartk.github.io/cleartk/) has been successfully used to automatically read clinical notes in the medical field (Albright et al. 2013, Styler et al. 2014). It is used on a daily basis to automatically process clinical notes and extract relevant information by dozens of medical institutions. ClearEarth is a collaborative project that brings together computational linguistics and domain scientists to port Natural Language Processing (NLP) modules trained on the same types of linguistic annotation to the fields of geology, cryology, and ecology. The goal for ClearEarth in the ecology domain is the extraction of ecologically-relevant terms, including eco-phenotypic traits from text and the assignment of those traits to taxa. Four annotators used Anafora (an annotation software; https://github.com/weitechen/anafora) to mark seven entity types (biotic, aggregate, abiotic, locality, quality, unit, value) and six reciprocal property types (synonym of/has synonym, part of/has part, subtype/supertype) in 133 documents from primarily Encyclopedia of Life (EOL) and Wikipedia according to project guidelines (https://github.com/ClearEarthProject/AnnotationGuidelines). Inter-annotator agreement ranged from 43% to 90%. Performance of ClearEarth on identifying named entities in biology text overall was good (precision: 85.56%; recall: 71.57%). The named entities with the best performance were organisms and their parts/products (biotic entities - precision: 72.09%; recall: 54.17%) and systems and environments (aggregate entities - precision: 79.23%; recall: 75.34%). Terms and their relationships extracted by ClearEarth can be embedded in the new ecocore ontology after vetting (http://www.obofoundry.org/ontology/ecocore.html). This project enables use of advanced industry and research software within natural sciences for downstream operations such as data discovery, assessment, and analysis. In addition, ClearEarth uses the NLP results to generate domain-specific ontologies and other semantic resources.
- Open Access
- 1 January 2017
- Published by Association for Computational Linguistics (ACL)
- p. 463-472
Abstract Meaning Representation Parsing using LSTM Recurrent Neural Networks
- 1 January 2017
- Published by Association for Computational Linguistics (ACL)
- p. 463-472
Open AccessAbstract Meaning Representation Parsing using LSTM Recurrent Neural Networks