Towards automated content analysis of discussion transcripts
Open Access
- 1 January 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 15-24
Abstract
In this paper, we present the results of an exploratory study that examined the problem of automating content analysis of student online discussion transcripts. We looked at the problem of coding discussion transcripts for the levels of cognitive presence, one of the three main constructs in the Community of Inquiry (CoI) model of distance education. Using Coh-Metrix and LIWC features, together with a set of custom features developed to capture discussion context, we developed a random forest classification system that achieved 70.3% classification accuracy and 0.63 Cohen's kappa, which is significantly higher than values reported in the previous studies. Besides improvement in classification accuracy, the developed system is also less sensitive to overfitting as it uses only 205 classification features, which is around 100 times less features than in similar systems based on bag-of-words features. We also provide an overview of the classification features most indicative of the different phases of cognitive presence that gives an additional insights into the nature of cognitive presence learning cycle. Overall, our results show great potential of the proposed approach, with an added benefit of providing further characterization of the cognitive presence coding scheme.Keywords
Funding Information
- JISC (36)
This publication has 25 references indexed in Scilit:
- Externally-facilitated regulation scaffolding and role assignment to develop cognitive presence in asynchronous online discussionsThe Internet and Higher Education, 2015
- Content analysis of computer conferencing transcriptsInteractive Learning Environments, 2011
- Coh-MetrixEducational Researcher, 2011
- What's It All About?2011
- Review of computer mediated communication research for educationInstructional Science, 2006
- Content analysis: What are they talking about?Computers & Education, 2006
- Creating cognitive presence in a blended faculty development communityThe Internet and Higher Education, 2005
- Critical thinking, cognitive presence, and computer conferencing in distance educationAmerican Journal of Distance Education, 2001
- The measurement of textual coherence with latent semantic analysisDiscourse Processes, 1998
- Indexing by latent semantic analysisJournal of the American Society for Information Science, 1990