Tradeoffs in Accuracy and Efficiency in Supervised Learning Methods
- 28 February 2012
- journal article
- research article
- Published by Taylor & Francis in Journal of Information Technology & Politics
- Vol. 9 (3) , 298-318
Abstract
Words are an increasingly important source of data for social science research. Automated classification methodologies hold the promise of substantially lowering the costs of analyzing large amounts of text. In this article, we consider a number of questions of interest to prospective users of supervised learning methods, which are used to automatically classify events based on a pre-existing classification system. Although information scientists devote considerable attention to assessing the performance of different supervised learning algorithms and feature representations, the questions asked are often less directly relevant to the more practical concerns of social scientists. The first question prospective social science users are likely to ask is, How well do such methods work? The second is, How much human labeling effort is required? The third is, How do we assess whether virgin cases have been automatically classified with sufficient accuracy? We address these questions in the context of a particular dataset—the Congressional Bills Project—which includes more than 400,000 bill titles that humans have classified into 20 policy topics. This corpus offers an unusual opportunity to assess the performance of different algorithms, the impact of sample size, and the benefits of ensemble learning as a means for estimating classification accuracy.Keywords
This publication has 16 references indexed in Scilit:
- A Method of Automated Nonparametric Content Analysis for Social ScienceAmerican Journal of Political Science, 2009
- Measuring the Happiness of Large-Scale Written Expression: Songs, Blogs, and PresidentsJournal of Happiness Studies, 2009
- Computational Social ScienceScience, 2009
- Text Annotation for Political Science ResearchJournal of Information Technology & Politics, 2008
- Computer-Assisted Topic Classification for Mixed-Methods Social Science ResearchJournal of Information Technology & Politics, 2008
- An Automated Information Extraction Tool for International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation DesignInternational Organization, 2003
- Extracting Policy Positions from Political Texts Using Words as DataAmerican Political Science Review, 2003
- 10.1162/jmlr.2003.3.4-5.993Applied Physics Letters, 2000
- Overfitting and undercomputing in machine learningACM Computing Surveys, 1995
- A training algorithm for optimal margin classifiers1992