Using the wayback machine to mine websites in the social sciences: A methodological resource
Open Access
- 5 May 2015
- journal article
- research article
- Published by Wiley in Journal of the Association for Information Science and Technology
- Vol. 67 (8) , 1904-1915
Abstract
Websites offer an unobtrusive data source for developing and analyzing information about various types of social science phenomena. In this paper, we provide a methodological resource for social scientists looking to expand their toolkit using unstructured web‐based text, and in particular, with the Wayback Machine, to access historical website data. After providing a literature review of existing research that uses the Wayback Machine, we put forward a step‐by‐step description of how the analyst can design a research project using archived websites. We draw on the example of a project that analyzes indicators of innovation activities and strategies in 300 U.S. small‐ and medium‐sized enterprises in green goods industries. We present six steps to access historical Wayback website data: (a) sampling, (b) organizing and defining the boundaries of the web crawl, (c) crawling, (d) website variable operationalization, (e) integration with other data sources, and (f) analysis. Although our examples draw on specific types of firms in green goods industries, the method can be generalized to other areas of research. In discussing the limitations and benefits of using the Wayback Machine, we note that both machine and human effort are essential to developing a high‐quality data set from archived web information.Keywords
Funding Information
- U.K. Economic and Social Research Council (ES/J008303/1)
- Center for Nanotechnology in Society (0531194, 0937591)
This publication has 32 references indexed in Scilit:
- Getting More Out of Biomedical Documents with GATE's Full Lifecycle Open Source Text AnalyticsPLOS Computational Biology, 2013
- Computational Social ScienceScience, 2009
- University entrepreneurship: a taxonomy of the literatureIndustrial and Corporate Change, 2007
- Constitutional Culture, Social Movement Conflict and Constitutional Change: The Case of the De Facto Era. 2005-06 Brennan Center Symposium LectureCalifornia Law Review, 2006
- A fresh look at the reliability of long-term digital storageACM SIGOPS Operating Systems Review, 2006
- Social Bookmarking Tools (I)D-Lib Magazine, 2005
- WebometricsAnnual Review of Information Science and Technology, 2005
- Non-regulated water contaminants: emerging researchEnvironmental Impact Assessment Review, 2004
- Collaboration Networks, Structural Holes, and Innovation: A Longitudinal StudyAdministrative Science Quarterly, 2000
- Toward a Theory of International New venturesJournal of International Business Studies, 1994