The Cost of a WARC: Analyzing Web Archives in the Cloud
- 1 June 2019
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 261-264
Abstract
The value of web archives to support scholarship in the humanities and social sciences is slowly being realized by the increasing availability of scalable tools and platforms. The cost of providing scholarly access is a critical component of developing a long-term sustainability strategy. This paper attempts to answer a straightforward question: How much does it cost to analyze web archives in the cloud? To make this question more concrete, we examine the creation of three derivatives (extraction of collection statistics, full text, and the webgraph) that serve as the starting points of many scholarly inquiries. Our analysis shows that these typical derivatives costs around US$7 per TB using our Archives Unleashed Toolkit. We describe in detail the methodology and assumptions made to arrive at this figure. To our knowledge, we are the first to quantify the economics of scholarly access to web archives, and we believe that this information is valuable for service planning by archives, libraries, and other institutions.Keywords
This publication has 7 references indexed in Scilit:
- Building Community and Tools for Analyzing Web Archives Through Datathons2019
- The Archived Web2018
- WarcbaseJournal on Computing and Cultural Heritage, 2017
- ArchiveSpark2016
- Content Selection and Curation for Web Archiving2016
- Evaluating Cost of Cloud Execution in a Data Repository2016
- Lost in the Infinite Archive: The Promise and Pitfalls of Web ArchivesInternational Journal of Humanities and Arts Computing, 2016