ArchiveSpark
Authors:
Open Access
- 19 June 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 83-92
Abstract
Web archives are a valuable resource for researchers of various disciplines. However, to use them as a scholarly source, researchers require a tool that provides efficient access to Web archive data for extraction and derivation of smaller datasets. Besides efficient access we identify five other objectives based on practical researcher needs such as ease of use, extensibility and reusability. Towards these objectives we propose ArchiveSpark, a framework for efficient, distributed Web archive processing that builds a research corpus by working on existing and standardized data formats commonly held by Web archiving institutions. Performance optimizations in ArchiveSpark, facilitated by the use of a widely available metadata index, result in significant speed-ups of data processing. Our benchmarks show that ArchiveSpark is faster than alternative approaches without depending on any additional data stores while improving usability by seamlessly integrating queries and derivations with external tools.Keywords
All Related Versions
Funding Information
- European Research Council (ERC 339233)
This publication has 8 references indexed in Scilit:
- Improving Access to Large-scale Digital Libraries ThroughSemantic-enhanced Search and Disambiguation2015
- Access and Scholarly Use of Web ArchivesAlexandria: The Journal of National and International Library and Information Issues, 2014
- Infrastructure for supporting exploration and discovery in web archives2014
- Temporal index sharding for space-time efficiency in archive search2011
- A Survey on Web Archiving Initiatives2011
- CassandraACM SIGOPS Operating Systems Review, 2010
- MapReduceCommunications of the ACM, 2010
- BigtableACM Transactions on Computer Systems, 2008