Ad-hoc data processing in the cloud
Authors:
Open Access
- 1 August 2008
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 1 (2) , 1472-1475
Abstract
Ad-hoc data processing has proven to be a critical paradigm for Internet companies processing large volumes of unstructured data. However, the emergence of cloud-based computing, where storage and CPU are outsourced to multiple third-parties across the globe, implies large collections of highly distributed and continuously evolving data. Our demonstration combines the power and simplicity of the MapReduce abstraction with a wide-scale distributed stream processor, Mortar. While our incremental MapReduce operators avoid data re-processing, the stream processor manages the placement and physical data flow of the operators across the wide area. We demonstrate a distributed web indexing engine against which users can submit and deploy continuous MapReduce jobs. A visualization component illustrates both the incremental indexing and index searches in real time.Keywords
This publication has 5 references indexed in Scilit:
- Map-reduce-merge2007
- Dryad2007
- Fault-tolerance in the Borealis distributed stream processing system2005
- UbiCrawler: a scalable fully distributed Web crawlerSoftware: Practice and Experience, 2004
- Parallel database systemsCommunications of the ACM, 1992