Towards building a scholarly big data platform: Challenges, lessons and opportunities
- 1 September 2014
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 117-126
Abstract
We introduce a Big Data platform that provides various services for harvesting scholarly information and enabling efficient scholarly applications. The core architecture of the platform is built on a secured private cloud, crawls data using a scholarly focused crawler that leverages a dynamic scheduler, processes by utilizing a map reduce based crawl-extraction-ingestion (CEI) workflow, and is stored in distributed repositories and databases. Services such as scholarly data harvesting, information extraction, and user information and log data analytics are integrated into the platform and provided by an OAI and RESTful API. We also introduce a set of scholarly applications built on top of this platform including citation recommendation and collaborator discovery.Keywords
This publication has 32 references indexed in Scilit:
- The Number of Scholarly Documents on the Public WebPLOS ONE, 2014
- A classification scheme for algorithm citation function in scholarly works2013
- Recommending citations2012
- The evolution of a crawling strategy for an academic document search engine2012
- An analysis of web proxy logs with query distribution pattern approach for search enginesComputer Standards & Interfaces, 2012
- ReVision2011
- Automated analysis of images in documents for intelligent document searchInternational Journal on Document Analysis and Recognition (IJDAR), 2009
- Time series analysis of a Web search engine transaction logInformation Processing & Management, 2009
- Exploring behavior of E-journal users in science and technology: Transaction log analysis of Elsevier's ScienceDirect OnSite in TaiwanLibrary & Information Science Research, 2002
- Real life information retrieval: a study of user queries on the WebACM SIGIR Forum, 1998