Clustering-based incremental web crawling
Authors:
Open Access
- 23 November 2010
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 28 (4) , 1-27
Abstract
When crawling resources, for example, number of machines, crawl-time, and so on, are limited, so a crawler has to decide an optimal order in which to crawl and recrawl Web pages. Ideally, crawlers should request only those Web pages that have changed since the last crawl; in practice, a crawler may not know whether a Web page has changed before downloading it. In this article, we identify features of Web pages that are correlated to their change frequency. We design a crawling algorithm that clusters Web pages based on features that correlate to their change frequencies obtained by examining past history. The crawler downloads a sample of Web pages from each cluster, and depending upon whether a significant number of these Web pages have changed in the last crawl cycle, it decides whether to recrawl the entire cluster. To evaluate the performance of our incremental crawler, we develop an evaluation framework that measures which crawling policy results in the best search results for the end-user. We run experiments on a real Web data set of about 300,000 distinct URLs distributed among 210 Web sites. The results demonstrate that the clustering-based sampling algorithm effectively clusters the pages with similar change patterns, and our clustering-based crawling algorithm outperforms existing algorithms in that it can improve the quality of the user experience for those who query the search engine.Keywords
Funding Information
- National Science Foundation (5.36E+12)
This publication has 30 references indexed in Scilit:
- Methods for comparing rankings of search engine resultsComputer Networks, 2006
- Effective page refresh policies for Web crawlersACM Transactions on Database Systems, 2003
- How dynamic is the Web?Computer Networks, 2000
- WTMS: a system for collecting and analyzing topic-specific Web informationComputer Networks, 2000
- Synchronizing a database to improve freshnessACM SIGMOD Record, 2000
- Keeping up with the changing WebComputer, 2000
- Analysis of a very large web search engine query logACM SIGIR Forum, 1999
- Focused crawling: a new approach to topic-specific Web resource discoveryComputer Networks, 1999
- The shark-search algorithm. An application: tailored Web site mappingComputer Networks and ISDN Systems, 1998
- The anatomy of a large-scale hypertextual Web search engineComputer Networks and ISDN Systems, 1998