Web‐crawling reliability
Author:
- 13 August 2004
- journal article
- research article
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 55 (14) , 1228-1238
Abstract
In this article, I investigate the reliability, in the social science sense, of collecting informetric data about the World Wide Web by Web crawling. The investigation includes a critical examination of the practice of Web crawling and contrasts the results of content crawling with the results of link crawling. It is shown that Web crawling by search engines is intentionally biased and selective. I also report the results of a large‐scale experimental simulation of Web crawling that illustrates the effects of different crawling policies on data collection. It is concluded that the reliability of Web crawling as a data collection technique is improved by fuller reporting of relevant crawling policies.Keywords
This publication has 28 references indexed in Scilit:
- Growth and structure of the World Wide Web: Towards realistic modelingComputer Physics Communications, 2002
- Methodologies for crawler based Web surveysInternet Research, 2002
- Scholarly communication and bibliometricsAnnual Review of Information Science and Technology, 2002
- Internet search engines – fluctuations in document accessibilityJournal of Documentation, 2001
- Bibliometrics and beyond: some thoughts on web-based citation analysisJournal of Information Science, 2001
- Hyperlink analysis for the WebIEEE Internet Computing, 2001
- Google's web page ranking applied to different topological web graph structuresJournal of the American Society for Information Science and Technology, 2001
- A technique for measuring the relative size and overlap of public Web search enginesComputer Networks and ISDN Systems, 1998
- The anatomy of a large-scale hypertextual Web search engineComputer Networks and ISDN Systems, 1998
- The RBSE spider — Balancing effective search against Web loadComputer Networks and ISDN Systems, 1994