The freshness of web search engine databases
Authors:
- Dirk LewandowskiDLDirk Lewandowski
- Henry WahligHWHenry Wahlig
- 1 April 2006
- journal article
- research article
- Published by SAGE Publications in Journal of Information Science
- Vol. 32 (2) , 131-148
Abstract
This study measures the frequency with which search engines update their indices. Therefore, 38 websites that are updated on a daily basis were analysed within a time-span of six weeks. The analysed search engines were Google, Yahoo and MSN. We find that Google performs best overall with the most pages updated on a daily basis, but only MSN is able to update all pages within a time-span of less than 20 days. Both other engines have outliers that are older. In terms of indexing patterns, we find different approaches at the different engines. While MSN shows clear update patterns, Google shows some outliers and the update process of the Yahoo index seems to be quite chaotic. Implications are that the quality of different search engine indices varies and more than one engine should be used when searching for current content.Keywords
This publication has 9 references indexed in Scilit:
- Date‐restricted queries in web search enginesOnline Information Review, 2004
- Web‐crawling reliabilityJournal of the American Society for Information Science and Technology, 2004
- Search engine coverage bias: evidence and possible causesInformation Processing & Management, 2004
- Web search strategies and retrieval effectiveness: an empirical studyJournal of Documentation, 2002
- Assessing bias in search enginesInformation Processing & Management, 2002
- A case study in web search using TREC algorithms2001
- Accessibility of information on the webNature, 1999
- First 20 precision among World Wide Web search services (search engines)Journal of the American Society for Information Science, 1999
- Searching the World Wide WebScience, 1998