A three-year study on the freshness of web search engine databases
- 8 May 2008
- journal article
- research article
- Published by SAGE Publications in Journal of Information Science
- Vol. 34 (6) , 817-831
Abstract
This paper deals with one aspect of the index quality of search engines: index freshness. The purpose is to analyse the update strategies of the major web search engines Google, Yahoo, and MSN/Live.com. We conducted a test of the updates of 40 daily updated pages and 30 irregularly updated pages. We used data from a time span of six weeks in the years 2005, 2006 and 2007. We found that the best search engine in terms of up-to-dateness changes over the years and that none of the engines has an ideal solution for index freshness. Indexing patterns are often irregular, and there seems to be no clear policy regarding when to revisit Web pages. A major problem identified in our research is the delay in making crawled pages available for searching, which differs from one engine to another.Keywords
This publication has 16 references indexed in Scilit:
- Equal Representation by Search Engines? A Comparison of Websites across Countries and DomainsJournal of Computer-Mediated Communication, 2007
- Web dynamics and their ramifications for the development of Web search enginesComputer Networks, 2006
- The freshness of web search engine databasesJournal of Information Science, 2006
- Date‐restricted queries in web search enginesOnline Information Review, 2004
- Search engine coverage bias: evidence and possible causesInformation Processing & Management, 2004
- A large‐scale study of the evolution of Web pagesSoftware: Practice and Experience, 2004
- Search Engine Ability to Cope With the Changing Web2004
- How Large Is the World Wide Web?2004
- Search engines and Web dynamicsComputer Networks, 2002
- Web page change and persistence—A four‐year longitudinal studyJournal of the American Society for Information Science and Technology, 2001