Web robot detection in the scholarly information environment
- 3 April 2008
- journal article
- Published by SAGE Publications in Journal of Information Science
- Vol. 34 (5) , 726-741
Abstract
An increasing number of robots harvest information on the world wide web for a wide variety of purposes. Protocols developed at the inception of the web laid out voluntary procedures in order to identify robot behaviour, and exclude it if necessary. Few robots now follow this protocol and it is now increasingly difficult to filter for this activity in reports of on-site activity. This paper seeks to demonstrate the issues involved in identifying robots and assessing their impact on usage in regard to a project which sought to establish the relative usage patterns of open access and non-open access articles in the Oxford University Press published journal Glycobiology, which offers in a single issue articles in both forms. A number of methods for identifying robots are compared and together these methods found that 40% of the raw logs of this journal could be attributed to robots.Keywords
This publication has 6 references indexed in Scilit:
- Open access in context: a user studyJournal of Documentation, 2007
- The impact of open access publishing (and other access initiatives) on use and users of digital scholarly journalsLearned Publishing, 2007
- An investigation of web crawler behavior: characterization and metricsComputer Communications, 2005
- The dark side of the WebACM SIGCOMM Computer Communication Review, 2004
- Discovery of Web Robot Sessions Based on their Navigational PatternsData Mining and Knowledge Discovery, 2002
- Ethical Web agentsComputer Networks and ISDN Systems, 1995