A focused crawler for Dark Web forums
- 11 May 2010
- journal article
- research article
- Published by Wiley in Journal of the American Society for Information Science and Technology
- Vol. 61 (6) , 1213-1231
Abstract
The unprecedented growth of the Internet has given rise to the Dark Web, the problematic facet of the Web associated with cybercrime, hate, and extremism. Despite the need for tools to collect and analyze Dark Web forums, the covert nature of this part of the Internet makes traditional Web crawling techniques insufficient for capturing such content. In this study, we propose a novel crawling system designed to collect Dark Web forum content. The system uses a human‐assisted accessibility approach to gain access to Dark Web forums. Several URL ordering features and techniques enable efficient extraction of forum postings. The system also includes an incremental crawler coupled with a recall‐improvement mechanism intended to facilitate enhanced retrieval and updating of collected content. Experiments conducted to evaluate the effectiveness of the human‐assisted accessibility approach and the recall‐improvement‐based, incremental‐update procedure yielded favorable results. The human‐assisted approach significantly improved access to Dark Web forums while the incremental crawler with recall improvement also outperformed standard periodic‐ and incremental‐update approaches. Using the system, we were able to collect over 100 Dark Web forums from three regions. A case study encompassing link and content analysis of collected forums was used to illustrate the value and importance of gathering and analyzing content from such online communities.Keywords
This publication has 26 references indexed in Scilit:
- A hybrid approach to Web forum interactional coherence analysisJournal of the American Society for Information Science and Technology, 2008
- Learning to crawlACM Transactions on Information Systems, 2005
- Applying Authorship Analysis to Extremist-Group Web Forum MessagesIEEE Intelligent Systems, 2005
- US Domestic Extremist Groups on the Web: Link and Content AnalysisIEEE Intelligent Systems, 2005
- Research feature - Comparison of three vertical search spidersComputer, 2003
- Tools for navigating large social cyberspacesCommunications of the ACM, 2002
- Studying Hate Crime with the Internet: What Makes Racists Advocate Racial Violence?Journal of Social Issues, 2002
- Information warfare: new battle fields Terrorists, propaganda and the InternetAslib Proceedings, 2001
- Database techniques for the World-Wide WebACM SIGMOD Record, 1998
- Efficient crawling through URL orderingComputer Networks and ISDN Systems, 1998