A Structure-Oriented Unsupervised Crawling Strategy for Social Media Sites
Authors:
- Keyang XuKXKeyang Xu
- Kyle Yingkai GaoKGKyle Yingkai Gao
- Jamie CallanJCJamie Callan
Preprint
Abstract
Existing techniques for efficiently crawling social media sites rely on URL patterns, query logs, and human supervision. This paper describes SOUrCe, a structure-oriented unsupervised crawler that uses page structures to learn how to crawl a social media site efficiently. SOUrCe consists of two stages. During its unsupervised learning phase, SOUrCe constructs a sitemap that clusters pages based on their structural similarity and generates a navigation table that describes how the different types of pages in the site are linked together. During its harvesting phase, it uses the navigation table and a crawling policy to guide the choice of which links to crawl next. Experiments show that this architecture supports different styles of crawling efficiently, and does a better job of staying focused on user-created contents than baseline methods.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Using XPaths of inbound links to cluster template-generated web pagesComputer Science and Information Systems, 2014
- Novel approaches to crawling important pages earlyKnowledge and Information Systems, 2012
- FoCUS: Learning to Crawl Web ForumsIEEE Transactions on Knowledge and Data Engineering, 2012
- Clustering Web pages based on their structureData & Knowledge Engineering, 2005
- Web page feature selection and classification using neural networksInformation Sciences, 2004
- Authoritative sources in a hyperlinked environmentJournal of the ACM, 1999
- Finding related pages in the World Wide WebComputer Networks, 1999
- Efficient crawling through URL orderingComputer Networks and ISDN Systems, 1998
- Term-weighting approaches in automatic text retrievalInformation Processing & Management, 1988