A Web Scraping Methodology for Bypassing Twitter API Restrictions
Preprint
Abstract
Retrieving information from social networks is the first and primordial step many data analysis fields such as Natural Language Processing, Sentiment Analysis and Machine Learning. Important data science tasks relay on historical data gathering for further predictive results. Most of the recent works use Twitter API, a public platform for collecting public streams of information, which allows querying chronological tweets for no more than three weeks old. In this paper, we present a new methodology for collecting historical tweets within any date range using web scraping techniques bypassing for Twitter API restrictions.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- Fuzzy Modelling for Human Dynamics Based on Online Social NetworksSensors, 2017
- RCrawler: An R package for parallel web crawling and scrapingSoftwareX, 2017
- A web-based, interactive visualization tool for social environmental survey dataEnvironmental Modelling & Software, 2016
- Analysing How People Orient to and Spread Rumours in Social Media by Looking at Conversational ThreadsPLOS ONE, 2016
- Tweets as impact indicators: Examining the implications of automated “bot” accounts on TwitterJournal of the Association for Information Science and Technology, 2015
- An Efficient Approach for Web Indexing of Big Data through Hyperlinks in Web CrawlingThe Scientific World Journal, 2015