BERT Based Topic-Specific Crawler
Authors:
- 6 October 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-5
Abstract
Nowadays, retrieving certain information using search engines is very popular and one of the main applications of the Internet. To speed up the process of getting the required information(web pages), having a topic-specific crawler is essential to fetch and index only the relevant ones. This paper presents a multi-thread web crawler using a Sentence Bidirectional Encoder Representations from Transformers (S-BERT). The S- BERT is used to calculate the similarity between the predefined classes and the text of the downloaded web pages. This provides a lightweight model compared to using a word embedding with deep learning for text classification.Keywords
This publication has 6 references indexed in Scilit:
- A Review of Focused Crawling Schemes for Search Engine2020
- Predictive Crawling for Commercial Web Content2019
- Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- An Efficient Focused Web Crawling Approach2018
- Keyword query based focused Web crawlerProcedia Computer Science, 2018