Bandit on the Hunt: Dynamic Crawling for Cyber Threat Intelligence
Preprint
Abstract
Public information contains valuable Cyber Threat Intelligence (CTI) that is used to prevent future attacks. While standards exist for sharing this information, much appears in non-standardized news articles or blogs. Monitoring online sources for threats is time-consuming and source selection is uncertain. Current research focuses on extracting Indicators of Compromise from known sources, rarely addressing new source identification. This paper proposes a CTI-focused crawler using multi-armed bandit (MAB) and various crawling strategies. It employs SBERT to identify relevant documents while dynamically adapting its crawling path. Our system ThreatCrawl achieves a harvest rate exceeding 25% and expands its seed by over 300% while maintaining topical focus. Additionally, the crawler identifies previously unknown but highly relevant overview pages, datasets, and domains.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- DomURLs_BERT: Pre-trained BERT-based Model for Malicious Domains and URLs Detection and ClassificationJournal of Network and Systems Management, 2026
- Common vulnerability scoring system prediction based on open source intelligence information sourcesComputers & Security, 2023
- A focused crawler based on semantic disambiguation vector space modelComplex & Intelligent Systems, 2022
- A Feature-driven Method for Automating the Assessment of OSINT Cyber Threat SourcesComputers & Security, 2022
- BERT Based Topic-Specific Crawler2021
- CySecAlert: An Alert Generation System for Cyber Security Events Using Open Source Intelligence Data2021
- Cyber Threat Intelligence: A Product Without a Process?International Journal of Intelligence and CounterIntelligence, 2020
- SIREN - Security Information Retrieval and Extraction eNgine2018
- Focused crawling: a new approach to topic-specific Web resource discoveryComputer Networks, 1999
- Surfing the Web backwardsComputer Networks, 1999