A scalable search index for binary files
Authors:
- 1 October 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 1741, 94-103
Abstract
The ability to locate specific byte-sequences in large collections of binary files is important in many applications, especially malware analysis. However, it can be a time consuming process. Researchers and analysts, such as those at CERT, often have to search terabytes of data for characteristic patterns and signatures, which can take upwards of days to complete. Although many search systems, designed specifically to expedite text and metadata queries, exist, these tools are unsuitable for searching files containing arbitrary bytes. By using probabilistic techniques to pre-filter likely search candidates, we present a scalable architecture for searching and indexing terabyte-size collections of binary files. Our implementation performs searches in minutes that would required days to complete using iterative techniques. It also reduces storage costs by balancing the amount of data indexed with the total time required to conduct and verify a query.Keywords
This publication has 8 references indexed in Scilit:
- Inverted index compression and query processing with optimized document ordering2009
- Introduction to Information Retrieval2008
- Performance of compressed inverted list caching in search engines2008
- Sorting Out the Document Identifier Assignment Problem2007
- Super-Scalar RAM-CPU Cache Compression2006
- An efficient document retrieval method using n‐gram indexingSystems and Computers in Japan, 2002
- Constructing the Suffix Tree of a Tree with a Large AlphabetLecture Notes in Computer Science, 1999
- Complete statistical indexing of text by overlapping word fragmentsACM SIGIR Forum, 1983