Word-based self-indexes for natural language text
Open Access
- 1 February 2012
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 30 (1) , 1-34
Abstract
The inverted index supports efficient full-text searches on natural language text collections. It requires some extra space over the compressed text that can be traded for search speed. It is usually fast for single-word searches, yet phrase searches require more expensive intersections. In this article we introduce a different kind of index. It replaces the text using essentially the same space required by the compressed text alone (compression ratio around 35%). Within this space it supports not only decompression of arbitrary passages, but efficient word and phrase searches. Searches are orders of magnitude faster than those over inverted indexes when looking for phrases, and still faster on single-word searches when little space is available. Our new indexes are particularly fast at counting the occurrences of words or phrases. This is useful for computing relevance of words or phrases. We adapt self-indexes that succeeded in indexing arbitrary strings within compressed space to deal with large alphabets. Natural language texts are then regarded as sequences of words, not characters, to achieve word-based self-indexes. We design an architecture that separates the searchable sequence from its presentation aspects. This permits applying case folding, stemming, removing stopwords, etc. as is usual on inverted indexes.Keywords
Funding Information
- Ministerio de Ciencia e Innovación (CDTI CEN-20091048, TIN2009-14560-C03-02TIN2010-21246-C02-01)
- Fondo Nacional de Desarrollo Científico y Tecnológico (1-0800191-110066)
- Xunta de Galicia (2010/17 (for the Spanish group))
This publication has 45 references indexed in Scilit:
- Engineering basic algorithms of an in-memory text search engineACM Transactions on Information Systems, 2010
- Efficient set intersection for inverted indexingACM Transactions on Information Systems, 2010
- An experimental investigation of set intersection algorithms for text searchingACM Journal of Experimental Algorithmics, 2009
- Compressed text indexesACM Journal of Experimental Algorithmics, 2009
- Dynamic entropy-compressed sequences and full-text indexesACM Transactions on Algorithms, 2008
- Alternation and redundancy analysis of the intersection problemACM Transactions on Algorithms, 2008
- Inverted files for text search enginesACM Computing Surveys, 2006
- New text indexing functionalities of the compressed suffix arraysJournal of Algorithms, 2003
- Compressing Integers for Fast File AccessThe Computer Journal, 1999
- On the implementation of minimum redundancy prefix codesIEEE Transactions on Communications, 1997