Text reuse in large historical corpora: insights from the optimization of a data science system
Open Access
- 7 April 2025
- journal article
- research article
- Published by Springer Nature in International Journal of Data Science and Analytics
- Vol. 20 (5) , 4631-4643
Abstract
Text reuse is of fundamental importance in humanities research, as near-verbatim pieces of text in different documents provide invaluable information about the historical spread, evolution of ideas and composition of cultural artifacts. Traditionally, scholars have studied text reuse at a very small scale, for example, when comparing the writings of two philosophers; however, modern digitized corpora spanning entire centuries promise to revolutionize humanities research through the detection of previously unobserved large-scale patterns. This paper presents insights from , a system for large-scale text reuse analysis over almost all known 18th-century books, articles, and newspapers. The system implements a data management pipeline for billions of text reuse instances and supports analysis tasks based on database queries (e.g., retrieving the most reused quotes from queried documents). The paper describes the principled and extensive evaluations across different normalization levels, query execution engines, and queries of interest that led to an optimized system—and offers insights from the observed trade-offs and how they were resolved to fit specific requirements. In summary, the paper explains how, for our system, (1) the row-store engine (MariaDB Aria) with denormalized relations emerged as the optimal choice for front-end interfaces, while (2) big data processing (Apache Spark) proved irreplaceable for data preprocessing.Keywords
Funding Information
- University of Helsinki
This publication has 22 references indexed in Scilit:
- The Reception of Hume’s Essays in Eighteenth-Century Britain2025
- Old Books and Digital Publishing: Eighteenth-Century Collections Online2020
- Quantifying the impact of dirty OCR on historical text analysis: Eighteenth Century Collections Online as a case studyDigital Scholarship in the Humanities, 2019
- Mass Digitization of Early Modern Texts With Optical Character RecognitionJournal on Computing and Cultural Heritage, 2017
- Computational Methods for Uncovering Reprinted Texts in Antebellum NewspapersAmerican Literary History, 2015
- Towards a Historical Text Re-use Detection2014
- Document Image Quality Assessment: A Brief Survey2013
- The Star Schema Benchmark and Augmented Fact Table Indexing2009
- Column-stores vs. row-stores2008
- Basic local alignment search toolJournal of Molecular Biology, 1990