Alexey Shigarov

Сегодня в мире имеется большое количество реляционных данных, доступных в неструктурированных источниках, а именно в таблицах произвольной формы, являющихся частью электронных документов различных редактируемых форматов. Последние, в отличие от баз данных, предназначены не для того, чтобы быть интерпретируемыми компьютерными программами, а исключительно для понимания человеком. Только будучи приведенными к структурированной форме, они станут доступными для использования в различных приложениях автоматической интерпретации данных. Настоящая работа посвящена комплексной научной задаче автоматизации процессов извлечения реляционных данных из произвольных таблиц электронных редактируемых документов. Предложен метод решения поставленной задачи на основе пользовательского программирования правил анализа и интерпретации таблиц, приводимых к формату так называемых рабочих книг (Excel/Sheets). Впервые такие правила отделены от применяемых моделей представления и алгоритмов обработки, при этом обеспечена поддержка произвольности структуры таблиц, а именно пользовательской компоновки, структурированности ячеек и иерархичности заголовков. Выполненная оценка производительности решений, реализованных на основе предлагаемого метода, показывает высокую эффективность их применения. Currently, a vast amount of relational data is available in unstructured sources, particularly as arbitrary tables within electronic documents in various editable formats. Unlike databases, these tables are primarily designed for human comprehension rather than for direct interpretation by computer programs. To be utilized in applications requiring automatic data interpretation, these tables have to be converted into a structured representation first. This work addresses the complex task of automating the extraction of relational data from arbitrary tables within electronic documents. A novel method is proposed to tackle this problem, leveraging end-user programming to create rules for analyzing and interpreting tables that have been converted into spreadsheet formats (Excel/Sheets). Unlike the existing rule-based methods devoted to the stated task, this approach decouples the rules from the underlying representation models and processing algorithms. Additionally, the proposed method accommodates arbitrary table layouts, structured cells, and header hierarchies. The tools implementing this method offer several notable advantages. The developed model of a document table supports arbitrary layouts, structured cells, and header hierarchies, providing flexibility in handling diverse table formats. The problem-oriented language, CRL, simplifies syntax specifically designed for end-user programming. Unlike general-purpose rule languages, CRL conceals irrelevant details, allowing users to focus exclusively on the logic of table analysis and interpretation. CRL rules can effectively replace specialized algorithms typically implemented using imperative programming or supervised learning. A single set of CRL rules can process a class of tables that share similar layout, formatting, and content characteristics. The complexity of implementing these rules varies depending on the diversity of these shared properties. The tools for table analysis and interpretation are integrated into the TabbyXL software package, which specializes in extracting data from tables. TabbyXL supports workbooks in formats (Excel/Sheets). It is important to note that document tables in other editable formats (Word/Docs, HTML, etc.) can be converted to the supported formats using existing conversion tools. The effectiveness of TabbyXL is substantiated by experimental data, both qualitative and quantitative comparisons with existing analogues. Additionally, the software has been successfully implemented in various scientific and industrial projects, addressing practical challenges such as data integration, constructing domain-specific ontologies, and managing the template document exchange.

This paper introduces ‘‘Regular Table Language’’ (RTL), a novel domain-specific language designed for extracting recordsets from arbitrary tables presented in machine-readable documents, such as spreadsheets, rich text, web pages (HTML), etc. The foundation of RTL rests on the hypothesis that any table can be aligned with a pattern that defines its structure. Furthermore, a whole class of tables designed by one template can be specified by one pattern. RTL offers a formalized way to express such patterns in a clear and concise manner. It is also proposed the ‘‘Interpretable Table Model’’ (ITM) as an intermediary between source tables and target recordsets. This model extends the conventional table structures found in widely used formats by adding semantics that provide the automatic inference of recordsets from tables. This semantics is recovered through matching appropriate RTL-patterns with ITM-instances generated from source tables. Both RTL and ITM have been implemented as core components of RegTab, an open-source software library designed to streamline the development of custom applications where data sources are various tables presented in electronic documents.

Tables are widely used to present related data in various formats and media. In particular, an immense number of HTML tables can be found on web pages. These tables serve as a valuable source of data for applications such as web mining, question-answering systems, and knowledge base development. However, not all HTML tables contain actual relational data. Many of them are used primarily for layout and navigation purposes. On the other hand, genuine tables may have different layouts, formatting, and content. One of the challenges in extracting data from web tables is to identify the main functional and layout types and properties of the tables that are commonly found on the internet. Currently, we have a wide range of taxonomies that classify table types and their properties. These taxonomies can be used to categorize tables and select the appropriate method for processing tabular data based on their specific type. Although the existing taxonomies provide a variety of table types, they often use confusing terminology. This paper aims to provide an overview of these taxonomies by aligning their terminology and comparing them in a qualitative manner.

Tables are probably the most natural way to represent relational data in various media and formats. They store a large number of valuable facts that could be utilized for question answering, knowledge base population, natural language generation, and other applications. However, many tables are not accompanied by semantics for the automatic interpretation of the information they present. Table Understanding (TU) aims at recovering the missing semantics that enables the extraction of facts from tables. This problem covers a range of issues from table detection in document images to semantic table interpretation with the help of external knowledge bases. To date, the TU research has been ongoing on for 30 years. Nevertheless, there is no common point of view on the scope of TU; the terminology still needs agreement and unification. In recent years, science and technology have shown a rapidly increasing interest in TU. Nowadays, it is especially important to check the meaning of this research problem once again. This article gives a comprehensive characterization of the TU problem, including a description of its subproblems, tasks, subtasks, and applications. It also discusses the common limitations used in the existing problem statements and proposes some directions for further research that would help overcome the corresponding limitations.This article is categorized under: Algorithmic Development > Text Mining Algorithmic Development > Web Mining

German-Russian Astroparticle Data Life Cycle Initiative is an international project whose aim is to develop a distributed data storage system that aggregates data from the storage systems of different astroparticle experiments. The prototype of such a system, which is called the Astroparticle Physics Distributed Data Storage (APPDS), has been being developed. In this paper, the Data Aggregation Service, one of the core services of APDDS, is presented. The Data Aggregation Service connects all distributed services of APPDS together to find the necessary data and deliver them to users on demand.