Enhancing Language Models via HTML DOM Tree for Text Structure Understanding
- 31 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1653-1663
Abstract
Understanding text structure, which enables the automated system to parse long text structure, is crucial for various natural language processing applications such as information extraction, summarization, and question answering. Although previous methods have advanced text structure parsing effectively, they face challenges such as not leveraging the abundance of unlabelled data and focusing mainly on content-inferred information. To address this deficiency, this paper introduces a novel Text Structure Language Model (TSLM), an LM pre-training framework that employs ubiquitous HTML documents and considers the text structure among text units. HTML documents are composed by experts and their hierarchies can reflect the structure of documents. Our learning framework is designed to equip the LM with awareness of two complementary kinds of structures from HTML documents. It encourages the model to learn local structure which helps in understanding the immediate connection between two units by reconstructing the structure of DOM tree, and global structure which shapes the overall organization and thematic development by predicting the optimal content-fitting tree. Extensive experiments with structure-related downstream tasks, including text segmentation and table of contents generation, validate the effectiveness of TSLM.Keywords
Funding Information
- National Natural Science Foundation of China (61972349)
- National Key R&D Program of China (2018YFC2002603)
- Alibaba-Zhejiang University Joint Institute of Frontier Technologies
This publication has 38 references indexed in Scilit:
- A Sequence-to-Sequence Approach with Mixed Pointers to Topic Segmentation and Segment Labeling2023
- Webformer2022
- WebKE2021
- Image-Based Table Recognition: Data, Model, and Evaluation2020
- FreeDOM: A Transferable Neural Architecture for Structured Information Extraction on Web Documents2020
- Table-of-Contents Generation on Contemporary Documents2019
- Learning Structured Text RepresentationsTransactions of the Association for Computational Linguistics, 2018
- Neural Motifs: Scene Graph Parsing with Global Context2018
- Fundamentals of Data Mining in Genomics and Proteomics2007
- Statistical Models for Text SegmentationMachine Learning, 1999