Qianren Mao

Timeline summarization (TLS) system involves creating a concise overview of extended daily summaries through sentence and date selection, employing predefined unlearnable representations for these elements. This poses a unique challenge, as it involves normal representations that limit the performance of the downstream daily summarization tasks. Previous research in this field separately encoded dates and sentences in independent learning representation spaces, hindering the integration of their relationships and the attainment of a globally optimal summary by the TLS system. In this study, our emphasis is on pretraining a document encoder for the TLS system, where we introduce the concept of representing each document as a plug-and-play document plugin (PlugIN). Specifically, the proposed MegraTLS (Meta-graph autoencoder based TimeLine Summarization System) is based on a modified meta-graph autoencoder network. By incorporating the prefabricated PlugIN into the backbone of the DateWise TLS framework for downstream timeline summarization, we streamline the encoding of a document, enabling efficient handling of date and sentence selection within a compact representation space in date and sentence embeddings. The experimental results demonstrate that our MegraTLS prominently promotes existing TLS frameworks in timeline summarization performance. Besides, the MegraTLS exhibits robustness and insensitivity to parameter changes. Case studies of financial events illustrate that our model accurately captures key timeline events when applied to specific fields, providing more industry-specific valuable information than ground truth while maintaining simplicity. Although our experiments are conducted exclusively on Chinese datasets, the methodological insights and empirical findings of this work offer transferable, actionable, and robust guidance for deploying timeline summarization technology in domain-specific industrial applications.

Retrieval-Augmented Generation (RAG) has recently emerged as a promising solution for enhancing the accuracy and credibility of Large Language Models (LLMs), particularly in Question & Answer tasks. This is achieved by incorporating proprietary and private data from integrated databases. However, private RAG systems face significant challenges due to the scarcity of private domain data and critical data privacy issues. These obstacles impede the deployment of private RAG systems, as developing privacy-preserving RAG systems requires a delicate balance between data security and data availability. To address these challenges, we regard federated learning (FL) as a highly promising technology for privacy-preserving RAG services. We propose a novel framework called Federated Retrieval-Augmented Generation (FedE4RAG). This framework facilitates collaborative training of client-side RAG retrieval models. The parameters of these models are aggregated and distributed on a central-server, ensuring data privacy without direct sharing of raw data. In FedE4RAG, knowledge distillation is employed for communication between the server and client models. This technique improves the generalization of local RAG retrievers during the federated learning process. Additionally, we apply homomorphic encryption within federated learning to safeguard model parameters and mitigate concerns related to data leakage. Extensive experiments conducted on the real-world dataset have validated the effectiveness of FedE4RAG. The results demonstrate that our proposed framework can markedly enhance the performance of private RAG systems while maintaining robust data privacy protection.

We propose a neuralized undirected graphical model called Neural-Hidden-CRF to solve the weakly-supervised sequence labeling problem. Under the umbrella of undirected graphical theory, the proposed Neural-Hidden-CRF embedded with a hidden CRF layer models the variables of word sequence, latent ground truth sequence, and weak label sequence with the global perspective that undirected graphical models particularly enjoy. In Neural-Hidden-CRF, we can capitalize on the powerful language model BERT or other deep models to provide rich contextual semantic knowledge to the latent ground truth sequence, and use the hidden CRF layer to capture the internal label dependencies. Neural-Hidden-CRF is conceptually simple and empirically powerful. It obtains new state-of-the-art results on one crowdsourcing benchmark and three weak-supervision benchmarks, including outperforming the recent advanced model CHMM by 2.80 F1 points and 2.23 F1 points in average generalization and inference performance, respectively.

Hashtag generation aims to generate short and informal topical tags from a microblog post, in which tokens or phrases form the hashtags. These tokens or phrases may originate from primary fragmental textual pieces (e.g., segments) in the original text and are separated into different segments. However, conventional sequence-to-sequence generation methods are hard to filter out secondary information from different textual granularity and are not good at selecting crucial tokens. Thus, they are suboptimal in generating more condensed hashtags. In this work, we propose a modified Transformer-based generation model with adding a segments-selection procedure for the original encoding and decoding phases. The segments-selection phase is based on a novel Segments Selection Mechanism (SSM) to model different textual granularity on global text, local segments, and tokens, contributing to generating condensed hashtags. Specifically, it first attends to primary semantic segments and then transforms discontinuous segments from the source text into a sequence of hashtags by selecting crucial tokens. Extensive evaluations on the two datasets reveal our approach's superiority with significant improvements to the extraction and generation baselines. The code and datasets are available at https://github.com/OpenSUM/HashtagGen.

Abstractive summarization generates a concise summary to capture the key ideas of the source text. This task underpins important applications like information retrieval, document comprehension, and event tracking. While much progress has been achieved, state-of-the-art summarization approaches often fail to generate high-quality summaries to reproduce factual details accurately. One of the key limitations of existing solutions is that they are primarily concerned about extracting facts from the source text but overlook other crucial factual information, such as the related time, locations, reasons, consequences, purposes, participants and involved parties. Furthermore, the current summarization frameworks are inadequate in modeling the complex semantic relations among facts and the corresponding factual information, leaving much room for improvement. This paper presents FFSum , a novel summarization framework for exploiting multi-grained factual information to improve text summarization. To this end, FFSum constructs an individual fine-grained factual graph with multiple relations among facts and the corresponding factual information. It employs a fact-driven graph attention network to integrate multi-granular factual representations at the encoding stage. It then uses a hybrid pointer network to retrieve factual pieces from the graph for the summary generation. We evaluate the FFSum by applying it to two real-world datasets. Experimental results show that the FFSum consistently outperforms a state-of-the-art approach across evaluation datasets.

Summarizing user reviews and classifying user sentiment are two critical tasks for modern e-commerce platforms. These two tasks can benefit each other by capturing the shared linguistic features. However, such a relationship has not been fully exploited by existing research on domain-specific contextual representations. This work explores a win-win strategy for a multi-task framework with three stages: general pre-training, adaptive pre-training, and collaborative fine-tuning. The task-adaptive continual pre-training on a language model can obtain domain-specific contextual representations, further used to improve two related tasks, sentiment classification and review summarization during the collaborative fine-tuning. Meanwhile, to effectively capture sentiment-oriented domain-specific contextual representations, we introduce a novel task-adaptive pre-training procedure, which adds a sentiment prediction task during the adaptive pre-training. Extensive experiments conducted on two adaption scenarios of a general-to-single domain and a general-to-multiple domain show that our framework outperforms state-of-the-art methods.