Chunxiao Fan

Visual retrievers such as visual retrieval-augmented generation (RAG) have recently emerged as a powerful model for retrieving multimodal documents without the need to convert page images into text. Existing visual retrievers typically encode every document page separately, ignoring the inherent rich context information across pages within multi-page documents. However, some crucial semantic information often spans multiple pages in a document, and should be effectively encoded for better retrieval. To address this problem, this paper proposes a novel approach utilizing dynamically fusing visual context (DFVC), which adaptively encodes the semantic information across pages. In the proposed DFVC approach, a lightweight plug-and-play adapter is designed; in addition, a contrastive loss function incorporating the positive fused embedding vectors and negative embedding vectors is designed to constrain the adapter, allowing it to learn the weights for the context pages. Together, the designed adapter and loss function allow the retriever to effectively encode useful semantic information across pages while excluding distracting noise. The proposed DFVC is validated on commonly used challenging multi-page document benchmarks. Extensive experimental results demonstrate that it significantly boosts retrieval performance. In addition, the proposed DFVC is highly parameter-efficient since it employs frozen vision-language backbones, allowing it to be easily integrated into existing visual RAG pipelines for finer document retrieval.

The rapid expansion of memes on social media has highlighted the urgent need for effective approaches to detect harmful content. However, traditional data-driven approaches struggle to detect new memes due to their evolving nature and the lack of up-to-date annotated data. To address this issue, we propose MIND, a multi-agent framework for zero-shot harmful meme detection that does not rely on annotated data. MIND implements three key strategies: 1) We retrieve similar memes from an unannotated reference set to provide contextual information. 2) We propose a bi-directional insight derivation mechanism to extract a comprehensive understanding of similar memes. 3) We then employ a multi-agent debate mechanism to ensure robust decision-making through reasoned arbitration. Extensive experiments on three meme datasets demonstrate that our proposed framework not only outperforms existing zero-shot approaches but also shows strong generalization across different model architectures and parameter scales, providing a scalable solution for harmful meme detection. The code is available at https://github.com/destroy-lonely/MIND.

With the popularity of mobile smart devices, speech recognition utilizes the speech signals received by sensors to quickly find the most likely text or command through non-autoregressive (NAR) inference, providing a solid foundation for human-computer interaction. However, NAR inference ignores the dependencies between words, which significantly degrades recognition accuracy. To alleviate this problem, we propose a novel Non-Autoregressive End-to-End Speech Recognition with Error Correction Decoder (NAR-SREC), which improves recognition accuracy and inference speed by closely collaborating three core components. Specifically, we first design a new Context-Association CTC (CACTC) to incorporate explicit linguistic information into the output condition term for NAR inference, which enhances inter-word relationship learning. Then, we propose a new BERT-augmented Module to simultaneously model the target sequence and source sequence, where the former provides linguistic information to the CACTC, and the latter performs error probability detection. Finally, we integrate a NAR Error Correction (NAR-EC) decoder for fast error correction, which focuses on correcting incorrect tokens to further reduce the insertion or substitution errors caused by the insufficient inter-word relationship. Extensive experiments on Aishell-1 and Librispeech show that NAR-SREC significantly improves the recognition accuracy compared with other advanced NAR methods, achieving a better trade-off between speed and performance.

Ziyan Liu, Chunxiao Fan, Haoran Lou, Yuexin Wu, Kaiwei Deng. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Machine reading comprehension (MRC) is a task that requires machines to answer relevant questions based on a given context. In recent years, it has attracted extensive attention with the development of deep learning and big data. Considering that human beings will associate some external relevant knowledge when understanding the text, researchers have proposed a method of introducing knowledge outside the given context to assist reading and this method is called Knowledge-Based Machine Reading Comprehension (KBMRC). However, the current research on this method is still scattered, and the retrieval and fusion of relevant knowledge are still two challenges in application, especially in Chinese MRC. The contribution of this paper mainly on the following three points: Firstly, in order to resolve the problem of related knowledge retrieval, we build up a related knowledge set. Secondly, in order to resolve the problem of related knowledge fusion, we propose a negative sample generation strategy and train a language model containing knowledge. Finally, a twin-tower fusion model is constructed based on this model. The experiments on Chinese reading comprehension dataset CMRC2018 show that our method has a certain improvement compared with the baseline method without external knowledge.

Visual object tracking is an important topic in computer vision, which has successfully utilized pretrained convolutional neural networks, such as VGG and ResNet. However, the features extracted by these pretrained models are high dimensional, and the redundant feature channels reduce target localization and scale estimation precision, leading to tracking drifting. In this paper, a novel visual object tracking method, called learning enhanced feature responses tracking (LEFRT), is proposed, which adopts the target-specific features to enhance target localization and scale estimation responses. First, a channel attention module, called target-specific network (TSNet), is presented to reduce the redundant feature channels. Secondly, the scale estimation network (SCENet) is introduced to extract spatial structural features to generate a more precise response for the scale estimation. Extensive experiments on six tracking benchmarks, including LaSOT, GOT-10k, TrackingNet, OTB-2013, OTB-2015, and TC-128, demonstrate that the proposed algorithm can effectively improve the precision and speed of visual object tracking. LEFRT achieves 90.4% precision and a 71.2% success rate on the OTB-2015 dataset, improving the tracking methods based on the pretrained features.