GenEn-MNER: Enhancing Nested Chinese NER With Multimodal Fusion and Alignment via Speech-to-Text Generation
- 26 March 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 1628-1640
Abstract
In recent years, the academic community has increasingly focused on multimodal Chinese Named Entity Recognition (NER) that utilizes speech cues. Existing methods typically rely solely on the NER objective function to guide the alignment and fusion of speech and text, overlooking the inherent alignment within speech-text pairs. Furthermore, these approaches generally employ sequence labeling techniques, which are inadequate for handling nested entities. To address these limitations, we introduce GenEn-MNER, a novel multimodal nested Chinese NER approach that enhances fusion and alignment through speech-to-text generation. This method leverages natural alignment information obtained from the speech-to-text task, using a cross-modal Transformer to integrate and align modalities. Additionally, the table-filling module redefines nested NER by conceptualizing it as the prediction of token pair relationships. Experimental results, as indicated by F1 scores, on CNERTA flat version (80.83%), CNERTA nest version (80.66%), and AISHELL-NER (94.52%) not only confirm the effectiveness of our approach but also demonstrate its superiority to existing state-of-the-art methods.Keywords
Funding Information
- National Natural Science Foundation of China (62276043)
- Fundamental Research Funds for the Central Universities (DUT22ZD205)
This publication has 33 references indexed in Scilit:
- Enhancing Multimodal Entity and Relation Extraction With Variational Information BottleneckIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024
- USAF: Multimodal Chinese named entity recognition using synthesized acoustic featuresInformation Processing & Management, 2023
- TOE: A Grid-Tagging Discontinuous NER Model Enhanced by Embedding Tag/Word Relations and More Fine-Grained TagsIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022
- AISHELL-NER: Named Entity Recognition from Chinese Speech2022
- PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition2022
- Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial LearningIEEE Transactions on Multimedia, 2020
- Speech-Transformer: A No-Recurrence Sequence-to-Sequence Model for Speech Recognition2018
- AISHELL-1: An open-source Mandarin speech corpus and a speech recognition baseline2017
- Core techniques of question answering systems over knowledge bases: a surveyKnowledge and Information Systems, 2017
- Character-Based LSTM-CRF with Radical-Level Features for Chinese Named Entity Recognition2016