GDN-CMCF

A Gated Disentangled Network With Cross-Modality Consensus Fusion for Multimodal Named Entity Recognition

Dados Bibliográficos

ID22108963
AutoresGuoheng Huang (0000-0002-3640-3229, Guangdong University of Technology), Qin He (0000-0002-7697-4034, Guangdong University of Technology), Zihao Dai (0000-0002-8921-2389, Guangdong University of Technology), Guo Zhong (0000-0002-1926-6381, Guangdong University of Foreign Studies), Xiaochen Yuan (0000-0002-7490-6695, Macao Polytechnic University), Chi-Man Pun (0000-0003-1788-3746, University of Macau)
Ano2024
Volume11
Fascículo3
Páginas3944-3954
Data de publicação2024-06-01
Peer ReviewedSim
Open AccessSim
TipoARTICLE
PeriódicoIEEE Transactions on Computational Social Systems (JOURNAL)
Identificadores do periódicoISSN: 2329-924X • E-ISSN: 2373-7476
EditoraInstitute of Electrical and Electronics Engineers (IEEE) (PUBLISHER)
DOI10.1109/tcss.2023.3323402
OpenAlexW4387682068
IdiomaEN
Citações recebidas2
Referências citadas42

Multimodal named entity recognition (MNER) is a crucial task in social systems of artificial intelligence that requires precise identification of named entities in sentences using both visual and textual information. Previous methods have focused on capturing fine-grained visual features and developing complex fusion procedures. However, these approaches overlook the heterogeneity gap and loss of original modality uniqueness that may occur during fusion, leading to incorrect entity identification. This article proposes a novel approach for MNER called a gated disentangled network with cross-modality consensus fusion (GDN-CMCF) to address the above challenges. Specifically, to eliminate cross-modality variation, we propose a cross-modality consensus fusion module that generates a consensus representation by learning inter-and intramodality interactions with a designed commonality constraint. We then introduce a gated disentanglement module to separate modality-relevant features from support and auxiliary modalities, which further filters out extraneous information while retaining the uniqueness of unimodal features. Experimental results on two real public datasets are provided to verify the effectiveness of our proposed GDN-CMCF. The source code of this article can be found at https://github.com/HaoDavis/ GDN-CMCF

Machine learning · Modalities · Natural language processing · Source code · Computer Science · Multimodal Machine Learning Applications · Natural Language Processing Techniques · Topic Modeling · Artificial Intelligence