Multimodal Fusion Network with Latent Topic Memory for Rumor Detection

Jiaxin CHEN; Zekai WU; Zhenguo YANG; Haoran XIE; Fu Lee WANG; Wenyin LIU

doi:10.1109/ICME51207.2021.9428404

Multimodal Fusion Network with Latent Topic Memory for Rumor Detection

Jiaxin CHEN, Zekai WU, Zhenguo YANG, Haoran XIE, Fu Lee WANG, Wenyin LIU

Lingnan University

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

12 Citations (Scopus)

Abstract

In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.

Original language	English
Title of host publication	2021 IEEE International Conference on Multimedia and Expo, ICME 2021
Publisher	IEEE Computer Society
Number of pages	6
ISBN (Electronic)	9781665438643
DOIs	https://doi.org/10.1109/ICME51207.2021.9428404
Publication status	Published - 9 Jun 2021
Event	2021 IEEE International Conference on Multimedia and Expo, ICME 2021 - Shenzhen, China Duration: 5 Jul 2021 → 9 Jul 2021

Publication series

Name	Proceedings - IEEE International Conference on Multimedia and Expo
ISSN (Print)	1945-7871
ISSN (Electronic)	1945-788X

Conference

Conference	2021 IEEE International Conference on Multimedia and Expo, ICME 2021
Country/Territory	China
City	Shenzhen
Period	5/07/21 → 9/07/21

Bibliographical note

Funding Information:
This work is supported by the National Natural Science Foundation of China (No.62076073), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010616), the Guangdong Innovative Research Team Program (No.2014ZT05G157), the Key-Area Research and Development Program of Guangdong Province (2019B010136001), and the Science and Technology Planning Project of Guangdong Province (LZC0023), a grant from the RGC of HKSAR, China (UGC/FDS16/E01/19), and the Faculty Research Fund (DB21A9), Lingnan University, Hong Kong.

Publisher Copyright:
© 2021 IEEE

Keywords

Multimodal fusion
Self-attentive
Rumor detection

Access to Document

10.1109/ICME51207.2021.9428404

Cite this

@inproceedings{e2ea167ba0964d019ac7c0832a6bca1f,

title = "Multimodal Fusion Network with Latent Topic Memory for Rumor Detection",

abstract = "In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.",

keywords = "Multimodal fusion, Self-attentive, Rumor detection",

author = "Jiaxin CHEN and Zekai WU and Zhenguo YANG and Haoran XIE and WANG, {Fu Lee} and Wenyin LIU",

note = "Funding Information: This work is supported by the National Natural Science Foundation of China (No.62076073), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010616), the Guangdong Innovative Research Team Program (No.2014ZT05G157), the Key-Area Research and Development Program of Guangdong Province (2019B010136001), and the Science and Technology Planning Project of Guangdong Province (LZC0023), a grant from the RGC of HKSAR, China (UGC/FDS16/E01/19), and the Faculty Research Fund (DB21A9), Lingnan University, Hong Kong. Publisher Copyright: {\textcopyright} 2021 IEEE; 2021 IEEE International Conference on Multimedia and Expo, ICME 2021 ; Conference date: 05-07-2021 Through 09-07-2021",

year = "2021",

month = jun,

day = "9",

doi = "10.1109/ICME51207.2021.9428404",

language = "English",

series = "Proceedings - IEEE International Conference on Multimedia and Expo",

publisher = "IEEE Computer Society",

booktitle = "2021 IEEE International Conference on Multimedia and Expo, ICME 2021",

address = "United States",

}

CHEN, J, WU, Z, YANG, Z, XIE, H, WANG, FL & LIU, W 2021, Multimodal Fusion Network with Latent Topic Memory for Rumor Detection. in 2021 IEEE International Conference on Multimedia and Expo, ICME 2021. Proceedings - IEEE International Conference on Multimedia and Expo, IEEE Computer Society, 2021 IEEE International Conference on Multimedia and Expo, ICME 2021, Shenzhen, China, 5/07/21. https://doi.org/10.1109/ICME51207.2021.9428404

Multimodal Fusion Network with Latent Topic Memory for Rumor Detection. / CHEN, Jiaxin; WU, Zekai; YANG, Zhenguo et al.
2021 IEEE International Conference on Multimedia and Expo, ICME 2021. IEEE Computer Society, 2021. (Proceedings - IEEE International Conference on Multimedia and Expo).

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

TY - GEN

T1 - Multimodal Fusion Network with Latent Topic Memory for Rumor Detection

AU - CHEN, Jiaxin

AU - WU, Zekai

AU - YANG, Zhenguo

AU - XIE, Haoran

AU - WANG, Fu Lee

AU - LIU, Wenyin

N1 - Funding Information: This work is supported by the National Natural Science Foundation of China (No.62076073), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010616), the Guangdong Innovative Research Team Program (No.2014ZT05G157), the Key-Area Research and Development Program of Guangdong Province (2019B010136001), and the Science and Technology Planning Project of Guangdong Province (LZC0023), a grant from the RGC of HKSAR, China (UGC/FDS16/E01/19), and the Faculty Research Fund (DB21A9), Lingnan University, Hong Kong. Publisher Copyright: © 2021 IEEE

PY - 2021/6/9

Y1 - 2021/6/9

N2 - In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.

AB - In this paper, we propose a multimodal fusion network (termed as MFN) to integrate the text and image data from social media for rumor detection. Given the multimodal features, MFN exploits self-attentive fusion (SAF) mechanism to conduct feature-level fusion by assigning corresponding weights to the complementary modalities. In particular, the textual features are combined with the fused features in a skip-connection manner, as textual features tend to be more distinguishable compared with visual features. Furthermore, MFN introduces latent topic memory (LTM) to store the semantic information about rumor and non-rumor events, benefiting to the identification of the upcoming posts. Extensive experiments conducted on two public datasets show that the proposed MFN outperforms the state-of-the-art approaches.

KW - Multimodal fusion

KW - Self-attentive

KW - Rumor detection

UR - http://www.scopus.com/inward/record.url?scp=85121919671&partnerID=8YFLogxK

U2 - 10.1109/ICME51207.2021.9428404

DO - 10.1109/ICME51207.2021.9428404

M3 - Conference paper (refereed)

T3 - Proceedings - IEEE International Conference on Multimedia and Expo

BT - 2021 IEEE International Conference on Multimedia and Expo, ICME 2021

PB - IEEE Computer Society

T2 - 2021 IEEE International Conference on Multimedia and Expo, ICME 2021

Y2 - 5 July 2021 through 9 July 2021

ER -

Multimodal Fusion Network with Latent Topic Memory for Rumor Detection

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Facilitate Tree-Structured Topic Modeling via Nonparametric Neural Inference

Multimodal fusion network with contrary latent topic memory for rumor detection

Cite this

Multimodal Fusion Network with Latent Topic Memory for Rumor Detection

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Projects

Facilitate Tree-Structured Topic Modeling via Nonparametric Neural Inference

Research output

Multimodal fusion network with contrary latent topic memory for rumor detection

Cite this