Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection

Zekai WU; Jiaxin CHEN; Zhenguo YANG; Haoran XIE; Fu Lee WANG; Wenyin LIU

doi:10.1007/978-3-030-90888-1_40

Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection

Zekai WU, Jiaxin CHEN, Zhenguo YANG^*, Haoran XIE, Fu Lee WANG, Wenyin LIU^*

^*Corresponding author for this work

Lingnan University

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

2 Citations (Scopus)

Abstract

In this paper, we design a cross-modal attention fusion network with orthogonal latent memory (CALM) to fuse multi-modal social media data for rumor detection. Given multimodal content features extracted from text and images, we devise a cross-modal attention fusion (CAF) mechanism to extract critical information underlying the modalities by intra-modality attention, and model the underlying relations among the modalities by inter-modality attention. In terms of the text, the natural sequential characteristics are critical to semantic understanding, while existing sequence models suffer from losing the information conveyed by the former words. To this end, we propose a Bi-GRU with orthogonal latent memory to extract the sequential features from the text, where the memory captures independent patterns. The fused content features and the sequential features can be used for rumor detection seamlessly. Extensive experiments conducted on two real-world datasets show the outperformance of the proposed CALM. (e.g., F1 -score is improved from 0.823 to 0.846 on Weibo dataset).

Original language	English
Title of host publication	Web Information Systems Engineering – WISE 2021
Subtitle of host publication	22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I
Editors	Wenjie ZHANG, Lei ZOU, Zakaria MAAMAR, Lu CHEN
Publisher	Springer, Cham
Pages	527-541
Number of pages	15
Edition	1
ISBN (Electronic)	9783030908881
ISBN (Print)	9783030908874
DOIs	https://doi.org/10.1007/978-3-030-90888-1_40
Publication status	Published - Jan 2022
Event	22nd International Conference on Web Information Systems Engineering - Melbourne, Australia Duration: 26 Oct 2021 → 29 Oct 2021 http://www.wise-conferences.org/2021/

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13080 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	22nd International Conference on Web Information Systems Engineering
Abbreviated title	WISE 2021
Country/Territory	Australia
City	Melbourne
Period	26/10/21 → 29/10/21
Internet address	http://www.wise-conferences.org/2021/

Bibliographical note

Funding Information:
Acknowledgement. This work is supported by the National Natural Science Foundation of China (No. 62076073), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010616), Science and Technology Program of Guangzhou (No. 202102020524), the Guangdong Innovative Research Team Program (No.2014Z T05G157), HKIBS Research Program Grant Application (HCRG-201-002) and the Faculty Research Grant (DB21B6) of Lingnan University, Hong Kong.

Publisher Copyright:
© 2021, Springer Nature Switzerland AG.

Keywords

Multi-modal
Rumor detection
Social media

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-90888-1_40

Cite this

WU, Z., CHEN, J., YANG, Z., XIE, H., WANG, F. L., & LIU, W. (2022). Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection. In W. ZHANG, L. ZOU, Z. MAAMAR, & L. CHEN (Eds.), Web Information Systems Engineering – WISE 2021: 22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I (1 ed., pp. 527-541). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13080 LNCS). Springer, Cham. https://doi.org/10.1007/978-3-030-90888-1_40

WU, Zekai ; CHEN, Jiaxin ; YANG, Zhenguo et al. / Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection. Web Information Systems Engineering – WISE 2021: 22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I. editor / Wenjie ZHANG ; Lei ZOU ; Zakaria MAAMAR ; Lu CHEN. 1. ed. Springer, Cham, 2022. pp. 527-541 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{3066f91923dd47e5a7ae604eb0719074,

title = "Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection",

abstract = "In this paper, we design a cross-modal attention fusion network with orthogonal latent memory (CALM) to fuse multi-modal social media data for rumor detection. Given multimodal content features extracted from text and images, we devise a cross-modal attention fusion (CAF) mechanism to extract critical information underlying the modalities by intra-modality attention, and model the underlying relations among the modalities by inter-modality attention. In terms of the text, the natural sequential characteristics are critical to semantic understanding, while existing sequence models suffer from losing the information conveyed by the former words. To this end, we propose a Bi-GRU with orthogonal latent memory to extract the sequential features from the text, where the memory captures independent patterns. The fused content features and the sequential features can be used for rumor detection seamlessly. Extensive experiments conducted on two real-world datasets show the outperformance of the proposed CALM. (e.g., F1 -score is improved from 0.823 to 0.846 on Weibo dataset).",

keywords = "Multi-modal, Rumor detection, Social media",

author = "Zekai WU and Jiaxin CHEN and Zhenguo YANG and Haoran XIE and WANG, {Fu Lee} and Wenyin LIU",

note = "Funding Information: Acknowledgement. This work is supported by the National Natural Science Foundation of China (No. 62076073), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010616), Science and Technology Program of Guangzhou (No. 202102020524), the Guangdong Innovative Research Team Program (No.2014Z T05G157), HKIBS Research Program Grant Application (HCRG-201-002) and the Faculty Research Grant (DB21B6) of Lingnan University, Hong Kong. Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 22nd International Conference on Web Information Systems Engineering, WISE 2021 ; Conference date: 26-10-2021 Through 29-10-2021",

year = "2022",

month = jan,

doi = "10.1007/978-3-030-90888-1_40",

language = "English",

isbn = "9783030908874",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer, Cham",

pages = "527--541",

editor = "Wenjie ZHANG and Lei ZOU and Zakaria MAAMAR and Lu CHEN",

booktitle = "Web Information Systems Engineering – WISE 2021",

edition = "1",

url = "http://www.wise-conferences.org/2021/",

}

WU, Z, CHEN, J, YANG, Z, XIE, H, WANG, FL & LIU, W 2022, Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection. in W ZHANG, L ZOU, Z MAAMAR & L CHEN (eds), Web Information Systems Engineering – WISE 2021: 22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I. 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13080 LNCS, Springer, Cham, pp. 527-541, 22nd International Conference on Web Information Systems Engineering, Melbourne, Australia, 26/10/21. https://doi.org/10.1007/978-3-030-90888-1_40

Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection. / WU, Zekai; CHEN, Jiaxin; YANG, Zhenguo et al.
Web Information Systems Engineering – WISE 2021: 22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I. ed. / Wenjie ZHANG; Lei ZOU; Zakaria MAAMAR; Lu CHEN. 1. ed. Springer, Cham, 2022. p. 527-541 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13080 LNCS).

Research output: Book Chapters | Papers in Conference Proceedings › Conference paper (refereed) › Research › peer-review

TY - GEN

T1 - Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection

AU - WU, Zekai

AU - CHEN, Jiaxin

AU - YANG, Zhenguo

AU - XIE, Haoran

AU - WANG, Fu Lee

AU - LIU, Wenyin

N1 - Funding Information: Acknowledgement. This work is supported by the National Natural Science Foundation of China (No. 62076073), the Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010616), Science and Technology Program of Guangzhou (No. 202102020524), the Guangdong Innovative Research Team Program (No.2014Z T05G157), HKIBS Research Program Grant Application (HCRG-201-002) and the Faculty Research Grant (DB21B6) of Lingnan University, Hong Kong. Publisher Copyright: © 2021, Springer Nature Switzerland AG.

PY - 2022/1

Y1 - 2022/1

N2 - In this paper, we design a cross-modal attention fusion network with orthogonal latent memory (CALM) to fuse multi-modal social media data for rumor detection. Given multimodal content features extracted from text and images, we devise a cross-modal attention fusion (CAF) mechanism to extract critical information underlying the modalities by intra-modality attention, and model the underlying relations among the modalities by inter-modality attention. In terms of the text, the natural sequential characteristics are critical to semantic understanding, while existing sequence models suffer from losing the information conveyed by the former words. To this end, we propose a Bi-GRU with orthogonal latent memory to extract the sequential features from the text, where the memory captures independent patterns. The fused content features and the sequential features can be used for rumor detection seamlessly. Extensive experiments conducted on two real-world datasets show the outperformance of the proposed CALM. (e.g., F1 -score is improved from 0.823 to 0.846 on Weibo dataset).

AB - In this paper, we design a cross-modal attention fusion network with orthogonal latent memory (CALM) to fuse multi-modal social media data for rumor detection. Given multimodal content features extracted from text and images, we devise a cross-modal attention fusion (CAF) mechanism to extract critical information underlying the modalities by intra-modality attention, and model the underlying relations among the modalities by inter-modality attention. In terms of the text, the natural sequential characteristics are critical to semantic understanding, while existing sequence models suffer from losing the information conveyed by the former words. To this end, we propose a Bi-GRU with orthogonal latent memory to extract the sequential features from the text, where the memory captures independent patterns. The fused content features and the sequential features can be used for rumor detection seamlessly. Extensive experiments conducted on two real-world datasets show the outperformance of the proposed CALM. (e.g., F1 -score is improved from 0.823 to 0.846 on Weibo dataset).

KW - Multi-modal

KW - Rumor detection

KW - Social media

UR - http://www.scopus.com/inward/record.url?scp=85121924889&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-90888-1_40

DO - 10.1007/978-3-030-90888-1_40

M3 - Conference paper (refereed)

SN - 9783030908874

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 527

EP - 541

BT - Web Information Systems Engineering – WISE 2021

A2 - ZHANG, Wenjie

A2 - ZOU, Lei

A2 - MAAMAR, Zakaria

A2 - CHEN, Lu

PB - Springer, Cham

T2 - 22nd International Conference on Web Information Systems Engineering

Y2 - 26 October 2021 through 29 October 2021

ER -

WU Z, CHEN J, YANG Z, XIE H, WANG FL, LIU W. Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection. In ZHANG W, ZOU L, MAAMAR Z, CHEN L, editors, Web Information Systems Engineering – WISE 2021: 22nd International Conference on Web Information Systems Engineering, WISE 2021, Melbourne, VIC, Australia, October 26–29, 2021, Proceedings, Part I. 1 ed. Springer, Cham. 2022. p. 527-541. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2022 Jan 1. doi: 10.1007/978-3-030-90888-1_40

Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection

Abstract

Publication series

Conference

Bibliographical note

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

A Label Extension Schema for Improved Text Emotion Classification

Cite this

Cross-modal Attention Network with Orthogonal Latent Memory for Rumor Detection

Abstract

Publication series

Conference

Bibliographical note

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Projects

A Label Extension Schema for Improved Text Emotion Classification

Cite this