Robust RGB-T Tracking via Adaptive Modality Weight Correlation Filters and Cross-modality Learning

Mingliang ZHOU, Xinwen ZHAO, Futing LUO, Jun LUO*, Huayan PU, Tao XIANG

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

1 Citation (Scopus)


RGBT tracking is gaining popularity due to its ability to provide effective tracking results in a variety of weather conditions. However, feature specificity and complementarity have not been fully used in existing models that directly fuse the correlation filtering response, which leads to poor tracker performance. In this article, we propose correlation filters with adaptive modality weight and cross-modality learning (AWCM) ability to solve multimodality tracking tasks. First, we use weighted activation to fuse thermal infrared and visible modalities, and the fusion modality is used as an auxiliary modality to suppress noise and increase the learning ability of shared modal features. Second, we design modal weights through average peak-to-correlation energy coefficients to improve model reliability. Third, we propose consistency in using the fusion modality as an intermediate variable for joint learning consistency, thereby increasing tracker robustness via interactive cross-modal learning. Finally, we use the alternating direction method of multipliers algorithm to produce a closed solution and conduct extensive experiments on the RGBT234, VOT-TIR2019, and GTOT tracking benchmark datasets to demonstrate the superior performance of the proposed AWCM against compared to existing tracking algorithms. The code developed in this study is available at the following website.1

Original languageEnglish
Article number95
Number of pages20
JournalACM Transactions on Multimedia Computing, Communications and Applications
Issue number4
Early online date11 Dec 2023
Publication statusPublished - Apr 2024
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.


  • adaptive modality weight
  • Correlation filters
  • cross-modality learning
  • RGBT tracking


Dive into the research topics of 'Robust RGB-T Tracking via Adaptive Modality Weight Correlation Filters and Cross-modality Learning'. Together they form a unique fingerprint.

Cite this