TogetherNet : Bridging Image Restoration and Object Detection Together via Dynamic Enhancement Learning

Yongzhen WANG, Xuefeng YAN*, Kaiwen ZHANG, Lina GONG, Haoran XIE, Fu Lee WANG, Mingqiang WEI

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

8 Citations (Scopus)

Abstract

Adverse weather conditions such as haze, rain, and snow often impair the quality of captured images, causing detection networks trained on normal images to generalize poorly in these scenarios. In this paper, we raise an intriguing question – if the combination of image restoration and object detection, can boost the performance of cutting-edge detectors in adverse weather conditions. To answer it, we propose an effective yet unified detection paradigm that bridges these two subtasks together via dynamic enhancement learning to discern objects in adverse weather conditions, called TogetherNet. Different from existing efforts that intuitively apply image dehazing/deraining as a pre-processing step, TogetherNet considers a multi-task joint learning problem. Following the joint learning scheme, clean features produced by the restoration network can be shared to learn better object detection in the detection network, thus helping TogetherNet enhance the detection capacity in adverse weather conditions. Besides the joint learning architecture, we design a new Dynamic Transformer Feature Enhancement module to improve the feature extraction and representation capabilities of TogetherNet. Extensive experiments on both synthetic and real-world datasets demonstrate that our TogetherNet outperforms the state-of-the-art detection approaches by a large margin both quantitatively and qualitatively. Source code is available at https://github.com/yz-wang/TogetherNet.

Original languageEnglish
Pages (from-to)465-476
Number of pages12
JournalComputer Graphics Forum
Volume41
Issue number7
DOIs
Publication statusPublished - Oct 2022

Bibliographical note

Funding Information:
This work was supported in part by the National Natural Science Foundation of China (No. 62172218), the Joint Fund of National Natural Science Foundation of China and Civil Aviation Administration of China (No. U2033202), the 14th Five-Year Planning Equipment Pre-Research Program (No. JCKY2020605C003), the Free Exploration of Basic Research Project, Local Science and Technology Development Fund Guided by the Central Government of China (No. 2021Szvup060), the Natural Science Foundation of Guangdong Province (No. 2022A1515010170).

Funding Information:
This work was supported in part by the National Natural Science Foundation of China (No. 62172218), the Joint Fund of National Natural Science Foundation of China and Civil Aviation Administration of China (No. U2033202), the 14th Five‐Year Planning Equipment Pre‐Research Program (No. JCKY2020605C003), the Free Exploration of Basic Research Project, Local Science and Technology Development Fund Guided by the Central Government of China (No. 2021Szvup060), the Natural Science Foundation of Guangdong Province (No. 2022A1515010170).

Publisher Copyright:
© 2022 The Author(s) Computer Graphics Forum © 2022 The Eurographics Association and John Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.

Keywords

  • Adverse weather
  • CCS Concepts
  • Dynamic transformer feature enhancement
  • Image restoration
  • Joint learning
  • Object detection
  • TogetherNet
  • • Computing methodologies → Object detection

Fingerprint

Dive into the research topics of 'TogetherNet : Bridging Image Restoration and Object Detection Together via Dynamic Enhancement Learning'. Together they form a unique fingerprint.

Cite this