Multi-Projection Fusion and Refinement Network for Salient Object Detection in 360° Omnidirectional Image

Runmin CONG, Ke HUANG, Jianjun LEI, Yao ZHAO, Qingming HUANG, Sam KWONG

Research output: Journal PublicationsJournal Article (refereed)peer-review

6 Citations (Scopus)

Abstract

Salient object detection (SOD) aims to determine the most visually attractive objects in an image. With the development of virtual reality (VR) technology, 360 ∘ omnidirectional image has been widely used, but the SOD task in 360 ∘ omnidirectional image is seldom studied due to its severe distortions and complex scenes. In this article, we propose a multi-projection fusion and refinement network (MPFR-Net) to detect the salient objects in 360 ∘ omnidirectional image. Different from the existing methods, the equirectangular projection (EP) image and four corresponding cube-unfolding (CU) images are embedded into the network simultaneously as inputs, where the CU images not only provide supplementary information for EP image but also ensure the object integrity of cube-map projection. In order to make full use of these two projection modes, a dynamic weighting fusion (DWF) module is designed to adaptively integrate the features of different projections in a complementary and dynamic manner from the perspective of inter and intrafeatures. Furthermore, in order to fully explore the way of interaction between encoder and decoder features, a filtration and refinement (FR) module is designed to suppress the redundant information of the feature itself and between the features. Experimental results on two omnidirectional datasets demonstrate that the proposed approach outperforms the state-of-the-art methods both qualitatively and quantitatively. The code and results can be found from the link of https://rmcong.github.io/proj_MPFRNet.html.
Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalIEEE Transactions on Neural Networks and Learning Systems
DOIs
Publication statusE-pub ahead of print - 9 Jan 2023
Externally publishedYes

Bibliographical note

Publisher Copyright:
IEEE

Keywords

  • 360∘ omnidirectional image
  • cube-unfolding (CU)
  • dynamic weighting
  • filtration and refinement (FR)
  • salient object detection (SOD)

Fingerprint

Dive into the research topics of 'Multi-Projection Fusion and Refinement Network for Salient Object Detection in 360° Omnidirectional Image'. Together they form a unique fingerprint.

Cite this