ImLiDAR: Cross-Sensor Dynamic Message Propagation Network for 3D Object Detection

Yiyang SHEN, Rongwei YU, Peng WU, Haoran XIE, Lina GONG, Jing QIN, Mingqiang WEI

Research output: Journal PublicationsJournal Article (refereed)peer-review

5 Citations (Scopus)

Abstract

LiDAR and camera, as two different sensors, supply geometric (point clouds) and semantic (RGB images) information of 3-D scenes. However, it is still challenging for existing methods to fuse data from the two cross sensors, making them complementary for quality 3-D object detection (3OD). We propose ImLiDAR, a new 3OD paradigm to narrow the cross-sensor discrepancies by progressively fusing the multiscale features of camera Images and LiDAR point clouds. ImLiDAR enables to provide the detection head with cross-sensor yet robustly fused features. To achieve this, two core designs exist in ImLiDAR. First, we propose a cross-sensor dynamic message propagation (CDMP) module to combine the best of the multiscale image and point features. Second, we raise a direct set prediction problem that allows designing an effective set-based detector (SD) to tackle the inconsistency of the classification and localization confidences, and the sensitivity of hand-tuned hyperparameters. Besides, the novel SD can be detachable and easily integrated into various detection networks. Comparisons on the KITTI, nuScenes, and SUN-RGBD datasets all show clear visual and numerical improvements of our ImLiDAR over 45 state-of-the-art 3OD methods.

Original languageEnglish
Article number5704613
Pages (from-to)1-13
Number of pages13
JournalIEEE Transactions on Geoscience and Remote Sensing
Volume61
Early online date2 Oct 2023
DOIs
Publication statusPublished - 2023

Bibliographical note

Publisher Copyright:
© 1980-2012 IEEE.

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant 42071431, Grant T2322012, and Grant 62172218; in part by the Shenzhen Science and Technology Program under Grant JCYJ20220818103401003 and Grant JCYJ20220530172403007; in part by the General Program of Natural Science Foundation of Guangdong Province under Grant 2022A1515010170; in part by the National Key Research and Development Program of China under Grant 2020YFB1805400 and Grant 2022YFB4500800; and in part by the Provincial Key Research and Development Program of Hubei, China, under Grant 2020BAB101.

Keywords

  • 3-D object detection (3OD)
  • ImLiDAR
  • cross sensors
  • dynamic message propagation
  • set-based detector (SD)

Fingerprint

Dive into the research topics of 'ImLiDAR: Cross-Sensor Dynamic Message Propagation Network for 3D Object Detection'. Together they form a unique fingerprint.

Cite this