LiDAR and camera, as two different sensors, supply geometric (point clouds) and semantic (RGB images) information of 3-D scenes. However, it is still challenging for existing methods to fuse data from the two cross sensors, making them complementary for quality 3-D object detection (3OD). We propose ImLiDAR, a new 3OD paradigm to narrow the cross-sensor discrepancies by progressively fusing the multiscale features of camera Images and LiDAR point clouds. ImLiDAR enables to provide the detection head with cross-sensor yet robustly fused features. To achieve this, two core designs exist in ImLiDAR. First, we propose a cross-sensor dynamic message propagation (CDMP) module to combine the best of the multiscale image and point features. Second, we raise a direct set prediction problem that allows designing an effective set-based detector (SD) to tackle the inconsistency of the classification and localization confidences, and the sensitivity of hand-tuned hyperparameters. Besides, the novel SD can be detachable and easily integrated into various detection networks. Comparisons on the KITTI, nuScenes, and SUN-RGBD datasets all show clear visual and numerical improvements of our ImLiDAR over 45 state-of-the-art 3OD methods.
|Number of pages||13|
|Journal||IEEE Transactions on Geoscience and Remote Sensing|
|Early online date||2 Oct 2023|
|Publication status||Published - 2023|
Bibliographical notePublisher Copyright:
© 1980-2012 IEEE.
- 3-D object detection (3OD)
- cross sensors
- dynamic message propagation
- set-based detector (SD)