A novel approach for the 3D localization of branch picking points based on deep learning applied to longan harvesting UAVs

Denghui LI, Xiaoxuan SUN, Shengping LV, Hamza ELKHOUCHLAA, Yuhang JIA, Zhongwei YAO, Peiyi LIN, Haobo ZHOU, Zhengqi ZHOU, Jiaxing SHEN*, Jun LI

*Corresponding author for this work

Research output: Journal PublicationsJournal Article (refereed)peer-review

Abstract

Longan is a famous speciality fruit and cultivated medicinal plant that has important edible and medicinal value; how to improve productivity in harvest is an important issue. At present, longan is mainly planted in hilly areas. For complex site conditions and tall trees, the ground harvesting machineries cannot work normally. In this study, aiming at harvesting longan fruit using unmanned aerial vehicles, a method combining an improved YOLOv5s, improved DeepLabv3+ model and depth image information is proposed, which is used for the three-dimensional (3D) positioning of branch picking points in complex natural environments. First, the improved YOLOv5s model is used to quickly detect longan fruit skewers and the main fruit branches from a complex orchard environment. The correct main fruit branch is obtained according to its relative position relationship and is extracted as the input to the semantic segmentation model. Second, using the improved DeepLabv3+ model, the image extracted in the previous step is semantically segmented to obtain the 2D coordinate information of the main longan fruit branches. Finally, combined with the growth characteristics of a longan fruit string, RGB-D information fusion is carried out on the main fruit branches in 3D space to obtain the central axis and pose information of the main fruit branches, and the 3D coordinates of the picking points are calculated, which provides destination information for a longan harvesting drone. To verify the effectiveness of the proposed method, an experiment for identifying and locating the main fruit branches and picking points was carried out in a longan orchard. The experimental results show that the longan string fruit and main fruit branch detection accuracy is 85.50%, and the main fruit branch semantic segmentation accuracy is 94.52%. The whole algorithm takes 0.58 s in the actual scene and can quickly and accurately locate the picking points. In summary, this paper fully exploits the advantages of the combination of a convolutional neural network and RGB-D image information, further improving the efficiency of longan harvesting drones in accurately positioning picking points in 3D space.

Original languageEnglish
Article number107191
JournalComputers and Electronics in Agriculture
Volume199
Early online date13 Jul 2022
DOIs
Publication statusPublished - Aug 2022
Externally publishedYes

Bibliographical note

Funding Information:
Funding: This work is supported by the earmarked fund for the Laboratory of Lingnan Modern Agriculture Project (NZ2021009), the China Agriculture Research System of MOF and MARA (No. CARS-32-13), the Special Project of Rural Vitalization Strategy of Guangdong Academy of Agricultural Sciences (No. TS-1-4), and the Guangdong Provincial Modern Agricultural Industry Technology System (No. 2021KJ123).

Publisher Copyright:
© 2022 Elsevier B.V.

Keywords

  • Convolutional Neural Network
  • Image analysis
  • Picking drones
  • Three-dimensional localization

Fingerprint

Dive into the research topics of 'A novel approach for the 3D localization of branch picking points based on deep learning applied to longan harvesting UAVs'. Together they form a unique fingerprint.

Cite this