Multiple caterpillar tracking and 3D positioning in orchard via YOLO plus DeepSORT

Caterpillars have been causing havoc to agriculture due to feeding aggressively on the foliage of the crops. The current methods of pest control like sticky traps or pheromone traps work on adult moths, but not on foraging caterpillars. Chemical means like pesticides are effective, but chemical residues on crops are concerning. Therefore, this study aims to primarily track and estimate the 3D position of the caterpillars in orchards in real-time. To this end, we have employed the state-of-the-art object detector YOLOv7, combined with Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT) algorithm. This combined approach, when compared to merely YOLOv7, has improved detection up to 25%, courtesy of the SORT embedded tracker. The RGB-D camera is utilized for this work is Intel Realsense D405. For the training data, 2,000 images captured in a jujube orchard with varying exposure, occlusion, and wind conditions were used. Inference was done from completely new images in real time. In the experiments, the YOLOv7+SORT approach makes detections within 17 ms per frame, with an average detection rate of 85%, indicative of is real-time applicability in orchards. The smallest object (around 2-cm length caterpillar) is recognized around 21×12 pixels, which is at a distance of 35 cm from the camera. Thus, this development of YOLOv7+SORT approach can be integrated with technologies like robot arms, that can pick the caterpillars, or even into stand-off techniques like laser pest targeting, which can help eradicate the pest problems in a physical manner efficiently.