Hanwen Kang

Accurate reconstruction of plant models for phenotyping analysis is critical for optimizing sustainable agricultural practices in precision agriculture. Traditional laboratory‐based phenotyping, while valuable, falls short of understanding how plants grow under uncontrolled conditions. Robotic technologies offer a promising avenue for large‐scale, direct phenotyping in real‐world environments. This study explores the deployment of emerging robotics and digital technology in plant phenotyping to improve performance and efficiency. Three critical functional modules, environmental understanding, robotic motion planning, and in situ phenotyping, are introduced to automate the entire process. Results demonstrate the effectiveness of the system in agricultural environments. The phenorobot system autonomously collects high‐quality data by navigating around plants. In addition, the in situ modeling model reconstructs high‐quality plant models from the data collected by the robot. The developed robotic system shows high efficiency and robustness, demonstrating its potential to advance plant science in real‐world agricultural environments.

Accurate collection of plant phenotyping is critical to optimising sustainable farming practices in precision agriculture. Traditional phenotyping in controlled laboratory environments, while valuable, falls short in understanding plant growth under real-world conditions. Emerging sensor and digital technologies offer a promising approach for direct phenotyping of plants in farm environments. This study investigates a learning-based phenotyping method using neural implicit surfaces reconstruction to achieve accurate in situ phenotyping of pepper plants in greenhouse environments. To quantitatively evaluate the performance of this method, traditional point cloud registration on 3D scanning data is implemented for comparison. Experimental result shows that NIR (neural implicit surfaces reconstruction) achieves competitive accuracy compared to the 3D scanning method. The mean distance error between the scanner-based method and the NeRF (neural radiance fields)-based method is 0.811 mm. This study shows that the learning-based NeRF method has similar accuracy to the 3D scanning-based method but with greater scalability and faster deployment capabilities.

This study presents the Fast Fruit 3D Detector (FF3D), a novel framework that contains a 3D neural network for fruit detection and an anisotropic Gaussian-based next-best view estimator. The proposed one-stage 3D detector, which utilizes an end-to-end 3D detection network, shows superior accuracy and robustness compared to traditional 2D methods. The core of the FF3D is a 3D object detection network based on a 3D convolutional neural network (3D CNN) followed by an anisotropic Gaussian-based next-best view estimation module. The innovative architecture combines point cloud feature extraction and object detection tasks, achieving accurate real-time fruit localization. The model is trained on a large-scale 3D fruit dataset and contains data collected from an apple orchard. Additionally, the proposed next-best view estimator improves accuracy and lowers the collision risk for grasping. Thorough assessments on the test set and in a simulated environment validate the efficacy of our FF3D. The experimental results show an AP of 76.3%, an AR of 92.3%, and an average Euclidean distance error of less than 6.2 mm, highlighting the framework’s potential to overcome challenges in orchard environments.

Environmental mapping and robot navigation are the basis for realizing robot automation in modern agricultural production. This study proposes a new autonomous mapping and navigation method for gardening scene robots. First, a new LiDAR slam-based semantic mapping algorithm is proposed to enable the robots to analyze structural information from point cloud images and generate roadmaps from them. Secondly, a general robot navigation framework is proposed to enable the robot to generate the shortest global path according to the road map, and consider the local terrain information to find the optimal local path to achieve safe and efficient trajectory tracking; this method is equipped in apple orchards. The LiDAR was evaluated on a differential drive robotic platform. Experimental results show that this method can effectively process orchard environmental information. Compared with vnf and pointnet++, the semantic information extraction efficiency and time are greatly improved. The map feature extraction time can be reduced to 0.1681 s, and its MIoU is 0.812. The resulting global path planning achieved a 100% success rate, with an average run time of 4ms. At the same time, the local path planning algorithm can effectively generate safe and smooth trajectories to execute the global path, with an average running time of 36 ms.

In an unmanned orchard, various tasks such as seeding, irrigation, health monitoring, and harvesting of crops are carried out by unmanned vehicles. These vehicles need to be able to distinguish which objects are fruit trees and which are not, rather than relying on human guidance. To address this need, this study proposes an efficient and robust method for fruit tree detection in orchard point cloud maps. Feature extraction is performed on the 3D point cloud to form a two-dimensional feature vector containing three-dimensional information of the point cloud and the tree target is detected through the customized deep learning network. The impact of various feature extraction methods such as average height, density, PCA, VFH, and CVFH on the detection accuracy of the network is compared in this study. The most effective feature extraction method for the detection of tree point cloud objects is determined. The ECA attention module and the EVC feature pyramid structure are introduced into the YOLOv8 network. The experimental results show that the deep learning network improves the precision, recall, and mean average precision by 1.5%, 0.9%, and 1.2%, respectively. The proposed framework is deployed in unmanned orchards for field testing. The experimental results demonstrate that the framework can accurately identify tree targets in orchard point cloud maps, meeting the requirements for constructing semantic orchard maps.

Digital modelling stands as a pivotal step in the realm of Digital Twinning. The future trend of Digital Twinning involves automated exploration and environmental modelling in complex scenes. In our study, we propose an innovative solution for robot odometry, path planning, and exploration in unknown outdoor environments, with a focus on Digital modelling. The approach uses a minimum cost formulation with pseudo-randomly generated objectives, integrating multi-path planning and evaluation, with emphasis on full coverage of unknown maps based on feasible boundaries of interest. The approach allows for dynamic changes to expected targets and behaviours. The evaluation is conducted on a robotic platform with a lightweight 3D LiDAR sensor model. The robustness of different types of odometry is compared, and the impact of parameters on motion planning is explored. The consistency and efficiency of exploring completely unknown areas are assessed in both indoor and outdoor scenarios. The experiment shows that the method proposed in this article can complete autonomous exploration and environmental modelling tasks in complex indoor and outdoor scenes. Finally, the study concludes by summarizing the reasons for exploration failures and outlining future focuses in this domain.

In the dynamic and unstructured environment where horticultural crops grow, obstacles and interference frequently occur but are rarely addressed, which poses significant challenges for robotic harvesting. This work proposed a tactile-enabled robotic grasping method that combines deep learning, tactile sensing, and soft robots. By integrating fin-ray fingers with embedded tactile sensing arrays and customized perception algorithms, the robot gains the ability to sense and handle branch interference during the harvesting process and thus reduce potential mechanical fruit damage. Through experimental validations, an overall 83.3–87.0% grasping status detection success rate, and a promising interference handling method have been demonstrated. The proposed grasping method can also be extended to broader robotic grasping applications wherever undesirable foreign object intrusion needs to be addressed.

Camellia oleifera fruits are randomly distributed in an orchard, and the fruits are easily blocked or covered by leaves. In addition, the colors of leaves and fruits are alike, and flowers and fruits grow at the same time, presenting many ambiguities. The large shock force will cause flowers to fall and affect the yield. As a result, accurate positioning becomes a difficult problem for robot picking. Therefore, studying target recognition and localization of Camellia oleifera fruits in complex environments has many difficulties. In this paper, a fusion method of deep learning based on visual perception and image processing is proposed to adaptively and actively locate fruit recognition and picking points for Camellia oleifera fruits. First, to adapt to the target classification and recognition of complex scenes in the field, the parameters of the You Only Live Once v7 (YOLOv7) model were optimized and selected to achieve Camellia oleifera fruits’ detection and determine the center point of the fruit recognition frame. Then, image processing and a geometric algorithm are used to process the image, segment, and determine the morphology of the fruit, extract the centroid of the outline of Camellia oleifera fruit, and then analyze the position deviation of its centroid point and the center point in the YOLO recognition frame. The frontlighting, backlight, partial occlusion, and other test conditions for the perceptual recognition processing were validated with several experiments. The results demonstrate that the precision of YOLOv7 is close to that of YOLOv5s, and the mean average precision of YOLOv7 is higher than that of YOLOv5s. For some occluded Camellia oleifera fruits, the YOLOv7 algorithm is better than the YOLOv5s algorithm, which improves the detection accuracy of Camellia oleifera fruits. The contour of Camellia oleifera fruits can be extracted entirely via image processing. The average position deviation between the centroid point of the image extraction and the center point of the YOLO recognition frame is 2.86 pixels; thus, the center point of the YOLO recognition frame is approximately considered to be consistent with the centroid point of the image extraction.