Zhengbo Zou

Intelligent construction robots are deemed the future of on-site construction for improved productivity and safety. To automate the construction process, imitation learning (IL) has been adopted to train construction robots in a repertoire of tasks. However, collecting demonstrations for robots to imitate from usually requires teleoperation setup and devices, such as virtual reality (VR) and gloves. To autonomously and efficiently generate demonstration data for imitation learning in construction tasks, we propose a large foundation model-based demonstration-generation framework, in which the Segment Anything Model (SAM) is used to extract geometric representations of objects of interest and a vision-language model (VLM), conditioned on mark-based visual prompting and languages, produces high-level action suggestions. As the demonstrator framework is imperfect and is computation-intensive to be fine-tuned, only successful episodes are retained automatically and structured into demonstrations for distilling a lightweight text-conditioned robot policy via behavioral cloning (BC). We evaluate the framework on a UR10 robot arm in Isaac Sim for a language-conditioned timber assembly task with frame pose randomization. The autonomous demonstrator achieves success rates ranging from 76.9% (small perturbations) to 33.7% (largest perturbations) but can be rolled out to curate balanced demonstration data sets. Policies distilled from these demonstrations attain 100% success across all three perturbation ranges and two placement scenarios, while retaining up to 70% success under modest out-of-distribution (OOD) frame poses. An ablation on the number of demonstrations shows that the distilled policies reach 100% success with only 70% of the data in low-variation settings but require about 90% of the data to achieve 100% success under larger workspace perturbations.

Construction safety inspections typically involve a human inspector identifying safety concerns on-site. With the rise of powerful vision language models (VLMs), researchers are exploring their use for tasks such as detecting safety rule violations from on-site images. However, there is a lack of open datasets to comprehensively evaluate and further fine-tune VLMs in construction safety inspection. Current applications of VLMs use small, supervised datasets, limiting their applicability in tasks they are not directly trained for. In this article, we propose the ConstructionSite 10 k , featuring 10,000 construction site images with annotations for three inter-connected tasks, including image captioning, safety rule violation visual question answering (VQA), and construction element visual grounding. Our subsequent evaluation of current state-of-the-art large pre-trained VLMs shows notable generalization abilities in zero-shot and few-shot settings, while additional training is needed to make them applicable to actual construction sites. This dataset allows researchers to train and evaluate their own VLMs with new architectures and techniques, providing a valuable benchmark for construction safety inspection.

Construction safety inspections typically involve a human inspector identifying safety concerns on-site. With the rise of powerful Vision Language Models (VLMs), researchers are exploring their use for tasks such as detecting safety rule violations from on-site images. However, there is a lack of open datasets to comprehensively evaluate and further fine-tune VLMs in construction safety inspection. Current applications of VLMs use small, supervised datasets, limiting their applicability in tasks they are not directly trained for. In this paper, we propose the ConstructionSite 10k, featuring 10,000 construction site images with annotations for three inter-connected tasks, including image captioning, safety rule violation visual question answering (VQA), and construction element visual grounding. Our subsequent evaluation of current state-of-the-art large pre-trained VLMs shows notable generalization abilities in zero-shot and few-shot settings, while additional training is needed to make them applicable to actual construction sites. This dataset allows researchers to train and evaluate their own VLMs with new architectures and techniques, providing a valuable benchmark for construction safety inspection.

Keeping an up-to-date three-dimensional (3D) representation of buildings is a crucial yet time-consuming step for Building Information Modeling (BIM) and digital twins. To address this issue, we propose ICON (Intelligent CONstruction) drone, an unmanned aerial vehicle (UAV) designed to navigate indoor environments autonomously and generate point clouds. ICON drone is constructed using a 250 mm quadcopter frame, a Pixhawk flight controller, and is equipped with an onboard computer, an Red Green Blue-Depth camera and an IMU (Inertial Measurement Unit) sensor. The UAV navigates autonomously using visual-inertial odometer and frontier-based exploration. The collected RGB images during the flight are used for 3D reconstruction and semantic segmentation. To improve the reconstruction accuracy in weak-texture areas in indoor environments, we propose depth-regularized planar-based Gaussian splatting reconstruction, where we use monocular-depth estimation as extra supervision for weak-texture areas. The final outputs are point clouds with building components and material labels. We tested the UAV in three scenes in an educational building: the classroom, the lobby, and the lounge. Results show that the ICON drone could: (1) explore all three scenes autonomously, (2) generate absolute scale point clouds with F1-score of 0.5806, 0.6638, and 0.8167 compared to point clouds collected using a high-fidelity terrestrial LiDAR scanner, and (3) label the point cloud with corresponding building components and material with mean intersection over union of 0.588 and 0.629. The reconstruction algorithm is further evaluated on ScanNet, and results show that our method outperforms previous methods by a large margin on 3D reconstruction quality.