Multi-Modal Excavator Activity Recognition Using Two-Stream CNN-LSTM with RGB and Point Cloud Inputs
Open Access
- 31 July 2025
- journal article
- research article
- Published by MDPI AG in Applied Sciences
- Vol. 15 (15) , 8505
Abstract
Recently, deep learning algorithms have been increasingly applied in construction for activity recognition, particularly for excavators, to automate processes and enhance safety and productivity through continuous monitoring of earthmoving activities. These deep learning algorithms analyze construction videos to classify excavator activities for earthmoving purposes. However, previous studies have solely focused on single-source external videos, which limits the activity recognition capabilities of the deep learning algorithm. This paper introduces a novel multi-modal deep learning-based methodology for recognizing excavator activities, utilizing multi-stream input data. It processes point clouds and RGB images using the two-stream long short-term memory convolutional neural network (CNN-LSTM) method to extract spatiotemporal features, enabling the recognition of excavator activities. A comprehensive dataset comprising 495,000 video frames of synchronized RGB and point cloud data was collected across multiple construction sites under varying conditions. The dataset encompasses five key excavator activities: Approach, Digging, Dumping, Idle, and Leveling. To assess the effectiveness of the proposed method, the performance of the two-stream CNN-LSTM architecture is compared with that of single-stream CNN-LSTM models on the same RGB and point cloud datasets, separately. The results demonstrate that the proposed multi-stream approach achieved an accuracy of 94.67%, outperforming existing state-of-the-art single-stream models, which achieved 90.67% accuracy for the RGB-based model and 92.00% for the point cloud-based model. These findings underscore the potential of the proposed activity recognition method, making it highly effective for automatic real-time monitoring of excavator activities, thereby laying the groundwork for future integration into digital twin systems for proactive maintenance and intelligent equipment management.Keywords
Funding Information
- National Research Foundation of Korea (NRF) grant funded by the Korean government (MEST) (No. NRF 2018R1A5A1025137, No. RS-2020-KA157089)
- Korea Agency for Infrastructure Technology Advancement (KAIA) grant funded by the Ministry of Land, Infrastructure, and Transport (National Research for Smart Construction Technology (No. NRF 2018R1A5A1025137, No. RS-2020-KA157089)
This publication has 71 references indexed in Scilit:
- AI-based technology to prognose and diagnose complex crack characteristics of railway concrete sleepersDiscover Applied Sciences, 2024
- Waste Lime Earthwork Management Using Drone and BIM Technology for Construction Projects: The Case Study of Urban Development ProjectKSCE Journal of Civil Engineering, 2023
- Human activity recognition from multiple sensors data using deep CNNsMultimedia Tools and Applications, 2023
- Computer vision-based deep learning for supervising excavator operations and measuring real-time earthwork productivityThe Journal of Supercomputing, 2022
- Human activity recognition via optical flow: decomposing activities into basic actionsNeural Computing and Applications, 2019
- Vision-based action recognition of construction workers using dense trajectoriesAdvanced Engineering Informatics, 2016
- Status quo and open challenges in vision-based sensing and tracking of temporary resources on infrastructure construction sitesAdvanced Engineering Informatics, 2015
- Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the futureAdvanced Engineering Informatics, 2015
- Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiersAdvanced Engineering Informatics, 2013
- Learning and classifying actions of construction workers and equipment using Bag-of-Video-Feature-Words and Bayesian network modelsAdvanced Engineering Informatics, 2011