Xi Wang

Multi-robot systems are emerging as a promising solution to the growing demand for productivity and safety across industrial sectors. However, effectively coordinating multiple robots in dynamic and uncertain environments, such as construction sites, remains a challenge, particularly due to unpredictable factors like material delays, unexpected site conditions, and weather-induced disruptions. To address these challenges, this study proposes an adaptive task allocation framework that strategically leverages the synergistic potential of Digital Twins, Integer Programming (IP), and Large Language Models (LLMs). The multi-robot task allocation problem is formally modeled using IP, considering task dependencies, robot heterogeneity, scheduling constraints, and replanning needs. A narrative-driven schedule adaptation mechanism is introduced, where unstructured natural language inputs are interpreted by an LLM to autonomously update optimization constraints, enabling human-in-the-loop flexibility without manual programming. A digital twin system enables real-time synchronization between physical operations and their digital representations and serves as the user interface, creating a closed-loop feedback mechanism that keeps the system responsive to site changes. A case study demonstrates both the computational efficiency of the optimization algorithm and the reasoning performance of LLMs, achieving over 97% accuracy in constraint and parameter extraction. The results confirm the practicality, adaptability, and cross-domain applicability of the proposed methods.

Purpose: Effective hazard recognition on construction sites relies heavily on workers’ visual attention; however, the presence of emerging systems such as unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs) may divert this attention away from hazards. Despite the growing integration of robots into construction operations, there remains a lack of empirical evidence on how these systems reallocate workers’ gaze between hazards and robotic elements. Design/methodology/approach: The present study employed a virtual reality (VR) environment integrated with eye-tracking to quantify the attentional trade-offs between hazards and robots. Twenty-eight participants completed hazard identification tasks in two controlled scenarios: one without robots (baseline) and one featuring UAVs and UGVs. Fixation count and duration metrics were analyzed using Wilcoxon signed-rank tests to assess attentional allocation differences. Findings: Results revealed significant changes in visual attention patterns when robots were present. Participants exhibited shorter fixation durations on hazards but increased fixation counts, indicating more frequent but briefer glances. Notably, within the robot-present scenario, participants fixated on robots five times longer than on hazards, highlighting a pronounced attentional shift toward robotic elements in the environment. Originality/value: These findings provide new insight into the cognitive implications of human–robot interaction in construction, demonstrating that robotic presence can unintentionally compromise hazard recognition through strong attentional capture. This work offers critical evidence to guide the design of attention-aware robotic deployment strategies, targeted worker training programs and safety protocols for future human–robot collaborative construction environments.

Human cognition plays a critical role in construction work, particularly in the context of high-level task planning and in-field improvisation. On the other hand, robots are adept at performing numerical computation and repetitive physical tasks with precise motion control. The unstructured and complex nature of construction environments and the inability to maintain tight tolerances in assembled workpieces pose several unique challenges to the wide application of robots in construction work. Thus, the robotization of field construction processes is best conceived as a collaborative human–robot endeavor that takes advantage of both human and robot intelligence as well as robots’ physical operation capabilities to overcome uncertainties and successfully perform useful construction work onsite. This paper proposes an interactive and immersive process-level digital twin (I2PL-DT) system in virtual reality (VR) that integrates visualization and supervision, task planning and execution, and bidirectional communication to enable collaborative human–robot construction work. In this work paradigm, the human worker is responsible for high-level task planning and work process supervision. The robot undertakes workspace sensing and monitoring, detailed motion planning, and physical execution of the work. A drywall installation case study involving imperfect rough carpentry (wall framing) is presented using a KUKA mobile industrial robotic arm emulator. A human-in-the-loop study involving 20 subjects was conducted for system verification and to collect feedback for future improvements. The experimental results show that users can use the system to specify work sequences, select optimal task plans, and perform robot trajectory guidance after simple training and felt positive about the system functions and user experience. The system demonstrates the potential of transitioning the role of construction workers from physical task performers to robot supervisors. In addition, the system establishes a promising framework for construction workers to remotely collaborate with onsite construction robots.

Vision for navigation had been an active area of research for more than three decades, and a vision-based navigation system always needs real-time image collecting and processing to acquire navigation information. In the indoor scenarios, illuminant reflection is often found in navigation images because of smooth surfaces in the environment, such as marble floor, planes of furniture and etc. The negative effect of illuminant reflection in navigation images is obvious and might lower the performance of the navigation system, As to resolve the above problem, the issue of how to detect illuminant reflection should be considered, This paper proposed an automatic detection algorithm to segment illuminant reflection regions in a color image using saturation and brightness characteristics as well as the brightness distribution of the illuminant reflective regions. As to verify the robustness and accuracy of this algorithm, experiments were carried out in different indoor environments where illuminant reflection is found in navigation images, the experiments results we obtained indicated that this algorithm is sufficient to handle the problem with providing good detection results as expected.