Francesco Argenziano

Task planning for robots in real-life settings is inherently complex due to challenges such as identifying grounded sequences of actions to achieve a goal, bridging the gap between high-level planning and low-level execution, and addressing the computational constraints of robotic hardware. Additionally, an essential, but frequently underestimated, aspect is the robot’s ability to recognize its own limitations and effectively delegate tasks beyond its abilities to human collaborators. Recent advancements in integrating Large Language Models (LLMs) into robotics have increased the likelihood of generating unfeasible plans, making this capability even more critical for any robot that needs to interact with a real environment. To address these challenges, this paper presents a framework that integrates open-vocabulary online grounding, planning, and embodiment, emphasizing self-awareness and self-analysis. The proposed self-awareness model enables robots to assess task complexity and map task requirements to their own capabilities. This can be used in several ways, including strategically requesting human assistance when needed. By leveraging pre-trained foundation models, our framework supports a multi-role mechanism that enhances adaptability and ensures effective collaboration between robots and humans in real-world scenarios. The experiments were conducted using a TIAGo robot with the framework’s ability to identify and delegate challenging tasks on a set of nine real-life scenarios, spanning different complexity levels. The results show a significant improvement in task success rate, achieving 89.1% compared to 37.0% of the baseline, demonstrating the effectiveness of incorporating self-awareness into the planning process. The code and the additional materials have been publicly released on the project website.

Task and motion planning are long-standing challenges in robotics, especially when robots have to deal with dynamic environments exhibiting long-term dynamics, such as households or warehouses. In these environments, long-term dynamics mostly stem from human activities, since previously detected objects can be moved or removed from the scene. This adds the necessity to find such objects again before completing the designed task, increasing the risk of failure due to missed relocalizations. However, in these settings, the nature of such human-object interactions is often overlooked, despite being governed by common habits and repetitive patterns. Our conjecture is that these cues can be exploited to recover the most likely objects' positions in the scene, helping to address the problem of unknown relocalization in changing environments. To this end we propose FlowMaps, a model based on Flow Matching that is able to infer multimodal object locations over space and time. Our results present statistical evidence to support our hypotheses, opening the way to more complex applications of our approach. The code is publically available at https://github.com/Fra-Tsuna/flowmaps

Recent years have witnessed a growing interest in automating labor-intensive and complex activities, i.e., those consisting of multiple atomic tasks, by deploying robots in dynamic and unpredictable environments such as industrial and agricultural settings. A key characteristic of these contexts is that activities are not predefined: while they involve a limited set of possible tasks, their combinations may vary depending on the situation. Moreover, despite recent advances in robotics, the ability for humans to monitor the progress of high-level activities - in terms of past, present, and future actions - remains fundamental to ensure the correct execution of safety-critical processes. In this paper, we introduce a general architecture that integrates Large Language Models (LLMs) with automated planning, enabling humans to specify high-level activities (also referred to as processes) using natural language, and to monitor their execution by querying a robot. We also present an implementation of this architecture using state-of-the-art components and quantitatively evaluate the approach in a real-world precision agriculture scenario.

Large Language Models (LLMs) and Visual Language Models (VLMs) are attracting increasing interest due to their improving performance and applications across various domains and tasks. However, LLMs and VLMs can produce erroneous results, especially when a deep understanding of the problem domain is required. For instance, when planning and perception are needed simultaneously, these models often struggle because of difficulties in merging multi-modal information. To address this issue, fine-tuned models are typically employed and trained on specialized data structures representing the environment. This approach has limited effectiveness, as it can overly complicate the context for processing. In this paper, we propose a multi-agent architecture for embodied task planning that operates without the need for specific data structures as input. Instead, it uses a single image of the environment, handling free-form domains by leveraging commonsense knowledge. We also introduce a novel, fully automatic evaluation procedure, PG2S, designed to better assess the quality of a plan. We validated our approach using the widely recognized ALFRED dataset, comparing PG2S to the existing KAS metric to further evaluate the quality of the generated plans.