Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videos
Authors:
- Mohaiminul IslamMIMohaiminul Islam
- Tushar NagarajanTNTushar Nagarajan
- Fu-Jen ChuFCFu-Jen Chu
- Kris KitaniKKKris Kitani
- Gedas BertasiusGBGedas Bertasius
- Xitong YangXYXitong Yang
Preprint
Abstract
Goal-oriented planning, or anticipating a series of actions that transition an agent from its current state to a predefined objective, is crucial for developing intelligent assistants aiding users in daily procedural tasks. The problem presents significant challenges due to the need for comprehensive knowledge of temporal and hierarchical task structures, as well as strong capabilities in reasoning and planning. To achieve this, prior work typically relies on extensive training on the target dataset, which often results in significant dataset bias and a lack of generalization to unseen tasks. In this work, we introduce VidAssist, an integrated framework designed for zero/few-shot goal-oriented planning in instructional videos. VidAssist leverages large language models (LLMs) as both the knowledge base and the assessment tool for generating and evaluating action plans, thus overcoming the challenges of acquiring procedural knowledge from small-scale, low-diversity datasets. Moreover, VidAssist employs a breadth-first search algorithm for optimal plan generation, in which a composite of value functions designed for goal-oriented planning is utilized to assess the predicted actions at each step. Extensive experiments demonstrate that VidAssist offers a unified framework for different goal-oriented planning setups, e.g., visual planning for assistance (VPA) and procedural planning (PP), and achieves remarkable performance in zero-shot and few-shot setups. Specifically, our few-shot model outperforms the prior fully supervised state-of-the-art method by +7.7% in VPA and +4.81% PP task on the COIN dataset while predicting 4 future actions. Code, and models are publicly available at https://sites.google.com/view/vidassist.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- Language Repository for Long Video Understanding2025
- Video ReCap: Recursive Captioning of Hour-Long Videos2024
- PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents2024
- Multimodal Procedural Planning via Dual Text-Image Prompting2024
- A Simple LLM Framework for Long-Range Video Question-Answering2024
- Pretrained Language Models as Visual Planners for Human Assistance2023
- PlaTe: Visually-Grounded Planning With Transformers in Procedural TasksIEEE Robotics and Automation Letters, 2022
- VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding2021
- The limits and potentials of deep learning for roboticsThe International Journal of Robotics Research, 2018
- Human problem solving: The state of the theory in 1970.American Psychologist, 1971