A Two-Stage Generate-Then-Reason Vision–Language Framework for Task-Conditioned Grasp Pose Selection

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Task-conditioned robotic manipulation requires grasps that are geometrically plausible and compatible with downstream function. Although vision–language models (VLMs) have shown promise for semantic reasoning in robotic manipulation, VLM-selected grasps may not consistently satisfy both requirements. To address this issue, we propose a two-stage generate-then-reason framework for task-conditioned 6-DoF grasp selection. In Stage 1, a VLM localizes task-relevant object regions, followed by segmentation and GraspNet-based generation of plausible grasp candidates. In Stage 2, another VLM ranks grounded grasp candidates according to their compatibility with the specified downstream task. The evaluation in this journal includes a 60-trial full-system physical study and four controlled replay experiments that examine Stage 1 localization, Stage 2 cross-model ranking, repeated-inference reliability, and cross-stage candidate-pool sensitivity. The physical system achieved an overall task success rate of 63.3%. In the replay analysis, Gemini 3 Flash Preview achieved the highest Stage 1 localization accuracy of 95.0% and the lowest median Stage 1 latency of 11.09 s. For Stage 2 ranking, GPT-5.6 Sol achieved the highest Top-1 task-valid rate of 92.9% on identical frozen candidate pools and showed the strongest repeated-inference behavior among the evaluated critics. The cross-stage sensitivity experiment yielded task-valid selection rates ranging from 70.8% to 87.5% across six model combinations, indicating sensitivity of downstream selection to upstream candidate evidence within the evaluated conditions. These results indicate that physical task completion and stage-specific VLM behavior capture different aspects of system performance and should be evaluated separately when assessing VLM-based grasp-selection systems.
Funding Information
  • National Taiwan Normal University (114-2221-E-003-022-MY2)
  • National Taiwan Normal University (114-2224-E-003-001)
  • National Taiwan Normal University (115-2222-E-032-006)

This publication has 15 references indexed in Scilit: