From Perception Errors to Unsafe Actions: A Survey of Behavioral Safety in MLLM Agents

Authors:
Preprint
Publisher website
Google Scholar
Abstract
Multimodal large language model (MLLM) agents connect perceptual inputs to tools and external actions, making safety a property of trajectories and retained state. This curated survey uses a constrained partially observable Markov decision process to organize six risk families: grounding failure, observation manipulation, miscalibrated authority, inference-time hijacking, unsafe commitment, and reusable-state corruption. We compare safeguards by mechanism, access requirements, and evidence scope, separating direct multimodal execution from transferred model-level and text-agent findings. The synthesis emphasizes image–text, GUI/web, and selected embodied settings; it is neither exhaustive nor independently coded. We distinguish response scores from executed violations and later-task reactivation, and propose containment and restoration outcomes for recovery evaluation. These distinctions motivate execution safeguards, calibrated autonomy, and integrity checks for reusable agent state.

This publication has 114 references indexed in Scilit: