SPIKE-RL: Video-LLMs meet Bayesian Surprise
Preprint
Abstract
Real-world videos often show routine activities punctuated by memorable, surprising events. However, most Video-LLMs process videos by sampling frames uniformly, likely missing critical moments that define a video's narrative. We introduce SPIKE, an inference-time framework that quantifies Bayesian Surprise as the belief update triggered by new visual evidence in the video stream, identifying moments where new visual evidence conflicts with prior beliefs. SPIKE effectively localizes surprise in videos, strongly correlated with humans on positive (FunQA) and negative (Oops!) surprise benchmarks. Since the beliefs of zero-shot Video-LLMs are often suboptimal, we develop SPIKE-RL, which leverages GRPO to optimize belief hypotheses based on a reward signal from the video caption. SPIKE and SPIKE-RL guide query-agnostic surprise-weighted frame sampling, which allocates more frames to interesting moments in the video. With this strategy, we achieve consistent performance gains on five downstream benchmarks over uniform sampling. By enabling Video-LLMs to track beliefs and register surprise, our work paves the way for more robust models that can revise their understanding in response to new information.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis2025
- Adaptive Keyframe Sampling for Long Video Understanding2025
- Enough Coin Flips Can Make LLMs Act Bayesian2025
- Robust Laughter Segmentation with Automatic Diverse Data Synthesis2024
- Improved Baselines with Visual Instruction Tuning2024
- UniVTG: Towards Unified Video-Language Temporal Grounding2023
- Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker2023
- Thinking Like a Skeptic: Defeasible Inference in Natural Language2020
- Rational quantitative attribution of beliefs, desires and percepts in human mentalizingNature Human Behaviour, 2017