Segmenting the Motion Components of a Video: A Long-Term Unsupervised Model
- 9 September 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (1) , 500-511
Abstract
Human beings have the ability to continuously analyze a video and immediately extract the motion components. We want to adopt this paradigm to provide a coherent and stable motion segmentation over the video sequence. In this perspective, we propose a novel long-term spatio-temporal model operating in a totally unsupervised way. It takes as input the volume of consecutive optical flow (OF) fields, and delivers a volume of segments of coherent motion over the video. More specifically, we have designed a transformer-based network, where we leverage a mathematically well-founded framework, the Evidence Lower Bound (ELBO), to derive the loss function. The loss function combines a flow reconstruction term involving spatio-temporal parametric motion models combining, in a novel way, polynomial (quadratic) motion models for the spatial dimensions and B-splines for the time dimension of the video sequence, and a regularization term enforcing temporal consistency on the segments. We report experiments on four VOS benchmarks, demonstrating competitive quantitative results while performing motion segmentation on a sequence in one go. We also highlight through visual results the key contributions on temporal consistency brought by our method.Keywords
All Related Versions
Funding Information
- BpiFrance through the LiChIE contract within a collaboration with Airbus Defence & Space
This publication has 43 references indexed in Scilit:
- Betrayed by Motion: Camouflaged Object Discovery via Motion Segmentation2021
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow2020
- A survey of variational and CNN-based optical flow techniquesSignal Processing: Image Communication, 2019
- Self-supervised Segmentation by Grouping Optical-Flow2019
- MoA-Net: Self-supervised Motion Segmentation2019
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- A Naturalistic Open Source Movie for Optical Flow Evaluation2012
- Motion Competition: A Variational Approach to Piecewise Parametric Motion SegmentationInternational Journal of Computer Vision, 2004
- Computation and analysis of image motion: A synopsis of current problems and methodsInternational Journal of Computer Vision, 1996
- On the limited memory BFGS method for large scale optimizationMathematical Programming, 1989