Optical Flow with Semantic Segmentation and Localized Layers
Authors:
- Laura Sevilla-LaraLSLaura Sevilla-Lara
- Deqing SunDSDeqing Sun
- Varun JampaniVJVarun Jampani
- Michael J. BlackMBMichael J. Black
- 1 June 2016
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3889-3898
Abstract
Existing optical flow methods make generic, spatially homogeneous, assumptions about the spatial structure of the flow. In reality, optical flow varies across an image depending on object class. Simply put, different objects move differently. Here we exploit recent advances in static semantic scene segmentation to segment the image into objects of different types. We define different models of image motion in these regions depending on the type of object. For example, we model the motion on roads with homographies, vegetation with spatially smooth flow, and independently moving objects like cars and planes with affine motion plus deviations. We then pose the flow estimation problem using a novel formulation of localized layers, which addresses limitations of traditional layered models for dealing with complex scene motion. Our semantic flow method achieves the lowest error of any published monocular method in the KITTI-2015 flow benchmark and produces qualitatively better flow and segmentation than recent top methods on a wide range of natural videos.Keywords
All Related Versions
This publication has 45 references indexed in Scilit:
- Vision meets robotics: The KITTI datasetThe International Journal of Robotics Research, 2013
- Semantic Video Segmentation from Occlusion Relations within a Convex Optimization FrameworkLecture Notes in Computer Science, 2013
- Joint Optimization for Object Class Segmentation and Dense Stereo ReconstructionInternational Journal of Computer Vision, 2011
- A Database and Evaluation Methodology for Optical FlowInternational Journal of Computer Vision, 2010
- Occlusion Boundaries from Motion: Low-Level Detection and Mid-Level ReasoningInternational Journal of Computer Vision, 2009
- Learning Layered Motion Segmentations of VideoInternational Journal of Computer Vision, 2007
- A unified approach to moving object detection in 2D and 3D scenesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1998
- Dense estimation and object-based segmentation of the optical flow with robust techniquesIEEE Transactions on Image Processing, 1998
- Estimating optical flow in segmented images using variable-order parametric models with local deformationsIEEE Transactions on Pattern Analysis and Machine Intelligence, 1996
- Representing moving images with layersIEEE Transactions on Image Processing, 1994