Spatial and temporal constrained ranked retrieval over videos
Authors:
Open Access
- 1 July 2022
- journal article
- conference paper
- Published by Association for Computing Machinery (ACM) in Proceedings of the VLDB Endowment
- Vol. 15 (11) , 3226-3239
Abstract
Recent advances in Computer Vision (CV) algorithms have improved accuracy and efficiency, making video annotations possible with high accuracy. In this paper, we utilize the annotated data provided by such algorithms and construct graph representations to capture both object labels and spatial-temporal relationships of objects in videos. We define the problem of Spatial and Temporal Constrained Ranked Retrieval (STAR Retrieval) over videos. Based on the graph representation, we propose a two-phase approach, consisting of the ingestion phase, where we construct and materialize the Graph Index (GI), and the query phase, where we compute the top ranked windows (video clips) according to the window matching score efficiently. We propose two algorithms to perform Spatial Matching (SMA) and Temporal Matching (TM) separately with an early-stopping mechanism. Our experiments demonstrate the effectiveness of the proposed methods, achieving orders of magnitude speedups on queries with high selectivity.Keywords
This publication has 34 references indexed in Scilit:
- You Only Look Once: Unified, Real-Time Object Detection2016
- Durable graph pattern queries on historical graphs2016
- Fast R-CNN2015
- Going deeper with convolutions2015
- Taming verification hardnessProceedings of the VLDB Endowment, 2008
- A review of text and image retrieval approaches for broadcast news videoInformation Retrieval Journal, 2007
- GraphScope2007
- Face Retrieval in Broadcasting News Video by Fusing Temporal and Intensity Information2006
- Person Spotting: Video Shot Retrieval for Face Sets2005
- Querying Video Data by Spatio-Temporal Relationships of Moving Object Traces2002