An End-to-End Two-Branch Network Towards Robust Video Fingerprinting
- 25 September 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 5 (5) , 2371-2384
Abstract
With the increasing number of edited videos, many robust video fingerprinting schemes have been proposed to solve the problem of video content authentication. However, most of them either deal with the temporal and spatial features symmetrically or insufficiently consider the temporal information. In this work, an end-to-end two-branch network towards robust video fingerprinting ( RVFNet ) is proposed, where the two branches focus on the temporal and spatial information, respectively. The temporal branch aims to comprehensively capture complex motion patterns by combining subtle motion changes with the overall motion trend. The spatial branch exploits the pixel-level information obtained by multiple receptive fields while preserving significant structural features. Deep metric learning is employed in the training process, and we adopt hard triplet loss to constrain the generation of fingerprints. Furthermore, we construct a large-scale and complex dataset for the robust video fingerprinting task based on multiple video content-preserving manipulations in actual scenarios. The size of our dataset exceeds most datasets adopted in the current robust video fingerprinting schemes. Based on the proposed dataset, experimental results demonstrate that our scheme achieves outstanding performance improvements compared to the state-of-the-art.Keywords
Funding Information
- National Natural Science Foundation of China (62172280, U20B2051)
- Natural Science Foundation of Shanghai Municipality (21ZR1444600)
- Shanghai Science and Technology Committee Capability Construction Project for Shanghai Municipal Universities (20060502300)
This publication has 37 references indexed in Scilit:
- Classification-enhancement deep hashing for large-scale video retrievalApplied Soft Computing, 2021
- Video hashing with secondary frames and invariant momentsJournal of Visual Communication and Image Representation, 2021
- High Accuracy Perceptual Video Hashing via Low-Rank Decomposition and DWT2019
- Geometrically robust video hashing based on ST-PCT for video copy detectionMultimedia Tools and Applications, 2019
- Perceptual video hashing based on temporal wavelet transform and random projections with application to indexing and retrieval of near-identical videosMultimedia Tools and Applications, 2019
- Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification2018
- Learning spatial-temporal features for video copy detection by the combination of CNN and RNNJournal of Visual Communication and Image Representation, 2018
- A robust hashing algorithm based on SURF for video copy detectionComputers & Security, 2012
- From Image Hashing to Video Hashing2010
- Convolutional Learning of Spatio-temporal Features2010