Toward Human Perception-Centric Video Thumbnail Generation
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 6653-6664
Abstract
Video thumbnail plays an essential role in summarizing video content into a compact and concise image for users to browse efficiently. However, automatically generating attractive and informative video thumbnails remains an open problem due to the difficulty of formulating human aesthetic perception and the scarcity of paired training data. This work proposes a novel Human Perception-Centric Video Thumbnail Generation (HPCVTG) to address these challenges. Specifically, our framework first generates a set of thumbnails using a principle-based system, which conforms to established aesthetic and human perception principles, such as visual balance in the layout and avoiding overlapping elements. Then rather than designing from scratch, we ask human annotators to evaluate some of these thumbnails and select their preferred ones. A Transformer-based Variational Auto-Encoder (VAE) model is firstly pre-trained with Model-Agnostic Meta-Learning (MAML) and then fine-tuned on these human-selected thumbnails. The exploration of combining the MAML pre-training paradigm with human feedback in training can reduce human involvement and make the training process more efficient. Extensive experimental results show that our HPCVTG framework outperforms existing methods in objective and subjective evaluations, highlighting its potential to improve the user experience when browsing videos and inspire future research in human perception-centric content generation tasks. The code and dataset will be released via https://github.com/yangtao2019yt/HPCVTG.Keywords
Funding Information
- Tencent PCG (ZGG9-P0043883)
This publication has 27 references indexed in Scilit:
- Layout Style Modeling for Automating Banner Design2017
- Automatic creation of magazine-page-like social media visual summary for mobile browsing2016
- Automatic Generation of Visual-Textual Presentation LayoutACM Transactions on Multimedia Computing, Communications, and Applications, 2016
- Automatic generation of social media snippets for mobile browsing2013
- Contextual Internet Multimedia AdvertisingProceedings of the IEEE, 2010
- Computational models and experimental investigations of effects of balance and symmetry on the aesthetics of text-overlaid imagesInternational Journal of Human-Computer Studies, 2010
- Visual snippets2009
- Improving relevance judgment of web search results with image excerpts2008
- Computational modeling and experimental investigation of effects of compositional elements on interface and design aestheticsInternational Journal of Human-Computer Studies, 2006
- Attention Capture and Transfer in Advertising: Brand, Pictorial, and Text-Size EffectsJournal of Marketing, 2004