Photo2Video: Semantic-Aware Deep Learning-Based Video Generation from Still Content
Open Access
- 10 March 2022
- journal article
- research article
- Published by MDPI AG in Journal of Imaging
- Vol. 8 (3) , 68
Abstract
Applying machine learning (ML), and especially deep learning, to understand visual content is becoming common practice in many application areas. However, little attention has been given to its use within the multimedia creative domain. It is true that ML is already popular for content creation, but the progress achieved so far addresses essentially textual content or the identification and selection of specific types of content. A wealth of possibilities are yet to be explored by bringing the use of ML into the multimedia creative process, allowing the knowledge inferred by the former to influence automatically how new multimedia content is created. The work presented in this article provides contributions in three distinct ways towards this goal: firstly, it proposes a methodology to re-train popular neural network models in identifying new thematic concepts in static visual content and attaching meaningful annotations to the detected regions of interest; secondly, it presents varied visual digital effects and corresponding tools that can be automatically called upon to apply such effects in a previously analyzed photo; thirdly, it defines a complete automated creative workflow, from the acquisition of a photograph and corresponding contextual data, through the ML region-based annotation, to the automatic application of digital effects and generation of a semantically aware multimedia story driven by the previously derived situational and visual contextual data. Additionally, it presents a variant of this automated workflow by offering to the user the possibility of manipulating the automatic annotations in an assisted manner. The final aim is to transform a static digital photo into a short video clip, taking into account the information acquired. The final result strongly contrasts with current standard approaches of creating random movements, by implementing an intelligent content- and context-aware video.Keywords
Funding Information
- The European Commission (H2020-ICT-20-2017-1-RIA-780612)
- National Funds through the Portuguese funding agency, FCT - Fundação para a Ciência e a Tecnologia (LA/P/0063/2020)
This publication has 19 references indexed in Scilit:
- Inferring Contextual Data from Real-World Photography2021
- MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object SegmentationIEEE Transactions on Image Processing, 2020
- The Open Images Dataset V4International Journal of Computer Vision, 2020
- Content-Based Visual Summarization for Image CollectionsIEEE Transactions on Visualization and Computer Graphics, 2019
- Efficient CIEDE2000-based Color Similarity Decision for Computer VisionIEEE Transactions on Circuits and Systems for Video Technology, 2019
- Deep Convolutional Neural Networks for Image Classification: A Comprehensive ReviewNeural Computation, 2017
- Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors2017
- A collaborative approach for semantic time-based video annotation using gamificationHuman-centric Computing and Information Sciences, 2017
- Using the crowd to boost video annotation processes2015
- The development of the CIE 2000 colour‐difference formula: CIEDE2000Color Research & Application, 2001