Tear the Image Into Strips for Style Transfer
- 14 September 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 3978-3988
Abstract
Recently, Deep Convolutional Neural Networks (DCNNs) have achieved remarkable progress in computer vision community, including in style transfer tasks. Normally, most methods feed the full image to the DCNN. Although highquality results can be achieved in this manner, several underlying problems arise. For one, with the increase in image resolution, the memory footprint will increase dramatically, leading to high latency and massive power consumption. Furthermore, these methods are usually unable to integrate with the commercial image signal processor (ISP), which processes the image in a line-sequential manner. To solve the above problems, we propose a novel ISP-friendly deep learning-based style transfer algorithm: SequentialStyle. A brand new line-sequential processing mode is proposed, where the image is torn into strips, and each strip is sequentially processed, contributing to less memory demand. We further propose a Spatial-Temporal Synergistic (STS) mechanism that decouples the previously simplex 2-D image style transfer into spatial feature processing (in-strip) and temporal correlation transmission (in-between strips). Compared with the SOTA style transfer algorithms, experimental results show that our SequentialStyle is competitive. Besides, SequentialStyle has less demand for memory consumption, even for the images whose resolutions are 4k or higher.Keywords
Funding Information
- National Natural Science Foundation of China (62031009)
- National Natural Science Foundation of China (61525401)
- Science and Technology Commission of Shanghai Municipality (19511104300)
- Alibaba Innovative Research
- Innovation Program of Shanghai Municipal Education Commission
- Fudan University-CIOMP Joint Fund (FC2019-001)
- Fudan-ZTE Joint Laboratory
This publication has 48 references indexed in Scilit:
- Perceptual Losses for Real-Time Style Transfer and Super-Resolution2016
- Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks2016
- A High-Throughput Neural Network AcceleratorIEEE Micro, 2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Speech recognition with deep recurrent neural networks2013
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- Graphcut texturesACM Transactions on Graphics, 2003
- Bilateral filtering for gray and color images2002
- Texture synthesis by non-parametric sampling1999
- Long Short-Term MemoryNeural Computation, 1997