TRTST: Arbitrary High-Quality Text-Guided Style Transfer With Transformers
- 23 January 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 34 (10577149) , 759-771
Abstract
Text-guided style transfer aims to repaint a content image with the target style described by a text prompt, offering greater flexibility and creativity compared to traditional image-guided style transfer. Despite the potential, existing text-guided style transfer methods often suffer from many issues, including insufficient visual quality, poor generalization ability, or a reliance on large amounts of paired training data. To address these limitations, we leverage the inherent strengths of transformers in handling multimodal data and propose a novel transformer-based framework called TRTST that not only achieves unpaired arbitrary text-guided style transfer but also significantly improves the visual quality. Specifically, TRTST explores combining a text transformer encoder with an image transformer encoder to project the input text prompt and content image into a joint embedding space and extract the desired style and content features. These features are then input into a multimodal co-attention module to stylize the image sequence based on the text sequence. We also propose a new adaptive parametric positional encoding (APPE) scheme which can adaptively produce different positional encodings to optimally match different inputs with a position encoder. In addition, to further improve content preservation, we introduce a text-guided identity loss to our model. Extensive results and comparisons are conducted to demonstrate the effectiveness and superiority of our method.Keywords
Funding Information
- Youth Science Foundation of Jiangsu Province (BK20230924)
- National Science Foundation of China (U24A20330, 62361166670)
This publication has 57 references indexed in Scilit:
- Language-Driven Artistic Style Transfer2022
- End-to-End Object Detection with Transformers2020
- A Style-Aware Content Loss for Real-Time HD Style Transfer2018
- Avatar-Net: Multi-scale Zero-Shot Style Transfer by Feature Decoration2018
- Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis2017
- Image Style Transfer Using Convolutional Neural Networks2016
- Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks2016
- Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis2016
- Describing Textures in the Wild2014
- Microsoft COCO: Common Objects in Context2014