RefinedStyle: Text-Conditioned Style Attention Reconstruction for Training-Free Style Transfer
Authors:
- Yilun LiuYLYilun Liu
- Jingyue WangJWJingyue Wang
- Meiqing WangMWMeiqing Wang
Preprint
- 26 June 2026
- preprint
- Published by Elsevier in SSRN Electronic Journal
Abstract
Recent advances in diffusion models have substantially improved image style transfer. However, text-driven style transfer remains challenging because models often overfit to the reference style and exhibit insufficient text-image alignment. We propose RefinedStyle, a training-free text-driven style-transfer framework that reconstructs reference-driven style attention into a text-conditioned style representation while balancing textual semantics with style guidance. Specifically, SFCA forms text-aware style attention by fusing modality-specific key/value representations, and Adapter-AdaIN further reconstructs the injected style attention by using text attention as the content carrier and SFCA-enhanced style attention as the source of style statistics. Built on IP-Adapter, RefinedStyle generates style-customized images that preserve text content while mitigating content leakage and style overfitting. We also analyze the effective denoising stages for style injection and show that injecting style information during the early denoising stage, e.g., the first 20 steps in our 50-step inference setting, is sufficient for effective style transfer. Extensive experiments demonstrate that RefinedStyle improves text-image alignment while preserving reference-style consistency. Code will be made available upon acceptance.Keywords
This publication has 8 references indexed in Scilit:
- An Item Is Worth a Prompt: Versatile Image Editing with Disentangled ControlProceedings of the AAAI Conference on Artificial Intelligence, 2025
- DesignEdit: Unify Spatial-Aware Image Editing via Training-free Inpainting with a Multi-Layered Latent Diffusion FrameworkProceedings of the AAAI Conference on Artificial Intelligence, 2025
- An Image is Worth Multiple Words: Multi-Attribute Inversion for Constrained Text-To-Image Synthesis2025
- A photo cartoonization method based on text-to-image diffusion modelNeurocomputing, 2024
- StyleAdapter: A Unified Stylized Image Generation ModelInternational Journal of Computer Vision, 2024
- CLIPScore: A Reference-free Evaluation Metric for Image Captioning2021
- Image Compression and Encryption Algorithm Based on Hyper-chaotic MapMobile Networks and Applications, 2019
- New image compression techniques using multiwavelets and multiwavelet packetsIEEE Transactions on Image Processing, 2001