Abstract
Recent advances in diffusion models have substantially improved image style transfer. However, text-driven style transfer remains challenging because models often overfit to the reference style and exhibit insufficient text-image alignment. We propose RefinedStyle, a training-free text-driven style-transfer framework that reconstructs reference-driven style attention into a text-conditioned style representation while balancing textual semantics with style guidance. Specifically, SFCA forms text-aware style attention by fusing modality-specific key/value representations, and Adapter-AdaIN further reconstructs the injected style attention by using text attention as the content carrier and SFCA-enhanced style attention as the source of style statistics. Built on IP-Adapter, RefinedStyle generates style-customized images that preserve text content while mitigating content leakage and style overfitting. We also analyze the effective denoising stages for style injection and show that injecting style information during the early denoising stage, e.g., the first 20 steps in our 50-step inference setting, is sufficient for effective style transfer. Extensive experiments demonstrate that RefinedStyle improves text-image alignment while preserving reference-style consistency. Code will be made available upon acceptance.

This publication has 8 references indexed in Scilit: