ArtCrafter: Text-Image Aligning Style Transfer via Embedding Reframing
Authors:
Preprint
Abstract
Recent years have witnessed significant advancements in text-guided style transfer, primarily attributed to innovations in diffusion models. These models excel in conditional guidance, utilizing text or images to direct the sampling process. However, despite their capabilities, direct conditional guidance approaches often face challenges in balancing the expressiveness of textual semantics with the diversity of output results while capturing stylistic features. To address these challenges, we introduce ArtCrafter, a novel framework for text-to-image style transfer. Specifically, we introduce an attention-based style extraction module, meticulously engineered to capture the subtle stylistic elements within an image. This module features a multi-layer architecture that leverages the capabilities of perceiver attention mechanisms to integrate fine-grained information. Additionally, we present a novel text-image aligning augmentation component that adeptly balances control over both modalities, enabling the model to efficiently map image and text embeddings into a shared feature space. We achieve this through attention operations that enable smooth information flow between modalities. Lastly, we incorporate an explicit modulation that seamlessly blends multimodal enhanced embeddings with original embeddings through an embedding reframing design, empowering the model to generate diverse outputs. Extensive experiments demonstrate that ArtCrafter yields impressive results in visual stylization, exhibiting exceptional levels of stylistic intensity, controllability, and diversity.Keywords
All Related Versions
This publication has 7 references indexed in Scilit:
- StyleShot: A Snapshot on Any StyleIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- IPAdapter-Instruct: Resolving Ambiguity in Image-Based Conditioning Using Instruct Prompts2025
- GRA: Detecting Oriented Objects Through Group-Wise Rotating and Attention2024
- Face-Adapter for Pre-trained Diffusion Models with Fine-Grained ID and Attribute Control2024
- DiffStyler: Controllable Dual Diffusion for Text-Driven Image StylizationIEEE Transactions on Neural Networks and Learning Systems, 2024
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance2024
- Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset TransferIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020