Hongtao Xie

Face swapping efforts strive to achieve high-fidelity and well-controlled generation effects. Owing to the remarkable generative capabilities, diffusion models deliver promising high-fidelity solutions. However, their intrinsic stochastic properties complicate the accurate modeling of facial representations, introducing new challenges for identity and attribute consistency of the generated faces. In this paper, we introduce a novel diffusion-based face-swapping framework, named SwapController, which achieves high-fidelity generation via careful facial identity and attribute modeling. Specifically, our facial modeling mainly involves facial structure and facial texture. For structure modeling, 3D facial priors are leveraged to provide explicit structure supervision, enabling accurate head structure control. On this basis, two novel components are proposed to deeply mine facial textural representations from identity and attribute aspects. To enhance identity control, multi-grained source identity embeddings are obtained from various functional encoders to convey critical global identity and fine-grained identity details. To improve attribute modeling, identity-shifted attribute embeddings are derived by applying identity modulation to the most salient textural attribute features of the target face. Moreover, in line with the diffusion denoising characteristics, a timestep-aware identity optimization objective is introduced to optimize identity consistency guidelines and overall fidelity. Extensive experiments demonstrate the effectiveness of our SwapController in generating identity-consistent portrait images while faithfully preserving target attributes, which obtains a 98.32 ID Retrieval, exceeding the SOTA DiffSFSR by 7.32 $\uparrow$. Codes will be available at https://github.com/tyrink/SwapController.

In the scenario of ubiquitous visual interaction within Internet of Things (IoT) applications, the reliability of Text-to-Image (T2I) models are challenged by inaccurate text rendering, which hampers the effective delivery of critical visual information. Contemporary T2I models typically rely on non-iterative generative paradigms without self-correction, yielding unreliable outputs that fall short of the high-precision demands in automated IoT systems. To address this limitation, we propose H-SMART, a training-free framework that introduces Hierarchical Self-correcting Multi-Agent Reasoning for Text Rendering (H-SMART). H-SMART shifts the generative process from static generation to a dynamic, agent-steered trajectory by decoupling high-level strategic reasoning from low-level execution. Within this architecture, a central orchestrator agent comprehends user intent and autonomously initiates the generative process, orchestrating an iterative refinement loop through the synergistic coordination of specialized agents. Specifically, a planning agent formulates diagnostic strategies by translating generative failures into actionable corrective plans, which the orchestrator then executes by invoking specialized tools for localized image manipulation. A critic agent subsequently performs verification to determine the efficacy of each refinement. Extensive experiments across diverse models and benchmarks demonstrate that H-SMART consistently yields substantial accuracy gains. Specifically, our framework achieves an average absolute improvement of 6.2% in word accuracy across various T2I models and benchmarks, validating the generalizability of our reflective paradigm and its necessity for achieving accurate visual text rendering.

Recent advances in score distillation sampling (SDS) have revolutionized the field of text-to-3D generation, enabling the distillation of prior knowledge from diffusion models for 3D generation. Although exhibiting impressive texture quality, these methods often suffer from geometric inconsistencies (“Janus” issue), as the prior 2D diffusion model inherently lacks 3D awareness. Recent work fine-tunes the 2D diffusion model on 3D data to obtain a multi-view diffusion model as the SDS prior, which addresses the Janus issue but is at the cost of sacrificing texture quality, as available 3D training data always have unrealistic texture. Thus, a natural question arises — Is there an ideal prior diffusion model for 3D generation that simultaneously has 3D awareness and high texture fidelity? In response, we present HiFi3D, a tuning-free method to establish a new hybrid diffusion model that can generate consistent multi-view images with photorealism appearances. We accomplish this by novelly marring a 3D multi-view diffusion model with a 2D image diffusion model through our unique designs. We find that such a high-fidelity multi-view diffusion model harbors an innate agency to serve as a strong prior for SDS optimization. Additionally, we introduce a depth-guided multi-view attention strategy to further improve the 3D consistency across views during optimization. Extensive experiments demonstrate that our HiFi3D outperforms previous state-of-the-art methods in faithfully generating 3D content with realistic textual details and consistent geometry.