Tong-Yee Lee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Tong-Yee Lee has a total of 221 co-authors. Here are the top co-authors by number of shared publications:
- Fan Tang
Institute of Computing Technique, Chinese Academy of Sciences, Beijing, China
18 shared publications - Weiming Dong
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China and School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
17 shared publications - Thi-Ngoc-Hanh Le
International University, VNU-HCM, Ho Chi Minh, Vietnam
13 shared publications - Changsheng Xu
Institute of Automation, Chinese Academy of Sciences, Beijing, China, University of Chinese Academy of Sciences, Beijing, China, and Peng Cheng Laboratory, Shenzhen, China
10 shared publications - Ping Li
The Hong Kong Polytechnic University
10 shared publications - Yuxin Zhang
MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, China and School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
10 shared publications
Latest Publications
- Open Access
- 8 October 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 22 (10) , 1-28
Integrating Visual and Contextual Cues for Ancient Chinese Rubbing Image Restoration
Rubbing served as a crucial cultural carrier, transferring characters from steles onto paper using ink, thereby facilitating greater dissemination and long-term preservation. However, prolonged corrosion, inclement weather, and other factors cause various rubbing deteriorations. Therefore, an automated rubbing image restoration method is highly desirable to reduce the complexity of manual restoration procedures. In particular, when we refer to restoring a rubbing image, we mean recreating the original text on the rubbing image to reflect its original appearance. Existing methods, such as image denoising, can address minor deterioration but cannot handle severe deterioration. Image inpainting methods attempt to handle large areas of deterioration but generally struggle to produce contextually consistent strokes due to a lack of control in the restoration process. In addition, currently, there is no reliable method for rubbing character recognition. In this article, we propose a novel 2-stage rubbing image restoration method that performs well on different levels of deterioration. In the first stage, we integrate visual and contextual information for text recognition in rubbing images, including those that are heavily deteriorated. The second stage includes a multi-prior conditional latent diffusion model for rubbing image restoration, utilizing the character identities recognized in the first stage and undeteriorated characters as priors to guide the restoration process. Extensive experiments have demonstrated the effectiveness of our method compared to existing approaches.
- 8 October 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 22 (10) , 1-28
Open AccessIntegrating Visual and Contextual Cues for Ancient Chinese Rubbing Image Restoration
Rubbing served as a crucial cultural carrier, transferring characters from steles onto paper using ink, thereby facilitating greater dissemination and long-term preservation. However, prolonged corrosion, inclement weather, and other factors cause various rubbing deteriorations. Therefore, an automated rubbing image restoration method is highly desirable to reduce the complexity of manual restoration procedures. In particular, when we refer to restoring a rubbing image, we mean recreating the original text on the rubbing image to reflect its original appearance. Existing methods, such as image denoising, can address minor deterioration but cannot handle severe deterioration. Image inpainting methods attempt to handle large areas of deterioration but generally struggle to produce contextually consistent strokes due to a lack of control in the restoration process. In addition, currently, there is no reliable method for rubbing character recognition. In this article, we propose a novel 2-stage rubbing image restoration method that performs well on different levels of deterioration. In the first stage, we integrate visual and contextual information for text recognition in rubbing images, including those that are heavily deteriorated. The second stage includes a multi-prior conditional latent diffusion model for rubbing image restoration, utilizing the character identities recognized in the first stage and undeteriorated characters as priors to guide the restoration process. Extensive experiments have demonstrated the effectiveness of our method compared to existing approaches.
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 6766-6780
GSDeformer: Direct, Real-Time and Extensible Cage-Based Deformation for 3D Gaussian Splatting
We present GSDeformer, a method that enables cage-based deformation on 3D Gaussian Splatting (3DGS). Our approach bridges cage-based deformation and 3DGS by using a proxy point-cloud representation. This point cloud is generated from 3D Gaussians, and deformations applied to the point cloud are translated into transformations on the 3D Gaussians. To handle potential bending caused by deformation, we incorporate a splitting process to approximate it. Our method does not modify or extend the core architecture of 3D Gaussian Splatting, making it compatible with any trained vanilla 3DGS or its variants. Additionally, we automate cage construction for 3DGS and its variants using a render-and-reconstruct approach. Experiments demonstrate that GSDeformer delivers superior deformation results compared to existing methods, is robust under extreme deformations, requires no retraining for editing, runs in real-time, and can be extended to other 3DGS variants.
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 6766-6780
GSDeformer: Direct, Real-Time and Extensible Cage-Based Deformation for 3D Gaussian Splatting
We present GSDeformer, a method that enables cage-based deformation on 3D Gaussian Splatting (3DGS). Our approach bridges cage-based deformation and 3DGS by using a proxy point-cloud representation. This point cloud is generated from 3D Gaussians, and deformations applied to the point cloud are translated into transformations on the 3D Gaussians. To handle potential bending caused by deformation, we incorporate a splitting process to approximate it. Our method does not modify or extend the core architecture of 3D Gaussian Splatting, making it compatible with any trained vanilla 3DGS or its variants. Additionally, we automate cage construction for 3DGS and its variants using a render-and-reconstruct approach. Experiments demonstrate that GSDeformer delivers superior deformation results compared to existing methods, is robust under extreme deformations, requires no retraining for editing, runs in real-time, and can be extended to other 3DGS variants.
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 4959-4972
ArtCrafter: Text-Image Aligning Artistic Attribute Transfer via Embedding Reframing
Recent years have witnessed significant advancements in text-guided style transfer, primarily attributed to innovations in diffusion models. These models excel in conditional guidance, utilizing text or images to direct the sampling process. Traditional style transfer focuses on low-level visual features, such as brushstroke textures and color distributions, and appears more like applying an artistic filter to an image. Artistic attribute transfer, however, transcends the limitations of traditional style transfer by achieving the transfer of visual concepts from color and brushstrokes to high level aesthetic attributes such as composition, pose, and key semantic elements, resulting in more natural outcomes. Therefore, we propose an innovative text-to-image artistic attribute transfer framework named ArtCrafter. Specifically, we introduce an attention-based style extraction module, meticulously engineered to capture the subtle artistic attribute elements within an image. This module features a multi-layer architecture that leverages the capabilities of perceiver attention mechanisms to integrate fine-grained information. Additionally, we present a novel text-image aligning augmentation component that adeptly balances control over both modalities, enabling the model to efficiently map image and text embeddings into a shared feature space. We achieve this through attention operations that enable smooth information flow between modalities. Lastly, we incorporate an explicit modulation that seamlessly blends multimodal enhanced embeddings with original embeddings through an embedding reframing design, empowering the model to generate diverse outputs. Extensive experiments demonstrate that ArtCrafter yields impressive results in visual stylization, exhibiting exceptional levels of artistic attribute intensity, controllability, and diversity.
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (7) , 4959-4972
ArtCrafter: Text-Image Aligning Artistic Attribute Transfer via Embedding Reframing
Recent years have witnessed significant advancements in text-guided style transfer, primarily attributed to innovations in diffusion models. These models excel in conditional guidance, utilizing text or images to direct the sampling process. Traditional style transfer focuses on low-level visual features, such as brushstroke textures and color distributions, and appears more like applying an artistic filter to an image. Artistic attribute transfer, however, transcends the limitations of traditional style transfer by achieving the transfer of visual concepts from color and brushstrokes to high level aesthetic attributes such as composition, pose, and key semantic elements, resulting in more natural outcomes. Therefore, we propose an innovative text-to-image artistic attribute transfer framework named ArtCrafter. Specifically, we introduce an attention-based style extraction module, meticulously engineered to capture the subtle artistic attribute elements within an image. This module features a multi-layer architecture that leverages the capabilities of perceiver attention mechanisms to integrate fine-grained information. Additionally, we present a novel text-image aligning augmentation component that adeptly balances control over both modalities, enabling the model to efficiently map image and text embeddings into a shared feature space. We achieve this through attention operations that enable smooth information flow between modalities. Lastly, we incorporate an explicit modulation that seamlessly blends multimodal enhanced embeddings with original embeddings through an embedding reframing design, empowering the model to generate diverse outputs. Extensive experiments demonstrate that ArtCrafter yields impressive results in visual stylization, exhibiting exceptional levels of artistic attribute intensity, controllability, and diversity.
- Open Access
- 28 May 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 46 (1) , 1-14
DealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer
Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose DealMaTe , using de pth, norm a l, and l ighting images for ma terial t ransf e r. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.
- 28 May 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 46 (1) , 1-14
Open AccessDealMaTe: Multi-Dimensional Material Transfer via Diffusion Transformer
Recently, diffusion-based material transfer methods rely on image fine-tuning or complex architectures with auxiliary networks but face challenges such as text dependency, additional computational costs, and feature misalignment. To address these limitations, we propose DealMaTe , using de pth, norm a l, and l ighting images for ma terial t ransf e r. DealMaTe is a simplified diffusion framework that eliminates text guidance and reference networks. We design a lightweight 3D information injection method, Multi-Dim 3D Shader LoRA, which, without modifying the base model weights, enables compatible control conditions and achieves harmonious and stable results. Additionally, we optimize the attention mechanism with Shader Causal Mutual Attention and key-value (KV) caching to reduce inference latency caused by multiple conditions, improve computational efficiency, and achieve high-quality material transfer results with low architectural complexity. Extensive experiments covering a wide variety of objects and lighting conditions consistently demonstrate that DealMaTe achieves remarkable high-fidelity material transfer under arbitrary input materials. The code is available at https://github.com/haha-lisa/DealMaTe.
- Top Cited PapersOpen Access
- 24 April 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (3) , 1-14
HeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads
Diffusion Transformers (DiTs) have exhibited robust capabilities in image generation tasks. However, accurate text-guided image editing for multimodal DiTs (MM-DiTs) still poses a significant challenge. Unlike UNet-based structures that could utilize self/cross-attention maps for semantic editing, MM-DiTs inherently lack support for explicit and consistent incorporated text guidance, resulting in semantic misalignment between the edited results and texts. In this study, we disclose the sensitivity of different attention heads to different image semantics within MM-DiTs and introduce HeadRouter , a training-free image editing framework that edits the source image by adaptively routing the text guidance to different attention heads in MM-DiTs. Furthermore, we propose a dual-token refinement module to refine text/image token representations for precise semantic guidance and accurate region expression. Experiments on multiple benchmarks demonstrate HeadRouter’s performance in terms of editing fidelity and image quality. The code is available at https://github.com/ICTMCG/HeadRouter .
- 24 April 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 45 (3) , 1-14
Top Cited PapersOpen AccessHeadRouter: A Training-free Image Editing Framework for MM-DiTs by Adaptively Routing Attention Heads
Diffusion Transformers (DiTs) have exhibited robust capabilities in image generation tasks. However, accurate text-guided image editing for multimodal DiTs (MM-DiTs) still poses a significant challenge. Unlike UNet-based structures that could utilize self/cross-attention maps for semantic editing, MM-DiTs inherently lack support for explicit and consistent incorporated text guidance, resulting in semantic misalignment between the edited results and texts. In this study, we disclose the sensitivity of different attention heads to different image semantics within MM-DiTs and introduce HeadRouter , a training-free image editing framework that edits the source image by adaptively routing the text guidance to different attention heads in MM-DiTs. Furthermore, we propose a dual-token refinement module to refine text/image token representations for precise semantic guidance and accurate region expression. Experiments on multiple benchmarks demonstrate HeadRouter’s performance in terms of editing fidelity and image quality. The code is available at https://github.com/ICTMCG/HeadRouter .
- 28 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (4) , 2944-2961
Interactive Visual Assessment for Text-to-Image Generation Models
Visual generation models have achieved remarkable progress in computer graphics applications but still face significant challenges in real-world deployment. Current assessment approaches for visual generation tasks typically follow an isolated three-phase framework: test input collection, model output generation, and user assessment. These fashions suffer from fixed coverage, evolving difficulty, and data leakage risks, limiting their effectiveness in comprehensively evaluating increasingly complex generation models. To address these limitations, we propose DyEval, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems. DyEval features an intuitive visual interface that enables users to interactively explore and analyze model behaviors, while adaptively generating hierarchical, fine-grained, and diverse textual inputs to continuously probe the capability boundaries of the models based on their feedback. Additionally, to provide interpretable analysis for users to further improve tested models, we develop a contextual reflection module that mines failure triggers of test inputs and reflects model potential failure patterns, supporting in-depth analysis using the logical reasoning ability of LLM. Qualitative and quantitative experiments demonstrate that DyEval can effectively help users identify max up to 2.56 timesmore generation failures than conventional methods, and uncover complex and rare failure patterns, such as issues with pronoun generation and specific cultural context generation. Our framework provides valuable insights for improving generative models and has broad implications for advancing the reliability and capabilities of visual generation systems across various domains.
- 28 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (4) , 2944-2961
Interactive Visual Assessment for Text-to-Image Generation Models
Visual generation models have achieved remarkable progress in computer graphics applications but still face significant challenges in real-world deployment. Current assessment approaches for visual generation tasks typically follow an isolated three-phase framework: test input collection, model output generation, and user assessment. These fashions suffer from fixed coverage, evolving difficulty, and data leakage risks, limiting their effectiveness in comprehensively evaluating increasingly complex generation models. To address these limitations, we propose DyEval, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems. DyEval features an intuitive visual interface that enables users to interactively explore and analyze model behaviors, while adaptively generating hierarchical, fine-grained, and diverse textual inputs to continuously probe the capability boundaries of the models based on their feedback. Additionally, to provide interpretable analysis for users to further improve tested models, we develop a contextual reflection module that mines failure triggers of test inputs and reflects model potential failure patterns, supporting in-depth analysis using the logical reasoning ability of LLM. Qualitative and quantitative experiments demonstrate that DyEval can effectively help users identify max up to 2.56 timesmore generation failures than conventional methods, and uncover complex and rare failure patterns, such as issues with pronoun generation and specific cultural context generation. Our framework provides valuable insights for improving generative models and has broad implications for advancing the reliability and capabilities of visual generation systems across various domains.
- 19 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (4) , 2883-2897
Make-Your-Anchor+: Temporal Consistent 2D Avatar Generation via Video Diffusion Prior
Despite the remarkable process of talking-head-based avatar-creating solutions, directly generating anchor-style videos with full-body motions remains challenging. In this study, we propose Make-Your-Anchor+, a novel system necessitating only a one-minute video clip of an individual for training, subsequently enabling the automatic generation of anchor-style videos with precise torso and hand movements. Specifically, we finetune a proposed structure-guided diffusion model on input video to render 3D mesh conditions into human appearances. We adopt a two-stage training strategy for the diffusion model, effectively mapping movements with specific appearances to create digital avatars for online streamers, live shopping hosts, and other applications. To produce arbitrary long temporal video, we extract human motion information from video diffusion prior by adapting the frame-wise diffusion model to pretrained video diffusion weights with lower cost, and a simple yet effective batch-overlapped temporal denoising module is proposed to bypass the constraints on video length during inference. Finally, a novel identity-specific face enhancement module is introduced to improve the visual quality of facial regions in the output videos. Comparative experiments demonstrate the system’s effectiveness and superiority in visual quality, temporal coherence, and identity preservation, outperforming SOTA diffusion/non-diffusion methods.
- 19 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (4) , 2883-2897
Make-Your-Anchor+: Temporal Consistent 2D Avatar Generation via Video Diffusion Prior
Despite the remarkable process of talking-head-based avatar-creating solutions, directly generating anchor-style videos with full-body motions remains challenging. In this study, we propose Make-Your-Anchor+, a novel system necessitating only a one-minute video clip of an individual for training, subsequently enabling the automatic generation of anchor-style videos with precise torso and hand movements. Specifically, we finetune a proposed structure-guided diffusion model on input video to render 3D mesh conditions into human appearances. We adopt a two-stage training strategy for the diffusion model, effectively mapping movements with specific appearances to create digital avatars for online streamers, live shopping hosts, and other applications. To produce arbitrary long temporal video, we extract human motion information from video diffusion prior by adapting the frame-wise diffusion model to pretrained video diffusion weights with lower cost, and a simple yet effective batch-overlapped temporal denoising module is proposed to bypass the constraints on video length during inference. Finally, a novel identity-specific face enhancement module is introduced to improve the visual quality of facial regions in the output videos. Comparative experiments demonstrate the system’s effectiveness and superiority in visual quality, temporal coherence, and identity preservation, outperforming SOTA diffusion/non-diffusion methods.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (9) , 1-15
Hiding in Plain Sight: Camouflaging Real-world Objects
Camouflage, a survival strategy perfected by nature, enables organisms to evade detection by blending seamlessly into their surroundings across multiple viewpoints. Replicating this remarkable ability in artificial settings remains a formidable challenge. While recent computational methods have advanced 2D camouflage, extending concealment into 3D is far more difficult: a single texture must reconcile drastically varying backgrounds, making effective camouflage across all views highly challenging. Prior approaches attempt to address this by designing multi view textures, but their appearance representations are too simplistic to cope with strongly conflicting backgrounds and fail to account for physical light transport, resulting in breakdowns under realistic illumination. We introduce a new paradigm that formulates 3D camouflage as a multiview inverse rendering problem. Instead of treating concealment as texture synthesis, we directly optimize appearance within a physically based rendering framework, explicitly modeling reflections, shadows, refractions, and other complex light interactions. Our approach expands the solution space by combining BSDF-based material representation with anamorphic surface displacement, enabling camouflage that adapts naturally to both viewpoint and illumination changes. Through tailored loss functions and optimization strategies, our method produces results that are not only perceptually convincing but also physically consistent, marking a significant step toward practical, real-world 3D camouflage. Experiments demonstrate that our physically grounded formulation achieves robust concealment across diverse viewpoints and lighting scenarios, substantially outperforming prior image-based methods. Project Website: https://dongee-w.github.io/camouflage website/.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 32 (9) , 1-15
Hiding in Plain Sight: Camouflaging Real-world Objects
Camouflage, a survival strategy perfected by nature, enables organisms to evade detection by blending seamlessly into their surroundings across multiple viewpoints. Replicating this remarkable ability in artificial settings remains a formidable challenge. While recent computational methods have advanced 2D camouflage, extending concealment into 3D is far more difficult: a single texture must reconcile drastically varying backgrounds, making effective camouflage across all views highly challenging. Prior approaches attempt to address this by designing multi view textures, but their appearance representations are too simplistic to cope with strongly conflicting backgrounds and fail to account for physical light transport, resulting in breakdowns under realistic illumination. We introduce a new paradigm that formulates 3D camouflage as a multiview inverse rendering problem. Instead of treating concealment as texture synthesis, we directly optimize appearance within a physically based rendering framework, explicitly modeling reflections, shadows, refractions, and other complex light interactions. Our approach expands the solution space by combining BSDF-based material representation with anamorphic surface displacement, enabling camouflage that adapts naturally to both viewpoint and illumination changes. Through tailored loss functions and optimization strategies, our method produces results that are not only perceptually convincing but also physically consistent, marking a significant step toward practical, real-world 3D camouflage. Experiments demonstrate that our physically grounded formulation achieves robust concealment across diverse viewpoints and lighting scenarios, substantially outperforming prior image-based methods. Project Website: https://dongee-w.github.io/camouflage website/.
- Open Access
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-12
In-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessIn-Context Brush: Zero-shot Customized Subject Insertion with Context-Aware Latent Space Manipulation
- Open Access
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-3
PolyArt: Customizable Multilingual Movie Poster Generation via Diffusion Transformer
- 14 December 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-3
Open AccessPolyArt: Customizable Multilingual Movie Poster Generation via Diffusion Transformer