Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models
Preprint
Abstract
Toon shading is a type of non-photorealistic rendering task of animation. Its primary purpose is to render objects with a flat and stylized appearance. As diffusion models have ascended to the forefront of image synthesis methodologies, this paper delves into an innovative form of toon shading based on diffusion models, aiming to directly render photorealistic videos into anime styles. In video stylization, extant methods encounter persistent challenges, notably in maintaining consistency and achieving high visual quality. In this paper, we model the toon shading problem as four subproblems: stylization, consistency enhancement, structure guidance, and colorization. To address the challenges in video stylization, we propose an effective toon shading approach called \textit{Diffutoon}. Diffutoon is capable of rendering remarkably detailed, high-resolution, and extended-duration videos in anime style. It can also edit the content according to prompts via an additional branch. The efficacy of Diffutoon is evaluated through quantitive metrics and human evaluation. Notably, Diffutoon surpasses both open-source and closed-source baseline approaches in our experiments. Our work is accompanied by the release of both the source code and example videos on Github (Project page: https://ecnu-cilab.github.io/DiffutoonProjectPage/).Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- A Survey on Video Diffusion ModelsACM Computing Surveys, 2024
- DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis2024
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation2023
- FateZero: Fusing Attentions for Zero-shot Text-based Video Editing2023
- Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators2023
- SRDiff: Single image super-resolution with diffusion probabilistic modelsNeurocomputing, 2022
- Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data2021
- Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset TransferIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020