BokehMe++: Harmonious Fusion of Classical and Neural Rendering for Versatile Bokeh Creation
- 18 November 2024
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (3) , 1530-1547
Abstract
Despite significant advancements in simulating the bokeh effect of Digital Single Lens Reflex Camera (DSLR) from an all-in-focus image, challenges remain in processing highlight points, preserving boundary details for in-focus objects and processing high-resolution images efficiently. To tackle these issues, we first develop a ray-tracing-based bokeh simulator. An innovative pipeline with weight redistribution is introduced to handle highlight rendering. By considering the front length of lens barrel, we can simulate realistic cat-eye effect. This bokeh simulator serves as the foundation for creating our training dataset. Building on this dataset, we introduce a hybrid framework BokehMe++, combining a classical renderer and a neural renderer. The classical renderer is implemented by a hierarchical scattering-based method, which suffers from boundary inaccuracies. These erroneous areas will be identified by an error map generator and be corrected by a two-stage neural renderer. Adaptive resizing and iterative upsampling are introduced in the neural renderer to process arbitrary blur size efficiently. Extensive experiments demonstrate that BokehMe++ outperforms existing methods and provides highly customizable rendering features, such as adjustable blur amount, focal plane, highlight mode and cat-eye effect. Furthermore, BokehMe++ can maintain the sharpness of hair details in portraits through an auxiliary alpha map input.Keywords
Funding Information
- National Natural Science Foundation of China (62406120)
This publication has 48 references indexed in Scilit:
- Realistic Bokeh Effect Rendering on Mobile GPUs, Mobile AI & AIM 2022 Challenge: Report2023
- MPIB: An MPI-Based Bokeh Rendering Framework for Realistic Partial Occlusion Effects2022
- BGGAN: Bokeh-Glass Generative Adversarial Network for Rendering Realistic Bokeh2020
- AIM 2020 Challenge on Rendering Realistic Bokeh2020
- Bokeh Rendering from Defocus Estimation2020
- Rendering Portraitures from Monocular Camera and Beyond2018
- Deep Image Matting2017
- Deep Automatic Portrait Matting2016
- Learning Recursive Filters for Low-Level Vision via a Hybrid Neural Network2016
- Fast bilateral-space stereo for synthetic defocus2015