E$^3$NeRF: Efficient Event-Enhanced Neural Radiance Fields from Blurry Images
Preprint
Abstract
Neural Radiance Fields (NeRF) achieve impressive rendering performance by learning volumetric 3D representation from several images of different views. However, it is difficult to reconstruct a sharp NeRF from blurry input as it often occurs in the wild. To solve this problem, we propose a novel Efficient Event-Enhanced NeRF (E$^3$NeRF) by utilizing the combination of RGB images and event streams. To effectively introduce event streams into the neural volumetric representation learning process, we propose an event-enhanced blur rendering loss and an event rendering loss, which guide the network via modeling the real blur process and event generation process, respectively. Specifically, we leverage spatial-temporal information from the event stream to evenly distribute learning attention over temporal blur while simultaneously focusing on blurry texture through the spatial attention. Moreover, a camera pose estimation framework for real-world data is built with the guidance of the events to generalize the method to practical applications. Compared to previous image-based or event-based NeRF, our framework makes more profound use of the internal relationship between events and images. Extensive experiments on both synthetic data and real-world data demonstrate that E$^3$NeRF can effectively learn a sharp NeRF from blurry images, especially in non-uniform motion and low-light scenes.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020
- Real-time high speed motion prediction using fast aperture-robust event-driven visual flowIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Event-Based Vision: A SurveyIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Low-Resolution Face Recognition in the Wild via Selective Knowledge DistillationIEEE Transactions on Image Processing, 2018
- Front and Back Illuminated Dynamic and Active Pixel Vision Sensors ComparisonIEEE Transactions on Circuits and Systems II: Express Briefs, 2018
- RankIQA: Learning from Rankings for No-Reference Image Quality Assessment2017
- A 240 × 180 130 dB 3 µs Latency Global Shutter Spatiotemporal Vision SensorIEEE Journal of Solid-State Circuits, 2014
- Total variation blind deconvolutionIEEE Transactions on Image Processing, 1998
- A 128× 128 120 dB 15 μs latency asynchronous temporal contrast vision sensor