Weihua He
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Weihua He has a total of 55 co-authors. Here are the top co-authors by number of shared publications:
- Guoqi Li
Institute of Automation, Chinese Academy of Sciences, Beijing, China
5 shared publications - Yaoyuan Wang
Huawei Technologies Co., Ltd., Shenzhen, China
4 shared publications - Ziyang Zhang
HUAWEI, Shenzhen, China
4 shared publications Ziyang Zhang
Huawei Technologies Company Limited, Shenzhen, P.R. China
3 shared publications- Bo Xu
Institute of Automation, Chinese Academy of Sciences, Beijing, China
2 shared publications Chen Yang
Huawei Technologies Co. Ltd
2 shared publications
Latest Publications
- 13 September 2026
- Published by Springer Nature
- p. 539-556
Intra-Class Consistency Guided Class-Agnostic Event Segmentation
- 13 September 2026
- Published by Springer Nature
- p. 539-556
Intra-Class Consistency Guided Class-Agnostic Event Segmentation
- 14 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (7) , 11977-11988
Event-Assisted Recurrent Network for Arbitrary-Temporal-Scale Blurry Image Unfolding
Recovering a sequence of latent sharp frames from a motion-blurred image is a challenging task. The bio-inspired event camera, which produces an event stream with high temporal resolution, has been exploited to promote the recovery performance. However, recovering sharp sequences with arbitrary temporal scales has been ignored for a long time. Existing works can only recover a fixed number of latent frames from a blurry image once they are trained. In this work, we propose an event-assisted blurry image unfolding framework that can work across arbitrary temporal scales. A bi-directional recurrent network is employed to encode events corresponding to each latent frame, which gathers information over all events in the exposure time. Features of both the blurry image and events are fused together and fed to a bi-directional latent sequence decoder (BiLSD) to produce a sequence of latent sharp frames. Extensive experiments show that the proposed method not only performs favorably against state-of-the-art methods in recovering a fixed number of frames from a blurry image but can be well generalized to arbitrary-temporal-scale blurry image unfolding.
- 14 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36 (7) , 11977-11988
Event-Assisted Recurrent Network for Arbitrary-Temporal-Scale Blurry Image Unfolding
Recovering a sequence of latent sharp frames from a motion-blurred image is a challenging task. The bio-inspired event camera, which produces an event stream with high temporal resolution, has been exploited to promote the recovery performance. However, recovering sharp sequences with arbitrary temporal scales has been ignored for a long time. Existing works can only recover a fixed number of latent frames from a blurry image once they are trained. In this work, we propose an event-assisted blurry image unfolding framework that can work across arbitrary temporal scales. A bi-directional recurrent network is employed to encode events corresponding to each latent frame, which gathers information over all events in the exposure time. Features of both the blurry image and events are fused together and fed to a bi-directional latent sequence decoder (BiLSD) to produce a sequence of latent sharp frames. Extensive experiments show that the proposed method not only performs favorably against state-of-the-art methods in recovering a fixed number of frames from a blurry image but can be well generalized to arbitrary-temporal-scale blurry image unfolding.
- 1 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 5824-5836
Event-Assisted Blurriness Representation Learning for Blurry Image Unfolding
The goal of blurry image deblurring and unfolding task is to recover a single sharp frame or a sequence from a blurry one. Recently, its performance is greatly improved with introduction of a bio-inspired visual sensor, event camera. Most existing event-assisted deblurring methods focus on the design of powerful network architectures and effective training strategy, while ignoring the role of blur modeling in removing various blur in dynamic scenes. In this work, we propose to implicitly model blur in an image by computing blurriness representation with an event-assisted blurriness encoder. The learning of blurriness representation is formulated as a ranking problem based on specially synthesized pairs. Blurriness-aware image unfolding is achieved by integrating blur relevant information contained in the representation into a base unfolding network. The integration is mainly realized by the proposed blurriness-guided modulation and multi-scale aggregation modules. Experiments on GOPRO and HQF datasets show favorable performance of the proposed method against state-of-the-art approaches. More results on real-world data validate its effectiveness in recovering a sequence of latent sharp frames from a blurry image.
- 1 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 33 (10577149) , 5824-5836
Event-Assisted Blurriness Representation Learning for Blurry Image Unfolding
The goal of blurry image deblurring and unfolding task is to recover a single sharp frame or a sequence from a blurry one. Recently, its performance is greatly improved with introduction of a bio-inspired visual sensor, event camera. Most existing event-assisted deblurring methods focus on the design of powerful network architectures and effective training strategy, while ignoring the role of blur modeling in removing various blur in dynamic scenes. In this work, we propose to implicitly model blur in an image by computing blurriness representation with an event-assisted blurriness encoder. The learning of blurriness representation is formulated as a ranking problem based on specially synthesized pairs. Blurriness-aware image unfolding is achieved by integrating blur relevant information contained in the representation into a base unfolding network. The integration is mainly realized by the proposed blurriness-guided modulation and multi-scale aggregation modules. Experiments on GOPRO and HQF datasets show favorable performance of the proposed method against state-of-the-art approaches. More results on real-world data validate its effectiveness in recovering a sequence of latent sharp frames from a blurry image.
- 16 August 2024
- Published by Springer Nature in Nature Computational Science
- Vol. 4 (8) , 584-599
Network model with internal complexity bridges artificial intelligence and neuroscience
Artificial intelligence (AI) researchers currently believe that the main approach to building more general model problems is the big AI model, where existing neural networks are becoming deeper, larger and wider. We term this the big model with external complexity approach. In this work we argue that there is another approach called small model with internal complexity, which can be used to find a suitable path of incorporating rich properties into neurons to construct larger and more efficient AI models. We uncover that one has to increase the scale of the network externally to stimulate the same dynamical properties. To illustrate this, we build a Hodgkin–Huxley (HH) network with rich internal complexity, where each neuron is an HH model, and prove that the dynamical properties and performance of the HH network can be equivalent to a bigger leaky integrate-and-fire (LIF) network, where each neuron is a LIF neuron with simple internal complexity.
- 16 August 2024
- Published by Springer Nature in Nature Computational Science
- Vol. 4 (8) , 584-599
Network model with internal complexity bridges artificial intelligence and neuroscience
Artificial intelligence (AI) researchers currently believe that the main approach to building more general model problems is the big AI model, where existing neural networks are becoming deeper, larger and wider. We term this the big model with external complexity approach. In this work we argue that there is another approach called small model with internal complexity, which can be used to find a suitable path of incorporating rich properties into neurons to construct larger and more efficient AI models. We uncover that one has to increase the scale of the network externally to stimulate the same dynamical properties. To illustrate this, we build a Hodgkin–Huxley (HH) network with rich internal complexity, where each neuron is an HH model, and prove that the dynamical properties and performance of the HH network can be equivalent to a bigger leaky integrate-and-fire (LIF) network, where each neuron is a LIF neuron with simple internal complexity.
- 10 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. 16 (6) , 2008-2018
Event-Based Depth Prediction With Deep Spiking Neural Network
Event cameras have gained popularity in depth estimation due to their superior features such as high temporal resolution, low latency, and low power consumption. Spiking Neural Network (SNN) is a promising approach for processing event camera inputs due to its spike-based event-driven nature. However, SNNs face performance degradation when the network becomes deeper, affecting their performance in depth estimation tasks. To address this issue, we propose a deep spiking U-Net model. Our spiking U-Net architecture leverages refined shortcuts and residual blocks to avoid performance degradation and boost task performance. We also propose a new event representation method designed for multi-step SNNs to effectively utilize depth information in the temporal dimension. Our experiments on MVSEC dataset show that the proposed method improves accuracy by 18.50% and 25.18% compared to current state-of-the- art ANN and SNN models, respectively. Moreover, the energy efficiency can be improved up to 58 times by our proposed SNN model compared with the corresponding ANN with the same network structure.
- 10 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. 16 (6) , 2008-2018
Event-Based Depth Prediction With Deep Spiking Neural Network
Event cameras have gained popularity in depth estimation due to their superior features such as high temporal resolution, low latency, and low power consumption. Spiking Neural Network (SNN) is a promising approach for processing event camera inputs due to its spike-based event-driven nature. However, SNNs face performance degradation when the network becomes deeper, affecting their performance in depth estimation tasks. To address this issue, we propose a deep spiking U-Net model. Our spiking U-Net architecture leverages refined shortcuts and residual blocks to avoid performance degradation and boost task performance. We also propose a new event representation method designed for multi-step SNNs to effectively utilize depth information in the temporal dimension. Our experiments on MVSEC dataset show that the proposed method improves accuracy by 18.50% and 25.18% compared to current state-of-the- art ANN and SNN models, respectively. Moreover, the energy efficiency can be improved up to 58 times by our proposed SNN model compared with the corresponding ANN with the same network structure.
- 21 April 2024
- Published by Elsevier in Neural Networks
- Vol. 176, 106330
Multi-scale full spike pattern for semantic segmentation
- 21 April 2024
- Published by Elsevier in Neural Networks
- Vol. 176, 106330
Multi-scale full spike pattern for semantic segmentation
- 3 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 1829-1842
Event-Based Semantic Segmentation With Posterior Attention
In the past years, attention-based Transformers have swept across the field of computer vision, starting a new stage of backbones in semantic segmentation. Nevertheless, semantic segmentation under poor light conditions remains an open problem. Moreover, most papers about semantic segmentation work on images produced by commodity frame-based cameras with a limited framerate, hindering their deployment to auto-driving systems that require instant perception and response at milliseconds. An event camera is a new sensor that generates event data at microseconds and can work in poor light conditions with a high dynamic range. It looks promising to leverage event cameras to enable perception where commodity cameras are incompetent, but algorithms for event data are far from mature. Pioneering researchers stack event data as frames so that event-based segmentation is converted to framebased segmentation, but characteristics of event data are not explored. Noticing that event data naturally highlight moving objects, we propose a posterior attention module that adjusts the standard attention by the prior knowledge provided by event data. The posterior attention module can be readily plugged into many segmentation backbones. Plugging the posterior attention module into a recently proposed SegFormer network, we get EvSegFormer (the event-based version of SegFormer) with state-of-the-art performance in two datasets (MVSEC and DDD-17) collected for event-based segmentation. Code is available at https://github.com/zexiJia/EvSegFormer to facilitate research on event-based vision.
- 3 March 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 32 (10577149) , 1829-1842
Event-Based Semantic Segmentation With Posterior Attention
In the past years, attention-based Transformers have swept across the field of computer vision, starting a new stage of backbones in semantic segmentation. Nevertheless, semantic segmentation under poor light conditions remains an open problem. Moreover, most papers about semantic segmentation work on images produced by commodity frame-based cameras with a limited framerate, hindering their deployment to auto-driving systems that require instant perception and response at milliseconds. An event camera is a new sensor that generates event data at microseconds and can work in poor light conditions with a high dynamic range. It looks promising to leverage event cameras to enable perception where commodity cameras are incompetent, but algorithms for event data are far from mature. Pioneering researchers stack event data as frames so that event-based segmentation is converted to framebased segmentation, but characteristics of event data are not explored. Noticing that event data naturally highlight moving objects, we propose a posterior attention module that adjusts the standard attention by the prior knowledge provided by event data. The posterior attention module can be readily plugged into many segmentation backbones. Plugging the posterior attention module into a recently proposed SegFormer network, we get EvSegFormer (the event-based version of SegFormer) with state-of-the-art performance in two datasets (MVSEC and DDD-17) collected for event-based segmentation. Code is available at https://github.com/zexiJia/EvSegFormer to facilitate research on event-based vision.
- 13 November 2022
- Published by Springer Nature
- p. 267-283
Video Interpolation by Event-Driven Anisotropic Adjustment of Optical Flow
- 13 November 2022
- Published by Springer Nature
- p. 267-283
Video Interpolation by Event-Driven Anisotropic Adjustment of Optical Flow
- 30 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 69 (3) , 1332-1336
An Event-Driven Spatiotemporal Domain Adaptation Method for DVS Gesture Recognition
This brief presents a novel spatio-temporal domain adaptation (DA) method that is unsupervised and event-driven for dynamic vision sensor (DVS) gesture recognition. This method realizes the transfer between the source domain and the target domain data in both the spatial and temporal dimension without the need of target domain data labels. Specifically, it consists of a deep spiking neural network (SNN)-based feature extractor, a label predictor and a domain discriminator. A time-space gradient reversal layer is responsible for building a spatio-temporal bridge between the domain discriminator and the feature extractor, which is essential to the alignment of the source domain spike features with the target ones and to achieve domain adaptation on both space and time dimensions. To demonstrate the effectiveness of our method, we adapted DVS hand gesture data from one temporal resolution to another and from original data to denoised data. Our method can provide up to 10.39% improvement in accuracy. Its accuracy improvement is more stable comparing with RNN-based and LSTM-based methods in this DA framework especially when the two domains have partial similarity in DVS data.
- 30 August 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems II: Express Briefs
- Vol. 69 (3) , 1332-1336
An Event-Driven Spatiotemporal Domain Adaptation Method for DVS Gesture Recognition
This brief presents a novel spatio-temporal domain adaptation (DA) method that is unsupervised and event-driven for dynamic vision sensor (DVS) gesture recognition. This method realizes the transfer between the source domain and the target domain data in both the spatial and temporal dimension without the need of target domain data labels. Specifically, it consists of a deep spiking neural network (SNN)-based feature extractor, a label predictor and a domain discriminator. A time-space gradient reversal layer is responsible for building a spatio-temporal bridge between the domain discriminator and the feature extractor, which is essential to the alignment of the source domain spike features with the target ones and to achieve domain adaptation on both space and time dimensions. To demonstrate the effectiveness of our method, we adapted DVS hand gesture data from one temporal resolution to another and from original data to denoised data. Our method can provide up to 10.39% improvement in accuracy. Its accuracy improvement is more stable comparing with RNN-based and LSTM-based methods in this DA framework especially when the two domains have partial similarity in DVS data.
- 1 December 2020
- Published by Elsevier in Neural Networks
- Vol. 132, 108-120
Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences
- 1 December 2020
- Published by Elsevier in Neural Networks
- Vol. 132, 108-120
Comparing SNNs and RNNs on neuromorphic vision datasets: Similarities and differences