NeuroCLIP: Neuromorphic Data Understanding by CLIP and SNN
Authors:
- Yufei GuoYGYufei Guo
- Yuanpei ChenYCYuanpei Chen
- Zhe MaZMZhe Ma
Preprint
Abstract
Recently, the neuromorphic vision sensor has received more and more interest. However, the neuromorphic data consists of asynchronous event spikes, which makes it difficult to construct a big benchmark to train a power general neural network model, thus limiting the neuromorphic data understanding for ``unseen" objects by deep learning. While for the frame image, since the training data can be obtained easily, the zero-shot and few-shot learning for ``unseen" task via the large Contrastive Vision-Language Pre-training (CLIP) model, which is pre-trained by large-scale image-text pairs in 2D, have shown inspirational performance. We wonder whether the CLIP could be transferred to neuromorphic data recognition to handle the ``unseen" problem. To this end, we materialize this idea with NeuroCLIP in the paper. The NeuroCLIP consists of 2D CLIP and two specially designed modules for neuromorphic data understanding. First, an event-frame module that could convert the event spikes to the sequential frame image with a simple discrimination strategy. Second, an inter-timestep adapter, which is a simple fine-tuned adapter based on a spiking neural network (SNN) for the sequential features coming from the visual encoder of CLIP to improve the few-shot performance. Various experiments on neuromorphic datasets including N-MNIST, CIFAR10-DVS, and ES-ImageNet demonstrate the effectiveness of NeuroCLIP. Our code is open-sourced at https://github.com/yfguo91/NeuroCLIP.git.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- RMP-Loss: Regularizing Membrane Potential Distribution for Spiking Neural Networks2023
- Membrane Potential Batch Normalization for Spiking Neural Networks2023
- Neuromorphic Data Augmentation for Training Spiking Neural Networks2022
- Learning to Prompt for Vision-Language ModelsInternational Journal of Computer Vision, 2022
- Object Detection with Spiking Neural Networks on Automotive Event Data2022
- PointCLIP: Point Cloud Understanding by CLIP2022
- Optimizing Deeper Spiking Neural Networks for Dynamic Vision SensingNeural Networks, 2021
- Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera2019
- HATS: Histograms of Averaged Time Surfaces for Robust Event-Based Object Classification2018
- Event-Based Stereo Matching Approaches for Frameless Address Event Stereo Data2011