Shengyu Gao
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Shengyu Gao has a total of 26 co-authors. Here are the top co-authors by number of shared publications:
- Xin Lou
ShanghaiTech University
4 shared publications - Chip-Hong Chang
Nanyang technological university, Singapore, Singapore
2 shared publications - Jiahua Jiang
School of Mathematics, University of Birmingham, Birmingham, UK
2 shared publications - Linlin Li
ShanghaiTech University
2 shared publications - Wuwei Ren
ShanghaiTech University
2 shared publications - Ying Tao
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore
2 shared publications
Latest Publications
- 20 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. PP (23798920) , 1-15
Object Detection with Spiking Neural Networks on Event Videos for Driving Scenarios
Event cameras excel in imaging objects under challenging lighting conditions with low latency. Their sparse and asynchronous event outputs closely align with the operational paradigm of Spiking Neural Networks (SNNs). An SNN mimics biological neurons of brain by transmitting information via binary spikes. The combination of biologically inspired event-based sensing with neuromorphic computing, while individually promising, faces significant challenges in their coherent integration to deliver better performance than existing solutions for real-time multi-scale, multi-rate object detection problems. In this paper, we propose a new Spike Tensor-based Multi-scale Object Detector (STMOD) for event video analytics in driving scenarios. STMOD features a cross-attention spatio-temporal spikes generator to convert asynchronous events into a high-dimension binary vector, a spiking backbone with three pyramid feature levels to fuse multi-scale spatio-temporal features and an asymmetric decoupled detection head to achieve high precision, energy efficient and responsive object detection without compromising the temporal resolution. Additionally, a new training method is proposed to overcome the mismatch in SNN’s state between training with independent samples and testing with video streaming to improve its performance on continuous video feed without impacting inference time resolution. Experimental results on the GEN1 and DSEC-Detection datasets show that STMOD achieves state-of-the-art mAP of 44.3% and 34.2%, respectively, with an inference time resolution of 5 ms. Finally, it also rivals existing event-based object detectors with significantly lower number of parameters and higher energy efficiency.
- 20 February 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cognitive and Developmental Systems
- Vol. PP (23798920) , 1-15
Object Detection with Spiking Neural Networks on Event Videos for Driving Scenarios
Event cameras excel in imaging objects under challenging lighting conditions with low latency. Their sparse and asynchronous event outputs closely align with the operational paradigm of Spiking Neural Networks (SNNs). An SNN mimics biological neurons of brain by transmitting information via binary spikes. The combination of biologically inspired event-based sensing with neuromorphic computing, while individually promising, faces significant challenges in their coherent integration to deliver better performance than existing solutions for real-time multi-scale, multi-rate object detection problems. In this paper, we propose a new Spike Tensor-based Multi-scale Object Detector (STMOD) for event video analytics in driving scenarios. STMOD features a cross-attention spatio-temporal spikes generator to convert asynchronous events into a high-dimension binary vector, a spiking backbone with three pyramid feature levels to fuse multi-scale spatio-temporal features and an asymmetric decoupled detection head to achieve high precision, energy efficient and responsive object detection without compromising the temporal resolution. Additionally, a new training method is proposed to overcome the mismatch in SNN’s state between training with independent samples and testing with video streaming to improve its performance on continuous video feed without impacting inference time resolution. Experimental results on the GEN1 and DSEC-Detection datasets show that STMOD achieves state-of-the-art mAP of 44.3% and 34.2%, respectively, with an inference time resolution of 5 ms. Finally, it also rivals existing event-based object detectors with significantly lower number of parameters and higher energy efficiency.
- 10 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 11 (25730436) , 955-966
High Resolution Diffuse Optical Tomography via Neural Fields
Light scattering imposes a major obstacle for imaging objects seated deeply in turbid media, such as biological tissues and foggy air. Diffuse optical tomography (DOT) tackles scattering by volumetrically recovering the optical absorbance and has shown significance in medical imaging, remote sensing and autonomous driving. A conventional DOT reconstruction paradigm necessitates discretizing the object volume into voxels at a pre-determined resolution for modelling diffuse light propagation and the resultant spatial resolution of the reconstruction is generally limited. We propose NeuDOT, a novel DOT scheme based on neural fields (NF) to continuously encode the optical absorbance within the volume and subsequently bridge the gap between model efficiency and high resolution. Comprehensive experiments demonstrate that NeuDOT affords to resolve complex 3D objects at 14 mm depth with submillimeter lateral resolution, outperforming the state-of-the-art methods. NeuDOT is a high-resolution and computationally efficient tomographic method, and also unlocks further applications of NF involving light scattering.
- 10 October 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Imaging
- Vol. 11 (25730436) , 955-966
High Resolution Diffuse Optical Tomography via Neural Fields
Light scattering imposes a major obstacle for imaging objects seated deeply in turbid media, such as biological tissues and foggy air. Diffuse optical tomography (DOT) tackles scattering by volumetrically recovering the optical absorbance and has shown significance in medical imaging, remote sensing and autonomous driving. A conventional DOT reconstruction paradigm necessitates discretizing the object volume into voxels at a pre-determined resolution for modelling diffuse light propagation and the resultant spatial resolution of the reconstruction is generally limited. We propose NeuDOT, a novel DOT scheme based on neural fields (NF) to continuously encode the optical absorbance within the volume and subsequently bridge the gap between model efficiency and high resolution. Comprehensive experiments demonstrate that NeuDOT affords to resolve complex 3D objects at 14 mm depth with submillimeter lateral resolution, outperforming the state-of-the-art methods. NeuDOT is a high-resolution and computationally efficient tomographic method, and also unlocks further applications of NF involving light scattering.
- 6 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biomedical Engineering
- Vol. 71 (4) , 1391-1403
Multifunctional Optical Tomography System With High-Fidelity Surface Extraction Based on a Single Programmable Scanner and Unified Pinhole Modeling
Objective: Macroscopic optical tomography is a non-invasive method that can visualize the 3D distribution of intrinsic optical properties or exogenous fluorophores, making it highly attractive for small animal imaging. However, reconstructing the images requires prior knowledge of surface information. To address this, existing systems often use additional hardware components or integrate multimodal information, which is expensive and introduces new issues such as image registration. Our goal is to develop a multifunctional optical tomography system that can extract surface information using a concise hardware design. Methods: Our proposed system uses a single programmable scanner to implement both surface extraction and optical tomography functions. A unified pinhole model is used to describe both the illumination and detection procedures for capturing 3D point cloud. Line-shaped scanning is adopted to improve both spatial resolution and speed of surface extraction. Finally, we integrate the extracted surface information into the optical tomographic reconstruction to more accurately map the fluorescence distribution. Result: Comprehensive phantom experiments with different levels of complexity were designed to evaluate the performance of surface extraction and fluorescence tomography. We also imaged the axillary lymph nodes in living mice after injection of fluorophore, demonstrating the proposed system facilitates more reliable fluorescence tomography. Conclusion: We have successfully developed a versatile optical tomography system by leveraging concise hardware design and unified pinhole modeling. Phantom validation demonstrates that our system provides high-precision surface information with a maximum error of 0.1 mm, while the surface-guided FMT reconstruction is more reliable than the blind reconstruction using simplified surface geometry, elevating several quantitative metrics including RMSE, CNR, and Dice. Significance: Our work explores the feasibility of obtaining additional surface information using existing components of standalone optical tomography. This makes the optical tomographic technique more accurate and more accessible to biomedical researchers.
- 6 December 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Biomedical Engineering
- Vol. 71 (4) , 1391-1403
Multifunctional Optical Tomography System With High-Fidelity Surface Extraction Based on a Single Programmable Scanner and Unified Pinhole Modeling
Objective: Macroscopic optical tomography is a non-invasive method that can visualize the 3D distribution of intrinsic optical properties or exogenous fluorophores, making it highly attractive for small animal imaging. However, reconstructing the images requires prior knowledge of surface information. To address this, existing systems often use additional hardware components or integrate multimodal information, which is expensive and introduces new issues such as image registration. Our goal is to develop a multifunctional optical tomography system that can extract surface information using a concise hardware design. Methods: Our proposed system uses a single programmable scanner to implement both surface extraction and optical tomography functions. A unified pinhole model is used to describe both the illumination and detection procedures for capturing 3D point cloud. Line-shaped scanning is adopted to improve both spatial resolution and speed of surface extraction. Finally, we integrate the extracted surface information into the optical tomographic reconstruction to more accurately map the fluorescence distribution. Result: Comprehensive phantom experiments with different levels of complexity were designed to evaluate the performance of surface extraction and fluorescence tomography. We also imaged the axillary lymph nodes in living mice after injection of fluorophore, demonstrating the proposed system facilitates more reliable fluorescence tomography. Conclusion: We have successfully developed a versatile optical tomography system by leveraging concise hardware design and unified pinhole modeling. Phantom validation demonstrates that our system provides high-precision surface information with a maximum error of 0.1 mm, while the surface-guided FMT reconstruction is more reliable than the blind reconstruction using simplified surface geometry, elevating several quantitative metrics including RMSE, CNR, and Dice. Significance: Our work explores the feasibility of obtaining additional surface information using existing components of standalone optical tomography. This makes the optical tomographic technique more accurate and more accessible to biomedical researchers.
- 17 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 22 (14) , 14306-14319
A Time-Frequency Bins Selection Pipeline for Direction-of-Arrival Estimation Using a Single Acoustic Vector Sensor
The case that multiple speakers are simultaneously active is usually regarded as adverse condition in the context of multi-source direction-of-arrival (DOA) estimation. Moreover, in adjacent sources scenarios, classifying DOA estimates by directions may merge adjacent sources and misclassify reverberation as sources, i.e., there is a mismatch between sources and reverberation. In order to address the open challenges mentioned above, this work introduces a pipeline for selecting reliable time-frequency (TF) bins and estimating multi-source DOAs using a single acoustic vector sensor (AVS). In the proposed pipeline, we show a viewpoint that the multiple simultaneously active sources condition can be exploited to select TF bins solely dominated by sources. The consistency of the distribution of intensity vectors (IVs) in multi-source scenarios is derived mathematically to support the proposed viewpoint. We address the mismatch between sources and reverberations in adjacent sources scenarios by focusing only on the direction with the highest intensity vector density (IVD), regardless of how many sources dominate the given TF window. Simulation and experimental results show that the proposed method can accurately and robustly estimate the multi-source DOAs in various scenarios.
- 17 June 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Sensors Journal
- Vol. 22 (14) , 14306-14319
A Time-Frequency Bins Selection Pipeline for Direction-of-Arrival Estimation Using a Single Acoustic Vector Sensor
The case that multiple speakers are simultaneously active is usually regarded as adverse condition in the context of multi-source direction-of-arrival (DOA) estimation. Moreover, in adjacent sources scenarios, classifying DOA estimates by directions may merge adjacent sources and misclassify reverberation as sources, i.e., there is a mismatch between sources and reverberation. In order to address the open challenges mentioned above, this work introduces a pipeline for selecting reliable time-frequency (TF) bins and estimating multi-source DOAs using a single acoustic vector sensor (AVS). In the proposed pipeline, we show a viewpoint that the multiple simultaneously active sources condition can be exploited to select TF bins solely dominated by sources. The consistency of the distribution of intensity vectors (IVs) in multi-source scenarios is derived mathematically to support the proposed viewpoint. We address the mismatch between sources and reverberations in adjacent sources scenarios by focusing only on the direction with the highest intensity vector density (IVD), regardless of how many sources dominate the given TF window. Simulation and experimental results show that the proposed method can accurately and robustly estimate the multi-source DOAs in various scenarios.
- 20 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 5122-5137
Gradient-Based Feature Extraction From Raw Bayer Pattern Images
In this paper, the impact of demosaicing on gradient extraction is studied and a gradient-based feature extraction pipeline based on raw Bayer pattern images is proposed. It is shown both theoretically and experimentally that the Bayer pattern images are applicable to the central difference gradient-based feature extraction algorithms with negligible performance degradation, as long as the arrangement of color filter array (CFA) patterns matches the gradient operators. The color difference constancy assumption, which is widely used in various demosaicing algorithms, is applied in the proposed Bayer pattern image-based gradient extraction pipeline. Experimental results show that the gradients extracted from Bayer pattern images are robust enough to be used in histogram of oriented gradients (HOG)-based pedestrian detection algorithms and shift-invariant feature transform (SIFT)-based matching algorithms. By skipping most of the steps in the image signal processing (ISP) pipeline, the computational complexity and power consumption of a computer vision system can be reduced significantly.
- 20 May 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 5122-5137
Gradient-Based Feature Extraction From Raw Bayer Pattern Images
In this paper, the impact of demosaicing on gradient extraction is studied and a gradient-based feature extraction pipeline based on raw Bayer pattern images is proposed. It is shown both theoretically and experimentally that the Bayer pattern images are applicable to the central difference gradient-based feature extraction algorithms with negligible performance degradation, as long as the arrangement of color filter array (CFA) patterns matches the gradient operators. The color difference constancy assumption, which is widely used in various demosaicing algorithms, is applied in the proposed Bayer pattern image-based gradient extraction pipeline. Experimental results show that the gradients extracted from Bayer pattern images are robust enough to be used in histogram of oriented gradients (HOG)-based pedestrian detection algorithms and shift-invariant feature transform (SIFT)-based matching algorithms. By skipping most of the steps in the image signal processing (ISP) pipeline, the computational complexity and power consumption of a computer vision system can be reduced significantly.
- Open Access
- 24 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 47786-47794
A Low-Complexity End-to-End Stereo Matching Pipeline From Raw Bayer Pattern Images to Disparity Maps
Conventional computer vision algorithms, including stereo matching algorithms, take finely rendered color images as input. However, existing image signal processing (ISP) pipelines for color image generation are designed for photography with a goal of generating pleasing images for human eyes. This paper describes a new end-to-end pipeline for stereo matching from raw Bayer pattern images to disparity maps with customized ISP. Unlike conventional stereo matching systems which need a complete ISP module to render full-size standard RGB (sRGB) images, a subsampling-based demosaicing-downsampling (SDD) operation is introduced in the proposed pipeline to demosaic and downsample the Bayer pattern images. The resultant half-size color image pairs are processed with simple denoising and tone mapping algorithms to generate the final input images of stereo matching algorithms. It is found that the simple nearest neighbor upsampling method is good enough to generate the final full-size disparity maps. Experimental results show that the proposed pipeline is capable of generating comparable or even better stereo matching results than the conventional pipeline. By skipping most of the unnecessary ISP steps and reducing the size of input images, the computational complexity of the end-to-end stereo matching pipeline is significantly reduced.
- 24 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 9, 47786-47794
Open AccessA Low-Complexity End-to-End Stereo Matching Pipeline From Raw Bayer Pattern Images to Disparity Maps
Conventional computer vision algorithms, including stereo matching algorithms, take finely rendered color images as input. However, existing image signal processing (ISP) pipelines for color image generation are designed for photography with a goal of generating pleasing images for human eyes. This paper describes a new end-to-end pipeline for stereo matching from raw Bayer pattern images to disparity maps with customized ISP. Unlike conventional stereo matching systems which need a complete ISP module to render full-size standard RGB (sRGB) images, a subsampling-based demosaicing-downsampling (SDD) operation is introduced in the proposed pipeline to demosaic and downsample the Bayer pattern images. The resultant half-size color image pairs are processed with simple denoising and tone mapping algorithms to generate the final input images of stereo matching algorithms. It is found that the simple nearest neighbor upsampling method is good enough to generate the final full-size disparity maps. Experimental results show that the proposed pipeline is capable of generating comparable or even better stereo matching results than the conventional pipeline. By skipping most of the unnecessary ISP steps and reducing the size of input images, the computational complexity of the end-to-end stereo matching pipeline is significantly reduced.