Lei Ma
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Lei Ma has a total of 310 co-authors. Here are the top co-authors by number of shared publications:
- Tiejun Huang
Peking University
13 shared publications Xiaofei Liu
School of Computer Science, Peking University
6 shared publications- Dong-Ming Yan
Chinese Academy of Sciences
5 shared publications - Kai Du
Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China
5 shared publications Kaiwen Sheng
Institute for Artificial Intelligence, Peking University
5 shared publicationsLiuyuan He
Peking Univeristy
5 shared publications
Latest Publications
- 13 November 2026
- Published by Wiley in Computer Graphics Forum
LiteTex‐GS: Fast and Lightweight Texturing for Gaussian Splatting
Gaussian Splatting has enabled real‐time novel view synthesis, but its tightly coupled geometry and appearance representation often require a large number of primitives to reproduce high‐frequency texture details, leading to substantial memory and optimization costs. Recent textured 2D Gaussian methods alleviate this limitation by attaching texture maps to Gaussian primitives. However, bridging the fundamental structural gap between discrete Gaussians and continuous 2D grids requires complex parameterizations that introduce severe computational overhead. This overhead fundamentally compromises the original efficiency of Gaussian Splatting, making the balance between detailed texturing and computational agility an unresolved challenge. To address these challenges, we propose LiteTex‐GS, a fast and lightweight texturing framework for Gaussian Splatting. Our method initializes an extremely compact representation—assigning minimal local texture to each Gaussian and progressively allocates higher resolution only to primitives with significant reconstruction errors. To maintain a streamlined geometric scaffold, we introduce a contribution‐ and area‐aware pruning strategy that eliminates low‐utility Gaussians. Furthermore, to mitigate the gradient dilution caused by texture upsampling, we design a resolution‐aware update rule that preserves rapid and stable convergence. Extensive experiments on standard novel view synthesis benchmarks demonstrate that our method achieves competitive or superior rendering quality while using substantially fewer parameters and less training time than existing textured Gaussian baselines. Rendering Computer vision
- 13 November 2026
- Published by Wiley in Computer Graphics Forum
LiteTex‐GS: Fast and Lightweight Texturing for Gaussian Splatting
Gaussian Splatting has enabled real‐time novel view synthesis, but its tightly coupled geometry and appearance representation often require a large number of primitives to reproduce high‐frequency texture details, leading to substantial memory and optimization costs. Recent textured 2D Gaussian methods alleviate this limitation by attaching texture maps to Gaussian primitives. However, bridging the fundamental structural gap between discrete Gaussians and continuous 2D grids requires complex parameterizations that introduce severe computational overhead. This overhead fundamentally compromises the original efficiency of Gaussian Splatting, making the balance between detailed texturing and computational agility an unresolved challenge. To address these challenges, we propose LiteTex‐GS, a fast and lightweight texturing framework for Gaussian Splatting. Our method initializes an extremely compact representation—assigning minimal local texture to each Gaussian and progressively allocates higher resolution only to primitives with significant reconstruction errors. To maintain a streamlined geometric scaffold, we introduce a contribution‐ and area‐aware pruning strategy that eliminates low‐utility Gaussians. Furthermore, to mitigate the gradient dilution caused by texture upsampling, we design a resolution‐aware update rule that preserves rapid and stable convergence. Extensive experiments on standard novel view synthesis benchmarks demonstrate that our method achieves competitive or superior rendering quality while using substantially fewer parameters and less training time than existing textured Gaussian baselines. Rendering Computer vision
- Open Access
- 3 November 2026
- Published by Wiley in Advanced Science
Accelerating High‐Fidelity Airfoil Design via Physics‐Informed Video Diffusion Model
Airfoil design remains a challenging multi‐objective optimization problem requiring precise geometric control for optimal aerodynamic performance. Data‐driven inverse design methods mitigate this complexity but often yield non‐physical results when physical constraints are not explicitly incorporated. Here, we construct a high‐fidelity NACA–Nek1000 datasets with 53 400 samples from direct numerical simulations to train a dual physics‐informed video diffusion model (PVDM). The governing equations of incompressible flow are intrinsically coupled with the diffusion learning process, enabling the reconstruction of full‐field velocity and pressure distributions from lift‐to‐drag response sequences. The PVDM achieves accurate low‐Reynolds number airfoil reconstruction and real‐time flow generation within 30 s per case, approximately 600 times faster than conventional simulations, while maintaining over 90% fidelity. The reconstructed airfoils are further modeled as propeller geometries and experimentally validated, which well fit the simulated lift‐to‐drag response, establishing a rapid and generalizable framework for high‐fidelity, physics‐informed inverse aerodynamic design.
- 3 November 2026
- Published by Wiley in Advanced Science
Open AccessAccelerating High‐Fidelity Airfoil Design via Physics‐Informed Video Diffusion Model
Airfoil design remains a challenging multi‐objective optimization problem requiring precise geometric control for optimal aerodynamic performance. Data‐driven inverse design methods mitigate this complexity but often yield non‐physical results when physical constraints are not explicitly incorporated. Here, we construct a high‐fidelity NACA–Nek1000 datasets with 53 400 samples from direct numerical simulations to train a dual physics‐informed video diffusion model (PVDM). The governing equations of incompressible flow are intrinsically coupled with the diffusion learning process, enabling the reconstruction of full‐field velocity and pressure distributions from lift‐to‐drag response sequences. The PVDM achieves accurate low‐Reynolds number airfoil reconstruction and real‐time flow generation within 30 s per case, approximately 600 times faster than conventional simulations, while maintaining over 90% fidelity. The reconstructed airfoils are further modeled as propeller geometries and experimentally validated, which well fit the simulated lift‐to‐drag response, establishing a rapid and generalizable framework for high‐fidelity, physics‐informed inverse aerodynamic design.
- 22 September 2026
- Published by Springer Nature
- p. 209-226
Learn to See the Unseen in Low-Light Spike Streams
- 22 September 2026
- Published by Springer Nature
- p. 209-226
Learn to See the Unseen in Low-Light Spike Streams
- Open Access
Structural–functional calibration corrects single-neuron identity errors in volumetric calcium imaging
Volumetric calcium imaging is increasingly used to capture larger neuronal populations at higher throughput, but high-speed axial sampling can compromise single-neuron identity. Here we identify cross-plane identity duplication as a structured error in volumetric imaging: anisotropic axial blurring and plane-wise functional segmentation can repeatedly detect the same neuron across adjacent planes, creating duplicate functional nodes that inflate neuronal counts and distort network phenotypes . We developed Comprehensive Label-Guided (CLG) volumetric imaging , a structural–functional calibration framework that uses nuclear labels as stable three-dimensional identity anchors for calcium signals . CLG combines nuclear labeling, deep-learning-based 3D segmentation, anatomical registration and identity-guided trace reassignment. In larval zebrafish whole-brain recordings, CLG resolved ∼30,000 redundant detections and reduced estimated neuronal counts by 37–46%. In mouse visual cortex, CLG consolidated ∼40% of putative duplicates and recovered over 2,000 active neurons missed by calcium-only analysis. Across baseline and perturbed conditions, calibration stabilized graph-derived measurements of hub organization, long-range correlations and network resilience. CLG therefore defines an anatomy-constrained identity-calibration layer for reliable single-neuron-resolved volumetric imaging.
Open AccessStructural–functional calibration corrects single-neuron identity errors in volumetric calcium imaging
Volumetric calcium imaging is increasingly used to capture larger neuronal populations at higher throughput, but high-speed axial sampling can compromise single-neuron identity. Here we identify cross-plane identity duplication as a structured error in volumetric imaging: anisotropic axial blurring and plane-wise functional segmentation can repeatedly detect the same neuron across adjacent planes, creating duplicate functional nodes that inflate neuronal counts and distort network phenotypes . We developed Comprehensive Label-Guided (CLG) volumetric imaging , a structural–functional calibration framework that uses nuclear labels as stable three-dimensional identity anchors for calcium signals . CLG combines nuclear labeling, deep-learning-based 3D segmentation, anatomical registration and identity-guided trace reassignment. In larval zebrafish whole-brain recordings, CLG resolved ∼30,000 redundant detections and reduced estimated neuronal counts by 37–46%. In mouse visual cortex, CLG consolidated ∼40% of putative duplicates and recovered over 2,000 active neurons missed by calcium-only analysis. Across baseline and perturbed conditions, calibration stabilized graph-derived measurements of hub organization, long-range correlations and network resilience. CLG therefore defines an anatomy-constrained identity-calibration layer for reliable single-neuron-resolved volumetric imaging.
- Top Cited Papers
- 1 July 2026
- Published by Springer Nature in Nature
- Vol. 655 (8121) , 59-70
Towards the construction of a virtual yeast
To advance the computational simulation of cellular life, we propose a virtual yeast, an artificial intelligence (AI)-driven agent that models eukaryotic cellular behaviours by integrating multimodal biological data, mechanistic reasoning and active experimentation using Saccharomyces cerevisiae as a genetically tractable and data-rich model system. Cellular complexity is decomposed into eight function-centred modules, spanning genetic, metabolic and structural systems, each realized as a domain-specific AI tool coordinated through a large language model-based orchestration layer. Built on three data pillars, namely, mechanistic knowledge, subcellular architecture and dynamic states, the system integrates representation learning and generative modelling within a closed-loop learning pipeline that autonomously designs and executes experiments. The virtual yeast serves as both a conceptual and an operational platform to optimize biosynthetic pathways, support the generation and prioritization of hypotheses across diverse cellular processes, and accelerate target discovery. By coupling biological realism with autonomous AI reasoning, the virtual yeast establishes a generalizable blueprint for constructing virtual eukaryotic cells and advancing synthetic biology.
- 1 July 2026
- Published by Springer Nature in Nature
- Vol. 655 (8121) , 59-70
Top Cited PapersTowards the construction of a virtual yeast
To advance the computational simulation of cellular life, we propose a virtual yeast, an artificial intelligence (AI)-driven agent that models eukaryotic cellular behaviours by integrating multimodal biological data, mechanistic reasoning and active experimentation using Saccharomyces cerevisiae as a genetically tractable and data-rich model system. Cellular complexity is decomposed into eight function-centred modules, spanning genetic, metabolic and structural systems, each realized as a domain-specific AI tool coordinated through a large language model-based orchestration layer. Built on three data pillars, namely, mechanistic knowledge, subcellular architecture and dynamic states, the system integrates representation learning and generative modelling within a closed-loop learning pipeline that autonomously designs and executes experiments. The virtual yeast serves as both a conceptual and an operational platform to optimize biosynthetic pathways, support the generation and prioritization of hypotheses across diverse cellular processes, and accelerate target discovery. By coupling biological realism with autonomous AI reasoning, the virtual yeast establishes a generalizable blueprint for constructing virtual eukaryotic cells and advancing synthetic biology.
- Open Access
- 25 June 2026
- Published by Elsevier in SSRN Electronic Journal
Dual-Identifiable Variational Autoencoder Reveals Cross-Subject Cyclic Variables in Neural Population Dynamics
Recovering cyclic variables that generalize across subjects remains a major challenge in neural population analysis, because standard representation learning methods often entangle shared dynamics with subject-specific variability. Here we introduce the dual-identifiable variational autoencoder (diVAE), a generative framework that jointly constrains latent representations using neuronal spatial coordinates and temporal labels to extract identifiable cyclic variables. Applied to large-scale Ca²⁺ imaging data from the suprachiasmatic nucleus, the mammalian central circadian pacemaker, diVAE recovered a conserved circular manifold encoding circadian time and enabled accurate hourly time decoding in held-out subjects without retraining. Gradient-based attribution analysis further revealed a takeshift scheme in which high-contribution neurons formed spatially clustered ensembles sequentially recruited across the daily cycle, suggesting that circadian time emerges from dynamic recruitment of spatially organized subpopulations rather than a fixed set of dedicated timing neurons. Broadly, diVAE provides an interpretable framework for extracting cross-subject cyclic variables from complex neural population dynamics.
- 25 June 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessDual-Identifiable Variational Autoencoder Reveals Cross-Subject Cyclic Variables in Neural Population Dynamics
Recovering cyclic variables that generalize across subjects remains a major challenge in neural population analysis, because standard representation learning methods often entangle shared dynamics with subject-specific variability. Here we introduce the dual-identifiable variational autoencoder (diVAE), a generative framework that jointly constrains latent representations using neuronal spatial coordinates and temporal labels to extract identifiable cyclic variables. Applied to large-scale Ca²⁺ imaging data from the suprachiasmatic nucleus, the mammalian central circadian pacemaker, diVAE recovered a conserved circular manifold encoding circadian time and enabled accurate hourly time decoding in held-out subjects without retraining. Gradient-based attribution analysis further revealed a takeshift scheme in which high-contribution neurons formed spatially clustered ensembles sequentially recruited across the daily cycle, suggesting that circadian time emerges from dynamic recruitment of spatially organized subpopulations rather than a fixed set of dedicated timing neurons. Broadly, diVAE provides an interpretable framework for extracting cross-subject cyclic variables from complex neural population dynamics.
- Open Access
Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector
Recent progress in generative modeling has enabled high-quality visual synthesis with diffusion-based frameworks, supporting controllable sampling and large-scale training. Inference-time guidance methods such as classifier-free and representative guidance enhance semantic alignment by modifying sampling dynamics; however, they do not fully exploit unsupervised feature representations. Although such visual representations contain rich semantic structure, their integration during generation is constrained by the absence of ground-truth reference images at inference. This work reveals semantic drift in the early denoising stages of diffusion transformers, where stochasticity results in inconsistent alignment even under identical conditioning. To mitigate this issue, we introduce a guidance scheme using a representation alignment projector that injects representations predicted by a projector into intermediate sampling steps, providing an effective semantic anchor without modifying the model architecture. Experiments on SiTs and REPAs show notable improvements in class-conditional ImageNet synthesis, achieving substantially lower FID scores; for example, REPA-XL/2 improves from 5.9 to 3.3, and the proposed method outperforms representative guidance when applied to SiT models. The approach further yields complementary gains when combined with classifier-free guidance, demonstrating enhanced semantic coherence and visual fidelity. These results establish representation-informed diffusion sampling as a practical strategy for reinforcing semantic preservation and image consistency.
Open AccessTraining-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector
Recent progress in generative modeling has enabled high-quality visual synthesis with diffusion-based frameworks, supporting controllable sampling and large-scale training. Inference-time guidance methods such as classifier-free and representative guidance enhance semantic alignment by modifying sampling dynamics; however, they do not fully exploit unsupervised feature representations. Although such visual representations contain rich semantic structure, their integration during generation is constrained by the absence of ground-truth reference images at inference. This work reveals semantic drift in the early denoising stages of diffusion transformers, where stochasticity results in inconsistent alignment even under identical conditioning. To mitigate this issue, we introduce a guidance scheme using a representation alignment projector that injects representations predicted by a projector into intermediate sampling steps, providing an effective semantic anchor without modifying the model architecture. Experiments on SiTs and REPAs show notable improvements in class-conditional ImageNet synthesis, achieving substantially lower FID scores; for example, REPA-XL/2 improves from 5.9 to 3.3, and the proposed method outperforms representative guidance when applied to SiT models. The approach further yields complementary gains when combined with classifier-free guidance, demonstrating enhanced semantic coherence and visual fidelity. These results establish representation-informed diffusion sampling as a practical strategy for reinforcing semantic preservation and image consistency.
- Open Access
- 22 January 2026
- Published by eLife Sciences Publications, Ltd in eLife preprints
Domain-adaptive matching bridges synthetic and in vivo neural dynamics for neural circuit connectivity inference
Accurately inferring neural circuit connectivity from in vivo recordings is essential for understanding the computations that support behavior and cognition. However, current deep learning approaches are limited by incomplete observability and the lack of ground-truth labels in real experiments. Consequently, models are often trained on synthetic data, which leads to the well-known “model mismatch” problem when simulated dynamics diverge from true neural activity. To overcome these challenges, we present Deep Domain-Adaptive Matching (DeepDAM), a training framework that adaptively matches synthetic and in vivo data domains for neural connectivity inference. Specifically, DeepDAM fine-tunes deep neural networks on a combined dataset of synthetic simulations and unlabeled in vivo recordings, aligning the model’s feature representations with real neural dynamics to mitigate model mismatch. We demonstrate this approach in rodent hippocampal CA1 circuits as a proof-of-concept, achieving near-perfect connectivity inference performance (Matthews correlation coefficient ∼0.97–1.0) and substantially surpassing classical methods (∼0.6–0.7). We further demonstrate robustness across multiple recording conditions within this hippocampal dataset. Additionally, to illustrate its broader applicability, we extend the framework to two distinct systems without altering the core methodology: a stomatogastric microcircuit in Cancer borealis (ex vivo) and single-neuron intracellular recordings in mouse, where DeepDAM significantly improves efficiency and accuracy over standard approaches. By effectively leveraging synthetic data for in vivo and ex vivo analysis, DeepDAM offers a generalizable strategy for overcoming model mismatch and represents a critical step towards data-driven reconstruction of functional neural circuits.
- 22 January 2026
- Published by eLife Sciences Publications, Ltd in eLife preprints
Open AccessDomain-adaptive matching bridges synthetic and in vivo neural dynamics for neural circuit connectivity inference
Accurately inferring neural circuit connectivity from in vivo recordings is essential for understanding the computations that support behavior and cognition. However, current deep learning approaches are limited by incomplete observability and the lack of ground-truth labels in real experiments. Consequently, models are often trained on synthetic data, which leads to the well-known “model mismatch” problem when simulated dynamics diverge from true neural activity. To overcome these challenges, we present Deep Domain-Adaptive Matching (DeepDAM), a training framework that adaptively matches synthetic and in vivo data domains for neural connectivity inference. Specifically, DeepDAM fine-tunes deep neural networks on a combined dataset of synthetic simulations and unlabeled in vivo recordings, aligning the model’s feature representations with real neural dynamics to mitigate model mismatch. We demonstrate this approach in rodent hippocampal CA1 circuits as a proof-of-concept, achieving near-perfect connectivity inference performance (Matthews correlation coefficient ∼0.97–1.0) and substantially surpassing classical methods (∼0.6–0.7). We further demonstrate robustness across multiple recording conditions within this hippocampal dataset. Additionally, to illustrate its broader applicability, we extend the framework to two distinct systems without altering the core methodology: a stomatogastric microcircuit in Cancer borealis (ex vivo) and single-neuron intracellular recordings in mouse, where DeepDAM significantly improves efficiency and accuracy over standard approaches. By effectively leveraging synthetic data for in vivo and ex vivo analysis, DeepDAM offers a generalizable strategy for overcoming model mismatch and represents a critical step towards data-driven reconstruction of functional neural circuits.
- Open Access
- 21 January 2026
- Published by Springer Nature in Research Square
OmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs
Clinical brain MRI analysis faces a fundamental challenge: bridging the gap between oversimplified research developments and the inherent heterogeneity of real-world clinical practice. Quantifying this gap, our analysis of 26 MRI attributes across 22 clinical datasets reveals substantial heterogeneity across institutions and patients. Current AI tools typically require rigid input formats, necessitating extensive data exclusion or preprocessing that severely limits their real-world utility. Here we present OmiMRI, a unified, format-free framework designed to bridge this gap by enabling adaptive processing of arbitrary MRI combinations. Rather than strictly defining a standalone architecture, OmiMRI functions as a universal framework that integrates diverse pretrained 2D/3D convolutional and Transformer-based networks as feature encoders. Through a self-attention mechanism and dynamic weighting to fuse features from variable inputs, OmiMRI decouples clinical performance from rigid input specifications and enables adaptive processing of arbitrary MRI combinations. Across 15 diverse classification, segmentation, and regression tasks, OmiMRI demonstrates robust input-scaling capabilities, yielding significant improvements over traditional fixed-input models. Notably, OmiMRI outperforms advanced medical imaging foundation models (e.g., BrainIAC and BrainSegFounder) in 94.4% of comparisons involving 2–4 input MRIs under consistent experimental conditions. Furthermore, the framework exhibits continuous performance gains through the incremental incorporation of multi-center data. In a clinically demanding, data-limited task distinguishing glioblastoma from metastasis, OmiMRI achieved diagnostic performance matching or exceeding that of senior neuroradiologists (AUROC 0.931 vs. 0.907, P > 0.05; AUPRC 0.973 vs. 0.930, P < 0.05), while providing interpretable attention maps aligned with radiological landmarks. Together, these results establish OmiMRI as a clinically adaptive AI paradigm that transform format-rigid modeling into flexible, expert-level systems capable of embracing the heterogeneity of real-world patient data.
- 21 January 2026
- Published by Springer Nature in Research Square
Open AccessOmiMRI: A Clinical-adaptive AI Framework for Format-Free Interpretation of Heterogeneous Brain MRIs
Clinical brain MRI analysis faces a fundamental challenge: bridging the gap between oversimplified research developments and the inherent heterogeneity of real-world clinical practice. Quantifying this gap, our analysis of 26 MRI attributes across 22 clinical datasets reveals substantial heterogeneity across institutions and patients. Current AI tools typically require rigid input formats, necessitating extensive data exclusion or preprocessing that severely limits their real-world utility. Here we present OmiMRI, a unified, format-free framework designed to bridge this gap by enabling adaptive processing of arbitrary MRI combinations. Rather than strictly defining a standalone architecture, OmiMRI functions as a universal framework that integrates diverse pretrained 2D/3D convolutional and Transformer-based networks as feature encoders. Through a self-attention mechanism and dynamic weighting to fuse features from variable inputs, OmiMRI decouples clinical performance from rigid input specifications and enables adaptive processing of arbitrary MRI combinations. Across 15 diverse classification, segmentation, and regression tasks, OmiMRI demonstrates robust input-scaling capabilities, yielding significant improvements over traditional fixed-input models. Notably, OmiMRI outperforms advanced medical imaging foundation models (e.g., BrainIAC and BrainSegFounder) in 94.4% of comparisons involving 2–4 input MRIs under consistent experimental conditions. Furthermore, the framework exhibits continuous performance gains through the incremental incorporation of multi-center data. In a clinically demanding, data-limited task distinguishing glioblastoma from metastasis, OmiMRI achieved diagnostic performance matching or exceeding that of senior neuroradiologists (AUROC 0.931 vs. 0.907, P > 0.05; AUPRC 0.973 vs. 0.930, P < 0.05), while providing interpretable attention maps aligned with radiological landmarks. Together, these results establish OmiMRI as a clinically adaptive AI paradigm that transform format-rigid modeling into flexible, expert-level systems capable of embracing the heterogeneity of real-world patient data.
- 13 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (5) , 5992-6007
Learn to Enhance Sparse Spike Streams
High-speed vision tasks have long been a challenge in computer vision. Recently, the spike camera has shown great potential in these tasks due to its high temporal resolution. Unlike traditional cameras, it emits asynchronous spike signals to capture visual information. However, under low-light conditions, spike signals become highly sparse, and the sparse spike stream severely hinders the effectiveness of existing spike-based methods in high-speed scenarios. To address this challenge, we introduce SS2DS, the first deep learning framework that enhances sparse spike streams into dense spike streams. SS2DS first estimates the spike firing frequency within sparse streams. Subsequently, the spike firing frequency is enhanced by a neural network. Finally, SS2DS decodes the enhanced spike stream from the enhanced spike firing frequency sequence. SS2DS can adjust the temporal distribution of sparse spike streams and improve the performance degradation of existing methods in low-light and high-speed scenarios. To evaluate sparse spike stream enhancement, we construct both synthetic and real sparse spike stream datasets. By comparing the reconstruction results, enhanced spike streams achieve an average improvement of +0.78 MA, −18.42 BRISQUE, and −1.42 NIQE over sparse spike streams. Moreover, the enhanced spike streams also benefit other spike-based vision tasks, such as 3D reconstruction (+1.325 dB PSNR, +0.005 SSIM, and −0.01 LPIPS) and superresolution (+0.63 MA, −13.67 BRISQUE, and −1.28 NIQE).
- 13 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (5) , 5992-6007
Learn to Enhance Sparse Spike Streams
High-speed vision tasks have long been a challenge in computer vision. Recently, the spike camera has shown great potential in these tasks due to its high temporal resolution. Unlike traditional cameras, it emits asynchronous spike signals to capture visual information. However, under low-light conditions, spike signals become highly sparse, and the sparse spike stream severely hinders the effectiveness of existing spike-based methods in high-speed scenarios. To address this challenge, we introduce SS2DS, the first deep learning framework that enhances sparse spike streams into dense spike streams. SS2DS first estimates the spike firing frequency within sparse streams. Subsequently, the spike firing frequency is enhanced by a neural network. Finally, SS2DS decodes the enhanced spike stream from the enhanced spike firing frequency sequence. SS2DS can adjust the temporal distribution of sparse spike streams and improve the performance degradation of existing methods in low-light and high-speed scenarios. To evaluate sparse spike stream enhancement, we construct both synthetic and real sparse spike stream datasets. By comparing the reconstruction results, enhanced spike streams achieve an average improvement of +0.78 MA, −18.42 BRISQUE, and −1.42 NIQE over sparse spike streams. Moreover, the enhanced spike streams also benefit other spike-based vision tasks, such as 3D reconstruction (+1.325 dB PSNR, +0.005 SSIM, and −0.01 LPIPS) and superresolution (+0.63 MA, −13.67 BRISQUE, and −1.28 NIQE).