Donghyun Lee

Building on the success of transformers, Spiking Neural Networks (SNNs) have increasingly been integrated with transformer architectures, leading to spiking transformers that demonstrate promising performance on event-based vision tasks. However, despite these empirical successes, there remains limited understanding of how spiking transformers fundamentally process event-based data. Current approaches primarily focus on architectural modifications without analyzing the underlying signal processing characteristics. In this work, we analyze spiking transformers through the frequency spectrum domain and discover that they behave as high-pass filters, contrasting with Vision Transformers (ViTs) that act as low-pass filters. This frequency domain analysis reveals why certain designs work well for event-based data, which contains valuable high-frequency information but is also sparse and noisy. Based on this observation, we propose SpikePool, which replaces spike-based self-attention with max pooling attention, a low-pass filtering operation, to create a selective band-pass filtering effect. This design preserves meaningful high-frequency content while capturing critical features and suppressing noise, achieving a better balance for event-based data processing. Our approach demonstrates competitive results on event-based datasets for both classification and object detection tasks while significantly reducing training and inference time by up to 42.5% and 32.8%, respectively.

Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike , a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By partitioning film clips into RGB image Key Frames , which primarily capture spatial information, and event-like Residual Frames , which emphasize temporal dynamics cues, ReSpike leverages ANN for processing spatial features and SNN for modeling temporal features. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g. > 30 % absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff. Code is shared at https://github.com/Intelligent-Computing-Lab-Yale/ReSpike .

Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.

Two-dimensional materials and their heterostructures have thus far been identified as leading candidates for nanoelectronics owing to the near-atom thickness, superior electrostatic control, and adjustable device architecture. These characteristics are indeed advantageous for neuro-inspired computing hardware where precise programming is strongly required. However, its successful demonstration fully utilizing all of the given benefits remains to be further developed. Herein, we present van der Waals (vdW) integrated synaptic transistors with multistacked floating gates, which are reconfigured upon surface oxidation. When compared with a conventional device structure with a single floating gate, our double-floating-gate (DFG) device exhibits better nonvolatile memory performance, including a large memory window (>100 V), high on-off current ratio (∼10⁷), relatively long retention time (>5000 s), and satisfactory cyclic endurance (>500 cycles), all of which can be attributed to its increased charge-storage capacity and spatial redistribution. This facilitates highly effective modulation of trapped charge density with a large dynamic range. Consequently, the DFG transistor exhibits an improved weight update profile in long-term potentiation/depression synaptic behavior for nearly ideal classification accuracies of up to 96.12% (MNIST) and 81.68% (Fashion-MNIST). Our work adds a powerful option to vdW-bonded device structures for highly efficient neuromorphic computing.

To implement artificial neural networks (ANNs) based on memristor devices, it is essential to secure the linearity and symmetry in weight update characteristics of the memristor, and reliability in the cycle-to-cycle and device-to-device variations. This study experimentally demonstrated and compared the filamentary and interface-type resistive switching (RS) behaviors of tantalum oxide (Ta₂O₅ and TaO₂)-based devices grown by atomic layer deposition (ALD) to propose a suitable RS type in terms of reliability and weight update characteristics. Although Ta₂O₅ is a strong candidate for memristor, the filament-type RS behavior of Ta₂O₅ does not fit well with ANNs demanding analog memory characteristics. Therefore, this study newly designed an interface-type TaO₂ memristor and compared it to a filament type of Ta₂O₅ memristor to secure the weight update characteristics and reliability. The TaO₂-based interface-type memristor exhibited gradual RS characteristics and area dependency in both high- and low-resistance states. In addition, compared to the filamentary memristor, the RS behaviors of the TaO₂-based interface-type device exhibited higher suitability for the neuromorphic, symmetric, and linear long-term potentiation (LTP) and long-term depression (LTD). These findings suggest better types of memristors for implementing ionic memristor-based ANNs among the two types of RS mechanisms.

As many deep neural network models become deeper and more complex, processing devices with stronger computing performance and communication capability are required. Following this trend, the dependence on multichip many-core systems that have high parallelism and reasonable transmission costs is on the rise. In this work, in order to improve routing performance of the system, such as routing runtime and power consumption, we propose a reinforcement learning (RL)based core placement optimization approach, considering application constraints, such as deadlock caused by multicast paths. We leverage the capability of deep RL from indirect supervision as a direct nonlinear optimizer, and the parameters of the policy network are updated by proximal policy optimization. We treat the routing topology as a network graph, so we utilize a graph convolutional network to embed the features into the policy network. One step size environment is designed, so all cores are placed simultaneously. To handle large dimensional action space, we use continuous values matching with the number of cores as the output of the policy network and discretize them again for obtaining the new placement. For multichip system mapping, we developed a community detection algorithm. We use several datasets of multilayer perceptron and convolutional neural networks to evaluate our agent. We compare the optimal results obtained by our agent with other baselines under different multicast conditions. Our approach achieves a significant reduction of routing runtime, communication cost, and average traffic load, along with deadlock-free performance for inner chip data transmission. The traffic of interchip routing is also significantly reduced after integrating the community detection algorithm to our agent.