Juntao Guan

While Vision Transformers (ViTs) demonstrate superior performance in computer vision tasks, their massive computational requirements create a prohibitive barrier to deployment on edge devices and real-time control systems. The quadratic complexity inherent in self-attention mechanisms results in substantial latency, rendering standard ViTs impractical for resource-constrained hardware environments. To address these computational bottlenecks, this paper proposes ATA-ViT, an efficient vision transformers framework that aggressively reduces redundancy across both spatial and weight dimensions. First, we present an Adaptive Token Aggregation (ATA) strategy that utilizes attention-based scoring and Gumbel-Softmax reparameterization to dynamically distill low-contribution tokens into representative aggregated features, thereby preserving the global context often lost in traditional token pruning. Second, we incorporate 2:4 fine-grained structured sparsity into the linear projections to directly leverage specific hardware acceleration on modern GPU architectures. Experimental results on ImageNet-1K demonstrate that our proposed ATA-ViT yields a superior trade-off between accuracy and computational efficiency. Specifically, the proposed method increases the inference throughput of the DeiT-Small model by 66.6% on NVIDIA RTX 3090 with only a marginal 1.2% decline in Top-1 accuracy, significantly enhancing the feasibility of ViT deployment in practical, efficiency-critical scenarios.

Event Cameras offer appealing advantages, including power efficiency and ultra-low latency, driving forward advancements in edge applications. In order to leverage mature frame-based algorithms, most approaches typically compute dense, image-like representations from sparse, asynchronous events. However, they are often unable to capture comprehensive information or are computationally intensive, which hinders the edge deployment of event-based vision. Meanwhile, pillar-based paradigms have been proven to be efficient and well established for dense representations of sparse data. Hence, from a novel pillar-based perspective, we present EventPillars, an efficient, comprehensive framework for dense event representations. To summarize, it (i) incorporates the Temporal Event Range to describe an intact temporal distribution, (ii) Activates the Event Polarities to explicitly record the scene dynamics, (iii) enhances the target awareness by a spatial attention prior from Normalized Event Density, (iv) can be plug-and-played into different downstream tasks. Extensive experiments show that our EventPillars records a new state-of-the-art precision on object recognition and detection datasets with surprisingly 9.2× and 4.5× lower computation and storage consumption. This brings a new insight into dense event representations and is promising to boost the edge deployment of event-based vision.

Current LiDAR-Camera fusion methods for 3D object detection achieve considerable accuracy at the immense cost of computation and storage, posing challenges for the deployment at the edge. To address this issue, we propose a lightweight 3D object detection framework, namely TinyFusion-Det. Specially, we put forward an ingenious Hybrid Scale Pillar Strategy in LiDAR point cloud feature extraction to efficiently improve the detection accuracy of small objects. Meanwhile, a low cost Cross-Modal Heatmap Attention module is presented to suppress background interference in image features for reducing false positives. Moreover, a Cross-Modal Feature Interaction module is designed to enhance the cross-modal information fusion among channels for further promoting the detection precision. Extensive experiments demonstrated that TinyFusionDet achieves competitive accuracy with the lowest memory consumption and inference latency, making it suitable for hardware constrained edge devices. Furthermore, TinyFusionDet is implemented on a customized FPGA-based prototype system, yielding a record high energy efficiency up to 114.97GOPS/W. To the best of our knowledge, this marks the first real-time LiDAR-Camera fusion detection framework for edge applications.

Due to the huge computational load, CNN-based super-resolution solutions are hard to be deployed on resource-constrained edge devices. In this brief, we proposed a Processing-in-Memory based SR accelerator (PIMSR) that leverages direct simple memory access operations to supersede intensive multiply-add in convolution operators, thus fundamentally improving the energy efficiency. To address the storage explosion problem, we presented an index reparameterized strategy for shrinking the memory requirement of LUT-based PIMSR framework by 19.58×. Furthermore, an address remapping design is putted forward to solve the access conflict in overlap configuration, which greatly boosts the efficiency of memory access. The prototype validation on low-end XC7A200T FPGA indicates that our design yields a new record of energy efficiency up to 671.61 Mpixels/J, over 5.36× higher than existing deep learning based image processors, with extremely low hardware costs.