Yie-Tarng Chen
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Yie-Tarng Chen has a total of 17 co-authors. Here are the top co-authors by number of shared publications:
- Wen-Hsien Fang
Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
13 shared publications - Rizard Renanda Adhi Pramono
Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
5 shared publications - Didik Purwanto
Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
3 shared publications Esteban Carlin
Indah Permata NTUST Taiwan NTUST Taiwan NTUST Taiwan Polytechnique Montréal Sari NTUST Taiwan NTUST Taiwan
2 shared publicationsEuhid Aman
Indah Permata NTUST Taiwan NTUST Taiwan NTUST Taiwan Polytechnique Montréal Sari NTUST Taiwan NTUST Taiwan
2 shared publicationsGhaluh Indah Permata Sari
2 shared publications
Latest Publications
- Open Access
- 31 July 2026
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 85 (8) , 668
CLARA: CLIP with adapters for robust video anomaly detection
- 31 July 2026
- Published by Springer Nature in Multimedia Tools and Applications
- Vol. 85 (8) , 668
Open AccessCLARA: CLIP with adapters for robust video anomaly detection
- Open Access
- 29 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 82836-82850
DANCE: A Lightweight Dynamic Adapter Network for Real-Time Traffic Accident Anticipation
Traffic accident anticipation is critical for proactive safety systems, yet remains challenging due to rapidly evolving scenes, transient accident precursors, and strict low-latency inference requirements. In this paper, we propose Dynamic Adapters for Neural CLIP Enhancement (DANCE), a lightweight end-to-end framework for real-time traffic accident anticipation. DANCE adopts a dual-branch architecture based on Contrastive Language–Image Pre-training (CLIP) and integrates a set of dynamic adapters to enable efficient task adaptation. To accommodate diverse driving scenarios, we introduce two input-aware mechanisms within the adapters: Dynamic Gating (DyG) and Dynamic Scaling (DyS). DyG acts as a content-conditioned soft routing mechanism that selectively emphasizes salient tokens while suppressing irrelevant noise, whereas DyS adaptively modulates the strength of residual contributions to balance pre-trained knowledge preservation and task-specific learning. Operating synergistically, these mechanisms enhance temporal sensitivity and semantic stability without compromising computational efficiency. In addition, lightweight adapters in the text encoder improve the alignment between risk-related textual queries and visual tokens, while a temporal refinement head stabilizes predictions over time. Two auxiliary objectives—a vision–text Contrastive Loss (CLoss) and a Gate Sparsity Loss (GSLoss)—are introduced to strengthen cross-modal consistency and improve feature selectivity.We also provide a discussion on framework properties, including the computational complexity, convergence, and stability of the proposed framework. Extensive experiments on standard benchmark datasets demonstrate that DANCE achieves competitive accuracy with improved early anticipation performance using only RGB inputs. Importantly, the proposed method achieves 41.19 FPS on the NVIDIA Jetson Orin Nano, demonstrating a favorable trade-off between accuracy and efficiency among CLIP-based RGB–text approaches for real-time edge deployment.
- 29 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 14, 82836-82850
Open AccessDANCE: A Lightweight Dynamic Adapter Network for Real-Time Traffic Accident Anticipation
Traffic accident anticipation is critical for proactive safety systems, yet remains challenging due to rapidly evolving scenes, transient accident precursors, and strict low-latency inference requirements. In this paper, we propose Dynamic Adapters for Neural CLIP Enhancement (DANCE), a lightweight end-to-end framework for real-time traffic accident anticipation. DANCE adopts a dual-branch architecture based on Contrastive Language–Image Pre-training (CLIP) and integrates a set of dynamic adapters to enable efficient task adaptation. To accommodate diverse driving scenarios, we introduce two input-aware mechanisms within the adapters: Dynamic Gating (DyG) and Dynamic Scaling (DyS). DyG acts as a content-conditioned soft routing mechanism that selectively emphasizes salient tokens while suppressing irrelevant noise, whereas DyS adaptively modulates the strength of residual contributions to balance pre-trained knowledge preservation and task-specific learning. Operating synergistically, these mechanisms enhance temporal sensitivity and semantic stability without compromising computational efficiency. In addition, lightweight adapters in the text encoder improve the alignment between risk-related textual queries and visual tokens, while a temporal refinement head stabilizes predictions over time. Two auxiliary objectives—a vision–text Contrastive Loss (CLoss) and a Gate Sparsity Loss (GSLoss)—are introduced to strengthen cross-modal consistency and improve feature selectivity.We also provide a discussion on framework properties, including the computational complexity, convergence, and stability of the proposed framework. Extensive experiments on standard benchmark datasets demonstrate that DANCE achieves competitive accuracy with improved early anticipation performance using only RGB inputs. Importantly, the proposed method achieves 41.19 FPS on the NVIDIA Jetson Orin Nano, demonstrating a favorable trade-off between accuracy and efficiency among CLIP-based RGB–text approaches for real-time edge deployment.
- Open Access
BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.
Open AccessBitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.
- 19 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (9) , 8989-9002
A Multi-Modal Architecture With Spatio-Temporal-Text Adaptation for Video-Based Traffic Accident Anticipation
Early and precise accident anticipation is critical for preventing road traffic incidents in advanced traffic systems. This paper presents a Multi-modal Architecture with Spatio-Temporal-Text Adaptation (MASTTA), featuring a Visual Encoder and a Text Encoder within a streamlined end-to-end framework for traffic accident anticipation. Both encoders leverage the CLIP model, pre-trained on large-scale text-image pairs, to utilize visual and textual information effectively. MASTTA captures complex traffic patterns and relationships by fine-tuning only the adapters, reducing retraining demands. In the Visual Encoder, spatio-temporal adaptation is achieved through a novel Temporal Adapter, a novel Spatial Adapter, and an MLP Adapter. The Temporal Adapter enhances temporal consistency in accident-prone areas, while the Spatial Adapter captures spatio-temporal interactions among visual cues. The Text Encoder, equipped with a Text Adapter and an MLP Adapter, aligns latent textual and visual features in a joint embedding space, refining semantic representation. This synergy of text and visual adapters enables MASTTA to model complex spatial interactions across long-range temporal context, improving accident anticipation. We validate MASTTA on DAD and CCD datasets, demonstrating significant improvements in both the earliness and correctness compared to state-of-the-art methods.
- 19 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 35 (9) , 8989-9002
A Multi-Modal Architecture With Spatio-Temporal-Text Adaptation for Video-Based Traffic Accident Anticipation
Early and precise accident anticipation is critical for preventing road traffic incidents in advanced traffic systems. This paper presents a Multi-modal Architecture with Spatio-Temporal-Text Adaptation (MASTTA), featuring a Visual Encoder and a Text Encoder within a streamlined end-to-end framework for traffic accident anticipation. Both encoders leverage the CLIP model, pre-trained on large-scale text-image pairs, to utilize visual and textual information effectively. MASTTA captures complex traffic patterns and relationships by fine-tuning only the adapters, reducing retraining demands. In the Visual Encoder, spatio-temporal adaptation is achieved through a novel Temporal Adapter, a novel Spatial Adapter, and an MLP Adapter. The Temporal Adapter enhances temporal consistency in accident-prone areas, while the Spatial Adapter captures spatio-temporal interactions among visual cues. The Text Encoder, equipped with a Text Adapter and an MLP Adapter, aligns latent textual and visual features in a joint embedding space, refining semantic representation. This synergy of text and visual adapters enables MASTTA to model complex spatial interactions across long-range temporal context, improving accident anticipation. We validate MASTTA on DAD and CCD datasets, demonstrating significant improvements in both the earliness and correctness compared to state-of-the-art methods.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 147-154
BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Euhid Aman, Esteban Carlin, Hsing-Kuo Kenneth Pao, Giovanni Beltrame, Ghaluh Indah Permata Sari, Yie-Tarng Chen. Proceedings of the First BabyLM Workshop. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 147-154
Open AccessBitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
Euhid Aman, Esteban Carlin, Hsing-Kuo Kenneth Pao, Giovanni Beltrame, Ghaluh Indah Permata Sari, Yie-Tarng Chen. Proceedings of the First BabyLM Workshop. 2025.
- 23 September 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 8184-8199
Relational Reasoning for Group Activity Recognition via Self-Attention Augmented Conditional Random Field
This paper presents a new relational network for group activity recognition. The essence of the network is to integrate conditional random fields (CRFs) with self-attention to infer the temporal dependencies and spatial relationships of the actors. This combination can take advantage of the capability of CRFs in modelling the actors’ features that depend on each other and the capability of self-attention in learning the temporal evolution and spatial relational contexts of every actor in videos. Additionally, there are two distinct facets of our CRF and self-attention. First, the pairwise energy of the new CRF relies on both of the temporal self-attention and spatial self-attention, which apply the self-attention mechanism to the features in time and space, respectively. Second, to address both local and non-local relationships in group activities, the spatial self-attention takes account of a collection of cliques with different scales of spatial locality. The associated mean-field inference thereafter can thus be reformulated as a self-attention network to generate the relational contexts of the actors and their individual action labels. Lastly, a bidirectional universal transformer encoder (UTE) is utilized to aggregate the forward and backward temporal context information, scene information and relational contexts for group activity recognition. A new loss function is also employed, consisting of not only the cost for the classification of individual actions and group activities, but also a contrastive loss to address the miscellaneous relational contexts between actors. Simulations show that the new approach can surpass previous works on four commonly used datasets.
- 23 September 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 8184-8199
Relational Reasoning for Group Activity Recognition via Self-Attention Augmented Conditional Random Field
This paper presents a new relational network for group activity recognition. The essence of the network is to integrate conditional random fields (CRFs) with self-attention to infer the temporal dependencies and spatial relationships of the actors. This combination can take advantage of the capability of CRFs in modelling the actors’ features that depend on each other and the capability of self-attention in learning the temporal evolution and spatial relational contexts of every actor in videos. Additionally, there are two distinct facets of our CRF and self-attention. First, the pairwise energy of the new CRF relies on both of the temporal self-attention and spatial self-attention, which apply the self-attention mechanism to the features in time and space, respectively. Second, to address both local and non-local relationships in group activities, the spatial self-attention takes account of a collection of cliques with different scales of spatial locality. The associated mean-field inference thereafter can thus be reformulated as a self-attention network to generate the relational contexts of the actors and their individual action labels. Lastly, a bidirectional universal transformer encoder (UTE) is utilized to aggregate the forward and backward temporal context information, scene information and relational contexts for group activity recognition. A new loss function is also employed, consisting of not only the cost for the classification of individual actions and group activities, but also a contrastive loss to address the miscellaneous relational contexts between actors. Simulations show that the new approach can surpass previous works on four commonly used datasets.
- 5 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 625-639
Spatial-Temporal Action Localization With Hierarchical Self-Attention
This paper proposes a novel architecture for spatial-temporal action localization in videos. The new architecture first employs a two-stream 3D convolutional neural network (3D-CNN) to provide initial action detection. Next, a new Hierarchical Self-Attention Network (HiSAN), the core of this architecture, is developed to learn the spatial-temporal relationships of key actors. Spatial Gaussian priors (SGP) are also imbued to the bidirectional self-attention to enhance HiSAN in modelling the relationships of neighboring actors. Such a combination of 3D-CNN and SGP augmented HiSAN allows us to effectively extract both of the spatial context information and the long-term temporal dependency to improve action localization accuracy. Afterwards, a new fusion strategy is employed, which first re-scores the bounding boxes to settle the inconsistent detection scores caused by background clutter or occlusion, and then aggregates the motion and appearance information from the two-stream network with the motion saliency to alleviate the impact of camera movement. Finally, a tube association network based on the self-similarity of the actors’ appearance and spatial information across frames is addressed to efficaciously construct the action tubes. Simulations on four widespread datasets reveal the efficacy of the new approach.
- 5 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 24 (15209210) , 625-639
Spatial-Temporal Action Localization With Hierarchical Self-Attention
This paper proposes a novel architecture for spatial-temporal action localization in videos. The new architecture first employs a two-stream 3D convolutional neural network (3D-CNN) to provide initial action detection. Next, a new Hierarchical Self-Attention Network (HiSAN), the core of this architecture, is developed to learn the spatial-temporal relationships of key actors. Spatial Gaussian priors (SGP) are also imbued to the bidirectional self-attention to enhance HiSAN in modelling the relationships of neighboring actors. Such a combination of 3D-CNN and SGP augmented HiSAN allows us to effectively extract both of the spatial context information and the long-term temporal dependency to improve action localization accuracy. Afterwards, a new fusion strategy is employed, which first re-scores the bounding boxes to settle the inconsistent detection scores caused by background clutter or occlusion, and then aggregates the motion and appearance information from the two-stream network with the motion saliency to alleviate the impact of camera movement. Finally, a tube association network based on the self-similarity of the actors’ appearance and spatial information across frames is addressed to efficaciously construct the action tubes. Simulations on four widespread datasets reveal the efficacy of the new approach.
- 29 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 27 (10709908) , 2188
Corrections to “Three-Stream Network With Bidirectional Self-Attention for Action Recognition in Extreme Low Resolution Videos”
Presents corrections to funding agency information for the above named paper.
- 29 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 27 (10709908) , 2188
Corrections to “Three-Stream Network With Bidirectional Self-Attention for Action Recognition in Extreme Low Resolution Videos”
Presents corrections to funding agency information for the above named paper.
- 3 November 2020
- Published by Springer Nature
- p. 71-90
Empowering Relational Network by Self-attention Augmented Conditional Random Fields for Group Activity Recognition
- 3 November 2020
- Published by Springer Nature
- p. 71-90
Empowering Relational Network by Self-attention Augmented Conditional Random Fields for Group Activity Recognition
- 19 June 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 26 (8) , 1187-1191
Three-Stream Network With Bidirectional Self-Attention for Action Recognition in Extreme Low Resolution Videos
This letter presents a novel three-stream network for action recognition in extreme low resolution (LR) videos. In contrast to the existing networks, the new network uses the trajectory-spatial network, which is robust against visual distortion, instead of the pose information to complement the two-stream network. Also, the new three-stream network is combined with the inflated 3D ConvNet (I3D) model pre-trained on kinetics to produce more discriminative spatio-temporal features in blurred LR videos. Moreover, a bidirectional self-attention network is aggregated with the three-stream network to further manifest various temporal dependence among the spatio-temporal features. A new fusion strategy is devised as well to integrate the information from the three different modalities. Simulations show that the new architecture outperforms the main state-of-the-art extreme LR action recognition methods on the HMDB-51 and IXMAS datasets.
- 19 June 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 26 (8) , 1187-1191
Three-Stream Network With Bidirectional Self-Attention for Action Recognition in Extreme Low Resolution Videos
This letter presents a novel three-stream network for action recognition in extreme low resolution (LR) videos. In contrast to the existing networks, the new network uses the trajectory-spatial network, which is robust against visual distortion, instead of the pose information to complement the two-stream network. Also, the new three-stream network is combined with the inflated 3D ConvNet (I3D) model pre-trained on kinetics to produce more discriminative spatio-temporal features in blurred LR videos. Moreover, a bidirectional self-attention network is aggregated with the three-stream network to further manifest various temporal dependence among the spatio-temporal features. A new fusion strategy is devised as well to integrate the information from the three different modalities. Simulations show that the new architecture outperforms the main state-of-the-art extreme LR action recognition methods on the HMDB-51 and IXMAS datasets.