Nguyen Anh Tu

Aerial video captioning (VC) facilitates the automatic interpretation of dynamic scenes in remote sensing (RS), supporting critical applications, such as disaster response, traffic monitoring, and environmental surveillance. However, challenges, such as extreme angles and continuous camera motion, require adaptive modeling of complex temporal relationships. To tackle these challenges, we leverage an image-language model as the vision encoder and introduce a temporal adaptation module that combines convolution with self-attention layers to both capture local semantics across neighboring frames and model global temporal dependencies. This design allows our model to exploit the multimodal knowledge of the vision encoder while effectively reasoning over the spatiotemporal dynamics. In addition, privacy concerns often restrict access to annotated aerial datasets, posing further challenges for model training. To address this, we develop a federated learning (FL) framework that enables collaborative model training across decentralized clients. Within this framework, we establish a unified benchmark for systematic comparison of temporal adapters, text decoders, and FL strategies, hence filling a gap in the existing literature. Extensive experiments validate the robustness of our approach and its potential for advancing aerial VC.

Vision-language models (VLMs), particularly contrastive language-image pre-training (CLIP), have recently demonstrated great success across various vision tasks. However, their potential in aerial video understanding, an increasingly active area of remote sensing (RS), remains underexplored. This is due to challenges posed by aerial data, such as UAV movement, extreme camera angles, and complex spatiotemporal dependencies. To tackle these challenges, we propose an effective method called CLIP-AVC, which adapts CLIP to classify aerial videos into predefined classes. Specifically, we leverage CLIP’s multimodal transferability by utilizing its encoders to extract robust visual and textual features. We then employ a Temporal Transformer to capture the interactions among the visual features. To address the lack of inductive bias in the CLIP’s visual encoder, we integrate the Temporal Transformer’s outputs with 3D features using a Cross-Transformer, thereby allowing the spatiotemporal locality of aerial videos. In addition, existing methods often fail to explore the semantic alignment between classes and video features. To further overcome these limitations, we propose a Context-enriched Transformer that employs self-attention mechanisms to adaptively refine visual and textual representations. Experimental results on two benchmark datasets validate the robustness of CLIP-AVC, demonstrating its potential to significantly advance VLMs for aerial scene understanding.

Few-shot action recognition aims to train a model to classify actions in videos using only a few examples, known as "shots," per action class. This learning approach is particularly useful but challenging due to the limited availability of labeled video data in practice. Although significant progress has been made in developing few-shot learners, existing methods still face several limitations. Firstly, current methods have not sufficiently explored the effectiveness of 3D feature extractors (e.g., 3D CNNs or Video Transformers), thereby failing to exploit spatiotemporal dynamics in videos. Secondly, the need for a large video dataset to train the model in a centralized manner raises privacy concerns and results in high storage costs and communication overheads. Thirdly, the existing solutions based on local deployment lack the capability to benefit global prior knowledge from a wide variety of real-world action samples. To address these limitations, we propose a federated learning (FL) framework named FedFSLAR++ to collaboratively train few-shot learners with 3D feature extractors. Specifically, we perform few-shot action recognition tasks under FL settings, enhancing privacy protection while maintaining efficient communication and storage. Moreover, FL allows us to effectively learn meta-knowledge from a large set of action videos among heterogeneous clients. Within our framework, we establish a unified benchmark to systematically and fairly compare different components, including feature extraction, meta-learning, and FL for model update and aggregation. This type of benchmark is still lacking in the literature. Notably, we thoroughly examine six 3D CNN and Transformer models for extracting spatiotemporal video features needed to adapt to new tasks quickly during the meta-learning process. We further propose a hybrid feature extractor that combines the advantages of 3D CNNs and Transformers to produce strong video representations. Additionally, we explore three meta-learning paradigms and three FL algorithms to investigate their effectiveness and suggest the optimal choices for performance improvement. Results from extensive experiments on four action datasets verify the robustness of the FedFSLAR++ framework. Our comprehensive study provides a solid foundation for future research advancements in action recognition.

The rising advancement in deep learning (DL) techniques has enabled machine learning (ML) models to assist practitioners in performing medical tasks with high accuracy. However, it also poses privacy concerns regarding how such models will proceed with medical data containing protected patient health information. Therefore, some efforts have been made to anonymize medical data to preserve data privacy while keeping the model performance high enough to avoid wrong decisions in the medical field. Nevertheless, the adversary can develop an ML model to re-identify a patient's identity by matching an arbitrary chest X-ray image with a public or leaked image dataset with high accuracy. This paper aims to find a trade-off between our privacy protection method and data utility for medical images. Specifically, we propose a solution to anonymize chest X-ray images by directly adding noise to the images to prevent verification attacks and evaluate how well those images can maintain good performance in the lung disease classification task. Simulation results on real-world datasets show that the proposed solution achieved a good trade-off between privacy protection and data utility.

Nowadays, enhancing the living standard with smart healthcare via Internet of Thing is one of the most critical goals of smart cities, in which Artificial Intelligence plays as the core technology. Many smart services, deployed according to wearable sensor-based physical activity (PA) recognition, have been able to early detect unhealthy daily behaviors and further medical risks. Numerous approaches have studied shallow handcrafted features coupled with traditional machine learning (ML) techniques, which find it difficult to model real-world activities. In this work, by revealing deep features from Deep Convolutional Neural Networks (DCNNs) in the fusion with conventional handcrafted features, we learn an intermediate fusion framework of human activity recognition (HAR). According to transforming raw signal value to pixel intensity value, a segmentation data acquired from a multi-sensor system is encoded to an activity image for deep model learning. Formulated by several novel Residual Triple-Convolutional blocks, the proposed DCNN allows extracting multi-scale spatiotemporal signal-and sensor-level correlations simultaneously from the activity image. In the fusion model, the hybrid feature merged from the handcrafted and deep features is learned by a multiclass Support Vector Machine (SVM) classifier. Based on several experiments of performance evaluation, our fusion approach for activity recognition has achieved the accuracy over 96.0% on three public benchmark datasets, including Daily and Sport Activities, Daily Life Activities, and RealWorld. Furthermore, the method outperforms several state-of-the-art HAR approaches and demonstrates the superiority of the proposed intermediate fusion model in multi-sensor systems.