Haibo Zhang

Patients with neurodegenerative diseases such as amyotrophic lateral sclerosis (ALS) frequently lose the ability to communicate through speech or writing. However, their cognitive and sensory functions are often relatively preserved. In Japan, the traditional method known as kuchimoji (mouth-based character communication) enables character-by-character communication using mouth shapes. This method relies heavily on caregiver skill and is challenging to implement consistently. This study introduces KUCHIMOJI, a Japanese text input system that uses mouth-shape recognition to support independent augmentative and alternative communication (AAC) without caregiver assistance. The system employs a lightweight convolutional neural network (MobileNetV2) to classify six mouth shapes. These shapes correspond to five vowels and a closed-lip state. To accommodate diverse user conditions, a multimodal input framework is designed. It supports three operation modes: facial-image-based signal input, button-based input, and key-based direct input. As an initial feasibility study, experiments with ten healthy participants were conducted to evaluate text entry performance in terms of text entry speed (TES) and miss entry rate (MER). Results indicate that the system achieves average input speeds of 3.86, 5.32, and 11.35 characters per minute (cpm) for the facial-image, button, and key-based modes, respectively. It maintains low error rates (2.96–5.05%). These findings suggest that the system offers a flexible trade-off between speed and accuracy depending on the input modality. The proposed approach provides a practical, low-cost, non-contact communication solution. This underscores its potential forpractical assistive communication applications.

Visual Automatic Speech Recognition (V-ASR) is a challenging task that involves interpreting spoken language solely from visual information, such as lip movements and facial expressions. This task is notably challenging due to the absence of auditory cues and the visual ambiguity of phonemes that exhibit similar visemes-distinct sounds that appear identical in lip motions. Existing methods often aim to predict words or characters directly from visual cues, but they commonly suffer from high error rates due to viseme ambiguity and require large amounts of pre-training data. We propose a novel phoneme-based two-stage framework that fuses visual and landmark motion features, followed by an LLM model for word reconstruction to address these challenges. Stage 1 consists of V-ASR, which outputs the predicted phonemes, thereby reducing training complexity. Meanwhile, the facial landmark features address speaker-specific facial characteristics. Stage 2 comprises an encoder-decoder LLM model, NLLB, that reconstructs the output phonemes back to words. Besides using a large visual dataset for deep learning fine-tuning, our PV-ASR method demonstrates superior performance by achieving 17.4% WER on the LRS2 and 21.0% WER on the LRS3 dataset.

Advanced Persistent Threats (APTs) involve attackers maintaining a long-term presence on victim systems, leading to the stealthy exfiltration of sensitive data during network transfers. Despite existing methods to detect and halt APT data exfiltration, these attacks continue to pose significant threats to sensitive information and result in substantial commercial losses. Current approaches primarily focus on preemptive measures, which are insufficient once early-stage detection fails due to a lack of continuous monitoring. We propose an effective and efficient network monitoring method to address this gap and detect APT exfiltration during data transfer. Our approach assumes the presence of an undetected APT attacker within the victim system. We examine data exfiltration across three exfiltration traffic environments: exfiltration over command control channels, exfiltration over transfer size limitations, and their combinations. We introduce two detection metrics: Package Transfer Rate and Byte Transfer Rate. Utilizing these metrics, we measure network traffic, categorize APT attack environments, and train deep neural network models, named EDXGB, using ensembled decision trees to predict APT exfiltration. Our method is validated on two public datasets and compared against six baseline methods. Additionally, we simulate real-world exfiltration scenarios by creating three exfiltration traffic environments for each dataset. The results demonstrate that our method effectively detects APT exfiltration across various network environments, enhancing data protection and secure transfer. The code is open source and available at https://github.com/cxjuan/EDXGB-for-APT.

Micro-expressions are very brief, involuntary facial expressions that reveal hidden emotions, lasting less than a second, while macro-expressions are more prolonged facial expressions that align with a person’s conscious emotions, typically lasting several seconds. Micro-expressions are difficult to detect in lengthy videos because they have tiny amplitudes, short durations, and frequently coexist alongside macro-expressions. Nevertheless, micro- and macro-expression analysis has sparked interest in researchers. Existing methods use optical flow features to capture the temporal differences. However, these optical flow features are limited to two successive images only. To address this limitation, this paper proposes LGNMNet-RF, which integrates a Lite General Network with MagFace CNN and a Random Forest classifier to predict micro-expression intervals. Our approach leverages Motion History Images (MHI) to capture temporal patterns across multiple frames, offering a more comprehensive representation of facial dynamics than optical flow-based methods, which are restricted to two successive frames. The novelty of our approach lies in the combination of MHI with MagFace CNN, which improves the discriminative power of facial micro-expression detection, and the use of a Random Forest classifier to enhance interval prediction accuracy. The evaluation results show that this method outperforms baseline techniques, achieving micro-expression F1-scores of 0.3019 on CAS(ME)² and 0.3604 on SAMM-LV. The results of our experiment indicate that MHI offers a viable alternative to optical flow-based methods for micro-expression detection.

Adversarial attacks present a significant security risk to image recognition tasks. Defending against these attacks in a real-life setting can be compared to the way antivirus software works, with a key consideration being how well the defense can adapt to new and evolving attacks. Another important factor is the resources involved in terms of time and cost for training defense models and updating the model database. Training many models that are specific to each type of attack can be time-consuming and expensive. Ideally, we should be able to train one single model that can handle a wide range of attacks. It appears that a defense method based on image-to-image translation may be capable of this. The proposed versatile defense approach in this paper only requires training one model to effectively resist various unknown adversarial attacks. The trained model has successfully improved the classification accuracy from nearly zero to an average of 86%, performing better than other defense methods proposed in prior studies. When facing the PGD attack and the MI-FGSM attack, versatile defense model even outperforms the attack-specific models trained based on these two attacks. The robustness check also shows that our versatile defense model performs stably regardless with the attack strength.