Zeshen Tang

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened interlevel communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting interlevel cooperation. Here, we propose a novel goal-conditioned HRL framework named Guided Cooperation via Model-Based Rollout (GCMR; code is available at https://github.com/HaoranWang-TJ/GCMR_ACLG_official), aiming to bridge interlayer information synchronization and cooperation by exploiting forward dynamics. First, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Second, to prevent disruption by the unseen subgoals and states, lower level $Q$ -function gradients are constrained using a gradient penalty with a model-inferred upper bound, leading to a more stable behavioral policy conducive to effective exploration. Third, we propose a one-step rollout-based planning, using higher level critics to guide the lower level policy. Specifically, we estimate the value of future states of the lower level policy using the higher level critic function, thereby transmitting global task information downward to avoid local pitfalls. These three critical components in GCMR are expected to facilitate interlevel cooperation significantly. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of hierarchical reinforcement learning guided by landmarks (HIGL), namely, adjacency constraint and landmark-guided planning (ACLG), yields more stable and robust policy improvement compared with various baselines and significantly outperforms previous state-of-the-art (SOTA) algorithms.

Background: Computer-aided detection of cognitive impairment garnered increasing attention, offering older adults in the community access to more objective, ecologically valid, and convenient cognitive assessments using multimodal sensing technology on digital devices. Methodology: In this study, we aimed to develop an automated method for screening cognitive impairment, building on paper- and electronic TMTs. We proposed a novel deep representation learning approach named Semi-Supervised Vector Quantised-Variational AutoEncoder (S2VQ-VAE). Within S2VQ-VAE, we incorporated intra- and inter-class correlation losses to disentangle class-related factors. These factors were then combined with various real-time obtainable features (including demographic, time-related, pressure-related, and jerk-related features) to create a robust feature engineering block. Finally, we identified the light gradient boosting machine as the optimal classifier. The experiments were conducted on a dataset collected from older adults in the community. Results: The experimental results showed that the proposed multi-type feature fusion method outperformed the conventional method used in paper-based TMTs and the existing VAE-based feature extraction in terms of screening performance. Conclusions: In conclusion, the proposed deep representation learning method significantly enhances the cognitive diagnosis capabilities of behavior-based TMTs and streamlines large-scale community-based cognitive impairment screening while reducing the workload of professional healthcare staff.

Background: Computer-aided detection, used in the screening and diagnosing of cognitive impairment, provides an objective, valid, and convenient assessment. Particularly, digital sensor technology is a promising detection method. Objective: This study aimed to develop and validate a novel Trail Making Test (TMT) using a combination of paper and electronic devices. Methods: This study included community-dwelling older adult individuals (n=297), who were classified into (1) cognitively healthy controls (HC; n=100 participants), (2) participants diagnosed with mild cognitive impairment (MCI; n=98 participants), and (3) participants with Alzheimer disease (AD; n=99 participants). An electromagnetic tablet was used to record each participant’s hand-drawn stroke. A sheet of A4 paper was placed on top of the tablet to maintain the traditional interaction style for participants who were not familiar or comfortable with electronic devices (such as touchscreens). In this way, all participants were instructed to perform the TMT-square and circle. Furthermore, we developed an efficient and interpretable cognitive impairment–screening model to automatically analyze cognitive impairment levels that were dependent on demographic characteristics and time-, pressure-, jerk-, and template-related features. Among these features, novel template-based features were based on a vector quantization algorithm. First, the model identified a candidate trajectory as the standard answer (template) from the HC group. The distance between the recorded trajectories and reference was computed as an important evaluation index. To verify the effectiveness of our method, we compared the performance of a well-trained machine learning model using the extracted evaluation index with conventional demographic characteristics and time-related features. The well-trained model was validated using follow-up data (HC group: n=38; MCI group: n=32; and AD group: n=22). Results: We compared 5 candidate machine learning methods and selected random forest as the ideal model with the best performance (accuracy: 0.726 for HC vs MCI, 0.929 for HC vs AD, and 0.815 for AD vs MCI). Meanwhile, the well-trained classifier achieved better performance than the conventional assessment method, with high stability and accuracy of the follow-up data. Conclusions: The study demonstrated that a model combining both paper and electronic TMTs increases the accuracy of evaluating participants’ cognitive impairment compared to conventional paper-based feature assessment.

BACKGROUND Computer-aided detection, used in the screening and diagnosing of cognitive impairment, provides an objective, valid, and convenient assessment. Particularly, digital sensor technology is a promising detection method. OBJECTIVE This study aimed to develop and validate a novel Trail Making Test (TMT) using a combination of paper and electronic devices. METHODS This study included community-dwelling older adult individuals (n=297), who were classified into (1) cognitively healthy controls (HC; n=100 participants), (2) participants diagnosed with mild cognitive impairment (MCI; n=98 participants), and (3) participants with Alzheimer disease (AD; n=99 participants). An electromagnetic tablet was used to record each participant’s hand-drawn stroke. A sheet of A4 paper was placed on top of the tablet to maintain the traditional interaction style for participants who were not familiar or comfortable with electronic devices (such as touchscreens). In this way, all participants were instructed to perform the TMT-square and circle. Furthermore, we developed an efficient and interpretable cognitive impairment–screening model to automatically analyze cognitive impairment levels that were dependent on demographic characteristics and time-, pressure-, jerk-, and template-related features. Among these features, novel template-based features were based on a vector quantization algorithm. First, the model identified a candidate trajectory as the standard answer (template) from the HC group. The distance between the recorded trajectories and reference was computed as an important evaluation index. To verify the effectiveness of our method, we compared the performance of a well-trained machine learning model using the extracted evaluation index with conventional demographic characteristics and time-related features. The well-trained model was validated using follow-up data (HC group: n=38; MCI group: n=32; and AD group: n=22). RESULTS We compared 5 candidate machine learning methods and selected random forest as the ideal model with the best performance (accuracy: 0.726 for HC vs MCI, 0.929 for HC vs AD, and 0.815 for AD vs MCI). Meanwhile, the well-trained classifier achieved better performance than the conventional assessment method, with high stability and accuracy of the follow-up data. CONCLUSIONS The study demonstrated that a model combining both paper and electronic TMTs increases the accuracy of evaluating participants’ cognitive impairment compared to conventional paper-based feature assessment.

Suspicious transactions are hidden in thousands of massive transaction data, causing incalculable losses and risks, but the detection is very difficult. In terms of how to effectively explore and identify suspicious transactions from massive transaction data accurately and quickly, this paper adopts the method based on the combination of k-means algorithm and random forest algorithm to solve the problem of data imbalance in the identification of suspicious transactions in bank accounts, and proposes an effective suspicious transaction detection model. At the same time, the AUC(Area Under Curve) and Recall indicators such as the unbalanced data classification standard of performance evaluation, and finally to Kaggle data platform for the bank account of suspicious transactions data set, the results show that the proposed detection model of performance evaluation index AUC increased by 5%, F1-measure increases by 1%, show that the method has some reference value to the suspicious transactions recognition, limited information utilization rate is higher, which makes all kinds of Banks prediction speed and accuracy of suspicious transactions events get improved, can to some extent, reduce the operating cost and risk of the banking sector.