Rui Liu
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Rui Liu has a total of 21 co-authors. Here are the top co-authors by number of shared publications:
- Kay Chen Tan
The Hong Kong Polytechnic University
10 shared publications - Yu-An Huang
Northwestern Polytechnical University
9 shared publications - Yao Hu
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, SAR
8 shared publications - Ka-Chun Wong
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China
4 shared publications - Zexuan Zhu
Shenzhen University
4 shared publications - Jibin Wu
The Hong Kong Polytechnic University
3 shared publications
Latest Publications
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48  (10) , 1-18
Enhancing X-ray Image Classification through Heterogeneous Federated Learning with Natural Image-Augmented Models
Deep learning-based computer-aided diagnosis (DL-CAD) models have achieved remarkable success in X-ray image analysis. Yet their effectiveness is often constrained by the tight regulations of governing sensitive X-ray data. Federated Learning (FL) offers a promising paradigm by facilitating the collaborative training of DL-CAD models across healthcare institutions without compromising data privacy. Despite this advancement, the limited local X-ray archives and the issue of heterogeneous model architectures bring distinctive challenges. To address these challenges, this work pioneers the utilization of natural images to develop a natural image-augmented heterogeneous FL framework (NatIMG-FL) for X-ray classification. For augmenting local training, NatIMG-FL leverages natural images as auxiliary supervised data to facilitate the alignment of feature distributions between natural and X-ray images. To tackle the model heterogeneity issue, NatIMG-FL introduces a novel dual weights-based fine-grained knowledge transfer method, enabling adaptive knowledge exchange between local and central models. The NatIMG-FL framework provides insights into exploiting natural images as proxy datasets to enhance knowledge transfer in heterogeneous FL for X-ray classification.
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48  (10) , 1-18
Enhancing X-ray Image Classification through Heterogeneous Federated Learning with Natural Image-Augmented Models
Deep learning-based computer-aided diagnosis (DL-CAD) models have achieved remarkable success in X-ray image analysis. Yet their effectiveness is often constrained by the tight regulations of governing sensitive X-ray data. Federated Learning (FL) offers a promising paradigm by facilitating the collaborative training of DL-CAD models across healthcare institutions without compromising data privacy. Despite this advancement, the limited local X-ray archives and the issue of heterogeneous model architectures bring distinctive challenges. To address these challenges, this work pioneers the utilization of natural images to develop a natural image-augmented heterogeneous FL framework (NatIMG-FL) for X-ray classification. For augmenting local training, NatIMG-FL leverages natural images as auxiliary supervised data to facilitate the alignment of feature distributions between natural and X-ray images. To tackle the model heterogeneity issue, NatIMG-FL introduces a novel dual weights-based fine-grained knowledge transfer method, enabling adaptive knowledge exchange between local and central models. The NatIMG-FL framework provides insights into exploiting natural images as proxy datasets to enhance knowledge transfer in heterogeneous FL for X-ray classification.
- 8 May 2025
- Published by Elsevier in Medical Image Analysis
- Vol. 103, 103581
CausalMixNet: A mixed-attention framework for causal intervention in robust medical image diagnosis
- 8 May 2025
- Published by Elsevier in Medical Image Analysis
- Vol. 103, 103581
CausalMixNet: A mixed-attention framework for causal intervention in robust medical image diagnosis
- 4 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 55  (3) , 1121-1134
Dynamic Graph Representation Learning for Spatio-Temporal Neuroimaging Analysis
Neuroimaging analysis aims to reveal the information-processing mechanisms of the human brain in a noninvasive manner. In the past, graph neural networks (GNNs) have shown promise in capturing the non-Euclidean structure of brain networks. However, existing neuroimaging studies focused primarily on spatial functional connectivity, despite temporal dynamics in complex brain networks. To address this gap, we propose a spatio-temporal interactive graph representation framework (STIGR) for dynamic neuroimaging analysis that encompasses different aspects from classification and regression tasks to interpretation tasks. STIGR leverages a dynamic adaptive-neighbor graph convolution network to capture the interrelationships between spatial and temporal dynamics. To address the limited global scope in graph convolutions, a self-attention module based on Transformers is introduced to extract long-term dependencies. Contrastive learning is used to adaptively contrast similarities between adjacent scanning windows, modeling cross-temporal correlations in dynamic graphs. Extensive experiments on six public neuroimaging datasets demonstrate the competitive performance of STIGR across different platforms, achieving state-of-the-art results in classification and regression tasks. The proposed framework enables the detection of remarkable temporal association patterns between regions of interest based on sequential neuroimaging signals, offering medical professionals a versatile and interpretable tool for exploring task-specific neurological patterns. Our codes and models are available at https://github.com/77YQ77/STIGR/.
- 4 February 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cybernetics
- Vol. 55  (3) , 1121-1134
Dynamic Graph Representation Learning for Spatio-Temporal Neuroimaging Analysis
Neuroimaging analysis aims to reveal the information-processing mechanisms of the human brain in a noninvasive manner. In the past, graph neural networks (GNNs) have shown promise in capturing the non-Euclidean structure of brain networks. However, existing neuroimaging studies focused primarily on spatial functional connectivity, despite temporal dynamics in complex brain networks. To address this gap, we propose a spatio-temporal interactive graph representation framework (STIGR) for dynamic neuroimaging analysis that encompasses different aspects from classification and regression tasks to interpretation tasks. STIGR leverages a dynamic adaptive-neighbor graph convolution network to capture the interrelationships between spatial and temporal dynamics. To address the limited global scope in graph convolutions, a self-attention module based on Transformers is introduced to extract long-term dependencies. Contrastive learning is used to adaptively contrast similarities between adjacent scanning windows, modeling cross-temporal correlations in dynamic graphs. Extensive experiments on six public neuroimaging datasets demonstrate the competitive performance of STIGR across different platforms, achieving state-of-the-art results in classification and regression tasks. The proposed framework enables the detection of remarkable temporal association patterns between regions of interest based on sequential neuroimaging signals, offering medical professionals a versatile and interpretable tool for exploring task-specific neurological patterns. Our codes and models are available at https://github.com/77YQ77/STIGR/.
- 17 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 8  (6) , 4046-4058
Spatio-Temporal Hybrid Attentive Graph Network for Diagnosis of Mental Disorders on fMRI Time-Series Data
Facing the prevalence of mental disorders around the world, the burden of healthcare services becomes increasingly imminent. To lessen patients' suffering, the timely diagnosis and therapy of mental disorders are particularly essential. Functional magnetic resonance imaging (fMRI), as the de facto non-invasive neuroimaging technique, can effectively examine the spatial and temporal patterns of brain activity. Recently, computer-aided diagnosis (CAD) approaches have emerged to assist doctors in interpreting fMRI images. However, existing CAD methods cannot fully exploit the spatio-temporal dependence in fMRI signals, possibly leading to inaccurate diagnosis. In this study, we propose a spatio-temporal hybrid attentive graph network (ST-HAG) for diagnosing autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD) from fMRI data. Specifically, a hybrid graph convolution network is developed to effectively capture complex spatio-temporal dynamics. Meanwhile, a Transformer-based self-attention module helps ST-HAG to extract the full-scale temporal correlation. Finally, we use a gated fusion unit to learn discriminative spatio-temporal graph representations for classification. Cross-validation experiments demonstrate that the proposed ST-HAG achieves state-of-the-art performance with a mean accuracy of 71.9% and 74.8% for ASD and ADHD on ABIDE (1035 subjects) and ADHD-200 (939 subjects) datasets, respectively. Moreover, thanks to the adopted dynamic graph attentive representation, the potent interpretability enables ST-HAG to detect the remarkable temporal association patterns among different brain regions based on dynamic functional connectivity networks.
- 17 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Emerging Topics in Computational Intelligence
- Vol. 8  (6) , 4046-4058
Spatio-Temporal Hybrid Attentive Graph Network for Diagnosis of Mental Disorders on fMRI Time-Series Data
Facing the prevalence of mental disorders around the world, the burden of healthcare services becomes increasingly imminent. To lessen patients' suffering, the timely diagnosis and therapy of mental disorders are particularly essential. Functional magnetic resonance imaging (fMRI), as the de facto non-invasive neuroimaging technique, can effectively examine the spatial and temporal patterns of brain activity. Recently, computer-aided diagnosis (CAD) approaches have emerged to assist doctors in interpreting fMRI images. However, existing CAD methods cannot fully exploit the spatio-temporal dependence in fMRI signals, possibly leading to inaccurate diagnosis. In this study, we propose a spatio-temporal hybrid attentive graph network (ST-HAG) for diagnosing autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD) from fMRI data. Specifically, a hybrid graph convolution network is developed to effectively capture complex spatio-temporal dynamics. Meanwhile, a Transformer-based self-attention module helps ST-HAG to extract the full-scale temporal correlation. Finally, we use a gated fusion unit to learn discriminative spatio-temporal graph representations for classification. Cross-validation experiments demonstrate that the proposed ST-HAG achieves state-of-the-art performance with a mean accuracy of 71.9% and 74.8% for ASD and ADHD on ABIDE (1035 subjects) and ADHD-200 (939 subjects) datasets, respectively. Moreover, thanks to the adopted dynamic graph attentive representation, the potent interpretability enables ST-HAG to detect the remarkable temporal association patterns among different brain regions based on dynamic functional connectivity networks.
- Open Access
- 13 November 2023
- Published by Springer Nature in Research Square
ST-ASDNET: A BLSTM-FCN-Transformer basedASD Classification Model for Time-series fMRI
Autism spectrum disorder (ASD) is a neurodevelopmental disorder that causesrepetitive and stereotyped behavior and social dysfunction. Early diagnosis andintervention can improve the effectiveness of treatment. However, the currentdiagnostic methods of ASD still use the subjective symptom criteria of clinicalobservation, which is time-consuming and labor-intensive. In recent years, theemergence of functional magnetic resonance imaging (fMRI) neuroimaging techniques has facilitated the identification of potential biomarkers for the diagnosisof ASD. In this study, we developed a deep learning framework called Spatio-Temporal Autism Spectrum Disorder Network (ST-ASDNet) to distinguish ASDsfrom normal controls (NCs) as well as to distinguish ASD subtypes based onthe fMRI data. Specifically, two modules: 1) bidirectional long and short-termmemory Transformer (BLSTM-Transformer) and 2) full convolutional networkTransformer (FCN-Transformer) are proposed to obtain the spatial and the temporal characteristics of the fMRI data, respectively. Additionally, we explored theinfluence of different brain atlas for preprocessing the fMRI data to recognize theASDs. The study verifies that a more detailed division of brain regions is helpfulfor the recognition of ASDs. The proposed model is evaluated on the Craddock200 (CC200) atlas and the Automated Anatomical Labeling (AAL) atlas with the robust accuracy of 70.26% and 67.80%, respectively. The effect of the proposed model is competitive in diagnosing ASD compared with other methods,which could provide a useful and effective way to assist the clinical diagnosis.
- 13 November 2023
- Published by Springer Nature in Research Square
Open AccessST-ASDNET: A BLSTM-FCN-Transformer basedASD Classification Model for Time-series fMRI
Autism spectrum disorder (ASD) is a neurodevelopmental disorder that causesrepetitive and stereotyped behavior and social dysfunction. Early diagnosis andintervention can improve the effectiveness of treatment. However, the currentdiagnostic methods of ASD still use the subjective symptom criteria of clinicalobservation, which is time-consuming and labor-intensive. In recent years, theemergence of functional magnetic resonance imaging (fMRI) neuroimaging techniques has facilitated the identification of potential biomarkers for the diagnosisof ASD. In this study, we developed a deep learning framework called Spatio-Temporal Autism Spectrum Disorder Network (ST-ASDNet) to distinguish ASDsfrom normal controls (NCs) as well as to distinguish ASD subtypes based onthe fMRI data. Specifically, two modules: 1) bidirectional long and short-termmemory Transformer (BLSTM-Transformer) and 2) full convolutional networkTransformer (FCN-Transformer) are proposed to obtain the spatial and the temporal characteristics of the fMRI data, respectively. Additionally, we explored theinfluence of different brain atlas for preprocessing the fMRI data to recognize theASDs. The study verifies that a more detailed division of brain regions is helpfulfor the recognition of ASDs. The proposed model is evaluated on the Craddock200 (CC200) atlas and the Automated Anatomical Labeling (AAL) atlas with the robust accuracy of 70.26% and 67.80%, respectively. The effect of the proposed model is competitive in diagnosing ASD compared with other methods,which could provide a useful and effective way to assist the clinical diagnosis.
- 24 July 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP  (01628828) , 1-18
Source Free Semi-Supervised Transfer Learning for Diagnosis of Mental Disorders on fMRI Scans
The high prevalence of mental disorders gradually poses a huge pressure on the public healthcare services. Deep learning-based computer-aided diagnosis (CAD) has emerged to relieve the tension in healthcare institutions by detecting abnormal neuroimaging-derived phenotypes. However, training deep learning models relies on sufficient annotated datasets, which can be costly and laborious. Semi-supervised learning (SSL) and transfer learning (TL) can mitigate this challenge by leveraging unlabeled data within the same institution and advantageous information from source domain, respectively. This work is the first attempt to propose an effective semi-supervised transfer learning (SSTL) framework dubbed S3TL for CAD of mental disorders on fMRI data. Within S3TL, a secure cross-domain feature alignment method is developed to generate target-related source model in SSL. Subsequently, we propose an enhanced dual-stage pseudo-labeling approach to assign pseudo-labels for unlabeled samples in target domain. Finally, an advantageous knowledge transfer method is conducted to improve the generalization capability of the target model. Comprehensive experimental results demonstrate that S3TL achieves competitive accuracies of 69.14%, 69.65%, and 72.62% on ABIDE-I, ABIDE-II, and ADHD-200 datasets, respectively. Furthermore, the simulation experiments also demonstrate the application potential of S3TL through model interpretation analysis and federated learning extension.
- 24 July 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP  (01628828) , 1-18
Source Free Semi-Supervised Transfer Learning for Diagnosis of Mental Disorders on fMRI Scans
The high prevalence of mental disorders gradually poses a huge pressure on the public healthcare services. Deep learning-based computer-aided diagnosis (CAD) has emerged to relieve the tension in healthcare institutions by detecting abnormal neuroimaging-derived phenotypes. However, training deep learning models relies on sufficient annotated datasets, which can be costly and laborious. Semi-supervised learning (SSL) and transfer learning (TL) can mitigate this challenge by leveraging unlabeled data within the same institution and advantageous information from source domain, respectively. This work is the first attempt to propose an effective semi-supervised transfer learning (SSTL) framework dubbed S3TL for CAD of mental disorders on fMRI data. Within S3TL, a secure cross-domain feature alignment method is developed to generate target-related source model in SSL. Subsequently, we propose an enhanced dual-stage pseudo-labeling approach to assign pseudo-labels for unlabeled samples in target domain. Finally, an advantageous knowledge transfer method is conducted to improve the generalization capability of the target model. Comprehensive experimental results demonstrate that S3TL achieves competitive accuracies of 69.14%, 69.65%, and 72.62% on ABIDE-I, ABIDE-II, and ADHD-200 datasets, respectively. Furthermore, the simulation experiments also demonstrate the application potential of S3TL through model interpretation analysis and federated learning extension.
- 17 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35  (8) , 10591-10605
Spatial–Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data
Neuroimaging techniques have been widely adopted to detect the neurological brain structures and functions of the nervous system. As an effective noninvasive neuroimaging technique, functional magnetic resonance imaging (fMRI) has been extensively used in computer-aided diagnosis (CAD) of mental disorders, e.g., autism spectrum disorder (ASD) and attention deficit/hyperactivity disorder (ADHD). In this study, we propose a spatial–temporal co-attention learning (STCAL) model for diagnosing ASD and ADHD from fMRI data. In particular, a guided co-attention (GCA) module is developed to model the intermodal interactions of spatial and temporal signal patterns. A novel sliding cluster attention module is designed to address global feature dependency of self-attention mechanism in fMRI time series. Comprehensive experimental results demonstrate that our STCAL model can achieve competitive accuracies of 73.0 $\pm$ 4.5%, 72.0 $\pm$ 3.8%, and 72.5 $\pm$ 4.2% on the ABIDE I, ABIDE II, and ADHD-200 datasets, respectively. Moreover, the potential for feature pruning based on the co-attention scores is validated by the simulation experiment. The clinical interpretation analysis of STCAL can allow medical professionals to concentrate on the discriminative regions of interest and key time frames from fMRI data.
- 17 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35  (8) , 10591-10605
Spatial–Temporal Co-Attention Learning for Diagnosis of Mental Disorders From Resting-State fMRI Data
Neuroimaging techniques have been widely adopted to detect the neurological brain structures and functions of the nervous system. As an effective noninvasive neuroimaging technique, functional magnetic resonance imaging (fMRI) has been extensively used in computer-aided diagnosis (CAD) of mental disorders, e.g., autism spectrum disorder (ASD) and attention deficit/hyperactivity disorder (ADHD). In this study, we propose a spatial–temporal co-attention learning (STCAL) model for diagnosing ASD and ADHD from fMRI data. In particular, a guided co-attention (GCA) module is developed to model the intermodal interactions of spatial and temporal signal patterns. A novel sliding cluster attention module is designed to address global feature dependency of self-attention mechanism in fMRI time series. Comprehensive experimental results demonstrate that our STCAL model can achieve competitive accuracies of 73.0 $\pm$ 4.5%, 72.0 $\pm$ 3.8%, and 72.5 $\pm$ 4.2% on the ABIDE I, ABIDE II, and ADHD-200 datasets, respectively. Moreover, the potential for feature pruning based on the co-attention scores is validated by the simulation experiment. The clinical interpretation analysis of STCAL can allow medical professionals to concentrate on the discriminative regions of interest and key time frames from fMRI data.
- 2 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35  (6) , 8161-8175
Multitask Learning for Joint Diagnosis of Multiple Mental Disorders in Resting-State fMRI
Facing the increasing worldwide prevalence of mental disorders, the symptom-based diagnostic criteria struggle to address the urgent public health concern due to the global shortfall in well-qualified professionals. Thanks to the recent advances in neuroimaging techniques, functional magnetic resonance imaging (fMRI) has surfaced as a new solution to characterize neuropathological biomarkers for detecting functional connectivity (FC) anomalies in mental disorders. However, the existing computer-aided diagnosis models for fMRI analysis suffer from unstable performance on large datasets. To address this issue, we propose an efficient multitask learning (MTL) framework for joint diagnosis of multiple mental disorders using resting-state fMRI data. A novel multiobjective evolutionary clustering algorithm is presented to group regions of interests (ROIs) into different clusters for FC pattern analysis. On the optimal clustering solution, the multicluster multigate mixture-of-expert model is used for the final classification by capturing the highly consistent feature patterns among related diagnostic tasks. Extensive simulation experiments demonstrate that the performance of the proposed framework is superior to that of the other state-of-the-art methods. Moreover, the potential for practical application of the framework is also validated in terms of limited computational resources, real-time analysis, and insufficient training data. The proposed model can identify the remarkable interpretative biomarkers associated with specific mental disorders for clinical interpretation analysis.
- 2 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35  (6) , 8161-8175
Multitask Learning for Joint Diagnosis of Multiple Mental Disorders in Resting-State fMRI
Facing the increasing worldwide prevalence of mental disorders, the symptom-based diagnostic criteria struggle to address the urgent public health concern due to the global shortfall in well-qualified professionals. Thanks to the recent advances in neuroimaging techniques, functional magnetic resonance imaging (fMRI) has surfaced as a new solution to characterize neuropathological biomarkers for detecting functional connectivity (FC) anomalies in mental disorders. However, the existing computer-aided diagnosis models for fMRI analysis suffer from unstable performance on large datasets. To address this issue, we propose an efficient multitask learning (MTL) framework for joint diagnosis of multiple mental disorders using resting-state fMRI data. A novel multiobjective evolutionary clustering algorithm is presented to group regions of interests (ROIs) into different clusters for FC pattern analysis. On the optimal clustering solution, the multicluster multigate mixture-of-expert model is used for the final classification by capturing the highly consistent feature patterns among related diagnostic tasks. Extensive simulation experiments demonstrate that the performance of the proposed framework is superior to that of the other state-of-the-art methods. Moreover, the potential for practical application of the framework is also validated in terms of limited computational resources, real-time analysis, and insufficient training data. The proposed model can identify the remarkable interpretative biomarkers associated with specific mental disorders for clinical interpretation analysis.
- 17 November 2022
- Published by Elsevier in Computers in Biology and Medicine
- Vol. 151  (Pt B) , 106320
Classifying ASD based on time-series fMRI using spatial–temporal transformer
- 17 November 2022
- Published by Elsevier in Computers in Biology and Medicine
- Vol. 151  (Pt B) , 106320
Classifying ASD based on time-series fMRI using spatial–temporal transformer
- 14 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35  (6) , 7627-7641
Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI
The globally rising prevalence of mental disorders leads to shortfalls in timely diagnosis and therapy to reduce patients’ suffering. Facing such an urgent public health problem, professional efforts based on symptom criteria are seriously overstretched. Recently, the successful applications of computer-aided diagnosis approaches have provided timely opportunities to relieve the tension in healthcare services. Particularly, multimodal representation learning gains increasing attention thanks to the high temporal and spatial resolution information extracted from neuroimaging fusion. In this work, we propose an efficient multimodality fusion framework to identify multiple mental disorders based on the combination of functional and structural magnetic resonance imaging. A multioutput conditional generative adversarial network (GAN) is developed to address the scarcity of multimodal data for augmentation. Based on the augmented training data, the multiheaded gating fusion model is proposed for classification by extracting the complementary features across different modalities. The experiments demonstrate that the proposed model can achieve robust accuracies of 75.1 $\pm$ 1.5%, 72.9 $\pm$ 1.1%, and 87.2 $\pm$ 1.5% for autism spectrum disorder (ASD), attention deficit/hyperactivity disorder, and schizophrenia, respectively. In addition, the interpretability of our model is expected to enable the identification of remarkable neuropathology diagnostic biomarkers, leading to well-informed therapeutic decisions.
- 14 November 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35  (6) , 7627-7641
Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI
The globally rising prevalence of mental disorders leads to shortfalls in timely diagnosis and therapy to reduce patients’ suffering. Facing such an urgent public health problem, professional efforts based on symptom criteria are seriously overstretched. Recently, the successful applications of computer-aided diagnosis approaches have provided timely opportunities to relieve the tension in healthcare services. Particularly, multimodal representation learning gains increasing attention thanks to the high temporal and spatial resolution information extracted from neuroimaging fusion. In this work, we propose an efficient multimodality fusion framework to identify multiple mental disorders based on the combination of functional and structural magnetic resonance imaging. A multioutput conditional generative adversarial network (GAN) is developed to address the scarcity of multimodal data for augmentation. Based on the augmented training data, the multiheaded gating fusion model is proposed for classification by extracting the complementary features across different modalities. The experiments demonstrate that the proposed model can achieve robust accuracies of 75.1 $\pm$ 1.5%, 72.9 $\pm$ 1.1%, and 87.2 $\pm$ 1.5% for autism spectrum disorder (ASD), attention deficit/hyperactivity disorder, and schizophrenia, respectively. In addition, the interpretability of our model is expected to enable the identification of remarkable neuropathology diagnostic biomarkers, leading to well-informed therapeutic decisions.