A multimodal deep architecture for traditional Chinese medicine diagnosis
- 27 April 2020
- journal article
- research article
- Published by Wiley in Concurrency and Computation: Practice and Experience
- Vol. 32 (19)
Abstract
Traditional Chinese medicine (TCM) is found on a long‐term medical practice in China. Rare human brains can fully grasp the deep TCM knowledge derived from a tremendous amount of experience. In this big data era, a big electronic brain might be competent via deep learning techniques. For this prospect, the electronic brain needs to process various heterogeneous data, such as images, texts, audio signals, and other sensory data. It used to be a challenge to analyze the heterogeneous data by the computer‐aided system until the advances of the powerful deep learning tools. We propose a multimodal deep learning framework to mimic a TCM practitioner to diagnose a patient on the basis of multimodal perceptions of see, listen, smell, ask, and touch. The framework learns common representations from various high‐dimensional sensory data, and fuse the information for final classification. We propose to use conceptual alignment deep neural networks to embed prior knowledge and obtain interpretable latent representations. We implement a multimodal deep architecture to process tongue image and description text data for TCM diagnosis. Experiments illustrate that the multimodal deep architecture can extract effective features from heterogeneous data, produce interpretable representations, and finally achieve a higher accuracy than either corresponding unimodal architectures.Keywords
Funding Information
- National Natural Science Foundation of China (61473259, 61632009, 61976089)
- Natural Science Foundation of Guangdong Province (2017A030308006)
This publication has 42 references indexed in Scilit:
- Diseases of the tongueClinics in Dermatology, 2016
- Deep learningNature, 2015
- Advances in Patient Classification for Traditional Chinese Medicine: A Machine Learning PerspectiveEvidence-Based Complementary and Alternative Medicine, 2015
- Integrative Data Analysis of Multi-Platform Cancer Data with a Multimodal Deep Learning ApproachIEEE/ACM Transactions on Computational Biology and Bioinformatics, 2014
- Design of a Breath Analysis System for Diabetes Screening and Blood Glucose Level PredictionIEEE Transactions on Biomedical Engineering, 2014
- Wrist pulse signals analysis based on Deep Convolutional Neural Networks2014
- Noninvasive Diabetes Mellitus Detection Using Facial Block Color With a Sparse Representation ClassifierIEEE Transactions on Biomedical Engineering, 2014
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Multilayer feedforward networks are universal approximatorsNeural Networks, 1989
- Hearing lips and seeing voicesNature, 1976