Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography

Authors:
  • CTChuanqi Tan
    Chuanqi Tan
  • FSFuchun Sun
    Fuchun Sun
  • WZWenchang Zhang
    Wenchang Zhang
  • JCJianhua Chen
    Jianhua Chen
  • CLChunfang Liu
    Chunfang Liu

Abstract
Electroencephalography (EEG) has become the most significant input signal for brain computer interface (BCI) based systems. However, it is very difficult to obtain satisfactory classification accuracy due to traditional methods can not fully exploit multimodal information. Herein, we propose a novel approach to modeling cognitive events from EEG data by reducing it to a video classification problem, which is designed to preserve the multimodal information of EEG. In addition, optical flow is introduced to represent the variant information of EEG. We train a deep neural network (DNN) with convolutional neural network (CNN) and recurrent neural network (RNN) for the EEG classification task by using EEG video and optical flow. The experiments demonstrate that our approach has many advantages, such as more robustness and more accuracy in EEG classification tasks. According to our approach, we designed a mixed BCI-based rehabilitation support system to help stroke patients perform some basic operations.
All Related Versions