Haibo Shi

Visual question answering (VQA) has been intensively studied as a multimodal task, requiring efforts to bridge vision and language for correct answer inference. Recent attempts have developed various attention-based modules for solving VQA tasks. However, the performance of model inference is largely bottlenecked by visual semantic comprehension. Most existing detection methods rely on bounding boxes, remaining a serious challenge for VQA models to comprehend and correctly infer the causal nexus of contextual object semantics in images. To this end, we propose a finer model framework without bounding boxes in this work, termed Looking Out of Instance Semantics (LOIS) to address this crucial issue. LOIS can achieve more fine-grained feature descriptions to generate visual facts. Furthermore, to overcome the label ambiguity caused by instance masks, two types of relation attention modules: 1) intra-modality and 2) inter-modality, are devised to infer the correct answers from different visual features. Specifically, we implement a mutual relation attention module to model sophisticated and deeper visual semantic relations between instance objects and background information. In addition, our proposed attention model can further analyze salient image regions by focusing on important word-related questions. Experimental results on four benchmark VQA datasets prove that our proposed method has favorable performance in improving visual reasoning capability. Our code is available on GitHub ( https://github.com/ArcherCYM/LOIS ).

Unsupervised Image-to-Image generation has obtained many studies recently, where generative adversarial networks (GANs) have developed as an effective model. Cycle consistency or dual learning guides the GAN from aligned image pairs to the unpaired training set and can be applied for many applications based on the image-to-image generation. However, for many tasks, error accumulation, which can be produced in the progress of image reconstruction, affects the realism and quality of the generated images. To eliminate error accumulation and guide the adversarial learning, in this paper, we propose dual-cycle consistent GAN (DucGAN), a novel approach for cross-domain image-to-image generation. The key idea is to introduce dual-cycle learning, which restrains error accumulation. In our method, inter-cycle and extra-cycle are based on the cycle consistency, while the output from intercycle is cast as a new augmented input in extra-cycle. On the one hand, reconstruction loss in extra-cycle can constrain and guide the training. On the other hand, dual-cycle learning is self-augmented. Our extensive experiments on image-to-image generation tasks show that DucGAN is effective and superior to several state-of-the-art GANs.

Computer Assisted Diagnosis (CAD) based on brain Magnetic Resonance Imaging (MRI) is a popular research field for the computer science and medical engineering. Traditional machine learning and deep learning methods were employed in the classification of brain MRI images in the previous studies. However, the current algorithms rarely take into consideration the influence of multi-scale brain connectivity disorders on some mental diseases. To improve this defect, a deep learning structure was proposed based on MRI images, which was designed to consider the brain's connections at different sizes and the attention of connections. In this work, a Multiscale View (MV) module was proposed, which was designed to detect multi-scale brain network disorders. On the basis of the MV module, the path attention module was also proposed to simulate the attention selection of the parallel paths in the MV module. Based on the two modules, we proposed a 3D Multiscale View Convolutional Neural Network with Attention (3D MVA-CNN) for classification of MRI images for mental disease. The proposed method outperformed the previous 3D CNN structures in the structural MRI data of ADHD-200 and the functional MRI data of schizophrenia. Finally, we also proposed a preliminary framework for clinical application using 3D CNN, and discussed its limitations on data accessing and reliability. This work promoted the assisted diagnosis of mental diseases based on deep learning and provided a novel 3D CNN method based on MRI data.

Neuronal oscillatory phase and phase synchronization are two main aspects of neuronal oscillation. Neurophysiological and computational studies have demonstrated that oscillatory phase for individual neurons has quantifiable relationships with neuronal excitation and input stimulus. In order to investigate the issue for neuronal groups, we constructed orientation columns by means of a spiking neural network and introduced six network activity states, pre-stimulus and stimulus periods for comparison. We proposed a new method of spike-LFP (Local Field Potential) phase based on vector addition of point spike-LFP phases to represent oscillatory phase. We also proposed a PPCG (Pairwise Phase Consistency for Group) method to quantify phase synchronization for neuronal groups. As illustrated in the simulation, the characteristics of oscillatory phase and phase synchronization for neuronal groups were consistent with the ones for individual neurons. Preferred orientations and stronger external inputs tended to result in smaller and more concentrated oscillatory phases. No matter individual neurons or neuronal groups, the oscillatory phase decreased monotonically as a function of neuronal excitation and input strength. More importantly, neuronal groups had a competitive advantage over individual neurons, because they can achieve reliable relationship quantification of oscillatory phase for all network activity states, even in weak oscillatory or non-oscillatory states.

It has been evidenced that the neural motor control exploits the hierarchical and intermittent representation. In this study we propose a hierarchical deep reinforcement learning method to learn the continuous control policy across multiple levels, by unifying the neuroscience principle of the minimum transition hypothesis. The control policies in the two levels of the hierarchy operate in different time scales. The high-level controller produces the intermittent actions to set a sequence of goals for the low-level controller, which in turn conducts the basic skills with the modulation of goals. The goal planning and the basic motor skills are trained jointly with the proposed algorithm: hierarchical intermittent deep deterministic policy gradient (HI-DDPG). The performance of the method is validated in two continuous control problems. The results show that the method successfully learns to temporally decompose compound tasks into sequences of basic motions with sparse transitions, and outperforms the previous deep reinforcement learning methods that lack a hierarchical continuous representation.

Conventional models of motor control exploit the spatial representation of the controlled system to generate control commands. Typically, the control command is gained with the feedback state of a specific instant in time, which behaves like an optimal regulator or spatial filter to the feedback state. Yet, recent neuroscience studies found that the motor network may constitute an autonomous dynamical system and the temporal patterns of the control command can be contained in the dynamics of the motor network, that is, the dynamical system hypothesis (DSH). Inspired by these findings, here we propose a computational model that incorporates this neural mechanism, in which the control command could be unfolded from a dynamical controller whose initial state is specified with the task parameters. The model is trained in a trial-and-error manner in the framework of deep deterministic policy gradient (DDPG). The experimental results show that the dynamical controller successfully learns the control policy for arm reaching movements, while the analysis of the internal activities of the dynamical controller provides the computational evidence to the DSH of the neural coding in motor cortices.