Mohammad Abuzar Shaikh
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Mohammad Abuzar Shaikh has a total of 28 co-authors. Here are the top co-authors by number of shared publications:
Jun Chu
Department of Computer Science and Engineering, University at Buffalo
5 shared publicationsMihir Chauhan
Department of Computer Science and Engineering, University at Buffalo
5 shared publicationsSargur N. Srihari
State University of New York at Buffalo
5 shared publications- Mihir Chauhan
Department of Computer Science and Engineering State University of New York at Buffalo
4 shared publications - Sargur N. Srihari
Department of Computer Science and Engineering University at Buffalo The State University of New York Buffalo NY USA
4 shared publications Mingchen Gao
Department of Computer Science and Engineering University at Buffalo The State University of New York Buffalo NY USA
3 shared publications
Latest Publications
- Open Access
LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation
Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space
Open AccessLAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation
Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space
- 21 September 2021
- Published by Springer Nature
- Vol. 12966, 110-119
Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR. Our code is available at https://github.com/mshaikh2/JoImTeR_MLMI_2021. Keywords Self-supervised learning Multi-modality Attention
- 21 September 2021
- Published by Springer Nature
- Vol. 12966, 110-119
Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR. Our code is available at https://github.com/mshaikh2/JoImTeR_MLMI_2021. Keywords Self-supervised learning Multi-modality Attention
- 21 September 2021
- Published by Springer Nature
- p. 13-23
Soft Attention Improves Skin Cancer Classification Performance
- 21 September 2021
- Published by Springer Nature
- p. 13-23
Soft Attention Improves Skin Cancer Classification Performance
- Open Access
- 8 June 2021
- Published by PubPub
Ultra Efficient Transfer Learning with Meta Update for Continuous EEG Classification Across Subjects
The pattern of Electroencephalogram(EEG) signal differs significantly across different subjects, and poses challenge for EEG classifiers in terms of 1) effectively adapting a learned classifier onto a new subject, 2) retaining knowledge of known subjects after the adaptation. We propose an efficient transfer learning method, named Meta UPdate Strategy (MUPS-EEG), for continuous EEG classification across different subjects. The model learns effective representations with meta update which accelerates adaptation on new subject and mitigate forgetting of knowledge on previous subjects at the same time. The proposed mechanism originates from meta learning and works to 1) find feature representation that is broadly suitable for different subjects, 2) maximizes sensitivity of loss function for fast adaptation on new subject. The method can be applied to all deep learning oriented models. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed model, outperforming current state of the arts by a large margin in terms of both adapting on new subject and retain knowledge of learned subjects.
- 8 June 2021
- Published by PubPub
Open AccessUltra Efficient Transfer Learning with Meta Update for Continuous EEG Classification Across Subjects
The pattern of Electroencephalogram(EEG) signal differs significantly across different subjects, and poses challenge for EEG classifiers in terms of 1) effectively adapting a learned classifier onto a new subject, 2) retaining knowledge of known subjects after the adaptation. We propose an efficient transfer learning method, named Meta UPdate Strategy (MUPS-EEG), for continuous EEG classification across different subjects. The model learns effective representations with meta update which accelerates adaptation on new subject and mitigate forgetting of knowledge on previous subjects at the same time. The proposed mechanism originates from meta learning and works to 1) find feature representation that is broadly suitable for different subjects, 2) maximizes sensitivity of loss function for fast adaptation on new subject. The method can be applied to all deep learning oriented models. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed model, outperforming current state of the arts by a large margin in terms of both adapting on new subject and retain knowledge of learned subjects.
- Open Access
Soft-Attention Improves Skin Cancer Classification Performance
In clinical applications, neural networks must focus on and highlight the most important parts of an input image. Soft-Attention mechanism enables a neural network to achieve this goal. This paper investigates the effectiveness of Soft-Attention in deep neural architectures. The central aim of Soft-Attention is to boost the value of important features and suppress the noise-inducing features. We compare the performance of VGG, ResNet, Inception ResNet v2 and DenseNet architectures with and without the Soft-Attention mechanism, while classifying skin lesions. The original network when coupled with Soft-Attention outperforms the baseline[16] by 4.7% while achieving a precision of 93.7% on HAM10000 dataset [25]. Additionally, Soft-Attention coupling improves the sensitivity score by 3.8% compared to baseline[31] and achieves 91.6% on ISIC-2017 dataset [2]. The code is publicly available at github 1 .
Open AccessSoft-Attention Improves Skin Cancer Classification Performance
In clinical applications, neural networks must focus on and highlight the most important parts of an input image. Soft-Attention mechanism enables a neural network to achieve this goal. This paper investigates the effectiveness of Soft-Attention in deep neural architectures. The central aim of Soft-Attention is to boost the value of important features and suppress the noise-inducing features. We compare the performance of VGG, ResNet, Inception ResNet v2 and DenseNet architectures with and without the Soft-Attention mechanism, while classifying skin lesions. The original network when coupled with Soft-Attention outperforms the baseline[16] by 4.7% while achieving a precision of 93.7% on HAM10000 dataset [25]. Additionally, Soft-Attention coupling improves the sensitivity score by 3.8% compared to baseline[31] and achieves 91.6% on ISIC-2017 dataset [2]. The code is publicly available at github 1 .
- Open Access
- 12 May 2021
- Published by Center for Open Science in OSF Preprints
Soft-Attention Improves Skin Cancer Classification Performance
In clinical applications, neural networks must focus on and highlight the most important parts of an input image. Soft-Attention mechanism enables a neural network to achieve this goal. This paper investigates the effectiveness of Soft-Attention in deep neural architectures. The central aim of Soft-Attention is to boost the value of important features and suppress the noise-inducing features. We com-pare the performance of VGG, ResNet, Inception ResNetv2 and DenseNet architectures with and without the Soft-Attention mechanism, while classifying skin lesions. The original network when coupled with Soft-Attention outperforms the baseline[16] by 4.7% while achieving a precision of 93.7% on HAM10000 dataset[25]. Additionally, Soft-Attention coupling improves the sensitivity score by 3.8% compared to baseline[31] and achieves 91.6% on ISIC-2017 dataset[2]. The code is publicly available at github
- 12 May 2021
- Published by Center for Open Science in OSF Preprints
Open AccessSoft-Attention Improves Skin Cancer Classification Performance
In clinical applications, neural networks must focus on and highlight the most important parts of an input image. Soft-Attention mechanism enables a neural network to achieve this goal. This paper investigates the effectiveness of Soft-Attention in deep neural architectures. The central aim of Soft-Attention is to boost the value of important features and suppress the noise-inducing features. We com-pare the performance of VGG, ResNet, Inception ResNetv2 and DenseNet architectures with and without the Soft-Attention mechanism, while classifying skin lesions. The original network when coupled with Soft-Attention outperforms the baseline[16] by 4.7% while achieving a precision of 93.7% on HAM10000 dataset[25]. Additionally, Soft-Attention coupling improves the sensitivity score by 3.8% compared to baseline[31] and achieves 91.6% on ISIC-2017 dataset[2]. The code is publicly available at github
- Open Access
Ultra Efficient Transfer Learning with Meta Update for Cross Subject EEG Classification
The pattern of Electroencephalogram (EEG) signal differs significantly across different subjects, and poses challenge for EEG classifiers in terms of 1) effectively adapting a learned classifier onto a new subject, 2) retaining knowledge of known subjects after the adaptation. We propose an efficient transfer learning method, named Meta UPdate Strategy (MUPS-EEG), for continuous EEG classification across different subjects. The model learns effective representations with meta update which accelerates adaptation on new subject and mitigate forgetting of knowledge on previous subjects at the same time. The proposed mechanism originates from meta learning and works to 1) find feature representation that is broadly suitable for different subjects, 2) maximizes sensitivity of loss function for fast adaptation on new subject. The method can be applied to all deep learning oriented models. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed model, outperforming current state of the arts by a large margin in terms of both adapting on new subject and retain knowledge of learned subjects.
Open AccessUltra Efficient Transfer Learning with Meta Update for Cross Subject EEG Classification
The pattern of Electroencephalogram (EEG) signal differs significantly across different subjects, and poses challenge for EEG classifiers in terms of 1) effectively adapting a learned classifier onto a new subject, 2) retaining knowledge of known subjects after the adaptation. We propose an efficient transfer learning method, named Meta UPdate Strategy (MUPS-EEG), for continuous EEG classification across different subjects. The model learns effective representations with meta update which accelerates adaptation on new subject and mitigate forgetting of knowledge on previous subjects at the same time. The proposed mechanism originates from meta learning and works to 1) find feature representation that is broadly suitable for different subjects, 2) maximizes sensitivity of loss function for fast adaptation on new subject. The method can be applied to all deep learning oriented models. Extensive experiments on two public datasets demonstrate the effectiveness of the proposed model, outperforming current state of the arts by a large margin in terms of both adapting on new subject and retain knowledge of learned subjects.
- Open Access
Self-Supervised Claim Identification for Automated Fact Checking
We propose a novel, attention-based self-supervised approach to identify "claim-worthy" sentences in a fake news article, an important first step in automated fact-checking. We leverage "aboutness" of headline and content using attention mechanism for this task. The identified claims can be used for downstream task of claim verification for which we are releasing a benchmark dataset of manually selected compelling articles with veracity labels and associated evidence. This work goes beyond stylistic analysis to identifying content that influences reader belief. Experiments with three datasets show the strength of our model. Data and code available at https://github.com/architapathak/Self-Supervised-ClaimIdentification
Open AccessSelf-Supervised Claim Identification for Automated Fact Checking
We propose a novel, attention-based self-supervised approach to identify "claim-worthy" sentences in a fake news article, an important first step in automated fact-checking. We leverage "aboutness" of headline and content using attention mechanism for this task. The identified claims can be used for downstream task of claim verification for which we are releasing a benchmark dataset of manually selected compelling articles with veracity labels and associated evidence. This work goes beyond stylistic analysis to identifying content that influences reader belief. Experiments with three datasets show the strength of our model. Data and code available at https://github.com/architapathak/Self-Supervised-ClaimIdentification
- Open Access
- 16 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 224791-224802
Meta Learn on Constrained Transfer Learning for Low Resource Cross Subject EEG Classification
Electroencephalogram (EEG) signal has large variance and its pattern differs significantly across subjects. Cross subject EEG classification is a challenging task due to such pattern variation and the limited target data available, as collecting and annotating EEG data for a new user is costly and involve efforts from human experts. We model the task as a transfer learning problem and propose to tackle it with meta learning on constrained transfer learning (MLCL). MLCL is an end to end trainable learning paradigm that trains on large standard datasets of known subjects and then quickly adapt to a new subject with minimal target data. The transfer process is accelerated by applying model-agnostic meta-learning (MAML) algorithm, performed under a novel constrained setting which keeps enough flexibility to adapt to new subject while significantly reducing number of parameters to transfer. This enables the adaptation done with a small amount of target data. The method can be applied to all deep learning oriented models. We performed extensive experiments across three public datasets. The proposed model outperforms current state of the arts in terms of both accuracy and AUC-ROC score for low target resource configurations. We further conducted interpretation analysis on the model, which reveals detailed information at the resolution of individual channels for the transfer process.
- 16 December 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 8, 224791-224802
Open AccessMeta Learn on Constrained Transfer Learning for Low Resource Cross Subject EEG Classification
Electroencephalogram (EEG) signal has large variance and its pattern differs significantly across subjects. Cross subject EEG classification is a challenging task due to such pattern variation and the limited target data available, as collecting and annotating EEG data for a new user is costly and involve efforts from human experts. We model the task as a transfer learning problem and propose to tackle it with meta learning on constrained transfer learning (MLCL). MLCL is an end to end trainable learning paradigm that trains on large standard datasets of known subjects and then quickly adapt to a new subject with minimal target data. The transfer process is accelerated by applying model-agnostic meta-learning (MAML) algorithm, performed under a novel constrained setting which keeps enough flexibility to adapt to new subject while significantly reducing number of parameters to transfer. This enables the adaptation done with a small amount of target data. The method can be applied to all deep learning oriented models. We performed extensive experiments across three public datasets. The proposed model outperforms current state of the arts in terms of both accuracy and AUC-ROC score for low target resource configurations. We further conducted interpretation analysis on the model, which reveals detailed information at the resolution of individual channels for the transfer process.
- Open Access
Attention based Writer Independent Handwriting Verification
The task of writer verification is to provide a likelihood score for whether the queried and known handwritten image samples belong to the same writer or not. Such a task calls for the neural network to make it's outcome interpretable, i.e. provide a view into the network's decision making process. We implement and integrate cross-attention and soft-attention mechanisms to capture the highly correlated and salient points in feature space of 2D inputs. The attention maps serve as an explanation premise for the network's output likelihood score. The attention mechanism also allows the network to focus more on relevant areas of the input, thus improving the classification performance. Our proposed approach achieves a precision of 86\% for detecting intra-writer cases in CEDAR cursive "AND" dataset. Furthermore, we generate meaningful explanations for the provided decision by extracting attention maps from multiple levels of the network.
Open AccessAttention based Writer Independent Handwriting Verification
The task of writer verification is to provide a likelihood score for whether the queried and known handwritten image samples belong to the same writer or not. Such a task calls for the neural network to make it's outcome interpretable, i.e. provide a view into the network's decision making process. We implement and integrate cross-attention and soft-attention mechanisms to capture the highly correlated and salient points in feature space of 2D inputs. The attention maps serve as an explanation premise for the network's output likelihood score. The attention mechanism also allows the network to focus more on relevant areas of the input, thus improving the classification performance. Our proposed approach achieves a precision of 86\% for detecting intra-writer cases in CEDAR cursive "AND" dataset. Furthermore, we generate meaningful explanations for the provided decision by extracting attention maps from multiple levels of the network.