Sargur N. Srihari
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Sargur N. Srihari has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
- Mohammad Abuzar Shaikh
Department of Computer Science and Engineering University at Buffalo The State University of New York Buffalo NY USA
4 shared publications Lisa Hanson
Bureau of Criminal Apprehension Laboratory, 1430 Maryland Ave. E., St. Paul, MN
2 shared publications- Mihir Chauhan
Department of Computer Science and Engineering State University of New York at Buffalo
2 shared publications Mingchen Gao
Department of Computer Science and Engineering University at Buffalo The State University of New York Buffalo NY USA
2 shared publicationsDana Moukheiber
Department of Computer Science and Engineering University at Buffalo The State University of New York Buffalo NY USA
1 shared publication- Dana Moukheiber
University at Buffalo, The State University of New York
1 shared publication
Latest Publications
- Open Access
LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation
Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space
Open AccessLAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation
Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space
- 21 September 2021
- Published by Springer Nature
- Vol. 12966, 110-119
Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR. Our code is available at https://github.com/mshaikh2/JoImTeR_MLMI_2021. Keywords Self-supervised learning Multi-modality Attention
- 21 September 2021
- Published by Springer Nature
- Vol. 12966, 110-119
Improving Joint Learning of Chest X-Ray and Radiology Report by Word Region Alignment
Self-supervised learning provides an opportunity to explore unlabeled chest X-rays and their associated free-text reports accumulated in clinical routine without manual supervision. This paper proposes a Joint Image Text Representation Learning Network (JoImTeRNet) for pre-training on chest X-ray images and their radiology reports. The model was pre-trained on both the global image-sentence level and the local image region-word level for visual-textual matching. Both are bidirectionally constrained on Cross-Entropy based and ranking-based Triplet Matching Losses. The region-word matching is calculated using the attention mechanism without direct supervision about their mapping. The pre-trained multi-modal representation learning paves the way for downstream tasks concerning image and/or text encoding. We demonstrate the representation learning quality by cross-modality retrievals and multi-label classifications on two datasets: OpenI-IU and MIMIC-CXR. Our code is available at https://github.com/mshaikh2/JoImTeR_MLMI_2021. Keywords Self-supervised learning Multi-modality Attention
- Open Access
Soft-Attention Improves Skin Cancer Classification Performance
In clinical applications, neural networks must focus on and highlight the most important parts of an input image. Soft-Attention mechanism enables a neural network toachieve this goal. This paper investigates the effectiveness of Soft-Attention in deep neural architectures. The central aim of Soft-Attention is to boost the value of important features and suppress the noise-inducing features. We compare the performance of VGG, ResNet, InceptionResNetv2 and DenseNet architectures with and without the Soft-Attention mechanism, while classifying skin lesions. The original network when coupled with Soft-Attention outperforms the baseline[16] by 4.7% while achieving a precision of 93.7% on HAM10000 dataset [25]. Additionally, Soft-Attention coupling improves the sensitivity score by 3.8% compared to baseline[31] and achieves 91.6% on ISIC-2017 dataset [2]. The code is publicly available at github.
Open AccessSoft-Attention Improves Skin Cancer Classification Performance
In clinical applications, neural networks must focus on and highlight the most important parts of an input image. Soft-Attention mechanism enables a neural network toachieve this goal. This paper investigates the effectiveness of Soft-Attention in deep neural architectures. The central aim of Soft-Attention is to boost the value of important features and suppress the noise-inducing features. We compare the performance of VGG, ResNet, InceptionResNetv2 and DenseNet architectures with and without the Soft-Attention mechanism, while classifying skin lesions. The original network when coupled with Soft-Attention outperforms the baseline[16] by 4.7% while achieving a precision of 93.7% on HAM10000 dataset [25]. Additionally, Soft-Attention coupling improves the sensitivity score by 3.8% compared to baseline[31] and achieves 91.6% on ISIC-2017 dataset [2]. The code is publicly available at github.
- Open Access
Attention based Writer Independent Handwriting Verification
The task of writer verification is to provide a likelihood score for whether the queried and known handwritten image samples belong to the same writer or not. Such a task calls for the neural network to make it's outcome interpretable, i.e. provide a view into the network's decision making process. We implement and integrate cross-attention and soft-attention mechanisms to capture the highly correlated and salient points in feature space of 2D inputs. The attention maps serve as an explanation premise for the network's output likelihood score. The attention mechanism also allows the network to focus more on relevant areas of the input, thus improving the classification performance. Our proposed approach achieves a precision of 86\% for detecting intra-writer cases in CEDAR cursive "AND" dataset. Furthermore, we generate meaningful explanations for the provided decision by extracting attention maps from multiple levels of the network.
Open AccessAttention based Writer Independent Handwriting Verification
The task of writer verification is to provide a likelihood score for whether the queried and known handwritten image samples belong to the same writer or not. Such a task calls for the neural network to make it's outcome interpretable, i.e. provide a view into the network's decision making process. We implement and integrate cross-attention and soft-attention mechanisms to capture the highly correlated and salient points in feature space of 2D inputs. The attention maps serve as an explanation premise for the network's output likelihood score. The attention mechanism also allows the network to focus more on relevant areas of the input, thus improving the classification performance. Our proposed approach achieves a precision of 86\% for detecting intra-writer cases in CEDAR cursive "AND" dataset. Furthermore, we generate meaningful explanations for the provided decision by extracting attention maps from multiple levels of the network.
- Open Access
Explanation based Handwriting Verification
Deep learning system have drawback that their output is not accompanied with ex-planation. In a domain such as forensic handwriting verification it is essential to provideexplanation to jurors. The goal of handwriting verification is to find a measure of confi-dence whether the given handwritten samples are written by the same or different writer.We propose a method to generate explanations for the confidence provided by convolu-tional neural network (CNN) which maps the input image to 15 annotations (features)provided by experts. Our system comprises of: (1) Feature learning network (FLN),a differentiable system, (2) Inference module for providing explanations. Furthermore,inference module provides two types of explanations: (a) Based on cosine similaritybetween categorical probabilities of each feature, (b) Based on Log-Likelihood Ratio(LLR) using directed probabilistic graphical model. We perform experiments using acombination of feature learning network (FLN) and each inference module. We evaluateour system using XAI-AND dataset, containing 13700 handwritten samples and 15 cor-responding expert examined features for each sample. The dataset is released for publicuse and the methods can be extended to provide explanations on other verification taskslike face verification and bio-medical comparison. This dataset can serve as the basis and benchmark for future research in explanation based handwriting verification. The code is available on github.
Open AccessExplanation based Handwriting Verification
Deep learning system have drawback that their output is not accompanied with ex-planation. In a domain such as forensic handwriting verification it is essential to provideexplanation to jurors. The goal of handwriting verification is to find a measure of confi-dence whether the given handwritten samples are written by the same or different writer.We propose a method to generate explanations for the confidence provided by convolu-tional neural network (CNN) which maps the input image to 15 annotations (features)provided by experts. Our system comprises of: (1) Feature learning network (FLN),a differentiable system, (2) Inference module for providing explanations. Furthermore,inference module provides two types of explanations: (a) Based on cosine similaritybetween categorical probabilities of each feature, (b) Based on Log-Likelihood Ratio(LLR) using directed probabilistic graphical model. We perform experiments using acombination of feature learning network (FLN) and each inference module. We evaluateour system using XAI-AND dataset, containing 13700 handwritten samples and 15 cor-responding expert examined features for each sample. The dataset is released for publicuse and the methods can be extended to provide explanations on other verification taskslike face verification and bio-medical comparison. This dataset can serve as the basis and benchmark for future research in explanation based handwriting verification. The code is available on github.
- 12 June 2018
- Published by Springer Nature
- p. 3031-3034
Synthetic Data Sets
- 10 August 2017
- Published by Springer Nature
- p. 1-4
Synthetic Data Sets
- 19 July 2016
- Published by Wiley in Journal of Forensic Sciences
- Vol. 61 (5) , 1292-1300
Development of Individuality in Children's Handwriting
Handwriting of children in early grades is studied from the viewpoint of quantitatively measuring the development of handwriting individuality. Handwriting samples of children, in grades 2–4, writing a paragraph of text in both handprinted and cursive, collected over a period of 3 years, were analyzed using two different approaches: (i) characteristics of the word “and” and (ii) entire paragraphs using an automated system. In the first approach, word characteristics were analyzed using statistical measures. In the second approach, pairs of paragraphs were compared. Both types of analysis, single word and complete writing, led to the same conclusions: (i) handwriting of each child remains relatively similar when handwriting has been just learnt and becomes markedly different from grades 3 to 4 and (ii) handwriting of different children becomes progressively more different from grades 2 to 4. The results provide strong support that handwriting becomes more individualistic with child development.
- 19 July 2016
- Published by Wiley in Journal of Forensic Sciences
- Vol. 61 (5) , 1292-1300
Development of Individuality in Children's Handwriting
Handwriting of children in early grades is studied from the viewpoint of quantitatively measuring the development of handwriting individuality. Handwriting samples of children, in grades 2–4, writing a paragraph of text in both handprinted and cursive, collected over a period of 3 years, were analyzed using two different approaches: (i) characteristics of the word “and” and (ii) entire paragraphs using an automated system. In the first approach, word characteristics were analyzed using statistical measures. In the second approach, pairs of paragraphs were compared. Both types of analysis, single word and complete writing, led to the same conclusions: (i) handwriting of each child remains relatively similar when handwriting has been just learnt and becomes markedly different from grades 3 to 4 and (ii) handwriting of different children becomes progressively more different from grades 2 to 4. The results provide strong support that handwriting becomes more individualistic with child development.
- 27 June 2015
- Published by Springer Nature
- p. 20-30
Studies in Individuality: Can Students, Teachers and Schools Be Determined from Children’s Handwriting?
- 27 June 2015
- Published by Springer Nature
- p. 20-30
Studies in Individuality: Can Students, Teachers and Schools Be Determined from Children’s Handwriting?
- 1 January 2014
- Published by Springer Nature
- p. 333-383
Computational Methods for the Analysis of Footwear Impression Evidence
- 1 January 2014
- Published by Springer Nature
- p. 333-383
Computational Methods for the Analysis of Footwear Impression Evidence