Saumik Bhattacharya

The rise of social media has amplified activities like uploading and downloading text and images, making it challenging to monitor and analyze the vast amount of data. Automatically studying user behavior from shared images and text is complex, which motivated us to develop a novel method for personality trait identification. Current methods typically depend on a single modality, such as text, images, or video, and fail to leverage the multimodal nature of data available on social media. In this article, we introduce a multimodal approach that integrates features from various image types (e.g., uploaded images, banners, and profile pictures) and textual content (e.g., posted text, text within images, and image descriptions) to classify social media images based on the Big Five Factors (BFF) of Personality Traits. Our model uses gradients in the frequency domain to detect focused image regions through a combination of quad trees and CNNs. Color features are extracted using a blend of a color synergy model and median cut clustering, while textual features are derived from a sentence inference model combining fuzzy vectors and CNNs. These features are fused using a Fully Connected Neural Network (FCNN) for personality trait classification. Experimental results on our dataset and two benchmark datasets demonstrate that our approach outperforms state-of-the-art methods in classification accuracy.

Hyperspectral unmixing (HSU) aims to separate each pixel into its constituent endmembers and estimate their corresponding abundance fractions. This work presents an algorithm-unrolling-based network for the HSU task, named the 3D Convolutional Sparse Coding Network (3D-CSCNet), built upon a 3D CSC model. Unlike existing unrolling-based networks, our 3D-CSCNet is designed within the powerful autoencoder (AE) framework. Specifically, to solve the 3D CSC problem, we propose a 3D CSC block (3D-CSCB) derived through deep algorithm unrolling. Given a hyperspectral image (HSI), 3D-CSCNet employs the 3D-CSCB to estimate the abundance matrix. The use of 3D CSC enables joint learning of spectral and spatial relationships in the 3D HSI data cube. The estimated abundance matrix is then passed to the AE decoder to reconstruct the HSI, and the decoder weights are extracted as the endmember matrix. Additionally, we propose a projected simplex volume maximization (PSVM) algorithm for endmember estimation, and the resulting endmembers are used to initialize the decoder weights of 3D-CSCNet. Extensive experiments on three real datasets and one simulated dataset with three different signal-to-noise ratio (SNR) levels demonstrate that our 3D-CSCNet outperforms state-of-the-art methods.

Non-contact electrocardiogram (ECG) reconstruction from radar signals offers a promising approach for unobtrusive cardiac monitoring. We present LifWavNet, a lifting wavelet network based on a multi-resolution analysis and synthesis (MRAS) model for radar-to-ECG reconstruction. Unlike prior models that use fixed wavelet approaches, LifWavNet employs learnable lifting wavelets with lifting and inverse lifting units to adaptively capture radar signal features and synthesize physiologically meaningful ECG waveforms. To improve reconstruction fidelity, we introduce a multi-resolution short-time Fourier transform (STFT) loss, that enforces consistency with the ground-truth ECG in both temporal and spectral domains. Evaluations on two public datasets demonstrate that LifWavNet outperforms state-of-the-art methods in ECG reconstruction and downstream vital sign estimation (heart rate and heart rate variability). Furthermore, intermediate feature visualization highlights the interpretability of multi-resolution decomposition and synthesis in radar-to-ECG reconstruction. These results establish LifWavNet as a robust framework for radar-based non-contact ECG measurement.

Self-supervised learning has developed rapidly over the last decade and has been applied in many areas of computer vision. Decorrelation-based self-supervised pretraining has shown great promise among non-contrastive algorithms, yielding performance at par with supervised and contrastive self-supervised baselines. In this work, we explore the decorrelation-based paradigm of self-supervised learning and apply the same to learning disentangled stroke features for writer identification. Here, we propose a modified formulation of the decorrelation-based framework named SWIS which was proposed for signature verification by standardizing the features along each dimension on top of the existing framework. We show that the proposed framework outperforms the contemporary self-supervised learning framework on the writer identification benchmark by 0.89%, 0.56%, and 1.23% on word-level images and 1.15%, 0.10%, and 0.39% on page-level images on IAM, CVL ,and Firemaker datasets, respectively. The proposed framework achieves word level accuracy of 87.94%, 84.80%, 93.32%, 74.24% and page level accuracy of 97.09%, 95.58%, 96.87%, 98.40% on AHAWP, IAM, CVL and Firemaker datasets, respectively, outperforming several recent supervised methods as well.

Vision-based Heart Rate (HR) estimation in adverse situations, such as changes in skin tone, arbitrary face movements, and complex backgrounds, etc., is challenging. Unlike state-of-the-art models that use color and spatial-temporal information, the present work exploits a Spatial-Frequency-Temporal Transformer (SFTT) for heart rate estimation. For extracting multi-scale contextual features, we propose an end-to-end transformer that encodes contextual information through a pyramidal structure-based approach. Furthermore, to strengthen the features, the proposed model introduces a new attention approach that performs mutual-sharing operations between spatial-temporal and frequency-temporal domains in an end-to-end fashion. Experimental results on four standard datasets, namely UBFC-rPPG, VIPL-HR, OBF, and MMSE-HR, show that the proposed model is generic and invariant to the aforementioned challenges. Further, a comparative study with the state-of-the-art models demonstrates the effectiveness of the proposed method over the existing methods on all four benchmark datasets. Besides, experiments on cross-dataset validation show that the proposed method is reliable and robust.

Human affordance learning investigates contextually relevant novel pose prediction such that the estimated pose represents a valid human action within the scene. While the task is fundamental to machine perception and automated interactive navigation agents, the exponentially large number of probable pose and action variations make the problem challenging and non-trivial. However, the existing datasets and methods for human affordance prediction in 2D scenes are significantly limited in the literature. In this paper, we propose a novel cross-attention mechanism to encode the scene context for affordance prediction by mutually attending spatial feature maps from two different modalities. The proposed method is disentangled among individual subtasks to efficiently reduce the problem complexity. First, we sample a probable location for a person within the scene using a variational autoencoder (VAE) conditioned on the global scene context encoding. Next, we predict a potential pose template from a set of existing human pose candidates using a classifier on the local context encoding around the predicted location. In the subsequent steps, we use two VAEs to sample the scale and deformation parameters for the predicted pose template by conditioning on the local context and template class. Our experiments show significant improvements over the previous baseline of human affordance injection into complex 2D scenes.

Decentralized federated learning enables learning of data representations from multiple sources without compromising the privacy of the clients. In applications like medical image segmentation, where obtaining a large annotated dataset from a single source is a distressing problem, federated self-supervised learning can provide some solace. In this work, we push the limits further by exploring a federated self-supervised one-shot segmentation task representing a more data-scarce scenario. We adopt a pre-existing self-supervised few-shot segmentation framework CoWPro and adapt it to the federated learning scenario. To the best of our knowledge, this work is the first to attempt a self-supervised few-shot segmentation task in the federated learning domain. Moreover, we consider the clients to be constituted of data from different modalities and imaging techniques like MR or CT, which makes the problem even harder. Additionally, we reinforce and improve the baseline CoWPro method using a fused dice loss which shows considerable improvement in performance over the baseline CoWPro. Finally, we evaluate this novel framework on a completely unseen held-out part of the local client dataset. We observe that the proposed framework can achieve performance at par or better than the FedAvg version of the CoWPro framework on the held-out validation dataset.