HKHimanshu Kumar

Himanshu Kumar

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Himanshu Kumar has a total of 3 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Cross-Modal Attention Fusion: A Deep Learning and Affective Computing Model for Emotion Recognition

    by Himanshu Kumar, Martin Aruldoss, Martin Wynn

    Artificial emotional intelligence is a sub-domain of human–computer interaction research that aims to develop deep learning models capable of detecting and interpreting human emotional states through various modalities. A major challenge in this domain is identifying meaningful correlations between heterogeneous modalities—for example, between audio and visual data—due to their distinct temporal and spatial properties. Traditional fusion techniques used in multimodal learning to combine data from different sources often fail to adequately capture meaningful and less computational cross-modal interactions, and struggle to adapt to varying modality reliability. Following a review of the relevant literature, this study adopts an experimental research method to develop and evaluate a mathematical cross-modal fusion model, thereby addressing a gap in the extant research literature. The framework uses the Tucker tensor decomposition to analyse the multi-dimensional array of data into a set of matrices to support the integration of temporal features from audio and spatiotemporal features from visual modalities. A cross-attention mechanism is incorporated to enhance cross-modal interaction, enabling each modality to attend to the relevant information from the other. The efficacy of the model is rigorously evaluated on three publicly available datasets and the results conclusively demonstrate that the proposed fusion technique outperforms conventional fusion methods and several more recent approaches. The findings break new ground in this field of study and will be of interest to researchers and developers in artificial emotional intelligence.


  • Open Access

    Gated Cross-Modal Fusion Mechanism for Audio-Video-based Emotion Recognition

    by Himanshu Kumar, Martin Aruldoss

    Due to its potential uses in security, surveillance, mental health monitoring, and human-computer interaction, artificial emotion recognition employing video and audio modalities has attracted a lot of attention. This study focuses on optimal cross-modal fusion techniques to enhance the precision and robustness of multimodal audio-video-based emotion recognition. Specifically, this study introduces a gated cross-modal fusion mechanism in audio-video-based emotion recognition, known as Compact Bilinear Gated Pooling (CBGP). The novelty of this work is that CBGP fusion is being applied to the emotion recognition task for the first time to integrate the extracted features and reduce the dimensionality of the audio and video modalities using 1DCNN and 3DCNN deep neural architectures, respectively. This novel approach was tested and verified on three benchmark datasets: CMU-MOSEI, RAVDESS, and IEMOCAP, each containing multimodal data representing a range of emotions, including happy, sad, fear, anger, neutral, and disgust. Experimental results show that CBGP consistently outperformed state-of-the-art fusion techniques, such as early fusion, late fusion, hybrid fusion, and others. CBGP extracts the relevant features, leading to higher accuracy and F1 scores due to its dynamic gating mechanism that selectively emphasizes relevant feature interactions. This study suggests that the integration of gating mechanisms within fusion processes is vital to improve emotion recognition. Future work will focus on extending these findings to real-time applications, exploring multitask learning frameworks, and enhancing the interpretability of multimodal emotion recognition systems.


  • Cross-Modal Harmony: Low-Rank Fusion for Enhanced Artificial Emotion Recognition

    by Himanshu Kumar, A Martin

    Emotion recognition has become a subject of considerable interest in recent times, owing to its diverse and far-reaching applications in various fields. These applications span from enhancing human-computer interactions to assessing mental health and improving entertainment systems. The proposed study presents a novel approach for emotion recognition by fusing audio and video modalities using low-rank fusion techniques. The proposed methodology leverages the complementary nature of audio and video data in capturing emotional cues. Audio data often encapsulates tone, speech patterns, and vocal nuances, while video data captures facial expressions, body language, and gestures. However, the challenge lies in effectively integrating these two modalities to enhance recognition accuracy. To address the challenge, it employs low-rank fusion, a dimensionality reduction technique that extracts the most informative features from both modalities while minimizing redundancy. Furthermore, it presents the implementation of the chosen low-rank fusion algorithm in a real-world emotion recognition system. The results can contribute to advancing the field of emotion recognition by providing a practical and efficient solution for combining audio and video data to achieve more robust and accurate emotion classification. Keywords: Deep Learning; Emotion Recognition; Human-Computer Interaction; Low-Rank Fusion; Multimodal Fusion.