Jainendra Shukla

Live commenting on video streams has surged in popularity on platforms like Twitch, enhancing viewer engagement through dynamic interactions. However, automatically generating contextually appropriate comments remains a challenging and exciting task. Video streams can contain a vast amount of data and extraneous content. Existing approaches tend to overlook an important aspect of prioritizing video frames that are most relevant to ongoing viewer interactions. This prioritization is crucial for producing contextually appropriate comments that align with viewer interests. To address this gap, we introduce a novel Semantic Frame Aggregation-based Transformer (SFAT) model for live video comment generation. This method not only leverages CLIP's visual-text multimodal knowledge to generate comments but also assigns weights to video frames based on their semantic relevance to ongoing viewer conversation. It employs an efficient weighted sum of frames technique to emphasize informative frames while focusing less on irrelevant ones. Finally, our comment decoder with cross-attention mechanism to attend to each modality ensures that the generated comment reflects contextual cues from both chats and video. Furthermore, to address the limitations of existing datasets, which predominantly focus on Chinese-language content with limited video categories, we have constructed a large-scale, diverse, multimodal English video comments dataset. Extracted from Twitch, this dataset covers 11 video categories, totaling 438 hours and 3.2 million comments. We demonstrate the effectiveness of our SFAT model by comparing it to existing methods for generating comments from live video and ongoing dialogue contexts.

Traditional mental health support systems often generate responses based solely on the user's current emotion and situations, resulting in superficial interventions that fail to address deeper emotional needs. This study introduces a novel framework by integrating spiritual wisdom from the Bhagavad Gita with advanced large language model GPT-4o to enhance emotional well-being. We present the GITes (Gita Integrated Therapy for Emotional Support) dataset, which enhances the existing ExTES mental health dataset by including 10,729 spiritually guided responses generated by GPT-4o and evaluated by domain experts. We benchmark GITes against 12 state-of-the-art LLMs, including both mental health specific and general purpose models. To evaluate spiritual relevance in generated responses beyond what conventional n-gram based metrics capture, we propose a novel Spiritual Insight metric and automate assessment via an LLM as jury framework using chain-of-thought prompting. Integrating spiritual guidance into AI driven support enhances both NLP and spiritual metrics for the best performing LLM Phi3-Mini 3.2B Instruct, achieving improvements of 122.71% in ROUGE, 126.53% in METEOR, 8.15% in BERT score, 15.92% in Spiritual Insight, 18.61% in Sufficiency and 13.22% in Relevance compared to its zero-shot counterpart. While these results reflect substantial improvements across automated empathy and spirituality metrics, further validation in real world patient populations remains a necessary step. Our findings indicate a strong potential for AI systems enriched with spiritual guidance to enhance user satisfaction and perceived support outcomes. The code and dataset will be publicly available to advance further research in this emerging area.

BACKGROUND Attention-deficit/hyperactivity disorder (ADHD) is a prevalent neurodevelopmental disorder characterized by persistent patterns of inattention and hyperactivity. Current diagnostic methods rely heavily on subjective measures, such as clinical interviews and behavior rating scales, which are prone to bias and variability. Objective biomarkers, essential for reliable and standardized diagnosis, remain elusive. Pupillometry, which measures dynamic pupil responses associated with cognitive and attentional processes, offers a promising avenue for objective ADHD diagnostics. However, existing studies often overlook clinically relevant features and fail to prioritize model interpretability, hindering their potential for clinical implementation. OBJECTIVE This study aims to develop interpretable machine learning models utilizing temporal pupil dynamics to classify ADHD and control groups, aiming for improved diagnostic accuracy and model explainability. METHODS This study utilized already published pupillometry data from 49 participants, including 21 controls and 28 ADHD-diagnosed children, with 17 assessed both on-medication and off-medication. Data were collected during a visuospatial working memory task designed to evaluate attentional processes. The data were preprocessed to remove noise and standardized for analysis. Pupil and behavioral features were first identified based on literature review of pupillometry studies conducted with the ADHD population. The final set of features was determined through statistical analyses using mixed and block-wise ANOVA to assess significance. Binary classification models were developed to differentiate ADHD from control participants. The models were evaluated progressively, starting with features derived only from pupil dynamics, then incorporating task performance metrics, and finally including reaction time metrics. Performance was assessed using accuracy and the area under the receiver operating characteristic curve. RESULTS The study ensured interpretability through the selection of statistically significant features, supported by literature review, that contribute meaningfully to the classification task. Key features included median pupil size (blocks 1 and 3), pupil dilation and contraction rates (blocks 4 and 8), reaction time, and task performance metrics, with each exhibiting distinct patterns of significance. Visualizations such as heatmaps and feature importance charts highlighted their relevance, providing transparency in the models' decision-making processes. The classification models demonstrated robust performance using these interpretable features. Models trained exclusively on pupil features achieved a best accuracy of 86.7% with an AUROC score of 0.884. Incorporating task performance metrics improved performance, achieving a best accuracy of 88.9% with an AUROC score of 0.931. The final integration of reaction time metrics resulted in the highest performance, achieving a best accuracy of 90%, an AUROC score of 0.93, sensitivity of 100%, and specificity of 80.8%. CONCLUSIONS This study highlights the potential of leveraging temporal pupil dynamics as an objective biomarker for ADHD. By focusing on interpretable and clinically relevant features, the proposed models offer a practical and trustworthy approach to ADHD diagnosis, advancing the development of diagnostic tools for clinical use.

The identification of Autism Spectrum Disorder (ASD) faces challenges due to the lack of reliable biomarkers and the subjectivity in diagnostic procedures, necessitating improved tools for objectivity and efficiency. Being a key characteristic of autism, language impairments are regarded as potential markers for identifying ASD. However, current research predominantly focuses on analyzing language characteristics in English, overlooking linguistic and contextual specificities in other resource-constrained languages. Motivated by these, we developed an Artificial Intelligence (AI)-based system to detect ASD, utilizing a range of acoustic and linguistic features extracted from dyadic conversations between a child and their communication partner. Validating our model on 76 English-speaking children (35 ASD and 41 typically developing) and 33 Hindi-speaking children (15 ASD and 18 typically developing), our extensive analysis of a diverse and comprehensive set of acoustic and linguistic speech attributes, including lexical, syntactic, semantic, and pragmatic elements revealed reliable speech attributes as predictors of ASD. This comprehensive analysis achieved a remarkable macro F1- score of approximately ∼91.30%. We further addressed the influence of linguistic diversity on speech-based ASD assessment by examining speech behaviours in both English and the low-resource language, Hindi. Specific features like adverbs and distinct roots contributed significantly to ASD classification in English, while the proportion of unintelligible utterances and adposition use held greater importance in Hindi. This study underscores the reliability of speech-based biomarkers in ASD assessment, emphasizing their effectiveness across diverse linguistic backgrounds and highlighting the need for language-specific research in this domain.

Automatic personality assessment (APA) has immense potential to improve decision-making and human-machine interaction. Numerous techniques for APA have been proposed in existing literature, with prior psychological studies demonstrating convergent validity between personality and trait emotional intelligence (EI). However, to the best of our knowledge, none in APA literature has leveraged this relationship. Further, the primary language for most APA studies is English. To this end, we propose VyaktitvaNirdharan , a multi-modal-multitask learning (MM-MTL) system for predicting personality and EI traits using dyadic conversations in a non-intrusive and near real-time fashion. While most of the previous research has focused on APA for the English language, VyaktitvaNirdharan is also amongst the first to use peer-to-peer conversations in Hindi, a low-resource language. We also show the convergent validity between the predictions of the developed system, confirming that the personality and EI traits estimated by our system preserve the properties of the original traits. Our system can identify the Big-Five and EI traits with average F1-scores of 0 . 88, and 0 . 82, respectively. The validity of our findings is further supported by a user study involving 7 stakeholders, including recruiters and clinical psychologists, conducted through a prototype. The proposed system demonstrates promising results and has the potential to aid in improving decision-making in recruitment, mental health wellness and enhance human-machine interaction.