Garima Sharma

Understanding social context and affect plays a key role in our day-to-day interaction and enables us to better understand the situation, our own self, and others. Classifying a social event and associated perceived affect in a visual scene involving humans is a step towards providing better scene understanding capability to socially interactive artificial intelligence agents. In this work, we perform Multi-Task Learning (MTL) to jointly learn and predict both social event and affect in group videos and explore whether perceived group affect has an influence on social event classification. Transformer based models in computer vision have gained a lot of attention in video classification tasks. We propose spatio-temporal transformer based models for our tasks and evaluate their performance using Video Group AFfect (VGAF) dataset, which consists of unconstrained YouTube video clips of social events and incorporates perceived affect information. Our contributions are as follows. We introduce ten social event category labels for the clips in the VGAF dataset from existing keywords. We propose a spatio-temporal transformer model and its variations for joint learning and prediction, and show improvements to social event prediction by utilizing affect information. Our proposed models improve the state-of-art for group affect prediction on VGAF dataset. Code is available at https://github.com/aarti9/VideoSocialContext

Labelling of human behavior analysis data is a complex and time consuming task. In this paper, a fully automatic technique for labelling an image based gaze behavior dataset for driver gaze zone estimation is proposed. Domain knowledge is added to the data recording paradigm and later labels are generated in an automatic manner using Speech To Text conversion (STT). In order to remove the noise in the STT process due to different illumination and ethnicity of subjects in our data, the speech frequency and energy are analysed. The resultant Driver Gaze in the Wild (DGW) dataset contains 586 recordings, captured during different times of the day including evenings. The large scale dataset contains 338 subjects with an age range of 18-63 years. As the data is recorded in different lighting conditions, an illumination robust layer is proposed in the Convolutional Neural Network (CNN). The extensive experiments show the variance in the dataset resembling real-world conditions and the effectiveness of the proposed CNN pipeline. The proposed network is also fine-tuned for the eye gaze prediction task, which shows the discriminativeness of the representation learnt by our network on the proposed DGW dataset. Project Page: https://sites.google.com/view/drivergazeprediction/home

Affective computing has progressed well due to methods, which can identify a person's posed and spontaneous perceived emotions with high accuracy. This paper focuses on group-level affect analysis on videos, which is one of the first few multimodal group-level affect analysis studies. There are many challenges on video-based group-level affect analysis as most of the work is focused on either a single persons affect recognition or image-based group affect analysis. To address this, first, we present an audio-visual perceived group affect dataset - Video level Group AFfect (VGAF). VGAF is a large-scale dataset consisting of 4,183 group videos. \textcolor{red}{The videos are collected from YouTube with large variations in the keywords for collecting data across different genders, group sizes, group settings, illuminations and poses}. The variety within the dataset will help the study of group emotion perception of emotions in a real environment. The data is manually annotated for three group emotion classes - positive, neutral, and negative. Further, a fusion based audio-visual method is proposed to set a benchmark performance on the proposed dataset. The experimental results show the effectiveness of holistic, facial and speech features for group-level affect analysis. \textcolor{red}{Baseline code, dataset, and pretrained models are available at [LINK]

Background: Neurocognitive disorders are often accompanied by behavioral symptoms such as anxiety, depression, and/or apathy. These symptoms can occur very early in the disease progression and are often difficult to detect and quantify in nonspecialized clinical settings. Objective: We focus in this study on apathy, one of the most common and debilitating neuropsychiatric symptoms in neurocognitive disorders. Specifically, we investigated whether facial expressivity extracted through computer vision software correlates with the severity of apathy symptoms in elderly subjects with neurocognitive disorders. Methods: A total of 63 subjects (38 females and 25 males) with neurocognitive disorder participated in the study. Apathy was assessed using the Apathy Inventory (AI), a scale comprising 3 domains of apathy: loss of interest, loss of initiation, and emotional blunting. The higher the scale score, the more severe the apathy symptoms. Participants were asked to recall a positive and a negative event of their life, while their voice and face were recorded using a tablet device. Action units (AUs), which are basic facial movements, were extracted using OpenFace 2.0. A total of 17 AUs (intensity and presence) for each frame of the video were extracted in both positive and negative storytelling. Average intensity and frequency of AU activation were calculated for each participant in each video. Partial correlations (controlling for the level of depression and cognitive impairment) were performed between these indexes and AI subscales. Results: Results showed that AU intensity and frequency were negatively correlated with apathy scale scores, in particular with the emotional blunting component. The more severe the apathy symptoms, the less expressivity in specific emotional and nonemotional AUs was displayed from participants while recalling an emotional event. Different AUs showed significant correlations depending on the sex of the participant and the task’s valence (positive vs negative story), suggesting the importance of assessing male and female participants independently. Conclusions: Our study suggests the interest of employing computer vision-based facial analysis to quantify facial expressivity and assess the severity of apathy symptoms in subjects with neurocognitive disorders. This may represent a useful tool for a preliminary apathy assessment in nonspecialized settings and could be used to complement classical clinical scales. Future studies including larger samples should confirm the clinical relevance of this kind of instrument.

BACKGROUND Neurocognitive disorders are often accompanied by behavioral symptoms such as anxiety, depression, and/or apathy. These symptoms can occur very early in the disease progression and are often difficult to detect and quantify in nonspecialized clinical settings. OBJECTIVE We focus in this study on apathy, one of the most common and debilitating neuropsychiatric symptoms in neurocognitive disorders. Specifically, we investigated whether facial expressivity extracted through computer vision software correlates with the severity of apathy symptoms in elderly subjects with neurocognitive disorders. METHODS A total of 63 subjects (38 females and 25 males) with neurocognitive disorder participated in the study. Apathy was assessed using the Apathy Inventory (AI), a scale comprising 3 domains of apathy: loss of interest, loss of initiation, and emotional blunting. The higher the scale score, the more severe the apathy symptoms. Participants were asked to recall a positive and a negative event of their life, while their voice and face were recorded using a tablet device. Action units (AUs), which are basic facial movements, were extracted using OpenFace 2.0. A total of 17 AUs (intensity and presence) for each frame of the video were extracted in both positive and negative storytelling. Average intensity and frequency of AU activation were calculated for each participant in each video. Partial correlations (controlling for the level of depression and cognitive impairment) were performed between these indexes and AI subscales. RESULTS Results showed that AU intensity and frequency were negatively correlated with apathy scale scores, in particular with the emotional blunting component. The more severe the apathy symptoms, the less expressivity in specific emotional and nonemotional AUs was displayed from participants while recalling an emotional event. Different AUs showed significant correlations depending on the sex of the participant and the task’s valence (positive vs negative story), suggesting the importance of assessing male and female participants independently. CONCLUSIONS Our study suggests the interest of employing computer vision-based facial analysis to quantify facial expressivity and assess the severity of apathy symptoms in subjects with neurocognitive disorders. This may represent a useful tool for a preliminary apathy assessment in nonspecialized settings and could be used to complement classical clinical scales. Future studies including larger samples should confirm the clinical relevance of this kind of instrument.

This paper proposes a database for group level emotion recognition in videos. The motivation is coming from the large number of information which the users are sharing online. This gives us the opportunity to use this perceived affect for various tasks. Most of the work in this area has been restricted to controlled environments. In this paper, we explore the group level emotion and cohesion in a real-world environment. There are several challenges involved in moving from a controlled environment to real-world scenarios such as face tracking limitations, illumination variations, occlusion and type of gatherings. As an attempt to address these challenges, we propose a ‘Video level Group AFfect (VGAF), database containing 1,004 videos downloaded from the web. The collected videos have a large variations in terms of gender, ethnicity, the type of social event, number of people, pose, etc. We have labelled our database for group level emotion and cohesion tasks and proposed a baseline based on the Inception V3 network on the database.

The paper explores the topic of Facial Action Unit (FAU) detection in the wild. In particular, we are interested in answering the following questions: (1) How useful are residual connections across dense blocks for face analysis? (2) How useful is the information from a network trained for categorical Facial Expression Recognition (FER) for the task of FAU detection? The proposed network (ResiDen) exploits dense blocks along with residual connections and uses auxiliary information from a FER network. The experiments are performed on the EmotionNet and DISFA datasets. The experiments show the usefulness of facial expression information for AU detection. The proposed network achieves state-of-the-art results on the two datasets. Analysis of the results for cross dataset protocol shows the effectiveness of the network.

The deep neural network shows a consequential performance for a set of specific tasks. A system designed for some correlated task altogether can be feasible for `in the wild' applications. This paper proposes a method for the face localization, Action Unit (AU) and emotion detection. The three different tasks are performed by a simultaneous hierarchical network which exploits the way of learning of neural networks. Such network can represent more relevant features than the individual network. Due to more complex structures and very deep networks, the deployment of neural networks for real life applications is a challenging task. The paper focuses to find an efficient trade-off between the performance and the complexity of the given tasks. This is done by exploring the advantages of optimization of the network for the given tasks by using separable convolutions, binarization and quantization. Four different databases (AffectNet, EmotioNet, RAF-DB and WiderFace) are used to evaluate the performance of our proposed approach by having a separate task specific database.