ESEssam Sleiman

Essam Sleiman

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Essam Sleiman has a total of 18 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Goldfish: Vision-Language Understanding of Arbitrarily Long Videos

    by Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, Mohamed Elhoseiny


  • Open Access

    Goldfish: Vision-Language Understanding of Arbitrarily Long Videos

    by Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, Mohamed Elhoseiny

    Most current LLM-based models for video understanding can process videos within minutes. However, they struggle with lengthy videos due to challenges such as "noise and redundancy", as well as "memory and computation" constraints. In this paper, we present Goldfish, a methodology tailored for comprehending videos of arbitrary lengths. We also introduce the TVQA-long benchmark, specifically designed to evaluate models' capabilities in understanding long videos with questions in both vision and text content. Goldfish approaches these challenges with an efficient retrieval mechanism that initially gathers the top-k video clips relevant to the instruction before proceeding to provide the desired response. This design of the retrieval mechanism enables the Goldfish to efficiently process arbitrarily long video sequences, facilitating its application in contexts such as movies or television series. To facilitate the retrieval process, we developed MiniGPT4-Video that generates detailed descriptions for the video clips. In addressing the scarcity of benchmarks for long video evaluation, we adapted the TVQA short video benchmark for extended content analysis by aggregating questions from entire episodes, thereby shifting the evaluation from partial to full episode comprehension. We attained a 41.78% accuracy rate on the TVQA-long benchmark, surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows exceptional performance in short video comprehension, exceeding existing state-of-the-art methods by 3.23%, 2.03%, 16.5% and 23.59% on the MSVD, MSRVTT, TGIF, and TVQA short video benchmarks, respectively. These results indicate that our models have significant improvements in both long and short-video understanding. Our models and code have been made publicly available at https://vision-cair.github.io/Goldfish_website/

  • Open Access

    Deep Learning-Based Autism Spectrum Disorder Detection Using Emotion Features From Video Recordings (Preprint)

    by Essam Sleiman, Onur Cezmi Mutlu, Saimourya Surabhi, Arman Husic, Aaron Kline, Peter Washington, Dennis P. Wall

    Journal of Medical Internet Research - International Scientific Journal for Medical Research, Information and Communication on the Internet #Preprint #PeerReviewMe: Warning: This is a unreviewed preprint. Readers are warned that the document has not been peer-reviewed by expert/patient reviewers or an academic editor, may contain misleading claims, and is likely to undergo changes before final publication, if accepted, or may have been rejected/withdrawn. Readers with interest and expertise are encouraged to sign up as peer-reviewer, if the paper is within an open peer-review period. Please cite this preprint only for review purposes or for grant applications and CVs (if you are the author). Background: Autism spectrum disorder (ASD) is a prevalent neurodevelopmental disorder encountered by 1 in 44 children in the United States. Patients with autism face difficulty effectively communicating with peers, articulating feelings and emotions, and controlling behaviors. Rapid and early diagnosis leads to improved treatment outcomes; however, current medical techniques often take years. Difficult, time-consuming diagnoses accompanied by the recent advances in computer vision have led to a surge of interest from researchers in developing models to streamline the autism diagnosis process. Objective: We aimed to assess the viability of at-home autism diagnosis by leveraging video data collected from a mobile game app to train a computer vision model using insights extracted from the change in emotion-related expression features over time. Methods: With our GuessWhat game-based mobile app, we collected a video data set of 74 children with ASD and neurotypical diagnoses actively playing in a natural home environment. To investigate and quantify the significance of facial emotion features for ASD detection, we developed a deep learning–based autism classifier with the following two components: a convolutional neural network (CNN) attached to a long short-term memory (LSTM). We pretrained our CNN backbone in the following two ways: (1) on ImageNet or (2) on a compilation of facial expression recognition data sets to analyze autism detection performance improvements using emotion features. The output of each CNN is fed into an LSTM model to diagnose ASD from video data. Results: Our top-performing architecture employing a CNN backbone trained on the facial expression recognition data sets and a unidirectional LSTM achieved a top accuracy of 91.2%, precision of 91.2%, recall of 89.8%, and an F1-score of 90.6%. Conclusions: We discovered the change in an individual’s facial expression features over time as a relevant marker for autism detection. Extracting emotional features from each frame resulted in a 27.8% F1-score improvement when compared to ImageNet weights. Our study demonstrates the capability of mobile apps to collect a natural, diverse data set for an improved autism diagnosis. These results demonstrate that deep learning and computer vision–based methods are instrumental for automated autism diagnosis from at-home recorded videos using unspecialized equipment.


  • Open Access

    Classifying Autism Spectrum Disorder Using Emotion Features From Video Recordings (Preprint)

    by Essam Sleiman, Onur Cezmi Mutlu, Saimourya Surabhi, Arman Husic, Aaron Kline, Peter Washington, Dennis P. Wall

    BACKGROUND Autism Spectrum Disorder (ASD) is a prevalent neurodevelopmental disorder encountered by 1 in 44 children in the United States of America. Autism patients face difficulty effectively communicating with peers, articulating feelings and emotions, and controlling behaviors. Rapid and early diagnosis leads to improved treatment outcomes, however current medical techniques often take years. Difficult, time-consuming diagnoses accompanied by the recent advances in computer vision have led to a surge of interest from researchers in developing models to streamline the autism diagnosis process. OBJECTIVE We aim to assess the viability of at-home autism diagnosis by leveraging video data collected from a mobile game app to train a computer vision model utilizing insights extracted from the change in emotion expression features over time. METHODS With our GuessWhat game-based mobile app, we collect a video dataset of 74 ASD and NT children actively playing in a natural home environment. To investigate and quantify the significance of facial emotion features for ASD detection, we develop a deep learning-based autism classifier with two components: a Convolutional Neural Network (CNN) attached to a Long Short Term Memory (LSTM). We pre-train our CNN backbone in two fashions: (1) on ImageNet or (2) on a compilation of Facial Expression Recognition (FER) datasets to analyze autism detection performance improvements utilizing emotion features. The output of each CNN is fed into an LSTM model to diagnose ASD from video data. RESULTS Our top-performing architecture utilizing a CNN backbone pre-trained on ImageNet obtained an accuracy of 45.8% and an F1 score of 62.8% while our corresponding top architecture employing a CNN backbone trained on the FER datasets achieved top accuracy of 91.2% and an F1 score of 90.6%. CONCLUSIONS We discovered the change in an individual's facial expression features over time as a relevant marker for autism detection. Extracting emotional features from each frame resulted in a 27.8% F1 score improvement when compared to non-emotion weights. Our study demonstrates the capability of mobile applications to collect a natural, diverse dataset for an improved autism diagnosis. These results demonstrate Deep Learning and Computer Vision based methods are instrumental for automated autism diagnosis from at-home recorded videos using unspecialized equipment.