IEItir Onal Ertugrul

Itir Onal Ertugrul

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Itir Onal Ertugrul has a total of 32 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Multimodal Depression Detection Varies with Cross-Dataset Differences

    by Maneesh Bilalpur, Saurabh Hinduja, Sonish Sivarajkumar, Yanshan Wang, Nicholas Allen, Itir Onal Ertugrul, Jeff Cohn

  • Open Access

    Smiling in time: the interactive development of infant smiling with mothers, fathers and strangers

    by Eliala Alice Salvadori, Daniel S. Messinger, Alan B. Cobo-Lewis, Itir Onal Ertugrul, Cristina Colonnesi

    A developmental examination of the temporal dynamics of infant smiling during early social interaction is lacking. We ask how infants and adults influence each other's expressions of positive affect by examining the timing of transitions among dyadic states defined by the presence and absence of each partner's smiles. Fifty-eight infants interacted with their mothers, fathers and a stranger at 4 and 8 months. A survival analysis of the time course of infant (and adult partner) smiles and non-smiles indicated that infants initiated smiles and reciprocated smiles at lower rates than their adult partners, yet ended smiles at higher rates. Nevertheless, adult smiles were associated with higher infant smile rates, while infant smiles prolonged adult smiles, suggesting a dyadic path to mutual positivity. Eight-month-olds initiated smiles at higher rates than 4-month-olds when neither partner was smiling, suggesting increasing infant smiling initiative. Partner-specific patterns revealed selective infant responsiveness. Although strangers initiated and sustained smiles at higher rates than mothers, infants were more probably to reciprocate mother than stranger smiles. In the first year of life, infants showed a tendency to down-regulate positive affect expressions across age and context, as well as partner-specific smile reciprocation proclivities and a developmental increase in smile initiation. This article is part of the theme issue ‘Mechanisms, development, phylogeny and functions of emotional expressions’.

  • Multimodal Prediction of Obsessive-Compulsive Disorder and Comorbid Depression Severity and Energy Delivered by Deep Brain Electrodes

    by Saurabh Hinduja, Ali Darzi, Itir Onal Ertugrul, Nicole Provenza, Ron Gadot, Eric A. Storch, Sameer A. Sheth, Wayne K. Goodman, Jeffrey F. Cohn

    To develop reliable, valid, and efficient measures of obsessive-compulsive disorder (OCD) severity, comorbid depression severity, and total electrical energy delivered (TEED) by deep brain stimulation (DBS), we trained and compared random forests regression models in a clinical trial of participants receiving DBS for refractory OCD. Six participants were recorded during open-ended interviews at pre- and post-surgery baselines and then at 3-month intervals following DBS activation. Ground-truth severity was assessed by clinical interview and self-report. Visual and auditory modalities included facial action units, head and facial landmarks, speech behavior and content, and voice acoustics. Mixed-effects random forest regression with Shapley feature reduction strongly predicted severity of OCD, comorbid depression, and total electrical energy delivered by the DBS electrodes (intraclass correlation, ICC, = 0.83, 0.87, and 0.81, respectively. When random effects were omitted from the regression, predictive power decreased to moderate for severity of OCD and comorbid depression and remained comparable for total electrical energy delivered (ICC = 0.60, 0.68, and 0.83, respectively). Multimodal measures of behavior outperformed ones from single modalities. Feature selection achieved large decreases in features and corresponding increases in prediction. The approach could contribute to closed-loop DBS that would automatically titrate DBS based on affect measures.


  • Open Access

    Do Body Expressions Leave Good Impressions? - Predicting Investment Decisions based on Pitcher's Body Expressions

    by Merel M. Jung, Mark Van Vlierden, Werner Liebregts, Itır Önal Ertuğrul

    This work aims to investigate whether displayed body expressions during venture pitches hold predictive value over the success of gathering funding. To this end the performance of several traditional regression models trained on aggregated body expression features is compared to that of deep recurrent models. Moreover, the impact of features and the effect of oversampling to counteract data imbalance are investigated. The results showed that temporal modeling of body expressions with GRU yielded the best performance (average MAE = 16.9). Oversampling was found to not improve models’ performance. The analysis of the impact of features suggest that higher levels of expressed body movements negatively affect investment scores whereas a more open body posture can have a positive effect. These findings show that body expressions can be predictive of investment decisions. Future work will be directed towards integrating these cues into a multimodal model to aid entrepreneurs in improving their pitching skills and further the understanding of the investment decision-making process.

  • Open Access

    D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection

    by Itir Onal Ertugrul, Le Yang, László A. Jeni, Jeffrey F. Cohn

    Facial action units (AUs) relate to specific local facial regions. Recent efforts in automated AU detection have focused on learning the facial patch representations to detect specific AUs. These efforts have encountered three hurdles. First, they implicitly assume that facial patches are robust to head rotation; yet non-frontal rotation is common. Second, mappings between AUs and patches are defined a priori, which ignores co-occurrences among AUs. And third, the dynamics of AUs are either ignored or modeled sequentially rather than simultaneously as in human perception. Inspired by recent advances in human perception, we propose a dynamic patch-attentive deep network, called D-PAttNet, for AU detection that i) controls for 3D head and face rotation, ii) learns mappings of patches to AUs, and iii) models spatiotemporal dynamics. D-PAttNet approach significantly improves upon existing state of the art.


  • Gender classification using mesh networks on multiresolution multitask fMRI data

    by Itir Onal Ertugrul, Mete Ozay, Fatos T. Yarman Vural

    Brain connectivity networks have been shown to represent gender differences under a number of cognitive tasks. Recently, it has been conjectured that fMRI signals decomposed into different resolutions embed different types of cognitive information. In this paper, we combine multiresolution analysis and connectivity networks to study gender differences under a variety of cognitive tasks, and propose a machine learning framework to discriminate individuals according to their gender. For this purpose, we estimate a set of brain networks, formed at different resolutions while the subjects perform different cognitive tasks. First, we decompose fMRI signals recorded under a sequence of cognitive stimuli into its frequency subbands using Discrete Wavelet Transform (DWT). Next, we represent the fMRI signals by mesh networks formed among the anatomic regions for each task experiment at each subband. The mesh networks are constructed by ensembling a set of local meshes, each of which represents the relationship of an anatomical region as a weighted linear combination of its neighbors. Then, we estimate the edge weights of each mesh by ridge regression. The proposed approach yields 2CL functional mesh networks for each subject, where C is the number of cognitive tasks and L is the number of subband signals obtained after wavelet decomposition. This approach enables one to classify gender under different cognitive tasks and different frequency subbands. The final step of the suggested framework is to fuse the complementary information of the mesh networks for each subject to discriminate the gender. We fuse the information embedded in mesh networks formed for different tasks and resolutions under a three-level fuzzy stacked generalization (FSG) architecture. In this architecture, different layers are responsible for fusion of diverse information obtained from different cognitive tasks and resolutions. In the experimental analyses, we use Human Connectome Project task fMRI dataset. Results reflect that fusing the mesh network representations computed at multiple resolutions for multiple tasks provides the best gender classification accuracy compared to the single subband task mesh networks or fusion of representations obtained using only multitask or only multiresolution data. Besides, mesh edge weights slightly outperform pairwise correlations between regions, and significantly outperform raw fMRI signals. In addition, we analyze the gender discriminative power of mesh edge weights for different tasks and resolutions.

  • Open Access

    Modeling and synthesis of kinship patterns of facial expressions

    by Itır Önal Ertuğrul, László A. Jeni, Hamdi Dibeklioğlu


  • Encoding the local connectivity patterns of fMRI for cognitive task and state classification

    by Itir Onal Ertugrul, Mete Ozay, Fatos T Yarman Vural

    In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.


  • FACSCaps: Pose-Independent Facial Action Coding with Capsules

    by Itir Onal Ertugrul, Laszlo A. Jeni, Jeffrey F. Cohn

    Most automated facial expression analysis methods treat the face as a 2D object, flat like a sheet of paper. That works well provided images are frontal or nearly so. In real-world conditions, moderate to large head rotation is common and system performance to recognize expression degrades. Multi-view Convolutional Neural Networks (CNNs) have been proposed to increase robustness to pose, but they require greater model sizes and may generalize poorly across views that are not included in the training set. We propose FACSCaps architecture to handle multi-view and multi-label facial action unit (AU) detection within a single model that can generalize to novel views. Additionally, FACSCaps's ability to synthesize faces enables insights into what is leaned by the model. FACSCaps models video frames using matrix capsules, where hierarchical pose relationships between face parts are built into internal representations. The model is trained by jointly optimizing a multi-label loss and the reconstruction accuracy. FACSCaps was evaluated using the FERA 2017 facial expression dataset that includes spontaneous facial expressions in a wide range of head orientations. FACSCaps outperformed both state-of-the-art CNNs and their temporal extensions.