Itir Onal Ertugrul
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Itir Onal Ertugrul has a total of 32 co-authors. Here are the top co-authors by number of shared publications:
- Fatos T. Yarman Vural
Department of Computer Engineering, Middle East Technical University, Ankara, Turkey
4 shared publications - Mete Ozay
Graduate School of Information Sciences Tohoku University Japan
3 shared publications - Jeffrey F. Cohn
Department of Psychology and Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA and Deliberate AI, New York, NY, USA
2 shared publications - Laszlo Attila Jeni
Carnegie Mellon University
2 shared publications - Merel Madeleine Jung
Tilburg University, Netherlands
2 shared publications Mete Ozay
Graduate School of Information Sciences, Tohoku University, Sendai, Japan
2 shared publications
Latest Publications
- Open Access
- 5 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 337-346
Multimodal Depression Detection Varies with Cross-Dataset Differences
- 5 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 337-346
Open AccessMultimodal Depression Detection Varies with Cross-Dataset Differences
- Open Access
- 24 September 2026
- Published by The Royal Society in Philosophical Transactions of the Royal Society B: Biological Sciences
- Vol. 381 (1959)
Smiling in time: the interactive development of infant smiling with mothers, fathers and strangers
A developmental examination of the temporal dynamics of infant smiling during early social interaction is lacking. We ask how infants and adults influence each other's expressions of positive affect by examining the timing of transitions among dyadic states defined by the presence and absence of each partner's smiles. Fifty-eight infants interacted with their mothers, fathers and a stranger at 4 and 8 months. A survival analysis of the time course of infant (and adult partner) smiles and non-smiles indicated that infants initiated smiles and reciprocated smiles at lower rates than their adult partners, yet ended smiles at higher rates. Nevertheless, adult smiles were associated with higher infant smile rates, while infant smiles prolonged adult smiles, suggesting a dyadic path to mutual positivity. Eight-month-olds initiated smiles at higher rates than 4-month-olds when neither partner was smiling, suggesting increasing infant smiling initiative. Partner-specific patterns revealed selective infant responsiveness. Although strangers initiated and sustained smiles at higher rates than mothers, infants were more probably to reciprocate mother than stranger smiles. In the first year of life, infants showed a tendency to down-regulate positive affect expressions across age and context, as well as partner-specific smile reciprocation proclivities and a developmental increase in smile initiation. This article is part of the theme issue ‘Mechanisms, development, phylogeny and functions of emotional expressions’.
- 24 September 2026
- Published by The Royal Society in Philosophical Transactions of the Royal Society B: Biological Sciences
- Vol. 381 (1959)
Open AccessSmiling in time: the interactive development of infant smiling with mothers, fathers and strangers
A developmental examination of the temporal dynamics of infant smiling during early social interaction is lacking. We ask how infants and adults influence each other's expressions of positive affect by examining the timing of transitions among dyadic states defined by the presence and absence of each partner's smiles. Fifty-eight infants interacted with their mothers, fathers and a stranger at 4 and 8 months. A survival analysis of the time course of infant (and adult partner) smiles and non-smiles indicated that infants initiated smiles and reciprocated smiles at lower rates than their adult partners, yet ended smiles at higher rates. Nevertheless, adult smiles were associated with higher infant smile rates, while infant smiles prolonged adult smiles, suggesting a dyadic path to mutual positivity. Eight-month-olds initiated smiles at higher rates than 4-month-olds when neither partner was smiling, suggesting increasing infant smiling initiative. Partner-specific patterns revealed selective infant responsiveness. Although strangers initiated and sustained smiles at higher rates than mothers, infants were more probably to reciprocate mother than stranger smiles. In the first year of life, infants showed a tendency to down-regulate positive affect expressions across age and context, as well as partner-specific smile reciprocation proclivities and a developmental increase in smile initiation. This article is part of the theme issue ‘Mechanisms, development, phylogeny and functions of emotional expressions’.
- 30 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 15 (4) , 2025-2041
Multimodal Prediction of Obsessive-Compulsive Disorder and Comorbid Depression Severity and Energy Delivered by Deep Brain Electrodes
To develop reliable, valid, and efficient measures of obsessive-compulsive disorder (OCD) severity, comorbid depression severity, and total electrical energy delivered (TEED) by deep brain stimulation (DBS), we trained and compared random forests regression models in a clinical trial of participants receiving DBS for refractory OCD. Six participants were recorded during open-ended interviews at pre- and post-surgery baselines and then at 3-month intervals following DBS activation. Ground-truth severity was assessed by clinical interview and self-report. Visual and auditory modalities included facial action units, head and facial landmarks, speech behavior and content, and voice acoustics. Mixed-effects random forest regression with Shapley feature reduction strongly predicted severity of OCD, comorbid depression, and total electrical energy delivered by the DBS electrodes (intraclass correlation, ICC, = 0.83, 0.87, and 0.81, respectively. When random effects were omitted from the regression, predictive power decreased to moderate for severity of OCD and comorbid depression and remained comparable for total electrical energy delivered (ICC = 0.60, 0.68, and 0.83, respectively). Multimodal measures of behavior outperformed ones from single modalities. Feature selection achieved large decreases in features and corresponding increases in prediction. The approach could contribute to closed-loop DBS that would automatically titrate DBS based on affect measures.
- 30 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 15 (4) , 2025-2041
Multimodal Prediction of Obsessive-Compulsive Disorder and Comorbid Depression Severity and Energy Delivered by Deep Brain Electrodes
To develop reliable, valid, and efficient measures of obsessive-compulsive disorder (OCD) severity, comorbid depression severity, and total electrical energy delivered (TEED) by deep brain stimulation (DBS), we trained and compared random forests regression models in a clinical trial of participants receiving DBS for refractory OCD. Six participants were recorded during open-ended interviews at pre- and post-surgery baselines and then at 3-month intervals following DBS activation. Ground-truth severity was assessed by clinical interview and self-report. Visual and auditory modalities included facial action units, head and facial landmarks, speech behavior and content, and voice acoustics. Mixed-effects random forest regression with Shapley feature reduction strongly predicted severity of OCD, comorbid depression, and total electrical energy delivered by the DBS electrodes (intraclass correlation, ICC, = 0.83, 0.87, and 0.81, respectively. When random effects were omitted from the regression, predictive power decreased to moderate for severity of OCD and comorbid depression and remained comparable for total electrical energy delivered (ICC = 0.60, 0.68, and 0.83, respectively). Multimodal measures of behavior outperformed ones from single modalities. Feature selection achieved large decreases in features and corresponding increases in prediction. The approach could contribute to closed-loop DBS that would automatically titrate DBS based on affect measures.
- Open Access
- 9 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 144-152
Deciphering Entrepreneurial Pitches: A Multimodal Deep Learning Approach to Predict Probability of Investment
- 9 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 144-152
Open AccessDeciphering Entrepreneurial Pitches: A Multimodal Deep Learning Approach to Predict Probability of Investment
- Open Access
- 9 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 36-40
Do Body Expressions Leave Good Impressions? - Predicting Investment Decisions based on Pitcher's Body Expressions
This work aims to investigate whether displayed body expressions during venture pitches hold predictive value over the success of gathering funding. To this end the performance of several traditional regression models trained on aggregated body expression features is compared to that of deep recurrent models. Moreover, the impact of features and the effect of oversampling to counteract data imbalance are investigated. The results showed that temporal modeling of body expressions with GRU yielded the best performance (average MAE = 16.9). Oversampling was found to not improve models’ performance. The analysis of the impact of features suggest that higher levels of expressed body movements negatively affect investment scores whereas a more open body posture can have a positive effect. These findings show that body expressions can be predictive of investment decisions. Future work will be directed towards integrating these cues into a multimodal model to aid entrepreneurs in improving their pitching skills and further the understanding of the investment decision-making process.
- 9 October 2023
- Published by Association for Computing Machinery (ACM)
- p. 36-40
Open AccessDo Body Expressions Leave Good Impressions? - Predicting Investment Decisions based on Pitcher's Body Expressions
This work aims to investigate whether displayed body expressions during venture pitches hold predictive value over the success of gathering funding. To this end the performance of several traditional regression models trained on aggregated body expression features is compared to that of deep recurrent models. Moreover, the impact of features and the effect of oversampling to counteract data imbalance are investigated. The results showed that temporal modeling of body expressions with GRU yielded the best performance (average MAE = 16.9). Oversampling was found to not improve models’ performance. The analysis of the impact of features suggest that higher levels of expressed body movements negatively affect investment scores whereas a more open body posture can have a positive effect. These findings show that body expressions can be predictive of investment decisions. Future work will be directed towards integrating these cues into a multimodal model to aid entrepreneurs in improving their pitching skills and further the understanding of the investment decision-making process.
- Open Access
- 29 November 2019
- Published by Frontiers Media SA in Frontiers in Computer Science
- Vol. 1
D-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection
Facial action units (AUs) relate to specific local facial regions. Recent efforts in automated AU detection have focused on learning the facial patch representations to detect specific AUs. These efforts have encountered three hurdles. First, they implicitly assume that facial patches are robust to head rotation; yet non-frontal rotation is common. Second, mappings between AUs and patches are defined a priori, which ignores co-occurrences among AUs. And third, the dynamics of AUs are either ignored or modeled sequentially rather than simultaneously as in human perception. Inspired by recent advances in human perception, we propose a dynamic patch-attentive deep network, called D-PAttNet, for AU detection that i) controls for 3D head and face rotation, ii) learns mappings of patches to AUs, and iii) models spatiotemporal dynamics. D-PAttNet approach significantly improves upon existing state of the art.
- 29 November 2019
- Published by Frontiers Media SA in Frontiers in Computer Science
- Vol. 1
Open AccessD-PAttNet: Dynamic Patch-Attentive Deep Network for Action Unit Detection
Facial action units (AUs) relate to specific local facial regions. Recent efforts in automated AU detection have focused on learning the facial patch representations to detect specific AUs. These efforts have encountered three hurdles. First, they implicitly assume that facial patches are robust to head rotation; yet non-frontal rotation is common. Second, mappings between AUs and patches are defined a priori, which ignores co-occurrences among AUs. And third, the dynamics of AUs are either ignored or modeled sequentially rather than simultaneously as in human perception. Inspired by recent advances in human perception, we propose a dynamic patch-attentive deep network, called D-PAttNet, for AU detection that i) controls for 3D head and face rotation, ii) learns mappings of patches to AUs, and iii) models spatiotemporal dynamics. D-PAttNet approach significantly improves upon existing state of the art.
- 22 January 2019
- Published by Springer Nature in Brain Imaging and Behavior
- Vol. 14 (2) , 460-476
Gender classification using mesh networks on multiresolution multitask fMRI data
Brain connectivity networks have been shown to represent gender differences under a number of cognitive tasks. Recently, it has been conjectured that fMRI signals decomposed into different resolutions embed different types of cognitive information. In this paper, we combine multiresolution analysis and connectivity networks to study gender differences under a variety of cognitive tasks, and propose a machine learning framework to discriminate individuals according to their gender. For this purpose, we estimate a set of brain networks, formed at different resolutions while the subjects perform different cognitive tasks. First, we decompose fMRI signals recorded under a sequence of cognitive stimuli into its frequency subbands using Discrete Wavelet Transform (DWT). Next, we represent the fMRI signals by mesh networks formed among the anatomic regions for each task experiment at each subband. The mesh networks are constructed by ensembling a set of local meshes, each of which represents the relationship of an anatomical region as a weighted linear combination of its neighbors. Then, we estimate the edge weights of each mesh by ridge regression. The proposed approach yields 2CL functional mesh networks for each subject, where C is the number of cognitive tasks and L is the number of subband signals obtained after wavelet decomposition. This approach enables one to classify gender under different cognitive tasks and different frequency subbands. The final step of the suggested framework is to fuse the complementary information of the mesh networks for each subject to discriminate the gender. We fuse the information embedded in mesh networks formed for different tasks and resolutions under a three-level fuzzy stacked generalization (FSG) architecture. In this architecture, different layers are responsible for fusion of diverse information obtained from different cognitive tasks and resolutions. In the experimental analyses, we use Human Connectome Project task fMRI dataset. Results reflect that fusing the mesh network representations computed at multiple resolutions for multiple tasks provides the best gender classification accuracy compared to the single subband task mesh networks or fusion of representations obtained using only multitask or only multiresolution data. Besides, mesh edge weights slightly outperform pairwise correlations between regions, and significantly outperform raw fMRI signals. In addition, we analyze the gender discriminative power of mesh edge weights for different tasks and resolutions.
- 22 January 2019
- Published by Springer Nature in Brain Imaging and Behavior
- Vol. 14 (2) , 460-476
Gender classification using mesh networks on multiresolution multitask fMRI data
Brain connectivity networks have been shown to represent gender differences under a number of cognitive tasks. Recently, it has been conjectured that fMRI signals decomposed into different resolutions embed different types of cognitive information. In this paper, we combine multiresolution analysis and connectivity networks to study gender differences under a variety of cognitive tasks, and propose a machine learning framework to discriminate individuals according to their gender. For this purpose, we estimate a set of brain networks, formed at different resolutions while the subjects perform different cognitive tasks. First, we decompose fMRI signals recorded under a sequence of cognitive stimuli into its frequency subbands using Discrete Wavelet Transform (DWT). Next, we represent the fMRI signals by mesh networks formed among the anatomic regions for each task experiment at each subband. The mesh networks are constructed by ensembling a set of local meshes, each of which represents the relationship of an anatomical region as a weighted linear combination of its neighbors. Then, we estimate the edge weights of each mesh by ridge regression. The proposed approach yields 2CL functional mesh networks for each subject, where C is the number of cognitive tasks and L is the number of subband signals obtained after wavelet decomposition. This approach enables one to classify gender under different cognitive tasks and different frequency subbands. The final step of the suggested framework is to fuse the complementary information of the mesh networks for each subject to discriminate the gender. We fuse the information embedded in mesh networks formed for different tasks and resolutions under a three-level fuzzy stacked generalization (FSG) architecture. In this architecture, different layers are responsible for fusion of diverse information obtained from different cognitive tasks and resolutions. In the experimental analyses, we use Human Connectome Project task fMRI dataset. Results reflect that fusing the mesh network representations computed at multiple resolutions for multiple tasks provides the best gender classification accuracy compared to the single subband task mesh networks or fusion of representations obtained using only multitask or only multiresolution data. Besides, mesh edge weights slightly outperform pairwise correlations between regions, and significantly outperform raw fMRI signals. In addition, we analyze the gender discriminative power of mesh edge weights for different tasks and resolutions.
- Open Access
- 22 September 2018
- Published by Elsevier in Image and Vision Computing
- Vol. 79, 133-143
Modeling and synthesis of kinship patterns of facial expressions
- 22 September 2018
- Published by Elsevier in Image and Vision Computing
- Vol. 79, 133-143
Open AccessModeling and synthesis of kinship patterns of facial expressions
- 15 June 2018
- Published by Springer Nature in Brain Imaging and Behavior
- Vol. 13 (4) , 893-904
Encoding the local connectivity patterns of fMRI for cognitive task and state classification
In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.
- 15 June 2018
- Published by Springer Nature in Brain Imaging and Behavior
- Vol. 13 (4) , 893-904
Encoding the local connectivity patterns of fMRI for cognitive task and state classification
In this work, we propose a novel framework to encode the local connectivity patterns of brain, using Fisher vectors (FV), vector of locally aggregated descriptors (VLAD) and bag-of-words (BoW) methods. We first obtain local descriptors, called mesh arc descriptors (MADs) from fMRI data, by forming local meshes around anatomical regions, and estimating their relationship within a neighborhood. Then, we extract a dictionary of relationships, called brain connectivity dictionary by fitting a generative Gaussian mixture model (GMM) to a set of MADs, and selecting codewords at the mean of each component of the mixture. Codewords represent connectivity patterns among anatomical regions. We also encode MADs by VLAD and BoW methods using k-Means clustering. We classify cognitive tasks using the Human Connectome Project (HCP) task fMRI dataset and cognitive states using the Emotional Memory Retrieval (EMR). We train support vector machines (SVMs) using the encoded MADs. Results demonstrate that, FV encoding of MADs can be successfully employed for classification of cognitive tasks, and outperform VLAD and BoW representations. Moreover, we identify the significant Gaussians in mixture models by computing energy of their corresponding FV parts, and analyze their effect on classification accuracy. Finally, we suggest a new method to visualize the codewords of the learned brain connectivity dictionary.
- 1 June 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 2018, 2211-221109
FACSCaps: Pose-Independent Facial Action Coding with Capsules
Most automated facial expression analysis methods treat the face as a 2D object, flat like a sheet of paper. That works well provided images are frontal or nearly so. In real-world conditions, moderate to large head rotation is common and system performance to recognize expression degrades. Multi-view Convolutional Neural Networks (CNNs) have been proposed to increase robustness to pose, but they require greater model sizes and may generalize poorly across views that are not included in the training set. We propose FACSCaps architecture to handle multi-view and multi-label facial action unit (AU) detection within a single model that can generalize to novel views. Additionally, FACSCaps's ability to synthesize faces enables insights into what is leaned by the model. FACSCaps models video frames using matrix capsules, where hierarchical pose relationships between face parts are built into internal representations. The model is trained by jointly optimizing a multi-label loss and the reconstruction accuracy. FACSCaps was evaluated using the FERA 2017 facial expression dataset that includes spontaneous facial expressions in a wide range of head orientations. FACSCaps outperformed both state-of-the-art CNNs and their temporal extensions.
- 1 June 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 2018, 2211-221109
FACSCaps: Pose-Independent Facial Action Coding with Capsules
Most automated facial expression analysis methods treat the face as a 2D object, flat like a sheet of paper. That works well provided images are frontal or nearly so. In real-world conditions, moderate to large head rotation is common and system performance to recognize expression degrades. Multi-view Convolutional Neural Networks (CNNs) have been proposed to increase robustness to pose, but they require greater model sizes and may generalize poorly across views that are not included in the training set. We propose FACSCaps architecture to handle multi-view and multi-label facial action unit (AU) detection within a single model that can generalize to novel views. Additionally, FACSCaps's ability to synthesize faces enables insights into what is leaned by the model. FACSCaps models video frames using matrix capsules, where hierarchical pose relationships between face parts are built into internal representations. The model is trained by jointly optimizing a multi-label loss and the reconstruction accuracy. FACSCaps was evaluated using the FERA 2017 facial expression dataset that includes spontaneous facial expressions in a wide range of head orientations. FACSCaps outperformed both state-of-the-art CNNs and their temporal extensions.