Henglin Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Henglin Shi has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Guoying Zhao
University of Oulu, Oulu, Finland
5 shared publications - Haoyu Chen
University of Oulu, Oulu, Finland
3 shared publications - Wei Peng
Stanford University, Stanford, USA
2 shared publications Abdelrahman Mostafa
Center for Machine Vision and Signal Analysis, University of Oulu, Oulu, Finland
1 shared publication- Andy Nguyen
University of Oulu
1 shared publication - Eija Vuorenmaa
Faculty of Education and Psychology, University of Oulu, Oulu, FINLAND
1 shared publication
Latest Publications
- Open Access
- 2 August 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Interactive Intelligent Systems
- Vol. 14 (3) , 1-34
Interactions for Socially Shared Regulation in Collaborative Learning: An Interdisciplinary Multimodal Dataset
Socially shared regulation plays a pivotal role in the success of collaborative learning. However, evaluating socially shared regulation of learning (SSRL) proves challenging due to the dynamic and infrequent cognitive and socio-emotional interactions, which constitute the focal point of SSRL. To address this challenge, this article gathers interdisciplinary researchers to establish a multimodal dataset with cognitive and socio-emotional interactions for SSRL study. Firstly, to induce cognitive and socio-emotional interactions, learning science researchers designed a special collaborative learning task with regulatory trigger events among triadic people for the SSRL study. Secondly, this dataset includes various modalities like video, Kinect data, audio, and physiological data (accelerometer, EDA, heart rate) from 81 high school students in 28 groups, offering a comprehensive view of the SSRL process. Thirdly, three-level verbal interaction annotations and nonverbal interactions including facial expression, eye gaze, gesture, and posture are provided, which could further contribute to interdisciplinary fields such as computer science, sociology, and education. In addition, comprehensive analysis verifies the dataset’s effectiveness. As far as we know, this is the first multimodal dataset for studying SSRL among triadic group members.
- 2 August 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Interactive Intelligent Systems
- Vol. 14 (3) , 1-34
Open AccessInteractions for Socially Shared Regulation in Collaborative Learning: An Interdisciplinary Multimodal Dataset
Socially shared regulation plays a pivotal role in the success of collaborative learning. However, evaluating socially shared regulation of learning (SSRL) proves challenging due to the dynamic and infrequent cognitive and socio-emotional interactions, which constitute the focal point of SSRL. To address this challenge, this article gathers interdisciplinary researchers to establish a multimodal dataset with cognitive and socio-emotional interactions for SSRL study. Firstly, to induce cognitive and socio-emotional interactions, learning science researchers designed a special collaborative learning task with regulatory trigger events among triadic people for the SSRL study. Secondly, this dataset includes various modalities like video, Kinect data, audio, and physiological data (accelerometer, EDA, heart rate) from 81 high school students in 28 groups, offering a comprehensive view of the SSRL process. Thirdly, three-level verbal interaction annotations and nonverbal interactions including facial expression, eye gaze, gesture, and posture are provided, which could further contribute to interdisciplinary fields such as computer science, sociology, and education. In addition, comprehensive analysis verifies the dataset’s effectiveness. As far as we know, this is the first multimodal dataset for studying SSRL among triadic group members.
- 27 April 2023
- Published by Springer Nature
- p. 129-139
Attention-guided Boundary Refinement on Anchor-free Temporal Action Detection
- 27 April 2023
- Published by Springer Nature
- p. 129-139
Attention-guided Boundary Refinement on Anchor-free Temporal Action Detection
- Top Cited PapersOpen Access
- 16 February 2023
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (6) , 1346-1366
SMG: A Micro-gesture Dataset Towards Spontaneous Body Gestures for Emotional Stress State Analysis
We explore using body gestures for hidden emotional state analysis. As an important non-verbal communicative fashion, human body gestures are capable of conveying emotional information during social communication. In previous works, efforts have been made mainly on facial expressions, speech, or expressive body gestures to interpret classical expressive emotions. Differently, we focus on a specific group of body gestures, called micro-gestures (MGs), used in the psychology research field to interpret inner human feelings. MGs are subtle and spontaneous body movements that are proven, together with micro-expressions, to be more reliable than normal facial expressions for conveying hidden emotional information. In this work, a comprehensive study of MGs is presented from the computer vision aspect, including a novel spontaneous micro-gesture (SMG) dataset with two emotional stress states and a comprehensive statistical analysis indicating the correlations between MGs and emotional states. Novel frameworks are further presented together with various state-of-the-art methods as benchmarks for automatic classification, online recognition of MGs, and emotional stress state recognition. The dataset and methods presented could inspire a new way of utilizing body gestures for human emotion understanding and bring a new direction to the emotion AI community. The source code and dataset are made available:https://github.com/mikecheninoulu/SMG.
- 16 February 2023
- Published by Springer Nature in International Journal of Computer Vision
- Vol. 131 (6) , 1346-1366
Top Cited PapersOpen AccessSMG: A Micro-gesture Dataset Towards Spontaneous Body Gestures for Emotional Stress State Analysis
We explore using body gestures for hidden emotional state analysis. As an important non-verbal communicative fashion, human body gestures are capable of conveying emotional information during social communication. In previous works, efforts have been made mainly on facial expressions, speech, or expressive body gestures to interpret classical expressive emotions. Differently, we focus on a specific group of body gestures, called micro-gestures (MGs), used in the psychology research field to interpret inner human feelings. MGs are subtle and spontaneous body movements that are proven, together with micro-expressions, to be more reliable than normal facial expressions for conveying hidden emotional information. In this work, a comprehensive study of MGs is presented from the computer vision aspect, including a novel spontaneous micro-gesture (SMG) dataset with two emotional stress states and a comprehensive statistical analysis indicating the correlations between MGs and emotional states. Novel frameworks are further presented together with various state-of-the-art methods as benchmarks for automatic classification, online recognition of MGs, and emotional stress state recognition. The dataset and methods presented could inspire a new way of utilizing body gestures for human emotion understanding and bring a new direction to the emotion AI community. The source code and dataset are made available:https://github.com/mikecheninoulu/SMG.
- 20 September 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Intelligent Systems
- Vol. 37 (4) , 103-110
Multiscale 3D-Shift Graph Convolution Network for Emotion Recognition From Human Actions
Emotion recognition from body gestures is challenging since similar emotions can be expressed by arbitrary spatial configurations of joints, which results in relying on modeling spatial-temporal patterns from a more global level. However, most recent powerful graph convolution networks (GCNs) separate the spatial and temporal modeling into isolated processes, where GCN models spatial interactions using partially fixed adjacent matrices and 1D convolution captures temporal dynamics, which is insufficient for emotion recognition. In this work, we propose the 3D-Shift GCN, which enables interactions of joints within a spatial-temporal volume for global feature extraction. Besides, we further develop a multiscale architecture, the MS-Shift GCN, to fuse features captured under different temporal ranges for modeling richer dynamics. After conducting evaluation on two regular action recognition benchmarks and two gesture based emotion recognition datasets, the results show that the proposed method outperforms several state-of-the-art methods.
- 20 September 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Intelligent Systems
- Vol. 37 (4) , 103-110
Multiscale 3D-Shift Graph Convolution Network for Emotion Recognition From Human Actions
Emotion recognition from body gestures is challenging since similar emotions can be expressed by arbitrary spatial configurations of joints, which results in relying on modeling spatial-temporal patterns from a more global level. However, most recent powerful graph convolution networks (GCNs) separate the spatial and temporal modeling into isolated processes, where GCN models spatial interactions using partially fixed adjacent matrices and 1D convolution captures temporal dynamics, which is insufficient for emotion recognition. In this work, we propose the 3D-Shift GCN, which enables interactions of joints within a spatial-temporal volume for global feature extraction. Besides, we further develop a multiscale architecture, the MS-Shift GCN, to fuse features captured under different temporal ranges for modeling richer dynamics. After conducting evaluation on two regular action recognition benchmarks and two gesture based emotion recognition datasets, the results show that the proposed method outperforms several state-of-the-art methods.
- Top Cited PapersOpen Access
- 21 September 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (12) , 10023-10044
Hyperbolic Deep Neural Networks: A Survey
This article investigates the stabilization of discrete-time stochastic neural networks with time-varying delay via aperiodically intermittent control (AIC). A comprehensive analysis of the stabilization of discrete-time delayed systems via AIC is provided, where the Lyapunov function method and the Lyapunov-Krasovskii functional method are investigated, respectively. Then, three stabilization criteria are given, which extend previous works from the continuous-time framework to the discrete-time one, and the average activation time ratio (AATR) of AIC is estimated. It is highlighted that for the Lyapunov-Krasovskii functional method, a more flexible estimation for the AATR can be obtained. Finally, the differences and the advantages of the three stabilization criteria are illustrated by numerical simulations.
- 21 September 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 44 (12) , 10023-10044
Top Cited PapersOpen AccessHyperbolic Deep Neural Networks: A Survey
This article investigates the stabilization of discrete-time stochastic neural networks with time-varying delay via aperiodically intermittent control (AIC). A comprehensive analysis of the stabilization of discrete-time delayed systems via AIC is provided, where the Lyapunov function method and the Lyapunov-Krasovskii functional method are investigated, respectively. Then, three stabilization criteria are given, which extend previous works from the continuous-time framework to the discrete-time one, and the average activation time ratio (AATR) of AIC is estimated. It is highlighted that for the Lyapunov-Krasovskii functional method, a more flexible estimation for the AATR can be obtained. Finally, the differences and the advantages of the three stabilization criteria are illustrated by numerical simulations.