Dong She
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Dong She has a total of 23 co-authors. Here are the top co-authors by number of shared publications:
- Yang Gao
South China University of Technology
7 shared publications - Zhanpeng Jin
South China University of Technology
7 shared publications - Junbin Ren
East China Normal University, Shanghai, China
3 shared publications - Yuting Bai
Design, South China University of Technology, GuangZhou, GuangDong, China
3 shared publications - Cong Liu
South China University of Technology, Guangzhou, China
2 shared publications - Jinghe Yu
School of Future Technology, South China University of Technology, Guangzhou, Guangdong, China
2 shared publications
Latest Publications
- 18 November 2026
- Published by Elsevier in Expert Systems with Applications
- Vol. 336
MHP-Eval and MaKE: A benchmark and retrieval-guided inference framework for mental health sensing with large language models
- 18 November 2026
- Published by Elsevier in Expert Systems with Applications
- Vol. 336
MHP-Eval and MaKE: A benchmark and retrieval-guided inference framework for mental health sensing with large language models
- Open Access
- 15 June 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 10 (2) , 1-33
HMotionGPT: Aligning Hand Motions and Natural Language for Activity Understanding with Smart Rings
Hand-object interactions are central to everyday activities, yet most intelligent assistants today remain blind to users' physical actions. Existing IMU-based recognition approaches focus on classifying predefined gestures, but they lack the semantic expressiveness required for contextual support in real-world scenarios such as office work and home routines. In this paper, we introduce a semantic tokenization pipeline that bridges continuous inertial signals and large language models (LLMs), enabling assistants to “read” hand movements as naturally as words. We first collected a multimodal dataset of dual-hand activities across office and home environments capturing long-horizon action chains that span multiple interrelated sub-tasks. Using self-supervised representation learning, we discretize IMU embeddings into action tokens that approximate a vocabulary of hand interactions. These tokens are then aligned with natural language through instruction-tuned LLMs, supporting tasks such as action captioning, intent inference, and contextual feedback. Evaluation shows that our tokenization improves semantic consistency with language distributions, and the LLM produces accurate, human-preferred descriptions of actions across diverse activities. We further demonstrate a proof-of-concept assistant prototype that generates contextual reminders. Our findings highlight the potential of transforming raw hand motions into a “language of actions,” paving the way for everyday intelligent assistants that are aware of users' physical interactions. The Project page, source code, and dataset are publicly available at https://scut-hai.github.io/HMotionGPT/.
- 15 June 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 10 (2) , 1-33
Open AccessHMotionGPT: Aligning Hand Motions and Natural Language for Activity Understanding with Smart Rings
Hand-object interactions are central to everyday activities, yet most intelligent assistants today remain blind to users' physical actions. Existing IMU-based recognition approaches focus on classifying predefined gestures, but they lack the semantic expressiveness required for contextual support in real-world scenarios such as office work and home routines. In this paper, we introduce a semantic tokenization pipeline that bridges continuous inertial signals and large language models (LLMs), enabling assistants to “read” hand movements as naturally as words. We first collected a multimodal dataset of dual-hand activities across office and home environments capturing long-horizon action chains that span multiple interrelated sub-tasks. Using self-supervised representation learning, we discretize IMU embeddings into action tokens that approximate a vocabulary of hand interactions. These tokens are then aligned with natural language through instruction-tuned LLMs, supporting tasks such as action captioning, intent inference, and contextual feedback. Evaluation shows that our tokenization improves semantic consistency with language distributions, and the LLM produces accurate, human-preferred descriptions of actions across diverse activities. We further demonstrate a proof-of-concept assistant prototype that generates contextual reminders. Our findings highlight the potential of transforming raw hand motions into a “language of actions,” paving the way for everyday intelligent assistants that are aware of users' physical interactions. The Project page, source code, and dataset are publicly available at https://scut-hai.github.io/HMotionGPT/.
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-20
RageSense: Leveraging Acoustic Sensing and LLM-Based Intervention for Emotion Regulation in Mobile Gaming
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Open AccessRageSense: Leveraging Acoustic Sensing and LLM-Based Intervention for Emotion Regulation in Mobile Gaming
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-7
EmoType-Zero: Adaptive Fonts to Render the Speech Emotion
Text messaging often lacks paralinguistic cues essential for interpreting emotional tone. To explore visual alternatives, we first conducted a co-design study with 9 participants to distill a design space mapping vocal intonations to visual parameters (Geometry, Texture, and Motion). We developed EmoType-Zero, a research probe to operationalize these concepts. We then deployed this probe in a user study with 25 participants to empirically evaluate user perceptions. Results reveal a critical design trade-off: while the Combo mode (combining geometry, texture, and motion) achieved the highest emotional fidelity (mean ranking: M = 1.60, rank 1=best), its visual complexity compromised text legibility (M = 4.56). Consequently, participants preferred the subtle Geometry mode (M = 2.24) for daily usage, prioritizing clarity over high-intensity effects. Subjective feedback validated the probe’s utility, with 80% of participants agreeing that it improved emotional understanding and 72% expressing willingness to adopt the system for situational impairments and social modulation.
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-7
Open AccessEmoType-Zero: Adaptive Fonts to Render the Speech Emotion
Text messaging often lacks paralinguistic cues essential for interpreting emotional tone. To explore visual alternatives, we first conducted a co-design study with 9 participants to distill a design space mapping vocal intonations to visual parameters (Geometry, Texture, and Motion). We developed EmoType-Zero, a research probe to operationalize these concepts. We then deployed this probe in a user study with 25 participants to empirically evaluate user perceptions. Results reveal a critical design trade-off: while the Combo mode (combining geometry, texture, and motion) achieved the highest emotional fidelity (mean ranking: M = 1.60, rank 1=best), its visual complexity compromised text legibility (M = 4.56). Consequently, participants preferred the subtle Geometry mode (M = 2.24) for daily usage, prioritizing clarity over high-intensity effects. Subjective feedback validated the probe’s utility, with 80% of participants agreeing that it improved emotional understanding and 72% expressing willingness to adopt the system for situational impairments and social modulation.
- Open Access
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 1209-1216
MindChat-R0: A Large Language Model for Emotionally Supportive Dialogue through Reinforcement Learning
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 1209-1216
Open AccessMindChat-R0: A Large Language Model for Emotionally Supportive Dialogue through Reinforcement Learning
- Open Access
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 1633-1639
NursingLLM: A Retrieval-Augmented Health Chatbot for Supporting Nursing Home Residents
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 1633-1639
Open AccessNursingLLM: A Retrieval-Augmented Health Chatbot for Supporting Nursing Home Residents
- Open Access
- 3 September 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 9 (3) , 1-33
Motion2Press: Cross Model Learning from IMU to Plantar Pressure for Gait Analysis
Human locomotion is a fundamental aspect of daily life and a critical indicator of overall health. This study proposes a cross-modal learning framework based on inertial measurement units (IMUs) to infer plantar pressure distribution, ground reaction force (GRF), and center of pressure (COP). Traditional methods for pressure measurement rely on expensive and complex laboratory equipment, limiting their applicability in real-world scenarios. By integrating IMU data with deep learning techniques, this research achieves real-time, cost-effective, and accurate pressure estimation. The proposed method leverages IMU sensors to reconstruct foot pressure distribution in real-time, with high precision and adaptability to individual characteristics. A robust data pipeline and pretrained model enable accurate inference of GRF and COP using minimal IMU data, validated through case studies. Furthermore, The study builds a high-quality training dataset and optimizes the system for lightweight deployment, enhancing suitability for low-resource environments. Experimental results demonstrate the superior performance of the proposed approach across multiple evaluation metrics, particularly in individual adaptability, spatiotemporal feature capture, and deployment in resource-constrained settings. This work provides a novel and practical solution for clinical gait analysis, sports training, and health monitoring, while paving the way for advanced applications of intelligent devices in motion analysis.
- 3 September 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 9 (3) , 1-33
Open AccessMotion2Press: Cross Model Learning from IMU to Plantar Pressure for Gait Analysis
Human locomotion is a fundamental aspect of daily life and a critical indicator of overall health. This study proposes a cross-modal learning framework based on inertial measurement units (IMUs) to infer plantar pressure distribution, ground reaction force (GRF), and center of pressure (COP). Traditional methods for pressure measurement rely on expensive and complex laboratory equipment, limiting their applicability in real-world scenarios. By integrating IMU data with deep learning techniques, this research achieves real-time, cost-effective, and accurate pressure estimation. The proposed method leverages IMU sensors to reconstruct foot pressure distribution in real-time, with high precision and adaptability to individual characteristics. A robust data pipeline and pretrained model enable accurate inference of GRF and COP using minimal IMU data, validated through case studies. Furthermore, The study builds a high-quality training dataset and optimizes the system for lightweight deployment, enhancing suitability for low-resource environments. Experimental results demonstrate the superior performance of the proposed approach across multiple evaluation metrics, particularly in individual adaptability, spatiotemporal feature capture, and deployment in resource-constrained settings. This work provides a novel and practical solution for clinical gait analysis, sports training, and health monitoring, while paving the way for advanced applications of intelligent devices in motion analysis.