Yunpeng Song
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Yunpeng Song has a total of 44 co-authors. Here are the top co-authors by number of shared publications:
- Zhongmin Cai
Xi'an Jiaotong University, Xi'an, China
12 shared publications - Rongrong Zhu
MOE KLINNS Lab, and Xi'an Jiaotong University, Xi'an, Shaanxi, China
3 shared publications - Yiheng Bian
MOE KLINNS Lab, Xi'an Jiaotong University, China
3 shared publications - Bin Wu
College of Culture and Sports, Korea University, Seoul, Republic of Korea
2 shared publications - Guiyu Ma
MOE KLINNS Lab, Xi'an Jiaotong University, Xi'an, China
2 shared publications - Jie Xu
Wuhan University
2 shared publications
Latest Publications
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-8
When Verse Listens Back: Classical Chinese Poetry as a Culturally Grounded Medium for Multimodal AI-Guided Emotional Support
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-8
Open AccessWhen Verse Listens Back: Classical Chinese Poetry as a Culturally Grounded Medium for Multimodal AI-Guided Emotional Support
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-24
DroidRetriever: A Transparent and Steerable Automation System for Collaborative Mobile Information Seeking
Information seeking on mobile devices is often fragmented, trapping users in repetitive cycles of context switching and data re-entry, which increases cognitive load and disrupts workflow. Existing mobile agents provide limited cross-source integration and are largely opaque, presenting progress as a linear feed with few opportunities to intervene, steer, or take control. We present DroidRetriever, a transparent, steerable system for cross-source mobile information seeking. It accepts voice or typed input and the multi-LLM system decomposes the task, navigates to target pages, takes screenshots, and synthesizes a concise report with citation-linked screenshots. We make the process transparent through a progress dashboard combining sub-task progress and real-time exploration maps for seamless takeover. DroidRetriever also pauses on detected privacy or high-risk screens and prompts intervention. Across 35 tasks over 24 apps, experiments and user studies demonstrate improvements in coverage, transparency, and reduced workload. We release our code at https://github.com/AkimotoAyako/DroidRetriever.
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-24
Open AccessDroidRetriever: A Transparent and Steerable Automation System for Collaborative Mobile Information Seeking
Information seeking on mobile devices is often fragmented, trapping users in repetitive cycles of context switching and data re-entry, which increases cognitive load and disrupts workflow. Existing mobile agents provide limited cross-source integration and are largely opaque, presenting progress as a linear feed with few opportunities to intervene, steer, or take control. We present DroidRetriever, a transparent, steerable system for cross-source mobile information seeking. It accepts voice or typed input and the multi-LLM system decomposes the task, navigates to target pages, takes screenshots, and synthesizes a concise report with citation-linked screenshots. We make the process transparent through a progress dashboard combining sub-task progress and real-time exploration maps for seamless takeover. DroidRetriever also pauses on detected privacy or high-risk screens and prompts intervention. Across 35 tasks over 24 apps, experiments and user studies demonstrate improvements in coverage, transparency, and reduced workload. We release our code at https://github.com/AkimotoAyako/DroidRetriever.
- Open Access
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-20
AReframedChair: Reframing the Empty Chair through Dyadic and Triadic AR-Mediated Self-Embodiment
- 13 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-20
Open AccessAReframedChair: Reframing the Empty Chair through Dyadic and Triadic AR-Mediated Self-Embodiment
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Network Science and Engineering
- Vol. 13, 11116-11132
ShadowGuard: An Agentic Edge Framework for On-Device Detection of Mobile Dark Patterns
- 1 January 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Network Science and Engineering
- Vol. 13, 11116-11132
ShadowGuard: An Agentic Edge Framework for On-Device Detection of Mobile Dark Patterns
- Open Access
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 419-424
Poemithy: Leveraging Multimodal LLMs for Emotional Healing through Classical Chinese Poetry
- 12 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 419-424
Open AccessPoemithy: Leveraging Multimodal LLMs for Emotional Healing through Classical Chinese Poetry
- Open Access
- 3 September 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 9 (3) , 1-41
XRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses
Human Action Recognition (HAR) plays a crucial role in applications such as health monitoring, smart home automation, and human-computer interaction. While HAR has been extensively studied, action summarization using Wi-Fi and IMU signals in smart-home environments, which involves identifying and summarizing continuous actions, remains an emerging task. This paper introduces the novel XRF V2 dataset, designed for indoor daily activity Temporal Action Localization (TAL) and Action Summarization. XRF V2 integrates multimodal data from Wi-Fi signals, IMU sensors (smartphones, smartwatches, earbuds, and smart glasses), and synchronized video recordings, offering a diverse collection of indoor activities from 16 volunteers across three environments. To tackle TAL and action summarization, we propose the XRFMamba neural network, which captures long-term dependencies in untrimmed sensory sequences and achieves the best performance with an average mAP of 78.74, outperforming the recent WiFiTAD by 5.49 points in mAP@avg while using 35% fewer parameters. In action summarization, we introduce a new metric, Response Meaning Consistency (RMC), to evaluate performance. It achieves an average RMC (mRMC) of 0.802. We envision XRF V2 as a valuable resource for advancing research in human action localization, action forecasting, pose estimation, multimodal foundation model pre-training, synthetic data generation, and more. The data and code are available at https://github.com/aiotgroup/XRFV2.
- 3 September 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 9 (3) , 1-41
Open AccessXRF V2: A Dataset for Action Summarization with Wi-Fi Signals, and IMUs in Phones, Watches, Earbuds, and Glasses
Human Action Recognition (HAR) plays a crucial role in applications such as health monitoring, smart home automation, and human-computer interaction. While HAR has been extensively studied, action summarization using Wi-Fi and IMU signals in smart-home environments, which involves identifying and summarizing continuous actions, remains an emerging task. This paper introduces the novel XRF V2 dataset, designed for indoor daily activity Temporal Action Localization (TAL) and Action Summarization. XRF V2 integrates multimodal data from Wi-Fi signals, IMU sensors (smartphones, smartwatches, earbuds, and smart glasses), and synchronized video recordings, offering a diverse collection of indoor activities from 16 volunteers across three environments. To tackle TAL and action summarization, we propose the XRFMamba neural network, which captures long-term dependencies in untrimmed sensory sequences and achieves the best performance with an average mAP of 78.74, outperforming the recent WiFiTAD by 5.49 points in mAP@avg while using 35% fewer parameters. In action summarization, we introduce a new metric, Response Meaning Consistency (RMC), to evaluate performance. It achieves an average RMC (mRMC) of 0.802. We envision XRF V2 as a valuable resource for advancing research in human action localization, action forecasting, pose estimation, multimodal foundation model pre-training, synthetic data generation, and more. The data and code are available at https://github.com/aiotgroup/XRFV2.
- Open Access
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
EyeSee: Enhancing Art Appreciation through Anthropomorphic Interpretations from Multiple Perspectives
- 25 April 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-23
Open AccessEyeSee: Enhancing Art Appreciation through Anthropomorphic Interpretations from Multiple Perspectives
- Open Access
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (1) , 764-772
Predicting User Behavior in Smart Spaces with LLM-Enhanced Logs and Personalized Prompts
Enhancing the intelligence of smart systems, such as smart homes, smart vehicles, and smart grids, critically depends on developing sophisticated planning capabilities that can anticipate the next desired function based on historical interactions. While existing methods view user behaviors as sequential data and apply models like RNNs and Transformers to predict future actions, they often fail to incorporate domain knowledge and capture personalized user preferences. In this paper, we propose a novel approach that incorporates LLM-enhanced logs and personalized prompts. Our approach first constructs a graph that captures individual behavior preferences derived from their interaction histories. This graph effectively transforms into a soft continuous prompt that precedes the sequence of user behaviors. Then our approach leverages the vast general knowledge and robust reasoning capabilities of a pretrained LLM to enrich the oversimplified and incomplete log records. By enhancing these logs semantically, our approach better understands the user's actions and intentions, especially for those rare events in the dataset. We evaluate the method across four real-world datasets from both smart vehicle and smart home settings. The findings validate the effectiveness of our LLM-enhanced description and personalized prompt, shedding light on potential ways to advance the intelligence of smart space.
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (1) , 764-772
Open AccessPredicting User Behavior in Smart Spaces with LLM-Enhanced Logs and Personalized Prompts
Enhancing the intelligence of smart systems, such as smart homes, smart vehicles, and smart grids, critically depends on developing sophisticated planning capabilities that can anticipate the next desired function based on historical interactions. While existing methods view user behaviors as sequential data and apply models like RNNs and Transformers to predict future actions, they often fail to incorporate domain knowledge and capture personalized user preferences. In this paper, we propose a novel approach that incorporates LLM-enhanced logs and personalized prompts. Our approach first constructs a graph that captures individual behavior preferences derived from their interaction histories. This graph effectively transforms into a soft continuous prompt that precedes the sequence of user behaviors. Then our approach leverages the vast general knowledge and robust reasoning capabilities of a pretrained LLM to enrich the oversimplified and incomplete log records. By enhancing these logs semantically, our approach better understands the user's actions and intentions, especially for those rare events in the dataset. We evaluate the method across four real-world datasets from both smart vehicle and smart home settings. The findings validate the effectiveness of our LLM-enhanced description and personalized prompt, shedding light on potential ways to advance the intelligence of smart space.
- 25 January 2025
- Published by Springer Nature
- p. 187-201
Sentiment Caption Generation from Visual Scene Using Pre-trained Language Model
- 25 January 2025
- Published by Springer Nature
- p. 187-201
Sentiment Caption Generation from Visual Scene Using Pre-trained Language Model
- Open Access
- 21 November 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 8 (4) , 1-28
Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information Seeking
Extracting and collecting information during web exploration is an arduous process, demanding substantial cognitive and physical effort from users. Users must not only determine which content is worth capturing but also manually extract and save it, often disrupting the flow of their learning and exploration. To mitigate it, we propose AIbM (Attentive Interaction Behavior Modeling), which encodes and integrates multiple implicit interaction data to identify fine-grained helpful web content, providing an algorithmic foundation to facilitate automatic information extraction and collection. AIbM captures dynamic patterns of user interactions---specifically eye and mouse movements---by encoding these into images and time series. It further employs graph modeling to exploit relationships between a content block and its adjacent blocks. By incorporating graph attention networks, AIbM efficiently processes both intra-block and inter-block features, improving the identification of helpful web content. Experimental results demonstrate that AIbM achieves the highest F1 score of 82.80% in general helpful content identification, marking improvements of 10.86% over state-of-the-art models like GBT and at least 2.49% over individual modality inputs. These results underscore the potential of our approach to advance tools and systems designed for automated information extraction and collection to facilitate more effective exploratory search experiences.
- 21 November 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 8 (4) , 1-28
Open AccessModeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information Seeking
Extracting and collecting information during web exploration is an arduous process, demanding substantial cognitive and physical effort from users. Users must not only determine which content is worth capturing but also manually extract and save it, often disrupting the flow of their learning and exploration. To mitigate it, we propose AIbM (Attentive Interaction Behavior Modeling), which encodes and integrates multiple implicit interaction data to identify fine-grained helpful web content, providing an algorithmic foundation to facilitate automatic information extraction and collection. AIbM captures dynamic patterns of user interactions---specifically eye and mouse movements---by encoding these into images and time series. It further employs graph modeling to exploit relationships between a content block and its adjacent blocks. By incorporating graph attention networks, AIbM efficiently processes both intra-block and inter-block features, improving the identification of helpful web content. Experimental results demonstrate that AIbM achieves the highest F1 score of 82.80% in general helpful content identification, marking improvements of 10.86% over state-of-the-art models like GBT and at least 2.49% over individual modality inputs. These results underscore the potential of our approach to advance tools and systems designed for automated information extraction and collection to facilitate more effective exploratory search experiences.