Hayley Hung
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Hayley Hung has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
- Chirag Raman
Delft University of Technology
7 shared publications - Ekin Gedik
Delft University of Technology Delft the Netherlands
7 shared publications - Laura Cabrera-Quiros
Intelligent Systems, Technische Universiteit Delft, 2860 Delft, Delft Netherlands
6 shared publications - Bernd Dudzik
Intelligent Systems/Pattern Recognition & Bioinformatics, Delft University of Technology, Delft, Netherlands
5 shared publications - Guangliang Li
Ocean University of China
4 shared publications - Shimon Whiteson
University of Oxford
4 shared publications
Latest Publications
- Open Access
- 4 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 428-436
I Don’t Always Do What I Am Told: The Case for Task Incongruence-Aware Systems for Technology-Supported Reflection on Social Interactions
Technology-supported Self-Reflection (TSR) systems aim to facilitate learning through self-reflection. In the social domain, these systems often nudge individuals to perform better towards a certain outcome in a targeted task. Typically, this idea of support assumes that users pursue specific goals, which system creators anticipate will govern users’ behavior in targeted tasks. However, how goals shape behaviors in social interactions is complex: individuals’ goals can be highly idiosyncratic and multifaceted, sometimes contradictory to their best interest, and they are often not explicitly formulated in advance. We believe that viewing this complex landscape through the lens of a simple task formulation imposes limitations on the type of support these systems can provide. In this paper, we take a first step towards investigating the incongruence between assumed task goals and observed users’ behaviors and decisions in a partner selection setting, where participants need to select a partner for a cooperation task, and the apparent task goal can co-exist with other personal or relationship-oriented goals. First, we identify task-incongruent decisions, showing that even in a highly structured and incentivized setting, some choices deviate from the assumed task goal. Second, using a Large Language Model (LLM) to analyze conversational behavior, we explore topic-based and interactional indicators of task incongruence. Although the behavioral evidence is mixed, these findings still illustrate that what appears to be a reasonably assumed goal by the system may not be what users end up pursuing. In designing TSR systems, we believe that it is crucial to have capabilities for Task-Incongruence Awareness (TIA), enabling meaningful reflections on how users’ consciously or unconsciously self-ascribed goals are aligned with their behaviors and decisions.
- 4 October 2026
- Published by Association for Computing Machinery (ACM)
- p. 428-436
Open AccessI Don’t Always Do What I Am Told: The Case for Task Incongruence-Aware Systems for Technology-Supported Reflection on Social Interactions
Technology-supported Self-Reflection (TSR) systems aim to facilitate learning through self-reflection. In the social domain, these systems often nudge individuals to perform better towards a certain outcome in a targeted task. Typically, this idea of support assumes that users pursue specific goals, which system creators anticipate will govern users’ behavior in targeted tasks. However, how goals shape behaviors in social interactions is complex: individuals’ goals can be highly idiosyncratic and multifaceted, sometimes contradictory to their best interest, and they are often not explicitly formulated in advance. We believe that viewing this complex landscape through the lens of a simple task formulation imposes limitations on the type of support these systems can provide. In this paper, we take a first step towards investigating the incongruence between assumed task goals and observed users’ behaviors and decisions in a partner selection setting, where participants need to select a partner for a cooperation task, and the apparent task goal can co-exist with other personal or relationship-oriented goals. First, we identify task-incongruent decisions, showing that even in a highly structured and incentivized setting, some choices deviate from the assumed task goal. Second, using a Large Language Model (LLM) to analyze conversational behavior, we explore topic-based and interactional indicators of task incongruence. Although the behavioral evidence is mixed, these findings still illustrate that what appears to be a reasonably assumed goal by the system may not be what users end up pursuing. In designing TSR systems, we believe that it is crucial to have capabilities for Task-Incongruence Awareness (TIA), enabling meaningful reflections on how users’ consciously or unconsciously self-ascribed goals are aligned with their behaviors and decisions.
- Open Access
- 7 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 213-222
Reflecti-Mate: A Conversational Agent for Adaptive Decision-Making Support Through System 1 and System 2 Thinking
- 7 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 213-222
Open AccessReflecti-Mate: A Conversational Agent for Adaptive Decision-Making Support Through System 1 and System 2 Thinking
- Open Access
Reflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making
When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.
Open AccessReflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making
When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.
- 19 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 16 (4) , 3481-3498
PARSEL: A Multimodal Dataset for Modeling Decision-Making Processes Involved in Selecting Partners for Joint Tasks
How people evaluate, select, and engage with others in cooperative settings significantly impacts their well-being, happiness, and success. However, navigating these processes is complex. Equipping systems with the ability to recognize, interpret, and even engage during such socio-cognitive processes can increase their potential to support humans in these socio-cognitive processes and be more successful in adjusting to the social environment they are embedded in (e.g., understanding human preferences and attitudes), leading to better quality interactions and decision-making for future partners. Yet, the developments of such systems depend on available datasets. However, based on our knowledge, no dataset exists that can be used to model partner selection for joint tasks. To support research focused on creating such intelligent systems, we introduce the PARSEL dataset – a comprehensive corpus of dyadic interactions designed for computational modeling of PARtner SELection processes and collaborative behavior. In total, 297 participants took part in the datasets. The dataset contains measurements of partner selection decisions over three different stages, as well as factors that may influence partner selection in the context of (online) social interactions. It includes audiovisual recordings that offer fine-grained behavioral cues used during these interactions, self-reported traits, and reported perceptions of person-, situation- and team-specific phenomena. By providing this resource, we aim to foster advancements in computational methods that can effectively model and augment socio-cognitive processes, contributing to socially aware intelligent systems and enhanced human-system interactions.
- 19 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Affective Computing
- Vol. 16 (4) , 3481-3498
PARSEL: A Multimodal Dataset for Modeling Decision-Making Processes Involved in Selecting Partners for Joint Tasks
How people evaluate, select, and engage with others in cooperative settings significantly impacts their well-being, happiness, and success. However, navigating these processes is complex. Equipping systems with the ability to recognize, interpret, and even engage during such socio-cognitive processes can increase their potential to support humans in these socio-cognitive processes and be more successful in adjusting to the social environment they are embedded in (e.g., understanding human preferences and attitudes), leading to better quality interactions and decision-making for future partners. Yet, the developments of such systems depend on available datasets. However, based on our knowledge, no dataset exists that can be used to model partner selection for joint tasks. To support research focused on creating such intelligent systems, we introduce the PARSEL dataset – a comprehensive corpus of dyadic interactions designed for computational modeling of PARtner SELection processes and collaborative behavior. In total, 297 participants took part in the datasets. The dataset contains measurements of partner selection decisions over three different stages, as well as factors that may influence partner selection in the context of (online) social interactions. It includes audiovisual recordings that offer fine-grained behavioral cues used during these interactions, self-reported traits, and reported perceptions of person-, situation- and team-specific phenomena. By providing this resource, we aim to foster advancements in computational methods that can effectively model and augment socio-cognitive processes, contributing to socially aware intelligent systems and enhanced human-system interactions.
- Open Access
- 9 April 2025
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 20 (4) , e0318137
Partner perceptions during brief online interactions shape partner selection and cooperation
Evolutionary theory suggests that partner selection – the ability to identify and preferentially interact with individuals willing (warmth) and able (competence) to work towards mutual benefits – is a key driver of cooperative behavior. However, partner selection is complex, requiring the integration of various information, such as impression formation and task affordances. Despite its importance, there is limited research on the effect of these factors on partner selection for cooperative tasks. Thus, this paper investigates how person perceptions (warmth and competence), task affordances, and facial and acoustic nonverbal behavior inform partner selection for cooperative tasks. For this purpose, we asked participants to select partners for a task that either expressed warmth- or competence-related traits. Participants had a 3-minute (online) conversation with up to five individuals, reported their evaluations, selected partners for the task, and then engaged in the task. Results indicate that person perceptions guide partner selection, with each trait being more predictive in relevant tasks. Additionally, we found that the perceptions of warmth, but not competence, can be predicted by facial and acoustic cues during conversations. Lastly, we find that in the context of online social interactions, individuals were more cooperative towards selected participants than unselected. We discuss these implications in the context of the theory of partner selection and offer insights on how these results can be used in future efforts for designing socially intelligent artificial systems that support partner selection decisions.
- 9 April 2025
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 20 (4) , e0318137
Open AccessPartner perceptions during brief online interactions shape partner selection and cooperation
Evolutionary theory suggests that partner selection – the ability to identify and preferentially interact with individuals willing (warmth) and able (competence) to work towards mutual benefits – is a key driver of cooperative behavior. However, partner selection is complex, requiring the integration of various information, such as impression formation and task affordances. Despite its importance, there is limited research on the effect of these factors on partner selection for cooperative tasks. Thus, this paper investigates how person perceptions (warmth and competence), task affordances, and facial and acoustic nonverbal behavior inform partner selection for cooperative tasks. For this purpose, we asked participants to select partners for a task that either expressed warmth- or competence-related traits. Participants had a 3-minute (online) conversation with up to five individuals, reported their evaluations, selected partners for the task, and then engaged in the task. Results indicate that person perceptions guide partner selection, with each trait being more predictive in relevant tasks. Additionally, we found that the perceptions of warmth, but not competence, can be predicted by facial and acoustic cues during conversations. Lastly, we find that in the context of online social interactions, individuals were more cooperative towards selected participants than unselected. We discuss these implications in the context of the theory of partner selection and offer insights on how these results can be used in future efforts for designing socially intelligent artificial systems that support partner selection decisions.
- Open Access
- 24 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 1354-1365
Technologies Supporting Self-Reflection on Social Interactions: A Systematic Review
- 24 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 1354-1365
Open AccessTechnologies Supporting Self-Reflection on Social Interactions: A Systematic Review
- Open Access
How Private is Low-Frequency Speech Audio in the Wild? An Analysis of Verbal Intelligibility by Humans and Machines
Low-frequency audio has been proposed as a promising privacy-preserving modality to study social dynamics in real-world settings. To this end, researchers have developed wearable devices that can record audio at frequencies as low as 1250 Hz to mitigate the automatic extraction of the verbal content of speech that may contain private details. This paper investigates the validity of this hypothesis, examining the degree to which low-frequency speech ensures verbal privacy. It includes simulating a potential privacy attack in various noise environments. Further, it explores the trade-off between the performance of voice activity detection, which is fundamental for understanding social behavior, and privacy-preservation. The evaluation incorporates subjective human intelligibility and automatic speech recognition performance, comprehensively analyzing the delicate balance between effective social behavior analysis and preserving verbal privacy.
Open AccessHow Private is Low-Frequency Speech Audio in the Wild? An Analysis of Verbal Intelligibility by Humans and Machines
Low-frequency audio has been proposed as a promising privacy-preserving modality to study social dynamics in real-world settings. To this end, researchers have developed wearable devices that can record audio at frequencies as low as 1250 Hz to mitigate the automatic extraction of the verbal content of speech that may contain private details. This paper investigates the validity of this hypothesis, examining the degree to which low-frequency speech ensures verbal privacy. It includes simulating a potential privacy attack in various noise environments. Further, it explores the trade-off between the performance of voice activity detection, which is fundamental for understanding social behavior, and privacy-preservation. The evaluation incorporates subjective human intelligibility and automatic speech recognition performance, comprehensively analyzing the delicate balance between effective social behavior analysis and preserving verbal privacy.
- Open Access
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-9
The Discontent with Intent Estimation In-the-Wild: The Case for Unrealized Intentions
- 11 May 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-9
Open AccessThe Discontent with Intent Estimation In-the-Wild: The Case for Unrealized Intentions
- Open Access
REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild
Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.
Open AccessREWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild
Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.
- Open Access
- 5 October 2023
- Published by SAGE Publications in Organizational Research Methods
- Vol. 27 (3) , 477-515
A Multimodal Social Signal Processing Approach to Team Interactions
Social signal processing develops automated approaches to detect, analyze, and synthesize social signals in human–human as well as human–machine interactions by means of machine learning and sensor data processing. Most works analyze individual or dyadic behavior, while the analysis of group or team interactions remains limited. We present a case study of an interdisciplinary work process for social signal processing that can develop automatized measures of complex team interaction dynamics, using team task and social cohesion as an example. In a field sample of 25 real project team meetings, we obtained sensor data from cameras, microphones, and a smart ID badge measuring acceleration. We demonstrate how fine-grained behavioral expressions of task and social cohesion in team meetings can be extracted and processed from sensor data by capturing dyadic coordination patterns that are then aggregated to the team level. The extracted patterns act as proxies for behavioral synchrony and mimicry of speech and body behavior which map onto verbal expressions of task and social cohesion in the observed team meetings. We reflect on opportunities for future interdisciplinary or collaboration that can move beyond a simple producer–consumer model.
- 5 October 2023
- Published by SAGE Publications in Organizational Research Methods
- Vol. 27 (3) , 477-515
Open AccessA Multimodal Social Signal Processing Approach to Team Interactions
Social signal processing develops automated approaches to detect, analyze, and synthesize social signals in human–human as well as human–machine interactions by means of machine learning and sensor data processing. Most works analyze individual or dyadic behavior, while the analysis of group or team interactions remains limited. We present a case study of an interdisciplinary work process for social signal processing that can develop automatized measures of complex team interaction dynamics, using team task and social cohesion as an example. In a field sample of 25 real project team meetings, we obtained sensor data from cameras, microphones, and a smart ID badge measuring acceleration. We demonstrate how fine-grained behavioral expressions of task and social cohesion in team meetings can be extracted and processed from sensor data by capturing dyadic coordination patterns that are then aggregated to the team level. The extracted patterns act as proxies for behavioral synchrony and mimicry of speech and body behavior which map onto verbal expressions of task and social cohesion in the observed team meetings. We reflect on opportunities for future interdisciplinary or collaboration that can move beyond a simple producer–consumer model.