HHHayley Hung

Hayley Hung

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Hayley Hung has a total of 47 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    I Don’t Always Do What I Am Told: The Case for Task Incongruence-Aware Systems for Technology-Supported Reflection on Social Interactions

    by Chenxu Hao, Tiffany Matej Hrkalovic, Bernd Dudzik, Hayley Hung

    Technology-supported Self-Reflection (TSR) systems aim to facilitate learning through self-reflection. In the social domain, these systems often nudge individuals to perform better towards a certain outcome in a targeted task. Typically, this idea of support assumes that users pursue specific goals, which system creators anticipate will govern users’ behavior in targeted tasks. However, how goals shape behaviors in social interactions is complex: individuals’ goals can be highly idiosyncratic and multifaceted, sometimes contradictory to their best interest, and they are often not explicitly formulated in advance. We believe that viewing this complex landscape through the lens of a simple task formulation imposes limitations on the type of support these systems can provide. In this paper, we take a first step towards investigating the incongruence between assumed task goals and observed users’ behaviors and decisions in a partner selection setting, where participants need to select a partner for a cooperation task, and the apparent task goal can co-exist with other personal or relationship-oriented goals. First, we identify task-incongruent decisions, showing that even in a highly structured and incentivized setting, some choices deviate from the assumed task goal. Second, using a Large Language Model (LLM) to analyze conversational behavior, we explore topic-based and interactional indicators of task incongruence. Although the behavioral evidence is mixed, these findings still illustrate that what appears to be a reasonably assumed goal by the system may not be what users end up pursuing. In designing TSR systems, we believe that it is crucial to have capabilities for Task-Incongruence Awareness (TIA), enabling meaningful reflections on how users’ consciously or unconsciously self-ascribed goals are aligned with their behaviors and decisions.

  • Open Access

    Reflecti-Mate: A Conversational Agent for Adaptive Decision-Making Support Through System 1 and System 2 Thinking

    by Morita Tarvirdians, Senthil Chandrasegaran, Hayley Hung, Catholijn M. Jonker, Catharine Oertel

  • Open Access

    Reflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making

    by Morita Tarvirdians, Senthil Chandrasegaran, Hayley Hung, Catholijn M. Jonker, Catharine Oertel

    When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.

  • PARSEL: A Multimodal Dataset for Modeling Decision-Making Processes Involved in Selecting Partners for Joint Tasks

    by Tiffany Matej Hrkalovic, Bernd Dudzik, Daniel Balliet, Hayley Hung

    How people evaluate, select, and engage with others in cooperative settings significantly impacts their well-being, happiness, and success. However, navigating these processes is complex. Equipping systems with the ability to recognize, interpret, and even engage during such socio-cognitive processes can increase their potential to support humans in these socio-cognitive processes and be more successful in adjusting to the social environment they are embedded in (e.g., understanding human preferences and attitudes), leading to better quality interactions and decision-making for future partners. Yet, the developments of such systems depend on available datasets. However, based on our knowledge, no dataset exists that can be used to model partner selection for joint tasks. To support research focused on creating such intelligent systems, we introduce the PARSEL dataset – a comprehensive corpus of dyadic interactions designed for computational modeling of PARtner SELection processes and collaborative behavior. In total, 297 participants took part in the datasets. The dataset contains measurements of partner selection decisions over three different stages, as well as factors that may influence partner selection in the context of (online) social interactions. It includes audiovisual recordings that offer fine-grained behavioral cues used during these interactions, self-reported traits, and reported perceptions of person-, situation- and team-specific phenomena. By providing this resource, we aim to foster advancements in computational methods that can effectively model and augment socio-cognitive processes, contributing to socially aware intelligent systems and enhanced human-system interactions.


  • Open Access

    Partner perceptions during brief online interactions shape partner selection and cooperation

    by Tiffany Matej Hrkalovic, Bernd Dudzik, Hayley Hung, Daniel Balliet

    Evolutionary theory suggests that partner selection – the ability to identify and preferentially interact with individuals willing (warmth) and able (competence) to work towards mutual benefits – is a key driver of cooperative behavior. However, partner selection is complex, requiring the integration of various information, such as impression formation and task affordances. Despite its importance, there is limited research on the effect of these factors on partner selection for cooperative tasks. Thus, this paper investigates how person perceptions (warmth and competence), task affordances, and facial and acoustic nonverbal behavior inform partner selection for cooperative tasks. For this purpose, we asked participants to select partners for a task that either expressed warmth- or competence-related traits. Participants had a 3-minute (online) conversation with up to five individuals, reported their evaluations, selected partners for the task, and then engaged in the task. Results indicate that person perceptions guide partner selection, with each trait being more predictive in relevant tasks. Additionally, we found that the perceptions of warmth, but not competence, can be predicted by facial and acoustic cues during conversations. Lastly, we find that in the context of online social interactions, individuals were more cooperative towards selected participants than unselected. We discuss these implications in the context of the theory of partner selection and offer insights on how these results can be used in future efforts for designing socially intelligent artificial systems that support partner selection decisions.


  • Open Access

    Technologies Supporting Self-Reflection on Social Interactions: A Systematic Review

    by Chenxu Hao, Tiffany Matej Hrkalovic, Daniel Balliet, Hayley Hung, Bernd Dudzik


  • Open Access

    How Private is Low-Frequency Speech Audio in the Wild? An Analysis of Verbal Intelligibility by Humans and Machines

    by Ailin Liu, Pepijn Vunderink, Jose Vargas Quiros, Chirag Raman, Hayley Hung

    Low-frequency audio has been proposed as a promising privacy-preserving modality to study social dynamics in real-world settings. To this end, researchers have developed wearable devices that can record audio at frequencies as low as 1250 Hz to mitigate the automatic extraction of the verbal content of speech that may contain private details. This paper investigates the validity of this hypothesis, examining the degree to which low-frequency speech ensures verbal privacy. It includes simulating a potential privacy attack in various noise environments. Further, it explores the trade-off between the performance of voice activity detection, which is fundamental for understanding social behavior, and privacy-preservation. The evaluation incorporates subjective human intelligibility and automatic speech recognition performance, comprehensively analyzing the delicate balance between effective social behavior analysis and preserving verbal privacy.

  • Open Access

    REWIND Dataset: Privacy-preserving Speaking Status Segmentation from Multimodal Body Movement Signals in the Wild

    by Jose Vargas Quiros, Chirag Raman, Stephanie Tan, Ekin Gedik, Laura Cabrera-Quiros, Hayley Hung

    Recognizing speaking in humans is a central task towards understanding social interactions. Ideally, speaking would be detected from individual voice recordings, as done previously for meeting scenarios. However, individual voice recordings are hard to obtain in the wild, especially in crowded mingling scenarios due to cost, logistics, and privacy concerns. As an alternative, machine learning models trained on video and wearable sensor data make it possible to recognize speech by detecting its related gestures in an unobtrusive, privacy-preserving way. These models themselves should ideally be trained using labels obtained from the speech signal. However, existing mingling datasets do not contain high quality audio recordings. Instead, speaking status annotations have often been inferred by human annotators from video, without validation of this approach against audio-based ground truth. In this paper we revisit no-audio speaking status estimation by presenting the first publicly available multimodal dataset with high-quality individual speech recordings of 33 subjects in a professional networking event. We present three baselines for no-audio speaking status segmentation: a) from video, b) from body acceleration (chest-worn accelerometer), c) from body pose tracks. In all cases we predict a 20Hz binary speaking status signal extracted from the audio, a time resolution not available in previous datasets. In addition to providing the signals and ground truth necessary to evaluate a wide range of speaking status detection methods, the availability of audio in REWIND makes it suitable for cross-modality studies not feasible with previous mingling datasets. Finally, our flexible data consent setup creates new challenges for multimodal systems under missing modalities.

  • Open Access

    A Multimodal Social Signal Processing Approach to Team Interactions

    by Nale Lehmann-Willenbrock, Hayley Hung

    Social signal processing develops automated approaches to detect, analyze, and synthesize social signals in human–human as well as human–machine interactions by means of machine learning and sensor data processing. Most works analyze individual or dyadic behavior, while the analysis of group or team interactions remains limited. We present a case study of an interdisciplinary work process for social signal processing that can develop automatized measures of complex team interaction dynamics, using team task and social cohesion as an example. In a field sample of 25 real project team meetings, we obtained sensor data from cameras, microphones, and a smart ID badge measuring acceleration. We demonstrate how fine-grained behavioral expressions of task and social cohesion in team meetings can be extracted and processed from sensor data by capturing dyadic coordination patterns that are then aggregated to the team level. The extracted patterns act as proxies for behavioral synchrony and mimicry of speech and body behavior which map onto verbal expressions of task and social cohesion in the observed team meetings. We reflect on opportunities for future interdisciplinary or collaboration that can move beyond a simple producer–consumer model.