Catholijn M. Jonker
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Catholijn M. Jonker has a total of 184 co-authors. Here are the top co-authors by number of shared publications:
- Koen V. Hindriks
Vrije University Amsterdam, Amsterdam, Netherlands
21 shared publications - Pradeep Kumar Murukannaiah
Delft University of Technology, Delft, The Netherlands
14 shared publications - Tim Baarslag
Centrum Wiskunde & Informatica
14 shared publications - Myrthe Lotte Tielman
Delft University of Technology, Delft, Netherlands
11 shared publications - Reyhan Aydoğan
Ozyegin University
10 shared publications - Carolina Centeio Jorge
Delft University of Technology
6 shared publications
Latest Publications
- Open Access
- 25 September 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Human-Robot Interaction
- Vol. 15 (5) , 1-22
“I Know You’re Capable, but Are You Willing?”: Allocating Tasks in Human–Machine Teams
In human–machine teams, task allocation algorithms typically prioritize competence and other performance-related factors, overlooking human willingness to perform specific tasks. A recent framework proposed incorporating both competence and willingness into task allocation, but its effects on teamwork outcomes remain untested. This article investigates the impact of willingness-based task allocation in a simulated search and rescue (SAR) setting. We conducted a 2 × 2 user study (N = 72) in a 2D grid-world SAR scenario, where human participants collaborated with two virtual robots, which either used a willingness-based or a balanced task allocation strategy. Half the participants were able to change the task allocation prior to the start of the mission. Participants preferred robots and plans that accounted for their preferences, yet willingness-based allocation did not significantly influence trust in the robot or perceived team fluency, as these remained high throughout. Moreover, allowing changes to task allocation improved human performance. Our findings also suggest that the benefits of incorporating willingness may be context-dependent, with greater potential in lower-stakes tasks and longer-term teams.
- 25 September 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Human-Robot Interaction
- Vol. 15 (5) , 1-22
Open Access“I Know You’re Capable, but Are You Willing?”: Allocating Tasks in Human–Machine Teams
In human–machine teams, task allocation algorithms typically prioritize competence and other performance-related factors, overlooking human willingness to perform specific tasks. A recent framework proposed incorporating both competence and willingness into task allocation, but its effects on teamwork outcomes remain untested. This article investigates the impact of willingness-based task allocation in a simulated search and rescue (SAR) setting. We conducted a 2 × 2 user study (N = 72) in a 2D grid-world SAR scenario, where human participants collaborated with two virtual robots, which either used a willingness-based or a balanced task allocation strategy. Half the participants were able to change the task allocation prior to the start of the mission. Participants preferred robots and plans that accounted for their preferences, yet willingness-based allocation did not significantly influence trust in the robot or perceived team fluency, as these remained high throughout. Moreover, allowing changes to task allocation improved human performance. Our findings also suggest that the benefits of incorporating willingness may be context-dependent, with greater potential in lower-stakes tasks and longer-term teams.
- Open Access
- 2 July 2026
- Published by IOS Press
Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human–AI workflows has stayed behind. This work evaluates a chatbot-style assistant based on Retrieval-Augmented Generation (RAG) in a realistic multi-turn information-seeking scenario inspired by workplace settings where compliance with local legislation and secure handling of sensitive data are often key. Specifically, we examine the performance of humans (N=112) assisted by RAG-assistants compared to LLM-only or LLM+RAG baselines. In this setting, we investigate how underlying model size (3B, 8B, and 70B) shapes the human-AI collaborative dynamic and how it influences perceived usability and satisfaction. Results show that the performance gain of human-AI collaboration over the model-only baselines is significant, irrespective of model size, suggesting that hybrid systems are beneficial in information-seeking scenarios. Interestingly, however, perceived usability and satisfaction among participants showed little difference across model sizes. This demonstrates a nuanced trade-off between model size, performance, and user perception. Our work highlights the added value of evaluating AI applications in actual multi-turn interactions with human users, looking at usability and satisfaction besides accuracy, rather than focusing on benchmark performance only.
- 2 July 2026
- Published by IOS Press
Open AccessSeeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?
Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human–AI workflows has stayed behind. This work evaluates a chatbot-style assistant based on Retrieval-Augmented Generation (RAG) in a realistic multi-turn information-seeking scenario inspired by workplace settings where compliance with local legislation and secure handling of sensitive data are often key. Specifically, we examine the performance of humans (N=112) assisted by RAG-assistants compared to LLM-only or LLM+RAG baselines. In this setting, we investigate how underlying model size (3B, 8B, and 70B) shapes the human-AI collaborative dynamic and how it influences perceived usability and satisfaction. Results show that the performance gain of human-AI collaboration over the model-only baselines is significant, irrespective of model size, suggesting that hybrid systems are beneficial in information-seeking scenarios. Interestingly, however, perceived usability and satisfaction among participants showed little difference across model sizes. This demonstrates a nuanced trade-off between model size, performance, and user perception. Our work highlights the added value of evaluating AI applications in actual multi-turn interactions with human users, looking at usability and satisfaction besides accuracy, rather than focusing on benchmark performance only.
- Open Access
- 7 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 213-222
Reflecti-Mate: A Conversational Agent for Adaptive Decision-Making Support Through System 1 and System 2 Thinking
- 7 June 2026
- Published by Association for Computing Machinery (ACM)
- p. 213-222
Open AccessReflecti-Mate: A Conversational Agent for Adaptive Decision-Making Support Through System 1 and System 2 Thinking
- Open Access
- 18 February 2026
- Published by Springer Nature in Autonomous Agents and Multi-Agent Systems
- Vol. 40 (1) , 10
From human teams to hybrid intelligence teams: identifying, characterizing, and evaluating foundational quality attributes
Hybrid Intelligence (HI) is an emerging paradigm in which artificial intelligence (AI) augments human intelligence. The current literature lacks systematic models that guide the design and evaluation of HI systems. Further, discussions around HI primarily focus on technology, neglecting the holistic human-AI ensemble. In this paper, we take the initial steps toward the development of a quality model for characterizing and evaluating HI systems from a human-AI teams perspective. We first conducted a study investigating the adequacy of properties commonly associated with effective human teams to describe HI. The study features the insights of 50 HI researchers, and shows that various human team properties, including boundedness, interdependence, competency, purposefulness, initiative, normativity, and effectiveness, are important for HI systems. Based on these results, we developed a quality model for HI teams composed of seven high-level quality attributes, further refined into 16 specific ones. To evaluate the relevance and understanding of the proposed attributes, we conducted a second empirical investigation by staging competitions in which participants used the quality model to develop and analyze HI usage scenarios. Our analysis of 48 collected scenarios, which we openly release, confirms the proposed attributes’ relevance and highlights insights that emerge when designers consider the quality model in HI system design.
- 18 February 2026
- Published by Springer Nature in Autonomous Agents and Multi-Agent Systems
- Vol. 40 (1) , 10
Open AccessFrom human teams to hybrid intelligence teams: identifying, characterizing, and evaluating foundational quality attributes
Hybrid Intelligence (HI) is an emerging paradigm in which artificial intelligence (AI) augments human intelligence. The current literature lacks systematic models that guide the design and evaluation of HI systems. Further, discussions around HI primarily focus on technology, neglecting the holistic human-AI ensemble. In this paper, we take the initial steps toward the development of a quality model for characterizing and evaluating HI systems from a human-AI teams perspective. We first conducted a study investigating the adequacy of properties commonly associated with effective human teams to describe HI. The study features the insights of 50 HI researchers, and shows that various human team properties, including boundedness, interdependence, competency, purposefulness, initiative, normativity, and effectiveness, are important for HI systems. Based on these results, we developed a quality model for HI teams composed of seven high-level quality attributes, further refined into 16 specific ones. To evaluate the relevance and understanding of the proposed attributes, we conducted a second empirical investigation by staging competitions in which participants used the quality model to develop and analyze HI usage scenarios. Our analysis of 48 collected scenarios, which we openly release, confirms the proposed attributes’ relevance and highlights insights that emerge when designers consider the quality model in HI system design.
- 15 October 2025
- Published by Polish Information Processing Society PTI in Annals of Computer Science and Information Systems
- Vol. 43, 13-24
SHARE: An Interactive Learning System for Improving Legal Consultation Services and Training
- 15 October 2025
- Published by Polish Information Processing Society PTI in Annals of Computer Science and Information Systems
- Vol. 43, 13-24
SHARE: An Interactive Learning System for Improving Legal Consultation Services and Training
- Open Access
Reflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making
When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.
Open AccessReflection Before Action: Designing a Framework for Quantifying Thought Patterns for Increased Self-awareness in Personal Decision Making
When making significant life decisions, people increasingly turn to conversational AI tools, such as large language models (LLMs). However, LLMs often steer users toward solutions, limiting metacognitive awareness of their own decision-making. In this paper, we shift the focus in decision support from solution-orientation to reflective activity, coining the term pre-decision reflection (PDR). We introduce PROBE, the first framework that assesses pre-decision reflections along two dimensions: breadth (diversity of thought categories) and depth (elaborateness of reasoning). Coder agreement demonstrates PROBE's reliability in capturing how people engage in pre-decision reflection. Our study reveals substantial heterogeneity across participants and shows that people perceived their unassisted reflections as deeper and broader than PROBE's measures. By surfacing hidden thought patterns, PROBE opens opportunities for technologies that foster self-awareness and strengthen people's agency in choosing which thought patterns to rely on in decision-making.
- Open Access
- 22 September 2025
- Published by IOS Press
- p. 2-15
Exploring Human-AI Synergy for Complex Claim Verification
Combating widespread misinformation requires scalable and reliable fact-checking methods. Fact-checking involves several steps, including question generation, evidence retrieval, and veracity prediction. Importantly, fact-checking is well-suited to exploit hybrid intelligence since it requires both human expertise and AI’s large-scale information processing abilities. Thus, constructing an effective fact-checking pipeline requires a systematic understanding of the relative strengths and weaknesses of humans and AI in different steps of the fact-checking process. We investigate the ability of LLMs to perform the first step of the process, i.e., to generate pertinent questions for analyzing a claim. To evaluate the quality of the LLM-generated questions, we crowdsource a dataset in which 150 claims are annotated with questions (1) a novice fact-checker would ask and (2) a professional fact-checker would ask when fact-checking those claims. We study the effects of the human- and LLM-generated questions on evidence retrieval and veracity prediction. We find that LLMs are able to generate nuanced questions to verify a complex claim, but the final label prediction depends on the quality of the evidence corpus. However, the evidence collected by automated methods yields lower accuracy in the veracity prediction task than the evidence curated by experts.
- 22 September 2025
- Published by IOS Press
- p. 2-15
Open AccessExploring Human-AI Synergy for Complex Claim Verification
Combating widespread misinformation requires scalable and reliable fact-checking methods. Fact-checking involves several steps, including question generation, evidence retrieval, and veracity prediction. Importantly, fact-checking is well-suited to exploit hybrid intelligence since it requires both human expertise and AI’s large-scale information processing abilities. Thus, constructing an effective fact-checking pipeline requires a systematic understanding of the relative strengths and weaknesses of humans and AI in different steps of the fact-checking process. We investigate the ability of LLMs to perform the first step of the process, i.e., to generate pertinent questions for analyzing a claim. To evaluate the quality of the LLM-generated questions, we crowdsource a dataset in which 150 claims are annotated with questions (1) a novice fact-checker would ask and (2) a professional fact-checker would ask when fact-checking those claims. We study the effects of the human- and LLM-generated questions on evidence retrieval and veracity prediction. We find that LLMs are able to generate nuanced questions to verify a complex claim, but the final label prediction depends on the quality of the evidence corpus. However, the evidence collected by automated methods yields lower accuracy in the veracity prediction task than the evidence curated by experts.
- Open Access
- 16 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-11
How Should Your Artificial Teammate Tell You How Much It Trusts You?
- 16 September 2025
- Published by Association for Computing Machinery (ACM)
- p. 1-11
Open AccessHow Should Your Artificial Teammate Tell You How Much It Trusts You?
- Open Access
Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.
Open AccessWill Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process.
- Open Access
- 3 September 2025
- Published by Springer Nature in Research Square
From Human Teams to Hybrid Intelligence Teams: Identifying, Characterizing, and Evaluating Foundational Quality Attributes
Hybrid Intelligence (HI) is an emerging paradigm in which artificial intelligence (AI) augments human intelligence.The current literature lacks systematic models that guide the design and evaluation of HI systems. Further, discussions around HI primarily focus on technology, neglecting the holistic human-AI ensemble. In this paper, we take the initial steps toward the development of a quality model for characterizing and evaluating HI systems from a human-AI teams perspective. We first conducted a study investigating the adequacy of properties commonly associated with effective human teams to describe HI. The study features the insights of 50 HI researchers, and shows that various human team properties, including boundedness, interdependence, competency, purposefulness, initiative,normativity, and effectiveness, are important for HI systems. Based on these results, we developed a quality model for HI teams composed of seven high-level quality attributes, further refined into 16 specific ones. To evaluate the relevance and understanding of the proposed attributes, we conducted a second empirical investigation by staging competitions in which participants used the quality model to develop and analyze HI usage scenarios. Our analysis of 48 collected scenarios, which we openly release, confirms the proposed attributes' relevance and highlights insights that emerge when designers consider the quality model in HI system design.
- 3 September 2025
- Published by Springer Nature in Research Square
Open AccessFrom Human Teams to Hybrid Intelligence Teams: Identifying, Characterizing, and Evaluating Foundational Quality Attributes
Hybrid Intelligence (HI) is an emerging paradigm in which artificial intelligence (AI) augments human intelligence.The current literature lacks systematic models that guide the design and evaluation of HI systems. Further, discussions around HI primarily focus on technology, neglecting the holistic human-AI ensemble. In this paper, we take the initial steps toward the development of a quality model for characterizing and evaluating HI systems from a human-AI teams perspective. We first conducted a study investigating the adequacy of properties commonly associated with effective human teams to describe HI. The study features the insights of 50 HI researchers, and shows that various human team properties, including boundedness, interdependence, competency, purposefulness, initiative,normativity, and effectiveness, are important for HI systems. Based on these results, we developed a quality model for HI teams composed of seven high-level quality attributes, further refined into 16 specific ones. To evaluate the relevance and understanding of the proposed attributes, we conducted a second empirical investigation by staging competitions in which participants used the quality model to develop and analyze HI usage scenarios. Our analysis of 48 collected scenarios, which we openly release, confirms the proposed attributes' relevance and highlights insights that emerge when designers consider the quality model in HI system design.