Jun Baba

Teleoperated robots augmented with autonomous assistance are increasingly used for conversational customer service in co-located deployments; we need a systematic method for determining which functions to automate and which to retain for the operator. We introduce a Level of Autonomy (LOA) framework with five levels and empirically examine how LOA and autonomy reliability jointly affect outcomes in co-located deployments. We conducted a laboratory study in which participants ( N = 32) acted as robot operators and, via a teleoperated robot in a simulated shop, engaged in conversational product recommendation interactions using a teleoperation interface. We compared five LOA conditions across multiple reliability conditions and measured the NASA-TLX, System Usability Scale, interpersonal effects, and task performance. The results show that when reliability is high, the most effective strategy was either to fully automate the target function or to keep operator control minimal and clearly bounded; designs that required operators to both supervise the automation and intervene frequently tended to increase workload. In addition, in early deployments, when operators are unfamiliar with teleoperation, avoid designs that require frequent operator decisions. When reliability is limited, a lower LOA is preferable, reducing mode-switching between error recovery and normal teleoperation. This study provides a five-level LOA framework for co-located multi-robot customer service and suggests that effective customer-facing HRI in such settings requires aligning LOA with autonomy reliability and clear human–robot role allocation.

We report a mixed-methods field experiment of a conversational service robot deployed under everyday staffing discretion in a live bedding store. Over 12 days we alternated three conditions--Baseline (no robot), Robot-only, and Robot+Fixture--and video-annotated the service funnel from passersby to purchase. An explanatory sequential design then used six post-experiment staff interviews to interpret the quantitative patterns. Quantitatively, the robot increased stopping per passerby (highest with the fixture), yet clerk-led downstream steps per stopper--clerk approach, store entry, assisted experience, and purchase--decreased. Interviews explained this divergence: clerks avoided interrupting ongoing robot-customer talk, struggled with ambiguous timing amid conversational latency, and noted child-centered attraction that often satisfied curiosity at the doorway. The fixture amplified visibility but also anchored encounters at the threshold, creating a well-defined micro-space where needs could "close" without moving inside. We synthesize these strands into an integrative account from the initial show of interest on the part of a customer to their entering the store and derive actionable guidance. The results advance the understanding of interactions between customers, staff members, and the robot and offer practical recommendations for deploying service robots in high-touch retail.

The rapid development of artificial intelligence (AI) has fundamentally transformed creative work practices in the design industry. Existing studies have identified both opportunities and challenges for creative practitioners in their collaboration with generative AI and explored ways to facilitate effective human-AI co-creation. However, there is still a limited understanding of designers' collaboration with AI that supports creative processes distinct from generative AI. To address these gaps, this study focuses on understanding designers' collaboration with decision-making AI, which supports the convergence process in the creative workflow, as opposed to the divergent process supported by generative AI. Specifically, we conducted a case study at an online advertising design company to explore how professional graphic designers at the company perceive the impact of decision-making AI on their creative work practices. The case company incorporated an AI system that predicts the effectiveness of advertising design into the design workflow as a decision-making support tool. Findings from interviews with 12 designers identified how designers trust and rely on AI, its perceived benefits and challenges, and their strategies for navigating the challenges. Based on the findings, we discuss design recommendations for integrating decision-making AI into the creative design workflow.

Service robots have demonstrated significant potential in retail environments, with research efforts increasingly focused on designing effective robot-customer interactions. However, the collaboration between service robots and human clerks is crucial for optimizing customer engagement in real-world retail settings. While robots excel at attracting passersby, human clerks bring essential emotional intelligence and deep acting skills necessary for building trust and delivering personalized service. In this paper, we propose a novel robot-clerk collaboration model that leverages the complementary strengths of robots and clerks. Specifically, robots are tasked with attracting passersby and persuading them to interact with the products. Once a customer engages with a product, the robot subtly notifies a human clerk to take over, ensuring a seamless transition into the store for more in-depth product exploration and engagement. To develop this approach, we carried out a pilot study to identify the most effective and practical interaction configurations, followed by a 12-day experiment to evaluate the proposed strategy. The experimental results demonstrate the effectiveness of the robot-clerk collaboration approach. Based on these findings, we present detailed design implications and discuss key challenges to guide future research in this area.

Chat-oriented dialogue systems designed to provide tangible benefits, such as sharing the latest news or preventing frailty in senior citizens, often require Proactive acquisition of specific user Information via chats on user-faVOred Topics (PIVOT). This study proposes the PIVOT task, designed to advance the technical foundation for these systems. In this task, a system needs to acquire the answers of a user to predefined questions without making the user feel abrupt while engaging in a chat on a predefined topic. We found that even recent large language models (LLMs) show a low success rate in the PIVOT task. We constructed a dataset suitable for the analysis to develop more effective systems. Finally, we developed a simple but effective system for this task by incorporating insights obtained through the analysis of this dataset.

Shiki Sato, Jun Baba, Asahi Hentona, Shinji Iwata, Akifumi Yoshimoto, Koichiro Yoshino. Findings of the Association for Computational Linguistics: EMNLP 2025. 2025.

User willingness is a crucial element in the sales talk process that affects the achievement of the salesperson's or sales system's objectives. Despite the importance of user willingness, to the best of our knowledge, no previous study has addressed the development of automated sales talk dialogue systems that explicitly consider user willingness. A major barrier is the lack of sales talk datasets with reliable user willingness data. Thus, in this study, we developed a user willingness-aware sales talk collection by leveraging the ecological validity concept, which is discussed in the field of human-computer interaction. Our approach focused on three types of user willingness essential in real sales interactions. We created a dialogue environment that closely resembles real-world scenarios to elicit natural user willingness, with participants evaluating their willingness at the utterance level from multiple perspectives. We analyzed the collected data to gain insights into practical user willingness-aware sales talk strategies. In addition, as a practical application of the constructed dataset, we developed and evaluated a sales dialogue system aimed at enhancing the user's intent to purchase.

Various customer service robots’ teleoperation interfaces (I/Fs) have been developed for human-robot collaboration. However, a previous study has indicated a lack of operator enjoyment. This study aims to design an I/F that elicits operator enjoyment and identifies the factors that contribute to this enjoyment. We developed two I/Fs: a high degree of freedom I/F and a gradual flexibility degree of freedom I/F based on gamification to elicit operator enjoyment. Using these I/Fs, we conducted a field study in a real shopping mall to investigate the operators’ experiences. As a result, the operators in this experiment greatly enjoyed using our I/Fs. The results showed that our I/Fs could elicit operator enjoyment with the following three I/F factors suggested as potentially important design guidelines: ease of use through anonymity, moderate restriction of freedom, and sharing experiences with a group of operators.

In this study, we report a 6-month empirical study on a service robot deployed in a bakery shop. Recently, potential applications of service robots have been increasingly explored. However, further empirical knowledge is required to determine the optimal approach to design service robots for useful applications. We also address “usefulness” from two perspectives: the effects of a robot on customers’ shopping behavior and the practical benefits the robot could provide for human workers in its working environment. The results show that our robot achieved long-term effects on product recommendations for customers who visited the bakery store on a regular basis (weekly) but not for other customers. A thematic analysis of the interviews reflected the practical values that the staff expected from the robot. Based on these findings, we we outline key considerations for designing effective long-term service robot applications.