Chayan Sarkar
Research Subjects
Chayan Sarkar has a total of 51 co-authors. Here are the top co-authors by number of shared publications:
- Pradip Pramanick
Interdepartmental Center for Advances in Robotic Surgery - ICAROS, University of Naples Federico II, Italy
6 shared publications - RangaRao Venkatesha Prasad
Department of Software Technology, Delft University of Technology, Delft, The Netherlands
4 shared publications Moumita Debnath
3 shared publications- Sayan Paul
TCS Research
3 shared publications Sucheta Tidke
3 shared publicationsSudha Jain
3 shared publications
Latest Publications
- 14 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (6) , 6991-6998
From Tool to Teammate: Balancing Task-Oriented and Social Talk During Human–Robot Collaboration
As robots become more integrated into shared work environments, effective communication must balance clear task directives with social engagement to support both efficiency and user satisfaction. This letter examines how people perceive task-oriented versus social dialogue during human–robot collaboration and how they judge the appropriateness of switching between them. We discuss a pilot, in-person data collection and two follow-up online studies with an autonomous robot in a collaborative pizza-making task. The pilot assesses the technical feasibility of an autonomous robot generating distinct task and social dialogue styles. Next, with a larger participant sample, we evaluate the salience of the two talk types and their impact on perceptions of the robot as a collaborative partner. The second study explores judgments of talk type switching using video clips from the pilot, revealing how timing, task state, and user behavior influence perceived appropriateness. Together, these findings highlight nuanced human expectations for robot communication and inform the design of collaborative robots that can adaptively balance task-oriented and social talk in contextually sensitive ways.
- 14 April 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 11 (6) , 6991-6998
From Tool to Teammate: Balancing Task-Oriented and Social Talk During Human–Robot Collaboration
As robots become more integrated into shared work environments, effective communication must balance clear task directives with social engagement to support both efficiency and user satisfaction. This letter examines how people perceive task-oriented versus social dialogue during human–robot collaboration and how they judge the appropriateness of switching between them. We discuss a pilot, in-person data collection and two follow-up online studies with an autonomous robot in a collaborative pizza-making task. The pilot assesses the technical feasibility of an autonomous robot generating distinct task and social dialogue styles. Next, with a larger participant sample, we evaluate the salience of the two talk types and their impact on perceptions of the robot as a collaborative partner. The second study explores judgments of talk type switching using video clips from the pilot, revealing how timing, task state, and user behavior influence perceived appropriateness. Together, these findings highlight nuanced human expectations for robot communication and inform the design of collaborative robots that can adaptively balance task-oriented and social talk in contextually sensitive ways.
- Open Access
- 16 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1040-1049
Learning Human Preferences over a Human-Robot Collaboration Based on Explicit and Implicit Human Feedback
There is significant interest in enabling robots to learn to perform tasks directly from interactions with non-expert users. Typically, a human serves as a teacher whose only task is to provide feedback to a robot learner. However, in real-world human-robot collaborations, the human often assists with the task while also offering feedback. Our key insight is that we can extract additional, implicit feedback from the human’s actions during the collaboration to augment the robot learning process. Under the assumption of fixed-role assignments, we first propose to formalize human preferences over a human-robot collaboration as a shared set of parameters encoding alignment between two reward functions: one that drives human behavior, and another that should direct robot behavior. This allows us to extract implicit feedback from an interaction by reasoning about the human’s actions in the task as actions that reveal the human’s preferences. Then, we combine this implicit feedback with traditional explicit human feedback to facilitate estimating the human’s preferences. We evaluated our proposed approach for Preference learning from Implicit and Explicit feedback (PIE) in simulations and with real users in a cooking scenario. Our simulation results indicate that combining multiple modalities of human feedback improves a robot’s ability to estimate human preferences over the collaboration, with a similar trend observed in real-world evaluations. These findings highlight a promising direction for enabling robots to adapt to a user’s preference model more quickly, thereby reducing the amount of time a person must spend teaching a robot.
- 16 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1040-1049
Open AccessLearning Human Preferences over a Human-Robot Collaboration Based on Explicit and Implicit Human Feedback
There is significant interest in enabling robots to learn to perform tasks directly from interactions with non-expert users. Typically, a human serves as a teacher whose only task is to provide feedback to a robot learner. However, in real-world human-robot collaborations, the human often assists with the task while also offering feedback. Our key insight is that we can extract additional, implicit feedback from the human’s actions during the collaboration to augment the robot learning process. Under the assumption of fixed-role assignments, we first propose to formalize human preferences over a human-robot collaboration as a shared set of parameters encoding alignment between two reward functions: one that drives human behavior, and another that should direct robot behavior. This allows us to extract implicit feedback from an interaction by reasoning about the human’s actions in the task as actions that reveal the human’s preferences. Then, we combine this implicit feedback with traditional explicit human feedback to facilitate estimating the human’s preferences. We evaluated our proposed approach for Preference learning from Implicit and Explicit feedback (PIE) in simulations and with real users in a cooking scenario. Our simulation results indicate that combining multiple modalities of human feedback improves a robot’s ability to estimate human preferences over the collaboration, with a similar trend observed in real-world evaluations. These findings highlight a promising direction for enabling robots to adapt to a user’s preference model more quickly, thereby reducing the amount of time a person must spend teaching a robot.
- Open Access
- 23 October 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Human-Robot Interaction
- Vol. 13 (4) , 1-48
Enabling Social Robots to Perceive and Join Socially Interacting Groups Using F-formation: A Comprehensive Overview
Social robots in our daily surroundings, like personal guides, waiter robots, home helpers, assistive robots, and telepresence/teleoperation robots, are increasing day by day. Their usability and acceptability largely depend on their explicit and implicit interaction capability with fellow human beings. As a result, social behavior is one of the most sought-after qualities that a robot can possess. However, there is no specific aspect and/or feature that defines socially acceptable behavior, and it largely depends on the situation, application, and society. In this article, we investigate one such social behavior for collocated robots. Imagine a group of people is interacting with each other, and we want to join the group. We as human beings do it in a socially acceptable manner, i.e., within the group, we do position ourselves in such a way that we can participate in the group activity without disturbing/obstructing anybody. To possess such a quality, first, a robot needs to determine the formation of the group and then determine a position for itself, which we humans do implicitly. There are many theories which study group formations and proxemics; one such theory is f-formation which could be utilized for this purpose. As the types of formations can be very diverse, detecting the social groups is not a trivial task. In this article, we provide a comprehensive survey of the existing work on social interaction and group detection using f-formation for robotics and other applications. We also put forward a novel holistic survey framework combining some of the possibly more important concerns and modules relevant to this problem. We define taxonomies based on methods, camera views, datasets, detection capabilities and scale, evaluation approaches, and application areas. We discuss certain open challenges and limitations in the current literature along with possible future research directions based on this framework. In particular, we discuss the existing methods/techniques and their relative merits and demerits, applications, and provide a set of unsolved but relevant problems in this domain. The official website for this work is available at: https://github.com/HrishavBakulBarua/Social-Robots-F-formation
- 23 October 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Human-Robot Interaction
- Vol. 13 (4) , 1-48
Open AccessEnabling Social Robots to Perceive and Join Socially Interacting Groups Using F-formation: A Comprehensive Overview
Social robots in our daily surroundings, like personal guides, waiter robots, home helpers, assistive robots, and telepresence/teleoperation robots, are increasing day by day. Their usability and acceptability largely depend on their explicit and implicit interaction capability with fellow human beings. As a result, social behavior is one of the most sought-after qualities that a robot can possess. However, there is no specific aspect and/or feature that defines socially acceptable behavior, and it largely depends on the situation, application, and society. In this article, we investigate one such social behavior for collocated robots. Imagine a group of people is interacting with each other, and we want to join the group. We as human beings do it in a socially acceptable manner, i.e., within the group, we do position ourselves in such a way that we can participate in the group activity without disturbing/obstructing anybody. To possess such a quality, first, a robot needs to determine the formation of the group and then determine a position for itself, which we humans do implicitly. There are many theories which study group formations and proxemics; one such theory is f-formation which could be utilized for this purpose. As the types of formations can be very diverse, detecting the social groups is not a trivial task. In this article, we provide a comprehensive survey of the existing work on social interaction and group detection using f-formation for robotics and other applications. We also put forward a novel holistic survey framework combining some of the possibly more important concerns and modules relevant to this problem. We define taxonomies based on methods, camera views, datasets, detection capabilities and scale, evaluation approaches, and application areas. We discuss certain open challenges and limitations in the current literature along with possible future research directions based on this framework. In particular, we discuss the existing methods/techniques and their relative merits and demerits, applications, and provide a set of unsolved but relevant problems in this domain. The official website for this work is available at: https://github.com/HrishavBakulBarua/Social-Robots-F-formation
- Open Access
- 9 July 2024
- Published by Springer Nature in Journal of Intelligent & Robotic Systems
- Vol. 110 (3) , 1-22
Teledrive: An Embodied AI Based Telepresence System
This article presents ‘Teledrive’, a telepresence robotic system with embodied AI features that empowers an operator to navigate the telerobot in any unknown remote place with minimal human intervention. We conceive Teledrive in the context of democratizing remote ‘care-giving’ for elderly citizens as well as for isolated patients, affected by contagious diseases. In particular, this paper focuses on the problem of navigating to a rough target area (like ‘bedroom’ or ‘kitchen’) rather than pre-specified point destinations. This ushers in a unique ‘AreaGoal’ based navigation feature, which has not been explored in depth in the contemporary solutions. Further, we describe an edge computing-based software system built on a WebRTC-based communication framework to realize the aforementioned scheme through an easy-to-use speech-based human-robot interaction. Moreover, to enhance the ease of operation for the remote caregiver, we incorporate a ‘person following’ feature, whereby a robot follows a person on the move in its premises as directed by the operator. Moreover, the system presented is loosely coupled with specific robot hardware, unlike the existing solutions. We have evaluated the efficacy of the proposed system through baseline experiments, user study, and real-life deployment.
- 9 July 2024
- Published by Springer Nature in Journal of Intelligent & Robotic Systems
- Vol. 110 (3) , 1-22
Open AccessTeledrive: An Embodied AI Based Telepresence System
This article presents ‘Teledrive’, a telepresence robotic system with embodied AI features that empowers an operator to navigate the telerobot in any unknown remote place with minimal human intervention. We conceive Teledrive in the context of democratizing remote ‘care-giving’ for elderly citizens as well as for isolated patients, affected by contagious diseases. In particular, this paper focuses on the problem of navigating to a rough target area (like ‘bedroom’ or ‘kitchen’) rather than pre-specified point destinations. This ushers in a unique ‘AreaGoal’ based navigation feature, which has not been explored in depth in the contemporary solutions. Further, we describe an edge computing-based software system built on a WebRTC-based communication framework to realize the aforementioned scheme through an easy-to-use speech-based human-robot interaction. Moreover, to enhance the ease of operation for the remote caregiver, we incorporate a ‘person following’ feature, whereby a robot follows a person on the move in its premises as directed by the operator. Moreover, the system presented is loosely coupled with specific robot hardware, unlike the existing solutions. We have evaluated the efficacy of the proposed system through baseline experiments, user study, and real-life deployment.
- Open Access
tagE: Enabling an Embodied Agent to Understand Human Instructions
Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as task and argument grounding for Embodied agents (tagE). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.
Open AccesstagE: Enabling an Embodied Agent to Understand Human Instructions
Natural language serves as the primary mode of communication when an intelligent agent with a physical presence engages with human beings. While a plethora of research focuses on natural language understanding (NLU), encompassing endeavors such as sentiment analysis, intent prediction, question answering, and summarization, the scope of NLU directed at situations necessitating tangible actions by an embodied agent remains limited. The inherent ambiguity and incompleteness inherent in natural language present challenges for intelligent agents striving to decipher human intention. To tackle this predicament head-on, we introduce a novel system known as task and argument grounding for Embodied agents (tagE). At its core, our system employs an inventive neural network model designed to extract a series of tasks from complex task instructions expressed in natural language. Our proposed model adopts an encoder-decoder framework enriched with nested decoding to effectively extract tasks and their corresponding arguments from these intricate instructions. These extracted tasks are then mapped (or grounded) to the robot's established collection of skills, while the arguments find grounding in objects present within the environment. To facilitate the training and evaluation of our system, we have curated a dataset featuring complex instructions. The results of our experiments underscore the prowess of our approach, as it outperforms robust baseline models.
- Open Access
- 13 March 2023
- Published by Association for Computing Machinery (ACM)
- p. 471-475
Utilizing Prior Knowledge to Improve Automatic Speech Recognition in Human-Robot Interactive Scenarios
The prolificacy of human-robot interaction not only depends on a robot's ability to understand the intent and content of the human utterance but also gets impacted by the automatic speech recognition (ASR) system. Modern ASR can provide highly accurate (grammatically and syntactically) translation. Yet, the general purpose ASR often misses out on the semantics of the translation by incorrect word prediction due to open-vocabulary modeling. ASR inaccuracy can have significant repercussions as this can lead to a completely different action by the robot in the real world. Can any prior knowledge be helpful in such a scenario? In this work, we explore how prior knowledge can be utilized in ASR decoding. Using our experiments, we demonstrate how our system can significantly improve ASR translation for robotic task instruction.
- 13 March 2023
- Published by Association for Computing Machinery (ACM)
- p. 471-475
Open AccessUtilizing Prior Knowledge to Improve Automatic Speech Recognition in Human-Robot Interactive Scenarios
The prolificacy of human-robot interaction not only depends on a robot's ability to understand the intent and content of the human utterance but also gets impacted by the automatic speech recognition (ASR) system. Modern ASR can provide highly accurate (grammatically and syntactically) translation. Yet, the general purpose ASR often misses out on the semantics of the translation by incorrect word prediction due to open-vocabulary modeling. ASR inaccuracy can have significant repercussions as this can lead to a completely different action by the robot in the real world. Can any prior knowledge be helpful in such a scenario? In this work, we explore how prior knowledge can be utilized in ASR decoding. Using our experiments, we demonstrate how our system can significantly improve ASR translation for robotic task instruction.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 8846-8857
tagE: Enabling an Embodied Agent to Understand Human Instructions
Chayan Sarkar, Avik Mitra, Pradip Pramanick, Tapas Nayak. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 8846-8857
Open AccesstagE: Enabling an Embodied Agent to Understand Human Instructions
Chayan Sarkar, Avik Mitra, Pradip Pramanick, Tapas Nayak. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
- 28 September 2022
- Published by Elsevier in Robotics and Autonomous Systems
- Vol. 159
Consensus-based fast and energy-efficient multi-robot task allocation
- 28 September 2022
- Published by Elsevier in Robotics and Autonomous Systems
- Vol. 159
Consensus-based fast and energy-efficient multi-robot task allocation
- 1 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (4) , 10826-10833
DoRO: Disambiguation of Referred Object for Embodied Agents
Robotic task instructions often involve a referred object that the robot must locate (ground) within the environment. While task intent understanding is an essential part of natural language understanding, less effort is made to resolve ambiguity that may arise while grounding the task. Existing works use vision-based task grounding and ambiguity detection, suitable for a fixed view and a static robot. However, the problem magnifies for a mobile robot, where the ideal view is not known beforehand. Moreover, a single view may not be sufficient to locate all the object instances in the given area, which leads to inaccurate ambiguity detection. Human intervention is helpful only if the robot can convey the kind of ambiguity it is facing. In this article, we present DoRO ( Disambiguation of Referred Object ), a system that can help an embodied agent to disambiguate the referred object by raising a suitable query whenever required. Given an area where the intended object is, DoRO finds all the instances of the object by aggregating observations from multiple views while exploring & scanning the area. It then raises a suitable query using the information from the grounded object instances. Experiments conducted with the AI2Thor simulator show that DoRO not only detects the ambiguity more accurately but also raises verbose queries with more accurate information from the visual-language grounding.
- 1 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 7 (4) , 10826-10833
DoRO: Disambiguation of Referred Object for Embodied Agents
Robotic task instructions often involve a referred object that the robot must locate (ground) within the environment. While task intent understanding is an essential part of natural language understanding, less effort is made to resolve ambiguity that may arise while grounding the task. Existing works use vision-based task grounding and ambiguity detection, suitable for a fixed view and a static robot. However, the problem magnifies for a mobile robot, where the ideal view is not known beforehand. Moreover, a single view may not be sufficient to locate all the object instances in the given area, which leads to inaccurate ambiguity detection. Human intervention is helpful only if the robot can convey the kind of ambiguity it is facing. In this article, we present DoRO ( Disambiguation of Referred Object ), a system that can help an embodied agent to disambiguate the referred object by raising a suitable query whenever required. Given an area where the intended object is, DoRO finds all the instances of the object by aggregating observations from multiple views while exploring & scanning the area. It then raises a suitable query using the information from the grounded object instances. Experiments conducted with the AI2Thor simulator show that DoRO not only detects the ambiguity more accurately but also raises verbose queries with more accurate information from the visual-language grounding.