A multi-user multi-robot multi-goal multi-device human-robot interaction manipulation benchmark
Authors:
Open Access
- 25 September 2025
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Robotics and AI
- Vol. 12, 1528754
Abstract
One weakness of human-robot interaction (HRI) research is the lack of reproducible results, due to the lack of standardised benchmarks. In this work we introduce a multi-user multi-robot multi-goal multi-device manipulation benchmark (M4Bench), a flexible HRI platform in which multiple users can direct either a single—or multiple—simulated robots to perform a multi-goal pick-and-place task. Our software exposes a web-based visual interface, with support for mouse, keyboard, gamepad, eye tracker and electromyograph/electroencephalograph (EMG/EEG) user inputs. It can be further extended using native browser libraries or WebSocket interfaces, allowing researchers to add support for their own devices. We also provide tracking for several HRI metrics, such as task completion and command selection time, enabling quantitative comparisons between different user interfaces and devices. We demonstrate the utility of our benchmark with a user study (n = 50) conducted to compare five different input devices, and also compare single-vs. multi-user control. In the pick-and-place task, we found that users performed worse when using the eye tracker + EMG device pair, as compared to mouse + keyboard or gamepad + gamepad, over four quantitative metrics (corrected p 0.001). Our software is available at https://github.com/arayabrain/m4bench.
Keywords
Funding Information
- Moonshot Research and Development Program
This publication has 57 references indexed in Scilit:
- Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A SurveyAdvanced Intelligent Systems, 2023
- Comparing alternative modalities in the context of multimodal human–robot interactionJournal on Multimodal User Interfaces, 2023
- Humans interacting with multi-robot systems: a natural affect-based approachAutonomous Robots, 2019
- Playbook for UAS: UX of Goal-Oriented Planning and Execution2018
- Intelligent agent supporting human–multi-robot team collaborationArtificial Intelligence, 2017
- Progress and prospects of the human–robot collaborationAutonomous Robots, 2017
- HRI usability evaluation of interaction modes for a teleoperated agricultural robotic sprayerApplied Ergonomics, 2017
- Investigating the effect of increasing robot group sizes on the human psychophysiological state in the context of human–swarm interactionSwarm Intelligence, 2016
- Expressive Lights for Revealing Mobile Service Robot State2015
- Review of Human Studies Methods in HRI and RecommendationsInternational Journal of Social Robotics, 2010