DaFoEs: Mixing Datasets Towards the Generalization of Vision-State Deep-Learning Force Estimation in Minimally Invasive Robotic Surgery
- 22 January 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 9 (3) , 2527-2534
Abstract
Precisely determining the contact force during safe interaction in Minimally Invasive Robotic Surgery (MIRS) is still an open research challenge. Inspired by post-operative qualitative analysis from surgical videos, the use of cross-modality data driven deep neural network models has been one of the newest approaches to predict sensorless force trends. However, these methods required for large and variable datasets which are not currently available. In this paper, we present a new vision-haptic dataset (DaFoEs) with variable soft environments for the training of deep neural models. In order to reduce the bias from a single dataset, we present a pipeline to generalize different vision and state data inputs for mixed dataset training, using a previously validated dataset with different setup. Finally, we present a variable encoder-decoder architecture to predict the forces done by the laparoscopic tool using single input or sequence of inputs. For input sequence, we use a recurrent decoder, named with the prefix R, and a new temporal sampling to represent the acceleration of the tool. During our training, we demonstrate that single dataset training tends to overfit to the training data domain, but has difficulties on translating the results across new domains. However, dataset mixing presents a good translation with a mean relative estimated force error of 5% and 12% for the recurrent and non-recurrent models respectively. Our method, also marginally increase the effectiveness of transformers for force estimation up to a maximum of $\simeq 15\%$ , as the volume of available data is increase by 150%. In conclusion, we demonstrate that mixing experimental set ups for vision-state force estimation in MIRS is a possible approach towards the general solution of the problem.Keywords
Funding Information
- U.K. Research and Innovation in the Engineering and Physical Sciences Research Council (2446549)
- Centre for Artificial Intelligence and Robotics
- Hong Kong Institute of Science & Innovation, Chinese Academy of Sciences
This publication has 17 references indexed in Scilit:
- A Neural Network-based Suture-tension Estimation Method Using Spatio-temporal Features of Visual Information and Robot-state Information for Robot-assisted SurgeryInternational Journal of Control, Automation, and Systems, 2023
- A data driven recurrent neural network approach for reproduction of variable visuo-haptic force feedback in surgical tool insertionExpert Systems with Applications, 2023
- Investigating the influence of dimensionality reduction on force estimation in robotic-assisted surgery using recurrent and convolutional networksEngineering Applications of Artificial Intelligence, 2023
- Adaptive Token Sampling for Efficient Vision Transformers2022
- Camera Configuration Models for Machine Vision Based Force Estimation in Robot-Assisted Soft Body Manipulation2022
- Stereovision based force estimation with stiffness mapping in surgical tool insertion using recurrent neural networkThe Journal of Supercomputing, 2022
- Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State2021
- Dynamic Modeling of the Da Vinci Research Kit Arm for the Estimation of Interaction Wrench2019
- A recurrent convolutional neural network approach for sensorless force estimation in robotic surgeryBiomedical Signal Processing and Control, 2019
- Multi-Modal Haptic Feedback for Grip Force Reduction in Robotic SurgeryScientific Reports, 2019