Bounded Future MS-TCN++ for surgical gesture recognition
Preprint
Abstract
In recent times there is a growing development of video based applications for surgical purposes. Part of these applications can work offline after the end of the procedure, other applications must react immediately. However, there are cases where the response should be done during the procedure but some delay is acceptable. In the literature, the online-offline performance gap is known. Our goal in this study was to learn the performance-delay trade-off and design an MS-TCN++-based algorithm that can utilize this trade-off. To this aim, we used our open surgery simulation data-set containing 96 videos of 24 participants that perform a suturing task on a variable tissue simulator. In this study, we used video data captured from the side view. The Networks were trained to identify the performed surgical gestures. The naive approach is to reduce the MS-TCN++ depth, as a result, the receptive field is reduced, and also the number of required future frames is also reduced. We showed that this method is sub-optimal, mainly in the small delay cases. The second method was to limit the accessible future in each temporal convolution. This way, we have flexibility in the network design and as a result, we achieve significantly better performance than in the naive approach.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- Open surgery tool classification and hand utilization using a multi-camera systemInternational Journal of Computer Assisted Radiology and Surgery, 2022
- Using open surgery simulation kinematic data for tool and gesture recognitionInternational Journal of Computer Assisted Radiology and Surgery, 2022
- Video-based fully automatic assessment of open surgery suturing skillsInternational Journal of Computer Assisted Radiology and Surgery, 2022
- Surgical data science – from concepts toward clinical translationMedical Image Analysis, 2021
- Multi-task temporal convolutional networks for joint recognition of surgical phases and steps in gastric bypass proceduresInternational Journal of Computer Assisted Radiology and Surgery, 2021
- Robotic Technology in Operating Rooms: a ReviewCurrent Robotics Reports, 2021
- Video-based surgical skill assessment using 3D convolutional neural networksInternational Journal of Computer Assisted Radiology and Surgery, 2019
- Temporal Convolutional Networks: A Unified Approach to Action Segmentation2016
- A novel tool for summarization of arthroscopic videosMultimedia Tools and Applications, 2009
- Online algorithms: a surveyMathematical Programming, 2003