Philippe Zanne

Optical Coherence Tomography (OCT) offers real-time, high-resolution imaging of tissue internal structures, presenting a potential enhancement over conventional endoscopic examination of the digestive system surface. However, miniaturized OCT probes typically present challenges with a limited field of view and require precise positioning to optimize tissue visualization. This research addresses the challenge by leveraging robotizing a surgical endoscope and a steerable OCT probe to provide closed-loop scanning of larger areas in the presence of tissue motion. We propose a robotic multi-pullback technique, supported by real-time, tactile feedback from OCT images. Deep learning algorithms are employed for OCT image stabilization and segmentation, enabling autonomous robotic volumetric OCT scanning. Compared to previous works, our closed-loop control approach considers, in addition to the probe flexibility, also the movements and deformations of the tissue during the robotic scanning. For experimental validation, we designed a moving soft tissue phantom with force sensing and compared the proposed method with fixed-angle approaches. Our method consistently maintained minimal contact forces (around 50 mN) and clear tissue visibility for tissues with different stiffness and motion patterns. Resulting imaging quality was further validated using a multi-layer optical phantom under dynamic conditions.

Flexible steerable intraluminal robot platforms allow treatment and screening of colorectal cancer at an early stage, potentially reducing the associated incidence and mortality rates. Such robotic platforms often rely on a tree-like flexible architecture, with a flexible robotized body carrying both the endoscope camera and two robotized flexible surgical arms at its distal end. Telemanipulating these robotic platforms to correctly perform surgical tasks is technically difficult due to their kinematic complexity and the demanding nature of the task, which leads to potential interruptions in the surgical workflow. In this paper, a technique to efficiently control the arms and body and correctly perform complex surgical steps during the endoscopic submucosal dissection procedure is proposed. The technique, referred to as semi-autonomous arm-body control, is based on a quadratic programming controller. Custom-defined tasks synergistically control the arms and body, while avoiding unsafe positions for the arms. Experiments in a mixed physical-simulated setup with eight users show an increased performance on the task and smoother movements compared to manual telemanipulation, at the expense of a slightly longer operating time. Further study will look at validating the approach in more realistic scenarios.

New technologies for monitoring grip forces during hand and finger movements in non-standard task contexts have provided unprecedented functional insights into somatosensory cognition. Somatosensory cognition is the basis of our ability to manipulate and transform objects of the physical world and to grasp them with the right amount of force. In previous work, the wireless tracking of grip-force signals recorded from biosensors in the palm of the human hand has permitted us to unravel some of the functional synergies that underlie perceptual and motor learning under conditions of non-standard and essentially unreliable sensory input. This paper builds on this previous work and discusses further, functionally motivated, analyses of individual grip-force data in manual robot control. Grip forces were recorded from various loci in the dominant and non-dominant hands of individuals with wearable wireless sensor technology. Statistical analyses bring to the fore skill-specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain-inspired neural network model that uses the output metric of a self-organizing pap with unsupervised winner-take-all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time and reliably captures the differences between novice and expert performance in terms of grip-force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip-force monitoring in real time. This will permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot-interaction, which represents unprecedented challenges for perceptual and motor adaptation in environmental contexts of high sensory uncertainty. Cross-disciplinary insights from systems neuroscience and cognitive behavioral science, and the predictive modeling of operator skills using parsimonious Artificial Intelligence (AI), will contribute towards improving the outcome of new types of surgery, in particular the single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single-Incision Laparoscopic Surgery).

Optical Coherence Tomography (OCT) is an emerging medical imaging modality for luminal organ diagnosis. The non-constant rotation speed of optical components in the OCT catheter tip causes rotational distortion in OCT volumetric scanning. By improving the scanning process, this instability can be partially reduced. To further correct the rotational distortion in the OCT image, a volumetric data stabilization algorithm is proposed. The algorithm first estimates the Non-Uniform Rotational Distortion (NURD) for each B-scan by using a Convolutional Neural Network (CNN). A correlation map between two successive B-scans is computed and provided as input to the CNN. To solve the problem of accumulative error in iterative frame stream processing, we deploy an overall rotation estimation between reference orientation and actual OCT image orientation. We train the network with synthetic OCT videos by intentionally adding rotational distortion into real OCT images. As part of this article we discuss the proposed method in two different scanning modes: the first is a conventional pullback mode where the optical components move along the protection sheath, and the second is a self-designed scanning mode where the catheter is globally translated by using an external actuator. The efficiency and robustness of the proposed method are evaluated with synthetic scans as well as real scans under two scanning modes.

Optical coherence tomography (OCT) endoscopic catheters provide efficient solutions for the non-invasive scan of malignant tissues in internal human organs. In this letter, we investigate the image-guided control of a robotic flexible endoscopic system equipped with an OCT probe for autonomous tissue scanning. The visual control strategy is achieved by combining the image feedback from both a monocular endoscopic camera (eye-to-hand) and an OCT probe (eye-in-hand). Our control strategy allows the OCT probe to automatically track a trajectory defined by the user on the 2D endoscopic image and keep contact with the scanned surface. The orientation, depth, and endoscopic 2D image position are controlled by solving an optimization problem, which is converted to a quadratic programming problem. For the implementation of the visual control strategy, we also consider visibility constraints and actuators limitations. In addition, a marker-based method to estimate the 3D pose of the continuum robot using only 2D images is proposed based on a Kalman filter and a registration technique. The proposed control strategy is validated using both simulation and laboratory experiments.

Deep learning has provided new ways of manipulating, processing and analyzing data. It sometimes may achieve results comparable to, or surpassing human expert performance, and has become a source of inspiration in the era of artificial intelligence. Another subfield of machine learning named reinforcement learning, tries to find an optimal behavior strategy through interactions with the environment. Combining deep learning and reinforcement learning permits resolving critical issues relative to the dimensionality and scalability of data in tasks with sparse reward signals, such as robotic manipulation and control tasks, that neither method permits resolving when applied on its own. In this paper, we present recent significant progress of deep reinforcement learning algorithms, which try to tackle the problems for the application in the domain of robotic manipulation control, such as sample efficiency and generalization. Despite these continuous improvements, currently, the challenges of learning robust and versatile manipulation skills for robots with deep reinforcement learning are still far from being resolved for real-world applications.

The temporal evolution of individual grip force profiles of a novice using a robotic system for minimally invasive endoscopic surgery is analyzed on the basis of thousands of individual sensor data recorded in real time through a wearable wireless sensor glove system. The spatio-temporal grip force profiles from specific sensor locations in the dominant hand performing a four-step pick-and-drop simulator task reveal skill-relevant differences in force deployment by the small finger (fine grip force control) and the middle finger (gross grip force contribution) by comparison with the profiles of a highly proficient expert. Cross-disciplinary insights from systems neuroscience, cognitive behavioral science, and robotics, with implications for biologically inspired AI for human–robot interactions, highlight the functional significance of spatio-temporal grip force profiling.

Wearable sensor systems with transmitting capabilities are currently employed for the biometric screening of exercise activities and other performance data. Such technology is generally wireless and enables the non-invasive monitoring of signals to track and trace user behaviors in real time. Examples include signals relative to hand and finger movements or force control reflected by individual grip force data. As will be shown here, these signals directly translate into task, skill, and hand-specific (dominant versus non-dominant hand) grip force profiles for different measurement loci in the fingers and palm of the hand. The present study draws from thousands of such sensor data recorded from multiple spatial locations. The individual grip force profiles of a highly proficient left-hander (expert), a right-handed dominant-hand-trained user, and a right-handed novice performing an image-guided, robot-assisted precision task with the dominant or the non-dominant hand are analyzed. The step-by-step statistical approach follows Tukey’s “detective work” principle, guided by explicit functional assumptions relating to somatosensory receptive field organization in the human brain. Correlation analyses (Person’s product moment) reveal skill-specific differences in co-variation patterns in the individual grip force profiles. These can be functionally mapped to from-global-to-local coding principles in the brain networks that govern grip force control and its optimization with a specific task expertise. Implications for the real-time monitoring of grip forces and performance training in complex task-user systems are brought forward.