Rongrong Liu

Soustraction d'arrière-plan par images multispectrales avec les méthodes Codebook et apprentissage profond: Cette thèse vise à étudier les images multispectrales pour la détection d'objets en mouvement par soustraction d'arrière-plan, à la fois avec des méthodes classiques et d’apprentissage profond. En tant qu'algorithme classique efficace et représentatif pour la soustraction de fond, l’algorithme Codebook traditionnel a d'abord été étendu au cas multispectral. Afin de rendre l'algorithme fiable et robuste, un mécanisme auto-adaptatif pour sélectionner les paramètres optimaux a ensuite été proposé. Dans ce cadre, de nouveaux critères dans le processus d'appariement sont employés et de nouvelles techniques pour construire le modèle d'arrière-plan sont conçues, y compris le Codebook de boîtes, le Codebook dynamique et la stratégie de fusion. La dernière tentative est d'étudier les avantages potentiels de l'utilisation d'images multispectrales via des réseaux de neurones convolutifs. Sur la base de l'algorithme impressionnant FgSegNet_v2, les principales contributions de ce travail reposent sur deux aspects : (1) extraire trois canaux sur sept de l'ensemble des données multispectrales du FluxData FD-1665 pour correspondre au nombre de canaux d'entrée du modèle profond, et (2) proposer un nouvel encodeur convolutionnel pour pouvoir utiliser tous les canaux multispectraux disponibles permettant d’explorer davantage les informations des images multispectrales.

Individual grip strength provides a functional window into somatosensory processes and their effects on motor behaviour in healthy, impaired, and ageing individuals. Variations in grip strength during hand–tool interaction are therefore exploited in a variety of experimental tasks to study the effects of pathology or ageing-related changes on sensory, motor, and cognitive ability. However, many different factors may influence individual grip strength systematically in a given task context without being explicitly identified and controlled for. Grip strength may vary as a function of the location of the measurement device (sensor) on the fingers/hand, the shape, weight and size of object(s) being gripped, the type of grip investigated (static versus dynamic grip), and the hand (dominant versus non-dominant) used for gripping. This study tests for additional factors such as sight, sound, and interactions with/between any of the other factors in a complex task context. A wearable biosensor system, designed for measuring grip strength variations in operators gripping cylindrical objects bi-manually, was used. Grip force signals were recorded from all sensors of the wearable (glove) system, including three directly task-relevant sensors for bi-manually gripping cylindrical objects with the dominant and non-dominant hands. Five young male participants were tested for the effects of sound, movement, and sight on grip strength. The participants had to pick up two cylindrical objects of identical size and weight, then hold them still (static grip) or move them upwards and downwards (dynamic grip) for ten seconds while listening to soft or hard music, with their eyes open or blindfolded. Significant effects of sensor location, hand, movement, sight, and sound on bi-manual grip strength were found. Stronger grip force signals were produced by task-relevant sensors in the dominant hand when moving the cylindrical handles (dynamic grip) in comparison with the static grip condition, depending, as expected, on whether grip signals were measured from the dominant or the non-dominant hand. Significantly weaker grip strength was produced blindfolded (sight condition), and grips were significantly stronger with exposure to harder music (sound factor). It is concluded that grip strength is significantly influenced by sensory factors and interactions between the other factors tested for, pointing towards the need for identifying and systematically controlling such potential sources of variation in complex study task contexts.

New technologies for monitoring grip forces during hand and finger movements in non-standard task contexts have provided unprecedented functional insights into somatosensory cognition. Somatosensory cognition is the basis of our ability to manipulate and transform objects of the physical world and to grasp them with the right amount of force. In previous work, the wireless tracking of grip-force signals recorded from biosensors in the palm of the human hand has permitted us to unravel some of the functional synergies that underlie perceptual and motor learning under conditions of non-standard and essentially unreliable sensory input. This paper builds on this previous work and discusses further, functionally motivated, analyses of individual grip-force data in manual robot control. Grip forces were recorded from various loci in the dominant and non-dominant hands of individuals with wearable wireless sensor technology. Statistical analyses bring to the fore skill-specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain-inspired neural network model that uses the output metric of a self-organizing pap with unsupervised winner-take-all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time and reliably captures the differences between novice and expert performance in terms of grip-force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip-force monitoring in real time. This will permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot-interaction, which represents unprecedented challenges for perceptual and motor adaptation in environmental contexts of high sensory uncertainty. Cross-disciplinary insights from systems neuroscience and cognitive behavioral science, and the predictive modeling of operator skills using parsimonious Artificial Intelligence (AI), will contribute towards improving the outcome of new types of surgery, in particular the single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single-Incision Laparoscopic Surgery).

Aerial drone technology is currently being investigated worldwide for the delivery of blood components. Although it has been demonstrated to be safe, the delivered medical substances still need to be analyzed at the end of the flight mission to assess the level of haemolysis and pH prior to the use in a patient. This process can last up to 30 min and prevent the time saved using drone delivery. Our study aims to integrating an innovative sensor for the haemolysis and pH detection into the Smart Capsule, an already demonstrated technology capable of managing transfusion transport through drones. In the proposed scenario, the haemolysis is evaluated optically by a minilysis device using LED–photodetector combination. The preliminary validation has been demonstrated for both the thermal stability of the Smart Capsule and the haemolysis detection of the minilysis device prototype. Firstly, the onboard temperature test has shown that the delivery system is capable of maintaining proper temperature, even though the samples have been manipulated to reach a higher temperature before inserting into the Smart Capsule. Then, in the laboratory haemolysis test, the trend of linear regression between the outputs from the spectrophotometer and the minilysis prototype confirmed the concept design of the minilysis device.

This paper proposes a device of sensing that could be integrated into the instruments of any surgical robot. Despite advances in robot-assisted laparoscopic surgery, the tools currently supplied to surgical robots have limited functions, due to the absence of sensorization. With this motivation, we present a preliminary work based on the design, development, and early stages of experimentation with smart and multifunctional devices of sensing for surgical tools. The proposed device of sensing has a proximity sensor, colorimetric sensor, and BLE connection for different surgical instruments to connect to each other. The proximity feedback allows the surgeon to know the distance of the instrument from a particular tissue, to operate in conditions of greater safety. With the colorimetric feedback, on the other hand, we intend to proceed to the identification of specific tissue areas with characteristics that are not typical of the physiological tissue. The results show that the device is promising and can be further developed for multiple clinical needs in robotic procedures. This system can effectively increase the functionality of surgical instruments by overcoming the sensing limitations introduced by using robots in laparoscopic surgery.

This concept paper draws from our previous research on individual grip force data collected from biosensors placed on specific anatomical locations in the dominant and non-dominant hand of operators performing a robot-assisted precision grip task for minimally invasive endoscopic surgery. The specificity of the robotic system on the one hand, and that of the 2D image-guided task performed in a real-world 3D space on the other, constrain the individual hand and finger movements during task performance in a unique way. Our previous work showed task-specific characteristics of operator expertise in terms of specific grip force profiles, which we were able to detect in thousands of highly variable individual data. This concept paper is focused on two complementary data analysis strategies that allow achieving such a goal. In contrast with other sensor data analysis strategies aimed at minimizing variance in the data, it is necessary to decipher the meaning of intra- and inter-individual variance in the sensor data on the basis of appropriate statistical analyses, as shown in the first part of this paper. Then, it is explained how the computation of individual spatio-temporal grip force profiles allows detecting expertise-specific differences between individual users. It is concluded that both analytic strategies are complementary and enable drawing meaning from thousands of biosensor data reflecting human performance measures while fully taking into account their considerable inter- and intra-individual variability.

Deep learning has provided new ways of manipulating, processing and analyzing data. It sometimes may achieve results comparable to, or surpassing human expert performance, and has become a source of inspiration in the era of artificial intelligence. Another subfield of machine learning named reinforcement learning, tries to find an optimal behavior strategy through interactions with the environment. Combining deep learning and reinforcement learning permits resolving critical issues relative to the dimensionality and scalability of data in tasks with sparse reward signals, such as robotic manipulation and control tasks, that neither method permits resolving when applied on its own. In this paper, we present recent significant progress of deep reinforcement learning algorithms, which try to tackle the problems for the application in the domain of robotic manipulation control, such as sample efficiency and generalization. Despite these continuous improvements, currently, the challenges of learning robust and versatile manipulation skills for robots with deep reinforcement learning are still far from being resolved for real-world applications.

The temporal evolution of individual grip force profiles of a novice using a robotic system for minimally invasive endoscopic surgery is analyzed on the basis of thousands of individual sensor data recorded in real time through a wearable wireless sensor glove system. The spatio-temporal grip force profiles from specific sensor locations in the dominant hand performing a four-step pick-and-drop simulator task reveal skill-relevant differences in force deployment by the small finger (fine grip force control) and the middle finger (gross grip force contribution) by comparison with the profiles of a highly proficient expert. Cross-disciplinary insights from systems neuroscience, cognitive behavioral science, and robotics, with implications for biologically inspired AI for human–robot interactions, highlight the functional significance of spatio-temporal grip force profiling.