Chee-Kong Chui

Instrument-tissue interaction detection, a task aimed at understanding surgical scenes from videos, holds immense importance in constructing computer-assisted surgery systems. Existing methods consist of two stages: instance detection and interaction prediction. This sequential and separate model structure limits both effectiveness and efficiency, making it difficult to deploy on surgical robotic platforms. In this paper, we propose an end-to-end Action-Instance Progressive Learning Network (AIPNet) for the task. The model operates in three steps: action detection, instance detection, and action class refinement. Starting with coarse-scale proposals, the model progressively refines them into coarse-grained actions, which then serve as proposals for instance detection. The action prediction results are further refined using instance features through late fusion. These progressive learning processes improve the performance of the end-to-end model. Additionally, we introduce Dynamic Proposal Generators (DPG) to create dynamic adaptive learnable proposals for each video frame. To address the training challenges of this multi-task model, semantic supervised training is introduced to transfer prior language knowledge, and a training label strategy is proposed to generate unrelated instrument-tissue pair labels for enhanced supervision. Experimental results on PhacoQ and CholecQ datasets show that the proposed method achieves superior accuracy and faster processing speed than state-of-the-art models.

This study introduces an Improved Sparrow Search Algorithm (ISSA) to address the challenges of Machine Learning Hyperparameter Optimization (HPO) and efficient modeling of Electric Coolant Pumps (ECP) in electric vehicles. By integrating Lévy flight and Bernoulli mapping, ISSA enhances global search capabilities and ability to escapes from local optima. Experimental validations across 12 benchmark functions demonstrate ISSA outperforming the standard Sparrow Search Algorithm (SSA) and other advanced optimization algorithms in terms of exploratory and exploitative effectiveness. Specifically, ISSA proves exceptionally effective in autonomously handling the HPO for three typical machine learning (ML) algorithms, demonstrating superior performance over SSA and random search methods. A novel ISSA-ML surrogate model for ECP, incorporating structural and operational parameters, showcases significant improvements in predictive accuracy and robustness over traditional polynomial regression and ML models under conventional HPO methods. Furthermore, the application of this surrogate model in multi-objective optimization design for ECPs significantly reduces development time and computational costs, offering a streamlined and cost-effective solution for optimizing ECP performance. This study highlights potential future research directions, including the integration of other ML enhancements and the inclusion of more comprehensive feature parameters, to further improve the model's universality and applicability.

Instrument-tissue interaction detection task, which helps understand surgical activities, is vital for constructing computer-assisted surgery systems but with many challenges. Firstly, most models represent instrument-tissue interaction in a coarse-grained way which only focuses on classification and lacks the ability to automatically detect instruments and tissues. Secondly, existing works do not fully consider relations between intra-and inter-frame of instruments and tissues. In the paper, we propose to represent instrument-tissue interaction as ⟨instrument class, instrument bounding box, tissue class, tissue bounding box, action class⟩ quintuple and present an Instrument-Tissue Interaction Detection Network (ITIDNet) to detect the quintuple for surgery videos understanding. Specifically, we propose a Snippet Consecutive Feature (SCF) Layer to enhance features by modeling relationships of proposals in the current frame using global context information in the video snippet. We also propose a Spatial Corresponding Attention (SCA) Layer to incorporate features of proposals between adjacent frames through spatial encoding. To reason relationships between instruments and tissues, a Temporal Graph (TG) Layer is proposed with intra-frame connections to exploit relationships between instruments and tissues in the same frame and inter-frame connections to model the temporal information for the same instance. For evaluation, we build a cataract surgery video (PhacoQ) dataset and a cholecystectomy surgery video (CholecQ) dataset. Experimental results demonstrate the promising performance of our model, which outperforms other state-of-the-art models on both datasets.

The predictive energy management (PEM) of four-wheel drive (4WD) battery electric vehicles (BEVs) requires high accuracy in the driving information forecast of velocity and road gradient. In this paper, a meta-learning-based driving information prediction (DIP) method is proposed aiming to improve the performances of real-time PEM of 4WD-BEVs with a dual-electric machine (EM) driving system in each wheel. The proposed meta-learning-based DIP method is demonstrated to have higher accuracy, better robustness and faster computation speed by testing on collected real-world driving cycles. Then, a novel PEM method is proposed based on the model predictive controller (MPC), the reference projection of which is generated by the proposed meta-learning predictor. Our PEM strategy applies dynamic programming (DP) to solve the optimal control problem. The practicability of the proposed PEM method is analyzed in regard to the model accuracy, robustness to disturbances, generalization ability and real-time applicability. The proposed PEM method takes both the power distribution between the front and rear axles and that between two motors in each wheel into consideration. The simulation results reveal that the state-of-charge (SOC) declination of the proposed PEM method is better than the benchmark by 2.66% to 3.64%.

Keypoint detection is one of the main focused fields in computer vision with various applications. Traditional fully-supervised deep learning methods currently dominate the field with impressive accuracy, but typically require careful, expensive, and laborious effort for keypoint annotations. To tackle this problem, recent semi-supervised methods have emerged and shown great potential in utilizing a large amount of available unlabeled data. In this work, we explore a novel semi-supervised keypoint detection method, aiming to reduce the annotations required while maintaining the accuracy of traditional fully-supervised methods. We further augment the method by integrating it with a robust backbone network that has been pre-trained through self-supervised learning, thereby enabling better utilization of unlabeled data. Experimental results on three different datasets show that models trained using our semi-supervised method outperform their fully-supervised counterparts in accuracy despite using the same amount of labeled data. Additionally, under specific settings, our method can match the performance of existing semi-supervised methods even when using a reduced set of labeled data.

With the continuous development of intelligent operating room systems, the segmentation and automatic recognition of surgical workflow have become challenging research fields. In recent years, an increasing number of models have been proposed to address this challenge, with deep learning becoming the mainstream approach. In this paper, we propose a multi-stage network for surgical step recognition by using surgical video and kinematic data. Firstly, a convolutional neural network (ResNet34) is used to extract visual features from video frames. Next, since surgical videos are a form of sequential data, a Temporal Convolutional Network (TCN) is employed as a temporal extractor to process temporal information between video frames for classification. Finally, a multi-stage TCN network, consisting of Encoder-Decoded TCN and Dilated TCN architectures, is used to refine the result. The proposed network is compared against a LSTM network from our prior work and is evaluated on a surgical dataset named MISAW in two modes - video data with and without kinematic data. Experimental results indicate that kinematic data is crucial for robot motion control in the operating rooms of the future. The technology will also find application in robotic labs for the development and optimization of chemical manufacturing processes.

Editorial: Psychological and cognitive evaluation and intervention based on physiological and behavioral computing