Guo Yang

The rapid development of large language models (LLMs) and multimodal large models (MLMs) has introduced transformative opportunities for autonomous driving systems. These advanced models provide robust support for the realization of more intelligent, safer, and efficient autonomous driving. In this paper, we present a systematic review on the integration of LLMs and MLMs in autonomous driving systems. First, we provide an overview of the evolution of LLMs and MLMs, along with a detailed analysis of the architecture of autonomous driving systems. Next, we explore the applications of LLMs and MLMs in key components such as perception, prediction, decision making, planning, multitask processing, and human–machine interaction. Additionally, this paper reviews the core technologies involved in integrating LLMs and MLMs with autonomous driving systems, including multimodal fusion, knowledge distillation, prompt engineering, and supervised fine tuning. Finally, we provide an in-depth analysis of the major challenges faced by autonomous driving systems powered by large models, offering new perspectives for future research. Compared to existing review articles, this paper not only systematically examines the specific applications of LLMs and MLMs in autonomous driving systems but also delves into the key technologies and potential challenges involved in their integration. By comprehensively organizing and analyzing the current literature, this review highlights the application potential of large models in autonomous driving and offers insights and recommendations for improving system safety and efficiency.

The existing fault diagnosis of harmonic drives is difficult to deploy vibration sensors, and the diagnosis accuracy is insufficient only using current signal. Therefore, we propose an intelligent fault diagnosis method using a multimodal collaborative meta network (MCMN) with severely missing modality. First, multimodal data of the harmonic drive were collected to analyze. Second, a feature reconstruction network is used to achieve a unified model to handle missing modalities in testing. Besides, the uncertainty assessment is used as a feature regularization network to overcome data bias. Third, the inference of the meta-learning is used to obtain the generated weights, and carried out the joint optimization. Finally, MCMN is optimized by using the approximate lower bound of Monte Carlo. Experimental results show that the accuracy of MCMN is 93.75% in the case of full-modalities training data and the severely missing modalities testing data, which is better than the existing method.

This paper designed and developed an online digital imaging excitation sensor for wind power gearbox wear condition monitoring based on an adaptive deep learning method. A digital imaging excitation sensing image information collection architecture for magnetic particles in lubricating oil was established to characterize the wear condition of mechanical equipment, achieving the real-time online collection of wear particles in lubricating oil. On this basis, a mechanical equipment wear condition diagnosis method based on online wear particle images is proposed, obtaining data from an engineering test platform based on a wind power gearbox. Firstly, a foreground segmentation preprocessing method based on the U-Net network can effectively eliminate the interference of bubbles and dark fields in online wear particle images, providing high-quality segmentation results for subsequent image processing, A total of 1960 wear particle images were collected in the experiment, the average intersection union ratio of the validation set is 0.9299, and the accuracy of the validation set is 0.9799. Secondly, based on the foreground segmentation preprocessing of wear particle images, by using the watered algorithm to obtain the number of particles in each size segment, we obtained the number of magnetic particle grades in three different ranges: 4–38 µm, 39–70 µm, and >70 µm. Thirdly, we proposed a method named multidimensional transformer (MTF) network. Mean Square Error (MSE), Root Mean Square Error (RMSE), and Mean Absolute Error (MAE) are used to obtain the error, and the maintenance strategy is formulated according to the predicted trend. The experimental results show that the predictive performance of our proposed model is better than that of LSTM and TCN. Finally, the online real-time monitoring system triggered three alarms, and at the same time, our offline sampling data analysis was conducted, the accuracy of online real-time monitoring alarms was verified, and the gearbox of the wind turbine was shut down for maintenance and repair.

Harmonic drive is a core component of the industrial robot, its failure will directly affect the robot's performance. Moreover, as the harmonic drive often works with excessive speed and load, it may fail unpredictably. Therefore, online fault diagnosis is quite significant. In this paper, we propose an online intelligent fault diagnosis method for harmonic drives using a Semi-supervised Contrastive Graph Generative Network (SCGGN) via multimodal data. First, multimodal data (including motor current signals and encoder signals) of the harmonic drive are collected online. The Euclidean distance is used to analyze the similarity of the data in the frequency domain. Second, Multiple Graph Convolution Network (MGCN) and Hierarchical Graph Convolution Network (HGCN) are used to obtain complementary fault features from local and global views, respectively. Third, the Contrastive Learning Network is constructed to obtain high-level information through unsupervised learning and perform data clustering to obtain the multi-classification output. Finally, a combination of learnable loss functions is used to optimize the SCGGN. The presented method is tested on an industrial robot. The experimental results show that the method can achieve 86.15% accuracy with 8% of the labeled training data and 79.9% accuracy with only 0.5% of the labeled training data, which is superior to the existing methods.

The harmonic drive is a key component of the industrial robot. Because of its large reduction ratio and excessive dynamic loading, various kinds of faults may occur. In particular, since the robot is an integrated system, it is not unusual to have multiple harmonic drive malfunctions simultaneously, which is difficult to diagnose. In practice, these kinds of compound faults are often mislabeled as single faults causing missing repair. In this article, we propose a deep capsule graph convolutional network (DCGCN) approach to diagnose compound faults of harmonic drives. First, the multisensor data are used to obtain the frequency spectrum of the fault signal and construct the label relationship map of the adjacency matrix. Second, the deep capsule network is used to learn the representation of the fault vector, and the graph convolutional network is used to learn the relationship between different single-label faults. Third, the two networks are combined to obtain diagnosing results. Finally, the dynamic routing algorithm and the margin loss function are used to optimize the DCGCN. The experimental results show that the proposed DCGCN can effectively diagnose compound faults under varying working conditions, outperforming other existing state-of-the-art methods.

Harmonic drive is the core component of the industrial robot, and its fault diagnosis is crucial to the reliability and performance of the equipment. Most machine learning methods achieve good results based on the assumption of data balance. However, the scarce fault data of harmonic drive is difficult to collect, resulting in various imbalanced health status samples, which has an adverse effect on fault diagnosis. In this article, we propose a data generation method based on generative adversarial networks (GANs) to solve the problem of data imbalance and utilize the multiscale convolutional neural network (MSCNN) to realize the fault diagnosis of the harmonic drive. First, the data collected from three vibration acceleration sensors are preprocessed by fast Fourier transform (FFT) to obtain the frequency spectrum of the vibration signal. Second, multiple GANs were adopted to generate various fault spectrum data and the data selection module (DSM) is elaborately designed to filter and purify these data. Third, the filtered generated data will be combined with the real data to form a balanced dataset, and then the MSCNN is used to achieve multiclassification of the health status of the harmonic drive. Finally, the experiments have been implemented on an industrial robot vibration test bench to validate the effectiveness of our approach. The results have shown the fault multiclassification accuracy as 98.49% under imbalanced fault data conditions, which outperforms that of the other compared methods.

Harmonic drive is a key component in industry robot. Its complex design is highly sensitive to the manufacturing and assembly errors. Even a small error could cause excessive vibration jeopardizing the performance of the robot. Moreover, the vibration is dependent on the operation conditions. As a result, it is necessary to carry out a comprehensive test to detect possible manufacturing and assembling faults. This paper presents an intelligent approach that can automatically identify the different health conditions of harmonic drives. First, the acquired data from multiple acceleration sensors are cascaded in sequence after several preprocessing steps. Then, the multiscale convolutional neural network (MSCNN) architecture is used for disposing the inherent multiscale characteristics of the harmonic drive vibration signal, which can simultaneously perform multiscale feature extraction and classification. Finally, a large number of experiments were conducted on a real industrial robot vibration test platform to evaluate our approach. Based on the experiment results, the fault detection classification accuracy is 96.79%, which is higher than the other compared methods. This demonstrates that the presented method is effective for shop floor applications.

A novel framework of optical image hiding based on deep learning (DL) is proposed in this paper, and hidden information can be reconstructed from an interferogram by using an end to end network with high-quality. By using the prior data between the hidden image and the object image, a generative adversarial network was trained so that it can learn the hiding model, which resulting in only an interferogram needs to be transmitted and recorded to reconstruct image. Moreover, reconstruction process can be obtained without the parameters in optical inverse diffraction and the reconstruction result will not be affected by the phase shifts deviation and noise, which is convenient for practical application. The feasibility and security of the proposed method are demonstrated by the optical experiment results.