Xiangteng Zang

Real-time quality monitoring using molten pool images is a critical focus in researching high-quality, intelligent automated welding. To address interference problems in molten pool images under complex welding scenarios (e.g., reflected laser spots from spatter misclassified as porosity defects) and the limited interpretability of deep learning models, this paper proposes a multi-granularity spatiotemporal representation learning algorithm based on the hybrid enhancement of handcrafted and deep learning features. A MobileNetV2 backbone network integrated with a Temporal Shift Module (TSM) is designed to progressively capture the short-term dynamic features of the molten pool and integrate temporal information across both low-level and high-level features. A multi-granularity attention-based feature aggregation module is developed to select key interference-free frames using cross-frame attention, generate multi-granularity features via grouped pooling, and apply the Convolutional Block Attention Module (CBAM) at each granularity level. Finally, these multi-granularity spatiotemporal features are adaptively fused. Meanwhile, an independent branch utilizes the Histogram of Oriented Gradient (HOG) and Scale-Invariant Feature Transform (SIFT) features to extract long-term spatial structural information from historical edge images, enhancing the model’s interpretability. The proposed method achieves an accuracy of 99.187% on a self-constructed dataset. Additionally, it attains a real-time inference speed of 20.983 ms per sample on a hardware platform equipped with an Intel i9-12900H CPU and an RTX 3060 GPU, thus effectively balancing accuracy, speed, and interpretability.

Fringe projection profilometry (FPP), with benefits such as high precision and a large depth of field, is a popular 3D optical measurement method widely used in precision reconstruction scenarios. However, the pixel brightness at reflective edges does not satisfy the conditions of the ideal pixel-wise phase-shifting model due to the influence of scene texture and system defocus, resulting in severe phase errors. To address this problem, we theoretically analyze the non-pixel-wise phase propagation model for texture edges and propose a reprojection strategy based on scene texture modulation. The strategy first obtains the reprojection weight mask by projecting typical FPP patterns and calculating the scene texture reflection ratio, then reprojects stripe patterns modulated by the weight mask to eliminate texture edge effects, and finally fuses coarse and refined phase maps to generate an accurate phase map. We validated the proposed method on various texture scenes, including a smooth plane, depth surface, and curved surface. Experimental results show that the root mean square error (RMSE) of the phase at the texture edge decreased by 53.32%, proving the effectiveness of the reprojection strategy in eliminating depth errors at texture edges.

Residual map-based defect detection methods have been widely concerned in the field of prints, because of their superiority in detecting defects of arbitrary shapes and extreme aspect ratios by taking advantage of pixel-level comparison, especially in detecting the tiny defects. However, residual map-based methods require extremely high structural similarity between images, so it is important to eliminate the structural differences by warping the tested image to template image via the displacement field. In practice, weak textures, large displacement, and deformation widely existed in the nonflat print images, which increased the difficulty of obtaining a satisfied displacement field. Inspired by the PatchMatch, our approach, adaptive coarse-to-fine PatchMatch (ACPM), blends an adaptive patch matching strategy with the coarse-to-fine scheme for displacement field estimation. We generate adaptive size patches according to the texture richness of local regions and propagate displacement information based on morphological operations, so the matching correspondences can be more reliable and accurate. Furthermore, the displacement field is interpolated from the matching result of ACPM and refined by the variational model to tackle nonrigid deformation (called ACPMFlow). Compared to the state-of-the-art method DeepFlow, ACPMFlow can achieve comparable performance with $20\times$ speed and obtain an excellent performance in the detection of tiny defects on nonflat prints, which is more suitable for real-time inspection.