Kaiyou Song
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Kaiyou Song has a total of 16 co-authors. Here are the top co-authors by number of shared publications:
- Hua Yang
School of Mechanical Science and Engineering, State Key Laboratory of Intelligent Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China
8 shared publications - Zhouping Yin
Flexible Electronics Research Center, State Key Laboratory of Intelligent Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, Wuhan, China
5 shared publications Zhouping Yin
State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China
3 shared publicationsChenghui Huang
State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China
1 shared publicationChenting Gong
State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, China
1 shared publication- Fangqin Mao
State Key Laboratory of Digital Manufacturing Equipment and Technology, School of Mechanical Science and Engineering, Huazhong University of Science and Technology, 1037 Luoyu Road, Wuhan, 430074, P.R. China
1 shared publication
Latest Publications
- 9 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (8) , 6200-6214
Bootstrap Masked Visual Modeling via Hard Patch Mining
Masked visual modeling has attracted much attention due to its promising potential in learning generalizable representations. Typical approaches urge models to predict specific contents of masked tokens, which can be intuitively considered as teaching a student (the model) to solve given problems (predicting masked contents). Under such settings, the performance is highly correlated with mask strategies (the difficulty of provided problems). We argue that it is equally important for the model to stand in the shoes of a teacher to produce challenging problems by itself. Intuitively, patches with high values of reconstruction loss can be regarded as hard samples, and masking those hard patches naturally becomes a demanding reconstruction task. To empower the model as a teacher, we propose Hard Patch Mining (HPM), predicting patch-wise losses and subsequently determining where to mask. Technically, we introduce an auxiliary loss predictor, which is trained with a relative objective to prevent overfitting to exact loss values. To gradually guide the training procedure, we propose an easy-to-hard mask strategy. Empirically, HPM brings significant improvements under both image and video benchmarks. Interestingly, solely incorporating the extra loss prediction objective leads to better representations, verifying the efficacy of determining where is hard to reconstruct.
- 9 April 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (8) , 6200-6214
Bootstrap Masked Visual Modeling via Hard Patch Mining
Masked visual modeling has attracted much attention due to its promising potential in learning generalizable representations. Typical approaches urge models to predict specific contents of masked tokens, which can be intuitively considered as teaching a student (the model) to solve given problems (predicting masked contents). Under such settings, the performance is highly correlated with mask strategies (the difficulty of provided problems). We argue that it is equally important for the model to stand in the shoes of a teacher to produce challenging problems by itself. Intuitively, patches with high values of reconstruction loss can be regarded as hard samples, and masking those hard patches naturally becomes a demanding reconstruction task. To empower the model as a teacher, we propose Hard Patch Mining (HPM), predicting patch-wise losses and subsequently determining where to mask. Technically, we introduce an auxiliary loss predictor, which is trained with a relative objective to prevent overfitting to exact loss values. To gradually guide the training procedure, we propose an easy-to-hard mask strategy. Empirically, HPM brings significant improvements under both image and video benchmarks. Interestingly, solely incorporating the extra loss prediction objective leads to better representations, verifying the efficacy of determining where is hard to reconstruct.
- 3 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 71 (00189456) , 1-14
Anomaly Composition and Decomposition Network for Accurate Visual Inspection of Texture Defects
Texture defect inspection remains challenging due to the extreme variations in various textures and defects. Current unsupervised-learning-based texture defect inspection methods cannot simultaneously inspect a wide variety of texture defects because they lack an explicit mechanism to encourage the model to create large anomaly scores for defects. In this study, we propose a novel anomaly composition and decomposition network (ACDN) for accurate inspection of various texture defects. In the proposed ACDN, a Gaussian-sampling-based anomaly composition (GSAC) method is proposed to perform the anomaly composition procedure, which composites a large number of defective images for training. Then, a novel anomaly decomposition network (ADN) is proposed to perform the anomaly decomposition procedure, which decomposes the defective images into texture background images and anomaly images by forcing the intrinsic texture features of abnormal images to share a common distribution with those of defect-free images. Through the GSAC and ADN, ACDN learns not only to accurately reconstruct texture background images to cause large reconstruction errors for defect regions but also accurately segment defects. In the testing phase, a defective image is decomposed into a texture background image and an anomaly image through the trained ADN. The residual image between the defective image and the texture background image is then fused with the anomaly image to obtain the defect inspection result. Extensive experimental results on mainstream texture defect datasets demonstrate that ACDN achieves the state-of-the-art texture defect inspection accuracy.
- 3 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 71 (00189456) , 1-14
Anomaly Composition and Decomposition Network for Accurate Visual Inspection of Texture Defects
Texture defect inspection remains challenging due to the extreme variations in various textures and defects. Current unsupervised-learning-based texture defect inspection methods cannot simultaneously inspect a wide variety of texture defects because they lack an explicit mechanism to encourage the model to create large anomaly scores for defects. In this study, we propose a novel anomaly composition and decomposition network (ACDN) for accurate inspection of various texture defects. In the proposed ACDN, a Gaussian-sampling-based anomaly composition (GSAC) method is proposed to perform the anomaly composition procedure, which composites a large number of defective images for training. Then, a novel anomaly decomposition network (ADN) is proposed to perform the anomaly decomposition procedure, which decomposes the defective images into texture background images and anomaly images by forcing the intrinsic texture features of abnormal images to share a common distribution with those of defect-free images. Through the GSAC and ADN, ACDN learns not only to accurately reconstruct texture background images to cause large reconstruction errors for defect regions but also accurately segment defects. In the testing phase, a defective image is decomposed into a texture background image and an anomaly image through the trained ADN. The residual image between the defective image and the texture background image is then fused with the anomaly image to obtain the defect inspection result. Extensive experimental results on mainstream texture defect datasets demonstrate that ACDN achieves the state-of-the-art texture defect inspection accuracy.
- 18 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 3858-3871
Weighted Feature Histogram of Multi-Scale Local Patch Using Multi-Bit Binary Descriptor for Face Recognition
Most face recognition methods employ single-bit binary descriptors for face representation. The information from these methods is lost in the process of quantization from real-valued descriptors to binary descriptors, which greatly limits their robustness for face recognition. In this study, we propose a novel weighted feature histogram (WFH) method of multi-scale local patches using multi-bit binary descriptors for face recognition. First, to obtain multi-scale information of the face image, the local patches are extracted using a multi-scale local patch generation (MSLPG) method. Second, with the goal of reducing the quantization information loss of binary descriptors, a novel multi-bit local binary descriptor learning (MBLBDL) method is proposed to extract multi-bit local binary descriptors (MBLBDs). In MBLBDL, a learned mapping matrix and novel multi-bit coding rules are employed to project pixel difference vectors (PDVs) into the MBLBDs in each local patch. Finally, a novel robust weight learning (RWL) method is proposed to learn a set of robust weights for each patch to integrate the MBLBDs into the final face representation. In RWL, a codebook is first constructed by clustering MBLBDs on each local patch to extract a feature histogram. Then, considering that different parts of the face have different degrees of robustness to local changes, a set of weights is learned to concatenate the feature histograms of all local patches into the final representation of a face image. In addition, to further improve the performance for heterogeneous face recognition, a coupled WFH (C-WFH) method is proposed. C-WFH maintains the similarity of the corresponding MBLBDs and feature histograms for a pair of heterogeneous face images by means of a novel coupled feature learning (CFL) method to reduce the modality gap. A series of experiments are conducted on widely used face datasets to analyze the performance of WFH and C-WFH. Extensive experimental results show that WFH and C-WFH outperform state-of-the-art face recognition methods.
- 18 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 30 (10577149) , 3858-3871
Weighted Feature Histogram of Multi-Scale Local Patch Using Multi-Bit Binary Descriptor for Face Recognition
Most face recognition methods employ single-bit binary descriptors for face representation. The information from these methods is lost in the process of quantization from real-valued descriptors to binary descriptors, which greatly limits their robustness for face recognition. In this study, we propose a novel weighted feature histogram (WFH) method of multi-scale local patches using multi-bit binary descriptors for face recognition. First, to obtain multi-scale information of the face image, the local patches are extracted using a multi-scale local patch generation (MSLPG) method. Second, with the goal of reducing the quantization information loss of binary descriptors, a novel multi-bit local binary descriptor learning (MBLBDL) method is proposed to extract multi-bit local binary descriptors (MBLBDs). In MBLBDL, a learned mapping matrix and novel multi-bit coding rules are employed to project pixel difference vectors (PDVs) into the MBLBDs in each local patch. Finally, a novel robust weight learning (RWL) method is proposed to learn a set of robust weights for each patch to integrate the MBLBDs into the final face representation. In RWL, a codebook is first constructed by clustering MBLBDs on each local patch to extract a feature histogram. Then, considering that different parts of the face have different degrees of robustness to local changes, a set of weights is learned to concatenate the feature histograms of all local patches into the final representation of a face image. In addition, to further improve the performance for heterogeneous face recognition, a coupled WFH (C-WFH) method is proposed. C-WFH maintains the similarity of the corresponding MBLBDs and feature histograms for a pair of heterogeneous face images by means of a novel coupled feature learning (CFL) method to reduce the modality gap. A series of experiments are conducted on widely used face datasets to analyze the performance of WFH and C-WFH. Extensive experimental results show that WFH and C-WFH outperform state-of-the-art face recognition methods.
- 12 January 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 31 (11) , 4269-4282
Multi-Scale Boosting Feature Encoding Network for Texture Recognition
Texture recognition remains a challenging visual task due to the complex appearance variations caused by scale changes in the real world. In most existing texture recognition methods, textures are represented at a single scale; thus, multi-scale texture information is not fully utilized, resulting in insufficient representation and inaccurate recognition. In this study, with the goal of addressing the challenge of scale changes, we propose a novel multi-scale boosting feature encoding network (MSBFEN) for accurate texture recognition. MSBFEN first extracts multi-scale features with multi-scale texture structure information under the guidance of texture priors using a novel prior-guided feature extraction (PFE) method. Then, a multi-scale texture encoding (MSTE) method is devised to capture discriminative multi-scale texture representations by encoding the extracted features. Finally, to fully utilize the multi-scale texture representations for accurate texture recognition, a novel multi-scale boosting learning (MSBL) method is proposed. In MSBL, the learning procedure for multi-scale texture recognition is boosted in a hierarchical, progressively reinforced manner, significantly addressing the challenge of scale changes and greatly enhancing the recognition accuracy. In addition, a novel outlier-aware texture encoding (OTE) method is proposed for robust texture encoding at each scale of MSTE. OTE can resist the influence of background interference and can further enhance the robustness of MSBFEN. In extensive experiments conducted on six challenging texture recognition datasets, namely, KTH-TIPS2b, FMD, DTD, MINC, GTOS and GTOS-mobile, MSBFEN achieves accuracies of 86.2%, 86.4%, 77.8%, 85.3%, 86.4% and 87.57%, respectively, representing state-of-the-art texture recognition performance.
- 12 January 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 31 (11) , 4269-4282
Multi-Scale Boosting Feature Encoding Network for Texture Recognition
Texture recognition remains a challenging visual task due to the complex appearance variations caused by scale changes in the real world. In most existing texture recognition methods, textures are represented at a single scale; thus, multi-scale texture information is not fully utilized, resulting in insufficient representation and inaccurate recognition. In this study, with the goal of addressing the challenge of scale changes, we propose a novel multi-scale boosting feature encoding network (MSBFEN) for accurate texture recognition. MSBFEN first extracts multi-scale features with multi-scale texture structure information under the guidance of texture priors using a novel prior-guided feature extraction (PFE) method. Then, a multi-scale texture encoding (MSTE) method is devised to capture discriminative multi-scale texture representations by encoding the extracted features. Finally, to fully utilize the multi-scale texture representations for accurate texture recognition, a novel multi-scale boosting learning (MSBL) method is proposed. In MSBL, the learning procedure for multi-scale texture recognition is boosted in a hierarchical, progressively reinforced manner, significantly addressing the challenge of scale changes and greatly enhancing the recognition accuracy. In addition, a novel outlier-aware texture encoding (OTE) method is proposed for robust texture encoding at each scale of MSTE. OTE can resist the influence of background interference and can further enhance the robustness of MSBFEN. In extensive experiments conducted on six challenging texture recognition datasets, namely, KTH-TIPS2b, FMD, DTD, MINC, GTOS and GTOS-mobile, MSBFEN achieves accuracies of 86.2%, 86.4%, 77.8%, 85.3%, 86.4% and 87.57%, respectively, representing state-of-the-art texture recognition performance.
- 19 October 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 70 (00189456) , 1-15
Autolabeling-Enhanced Active Learning for Cost-Efficient Surface Defect Visual Classification
Active learning can reduce the human effort required for labeling training samples while preserving the performance of visual classifiers. However, existing active learning frameworks cannot be used to perform visual classification of industrial product surface defects because they still require intensive manual annotation efforts. In this study, we propose a cost-efficient autolabeling-enhanced active learning (ALEAL) framework to reduce the human annotation effort required for surface defect visual classification. The proposed ALEAL framework employs a deep convolutional neural network (CNN) as a visual classifier trained from an initial set of humanlabeled training samples. Then, the collected unlabeled training samples are input into the classifier for category confidence estimation. Next, a novel diverse cost-effective query strategy (DCEQS) is proposed to select some high-confidence samples for autolabeling and to select some informative samples for sample proposals that need labeling. Subsequently, to further reduce the human annotation effort, a novel autolabeling module is proposed and introduced in ALEAL that can automatically label a portion of the informative unlabeled training samples selected by the DCEQS. In this study, a novel attention-based similarity measurement network (ASMN) is proposed as an implementation of this autolabeling module by measuring the similarity between unlabeled and labeled samples. Finally, the remaining unlabeled samples are annotated by human experts, and all the newly labeled samples are used to retrain the classifier. Through the autolabeling process from the DCEQS and ASMN, ALEAL can automatically label additional training samples and achieve a competitive performance while requiring few humanlabeled training samples, which is highly important in industrial applications. Extensive experimental results show that compared with popular active learning methods, ALEAL can dramatically reduce the effort involved in human annotation and achieve stateof-the-art cost efficiency for the visual classification of industrial product surface defects.
- 19 October 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Instrumentation and Measurement
- Vol. 70 (00189456) , 1-15
Autolabeling-Enhanced Active Learning for Cost-Efficient Surface Defect Visual Classification
Active learning can reduce the human effort required for labeling training samples while preserving the performance of visual classifiers. However, existing active learning frameworks cannot be used to perform visual classification of industrial product surface defects because they still require intensive manual annotation efforts. In this study, we propose a cost-efficient autolabeling-enhanced active learning (ALEAL) framework to reduce the human annotation effort required for surface defect visual classification. The proposed ALEAL framework employs a deep convolutional neural network (CNN) as a visual classifier trained from an initial set of humanlabeled training samples. Then, the collected unlabeled training samples are input into the classifier for category confidence estimation. Next, a novel diverse cost-effective query strategy (DCEQS) is proposed to select some high-confidence samples for autolabeling and to select some informative samples for sample proposals that need labeling. Subsequently, to further reduce the human annotation effort, a novel autolabeling module is proposed and introduced in ALEAL that can automatically label a portion of the informative unlabeled training samples selected by the DCEQS. In this study, a novel attention-based similarity measurement network (ASMN) is proposed as an implementation of this autolabeling module by measuring the similarity between unlabeled and labeled samples. Finally, the remaining unlabeled samples are annotated by human experts, and all the newly labeled samples are used to retrain the classifier. Through the autolabeling process from the DCEQS and ASMN, ALEAL can automatically label additional training samples and achieve a competitive performance while requiring few humanlabeled training samples, which is highly important in industrial applications. Extensive experimental results show that compared with popular active learning methods, ALEAL can dramatically reduce the effort involved in human annotation and achieve stateof-the-art cost efficiency for the visual classification of industrial product surface defects.
- 11 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 17 (3) , 2220-2230
An Anomaly Feature-Editing-Based Adversarial Network for Texture Defect Visual Inspection
Establishing a unified model for defect inspection remains a challenge in the industrial automation field due to the various texture surfaces. Current unsupervised methods cannot address anomalies, which causes these methods cannot inspect various texture defects. In this paper, we propose a novel unsupervised anomaly-feature-editing-based adversarial network (AFEAN) for accurate inspection of various texture defects. First, a paired input, consisting of defect-free image and its artificially defective image, is utilized to train AFEAN. Subsequently, a novel anomaly feature detection module is proposed to detect anomaly features in the latent space. Next, a global context feature editing module is proposed to suppress the reconstruction of defects. Finally, a feature decoding module utilizes the edited features to reconstruct the texture background. In the testing phase, the defects are segmented by the residuals between the input and the reconstructed texture background. The extensive experimental results demonstrate that AFEAN achieves state-of-the-art inspection accuracy.
- 11 August 2020
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Industrial Informatics
- Vol. 17 (3) , 2220-2230
An Anomaly Feature-Editing-Based Adversarial Network for Texture Defect Visual Inspection
Establishing a unified model for defect inspection remains a challenge in the industrial automation field due to the various texture surfaces. Current unsupervised methods cannot address anomalies, which causes these methods cannot inspect various texture defects. In this paper, we propose a novel unsupervised anomaly-feature-editing-based adversarial network (AFEAN) for accurate inspection of various texture defects. First, a paired input, consisting of defect-free image and its artificially defective image, is utilized to train AFEAN. Subsequently, a novel anomaly feature detection module is proposed to detect anomaly features in the latent space. Next, a global context feature editing module is proposed to suppress the reconstruction of defects. Finally, a feature decoding module utilizes the edited features to reconstruct the texture background. In the testing phase, the defects are segmented by the residuals between the input and the reconstructed texture background. The extensive experimental results demonstrate that AFEAN achieves state-of-the-art inspection accuracy.
- 17 January 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (6) , 3061-3074
Robust Semantic Template Matching Using a Superpixel Region Binary Descriptor
Almost all conventional template-matching methods employ low-level image features to measure the similarity between a template image and a scene image using similarity measures, such as pixel intensity and pixel gradient. Although these methods have been widely used in many applications, they cannot simultaneously address all types of robustness challenges. In this paper, with the goal of simultaneously addressing the various challenges, we present a robust semantic template-matching (RSTM) approach. Inspired by the local binary descriptor, we propose a novel superpixel region binary descriptor (SRBD) to construct a multilevel semantic fusion feature vector for RSTM. SRBD uses a new kernel-distance-based simple linear iterative clustering method to extract the stable superpixels from the template image. Then, based on the average intensity difference between each superpixel region and its neighbors, the dominant gradient orientation of each superpixel can be obtained, and the semantic features of each superpixel can be described as the dominant orientation difference vector, which is coded as the rotation-invariant SRBD. In the offline matching phase, the fusion semantic feature vector of RSTM combines the multilevel SRBD features with different numbers of superpixels. In the online matching phase, to cope with rotation invariance, a marginal probability model is proposed and applied to locate the positions of template images in the scene image. Moreover, to accelerate computation, an image pyramid is employed. We conduct a series of experiments on a large dataset randomly selected from the MS COCO dataset to fully analyze the robustness of this approach. The experimental results show that RSTM simultaneously addresses rotation changes, scale changes, noise, occlusions, blur, nonlinear illumination changes, and deformation with high time efficiency while also outperforming the previous state-of-the-art template-matching methods.
- 17 January 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Image Processing
- Vol. 28 (6) , 3061-3074
Robust Semantic Template Matching Using a Superpixel Region Binary Descriptor
Almost all conventional template-matching methods employ low-level image features to measure the similarity between a template image and a scene image using similarity measures, such as pixel intensity and pixel gradient. Although these methods have been widely used in many applications, they cannot simultaneously address all types of robustness challenges. In this paper, with the goal of simultaneously addressing the various challenges, we present a robust semantic template-matching (RSTM) approach. Inspired by the local binary descriptor, we propose a novel superpixel region binary descriptor (SRBD) to construct a multilevel semantic fusion feature vector for RSTM. SRBD uses a new kernel-distance-based simple linear iterative clustering method to extract the stable superpixels from the template image. Then, based on the average intensity difference between each superpixel region and its neighbors, the dominant gradient orientation of each superpixel can be obtained, and the semantic features of each superpixel can be described as the dominant orientation difference vector, which is coded as the rotation-invariant SRBD. In the offline matching phase, the fusion semantic feature vector of RSTM combines the multilevel SRBD features with different numbers of superpixels. In the online matching phase, to cope with rotation invariance, a marginal probability model is proposed and applied to locate the positions of template images in the scene image. Moreover, to accelerate computation, an image pyramid is employed. We conduct a series of experiments on a large dataset randomly selected from the MS COCO dataset to fully analyze the robustness of this approach. The experimental results show that RSTM simultaneously addresses rotation changes, scale changes, noise, occlusions, blur, nonlinear illumination changes, and deformation with high time efficiency while also outperforming the previous state-of-the-art template-matching methods.
- 1 January 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 16 (3) , 1450-1467
Multiscale Feature-Clustering-Based Fully Convolutional Autoencoder for Fast Accurate Visual Inspection of Texture Surface Defects
Visual inspection of texture surface defects is still a challenging task in the industrial automation field due to the tremendous changes in the appearance of various surface textures. Current visual inspection methods cannot simultaneously and efficiently inspect various types of texture defects due to either the low discriminative capabilities of handcrafted features or their time-consuming sliding-window strategy. In this paper, we present a novel unsupervised multiscale feature-clustering-based fully convolutional autoencoder (MS-FCAE) method that efficiently and accurately inspects various types of texture defects based on a small number of defect-free texture samples. The proposed MS-FCAE method utilizes multiple FCAE subnetworks at different scale levels to reconstruct several textured background images. The residual images are obtained by subtracting these texture backgrounds from the input image individually; then, they are fused into one defect image. To maximize the efficiency, each FCAE subnetwork utilizes fully convolutional neural networks to extract the original feature maps directly from the input images. Meanwhile, each FCAE subnetwork performs feature clustering to improve the discriminant power of the encoded feature maps. The proposed MS-FCAE method is evaluated on several texture surface inspection data sets both qualitatively and quantitatively. This method achieves a Precision of 92.0% while requiring only 82 ms for input images of 1920x 1080 pixels. The extensive experimental results demonstrate that MS-FCAE achieves highly efficient and state-of-the-art inspection accuracy.
- 1 January 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Automation Science and Engineering
- Vol. 16 (3) , 1450-1467
Multiscale Feature-Clustering-Based Fully Convolutional Autoencoder for Fast Accurate Visual Inspection of Texture Surface Defects
Visual inspection of texture surface defects is still a challenging task in the industrial automation field due to the tremendous changes in the appearance of various surface textures. Current visual inspection methods cannot simultaneously and efficiently inspect various types of texture defects due to either the low discriminative capabilities of handcrafted features or their time-consuming sliding-window strategy. In this paper, we present a novel unsupervised multiscale feature-clustering-based fully convolutional autoencoder (MS-FCAE) method that efficiently and accurately inspects various types of texture defects based on a small number of defect-free texture samples. The proposed MS-FCAE method utilizes multiple FCAE subnetworks at different scale levels to reconstruct several textured background images. The residual images are obtained by subtracting these texture backgrounds from the input image individually; then, they are fused into one defect image. To maximize the efficiency, each FCAE subnetwork utilizes fully convolutional neural networks to extract the original feature maps directly from the input images. Meanwhile, each FCAE subnetwork performs feature clustering to improve the discriminant power of the encoded feature maps. The proposed MS-FCAE method is evaluated on several texture surface inspection data sets both qualitatively and quantitatively. This method achieves a Precision of 92.0% while requiring only 82 ms for input images of 1920x 1080 pixels. The extensive experimental results demonstrate that MS-FCAE achieves highly efficient and state-of-the-art inspection accuracy.
- 11 October 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 29 (10) , 2972-2985
Multi-Scale Attention Deep Neural Network for Fast Accurate Object Detection
Object detection remains a challenging task in computer vision due to the tremendous extent of changes in the appearances of objects caused by clustered backgrounds, occlusion, truncation and scale change. Current deep neural network (DNN)-based object detection methods cannot simultaneously achieve a high accuracy and a high efficiency. To overcome this limitation, in this study, we propose a novel multi-scale attention deep neural network (MSA-DNN) for accurate object detection with high efficiency. The proposed MSA-DNN method utilizes a novel multi-scale feature fusion module (MSFFM) to construct high-level semantic features. Subsequently, a novel multi-scale attention module (MSAM) based on the fused layers of the MSFFM is introduced to exploit the global semantic information of image-level labels to guide detection. On the one hand, MSAM can capture global semantic information to further enhance the semantic feature representation of the fused layers constructed by the MSFFM, thereby improving the detection accuracy. On the other hand, the multi-scale attention maps generated by MSAM can be employed to rapidly and coarsely locate objects at different scales. In addition, an attention-based hard negative mining strategy (AHNM) is introduced to filter out negative samples to reduce the search space, dramatically alleviating the severe class imbalance problem. Extensive experimental results on the challenging PASCAL VOC 2007, PASCAL VOC 2012 and MS COCO datasets demonstrate that MSA-DNN achieves a stateof- the-art detection accuracy while maintaining a high efficiency. Furthermore, MSA-DNN significantly improves the small-object detection accuracy.
- 11 October 2018
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 29 (10) , 2972-2985
Multi-Scale Attention Deep Neural Network for Fast Accurate Object Detection
Object detection remains a challenging task in computer vision due to the tremendous extent of changes in the appearances of objects caused by clustered backgrounds, occlusion, truncation and scale change. Current deep neural network (DNN)-based object detection methods cannot simultaneously achieve a high accuracy and a high efficiency. To overcome this limitation, in this study, we propose a novel multi-scale attention deep neural network (MSA-DNN) for accurate object detection with high efficiency. The proposed MSA-DNN method utilizes a novel multi-scale feature fusion module (MSFFM) to construct high-level semantic features. Subsequently, a novel multi-scale attention module (MSAM) based on the fused layers of the MSFFM is introduced to exploit the global semantic information of image-level labels to guide detection. On the one hand, MSAM can capture global semantic information to further enhance the semantic feature representation of the fused layers constructed by the MSFFM, thereby improving the detection accuracy. On the other hand, the multi-scale attention maps generated by MSAM can be employed to rapidly and coarsely locate objects at different scales. In addition, an attention-based hard negative mining strategy (AHNM) is introduced to filter out negative samples to reduce the search space, dramatically alleviating the severe class imbalance problem. Extensive experimental results on the challenging PASCAL VOC 2007, PASCAL VOC 2012 and MS COCO datasets demonstrate that MSA-DNN achieves a stateof- the-art detection accuracy while maintaining a high efficiency. Furthermore, MSA-DNN significantly improves the small-object detection accuracy.