Deep spectral improvement for unsupervised image instance segmentation
Authors:
Open Access
- 7 October 2024
- journal article
- research article
- Published by Public Library of Science (PLoS) in PLOS ONE
- Vol. 19 (10) , e0307432
Abstract
Recently, there has been growing interest in deep spectral methods for image localization and segmentation, influenced by traditional spectral segmentation approaches. These methods reframe the image decomposition process as a graph partitioning task by extracting features using self-supervised learning and utilizing the Laplacian of the affinity matrix to obtain eigensegments. However, instance segmentation has received less attention than other tasks within the context of deep spectral methods. This paper addresses that not all channels of the feature map extracted from a self-supervised backbone contain sufficient information for instance segmentation purposes. Some channels are noisy and hinder the accuracy of the task. To overcome this issue, this paper proposes two channel reduction modules, Noise Channel Reduction (NCR) and Deviation-based Channel Reduction (DCR). The NCR retains channels with lower entropy, as they are less likely to be noisy, while DCR prunes channels with low standard deviation, as they lack sufficient information for effective instance segmentation. Furthermore, the paper demonstrates that the dot product, commonly used in deep spectral methods, is not suitable for instance segmentation due to its sensitivity to feature map values, potentially leading to incorrect instance segments. A novel similarity metric called Bray-curtis over Chebyshev (BoC) is proposed to address this issue. This metric considers the distribution of features in addition to their values, providing a more robust similarity measure for instance segmentation. Quantitative and qualitative results on the Youtube-VIS 2019 and OVIS datasets highlight the improvements achieved by the proposed channel reduction methods and using BoC instead of the conventional dot product for creating the affinity matrix. These improvements regarding mean Intersection over Union (mIoU) and extracted instance segments are observed, demonstrating enhanced instance segmentation performance. The code is available on: https://github.com/farnooshar/SpecUnIIS.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Occluded Video Instance Segmentation: A BenchmarkInternational Journal of Computer Vision, 2022
- Normalized Cut Loss for Weakly-Supervised CNN Segmentation2018
- Simple Does It: Weakly Supervised Instance and Semantic Segmentation2017
- Colorful Image Colorization2016
- Context Encoders: Feature Learning by Inpainting2016
- Instance-Aware Semantic Segmentation via Multi-task Network Cascades2016
- A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation2016
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Distinctive Image Features from Scale-Invariant KeypointsInternational Journal of Computer Vision, 2004
- Normalized cuts and image segmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000