SEMat: Semantic Enhanced Natural Image Interactive Matting
- 26 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 36 (4) , 5329-5343
Abstract
Recent approaches attempt to adapt powerful interactive segmentation models, such as SAM, to interactive matting and fine-tune the models based on synthetic matting datasets. However, models trained on synthetic data fail to generalize to complex and occlusion scenes. We address this challenge by proposing a new matting dataset based on the COCO dataset, namely COCO-Matting. It selects real-world complex images from COCO and converts semantic segmentation masks to matting labels. The built COCO-Matting comprises an extensive collection of 36,980 human instance-level alpha mattes in complex natural scenarios. Furthermore, existing SAM-based matting methods extract intermediate features and masks from a frozen SAM and only train a lightweight matting decoder by end-to-end matting losses, which do not fully exploit the potential of the pre-trained SAM. Thus, we propose SEMat which revamps the network architecture and training objectives. For network architecture, the proposed feature-aligned transformer learns to extract fine-grained edge and transparency features. The proposed matte-aligned decoder aims to segment matting-specific objects and convert coarse masks into high-precision mattes. For training objectives, the proposed regularization and trimap loss aim to retain the prior from the pre-trained model and push the matting logits extracted from the mask decoder to contain trimap-based semantic information. Extensive experiments across seven diverse datasets demonstrate the superior performance of our method, proving its efficacy in interactive natural image matting. Code is available at https://github.com/XiaRho/SEMat.Keywords
Funding Information
- National Natural Science Foundation of China (62233005)
- National Natural Science Foundation of China (U25B6002)
- National Natural Science Foundation of China (U2441245)
- Ministry of Education, Singapore, under its AcRF Tier 1 Funding (23-SIS-SMU-070)
- Ministry of Education, Singapore, under its AcRF Tier 1 Funding (24-SIS-SMU-040)
This publication has 39 references indexed in Scilit:
- LLDiffusion: Learning degradation representations in diffusion models for low-light image enhancementPattern Recognition, 2025
- Matte anything: Interactive natural image matting with segment anything modelImage and Vision Computing, 2024
- ViTMatte: Boosting image matting with pre-trained plain vision transformersInformation Fusion, 2023
- Bridging Composite and Real: Towards End-to-End Deep Image MattingInternational Journal of Computer Vision, 2022
- Attention-Guided Hierarchical Structure Aggregation for Image Matting2020
- Deep Image Matting2017
- Deep Residual Learning for Image Recognition2016
- Microsoft COCO: Common Objects in Context2014
- A perceptually motivated online benchmark for image matting2009
- Alpha estimation in natural images2002