Cross-Attention and Cycle-Consistency-Based Haptic to Image Inpainting
- 14 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Signal Processing Letters
- Vol. 31 (10709908) , 1650-1654
Abstract
With the rapid advancement of deep learning and multimedia technologies, image inpainting has made significant progress in generating desirable content for damaged images. To fill in the corrupted part of the image, existing methods either utilize intrinsic information within the visual modality for single-modal image inpainting or consider semantic correlations from non-visual modalities for cross-modal image inpainting. However, when the corrupted part is extensive, intrinsic information alone within visual modality is insufficient to inpaint the entire image. Cross-modal image inpainting methods cannot guarantee image quality, as they do not explore the relationship among the corrupted part of the image, the remaining part of the image, and the non-visual modality. In order to handle this issue, a haptic to image inpainting scheme is proposed. Specifically, feature extraction and cross-attention-based feature index are firstly constructed to find the correspondence between the corrupted part of the image and the haptic modality. Then, cycle-consistency-based feature translation is implemented to explore the intrinsic correlations among the above two modalities, facilitating the formation of the corrupted part. Finally, the formed corrupted part is combined with the remaining part to realize image inpainting. Experimental results show the effectiveness of the proposed scheme.Keywords
Funding Information
- National Natural Science Foundation of China (62071254, 62277032)
- Open Foundation of Anhui Province Key Laboratory of Intelligent Building & Building Energy Saving (IBES2021KF09)
- Jiangsu Provincial Qinglan Project
- Special Fund for Urban and Rural Construction and Development in Jiangsu Province
This publication has 15 references indexed in Scilit:
- Exploring the Benefits of Cross-Modal CodingIEEE Transactions on Circuits and Systems for Video Technology, 2022
- FASNet: Feature Aggregation and Sharing Network for Image InpaintingIEEE Signal Processing Letters, 2022
- MAT: Mask-Aware Transformer for Large Hole Image Inpainting2022
- Cross-Modal Transmission StrategyIEEE Transactions on Circuits and Systems for Video Technology, 2021
- Positional Encoding as Spatial Inductive Bias in GANs2021
- EDBGAN: Image Inpainting via an Edge-Aware Dual Branch Generative Adversarial NetworkIEEE Signal Processing Letters, 2021
- Image Inpainting for Irregular Holes Using Partial Convolutions2018
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- Generative Image Inpainting with Contextual Attention2018
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017