FedPAM: Federated Personalized Augmentation Model for Text-to-Image Retrieval
Open Access
- 30 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1185-1189
Abstract
CLIP-based models have made significant advancements in text-to-image retrieval tasks. However, these retrieval models are typically trained on public datasets with optimizing all parameters, which limits their ability to generalize and adapt quickly to personalized private datasets. In this paper, we introduce a lightweight personalized federated learning solution, namely Federated Personalized Augmentation Model (FedPAM), to achieve personalized text-to-image retrieval from multiple private database. Specifically, for the query text, we fetch the top-k most similar text-image pairs from the private database. We then use an attention-based module to generate personalized representations for different clients. The updated representation includes client-specific information for text-to-image matching, resolving issues of data heterogeneity. Additionally, we ensure efficient and secure communication by fine-tuning a small portion of network parameters. Our experiments demonstrate the effectiveness of the proposed framework, exhibiting a significant performance improvement over recently proposed methods: +5.36 on IAPR TC-12, +2.86 on CC3M, and +1.72 on Flickr30k.Keywords
Funding Information
- the Key Research and Development Program of Zhejiang Province (2023C03192,2024C03270)
- China Postdoctoral Science Foundation (524000-X92302)
This publication has 14 references indexed in Scilit:
- PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models – Federated Learning in Age of Foundation ModelIEEE Transactions on Mobile Computing, 2023
- FedAdapter: Efficient Federated Learning for Mobile NLP2023
- FedPrompt: Communication-Efficient and Privacy-Preserving Prompt Tuning in Federated Learning2023
- Reproducible Scaling Laws for Contrastive Language-Image Learning2023
- pFedPrompt: Learning Personalized Prompt for Vision-Language Models in Federated Learning2023
- VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks2022
- Deep Leakage from Gradients2020
- Billion-Scale Similarity Search with GPUsIEEE Transactions on Big Data, 2019
- Exploiting Unintended Feature Leakage in Collaborative Learning2019
- From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptionsTransactions of the Association for Computational Linguistics, 2014