PGTuner: An Efficient Framework for Automatic and Transferable Configuration Tuning of Proximity Graphs
Preprint
Abstract
Approximate Nearest Neighbor Search (ANNS) plays a crucial role in many key areas. Proximity graphs (PGs) are the leading method for ANNS, offering the best balance between query efficiency and accuracy. However, their performance heavily depends on various construction and query parameters, which are difficult to optimize due to their complex inter-dependencies. Given that users often prioritize specific accuracy levels, efficiently identifying the optimal PG configurations to meet these targets is essential. Although some studies have explored automatic configuration tuning for PGs, they are limited by inefficiencies and suboptimal results. These issues stem from the need to construct numerous PGs for searching and re-tuning from scratch whenever the dataset changes, as well as the failure to capture the complex dependencies between configurations, query performance, and tuning objectives. To address these challenges, we propose PGTuner, an efficient framework for automatic PG configuration tuning leveraging pre-training knowledge and model transfer techniques. PGTuner improves efficiency through a pre-trained query performance prediction (QPP) model, eliminating the need to build multiple PGs. It also features a deep reinforcement learning-based parameter configuration recommendation (PCR) model to recommend optimal configurations for specific datasets and accuracy targets. Additionally, PGTuner incorporates out-of-distribution detection and deep active learning for efficient tuning in dynamic scenarios and transferring to new datasets. Extensive experiments demonstrate that PGTuner can stably achieve the top-level tuning effect across different datasets while significantly improving tuning efficiency by up to 14.69X, with a 14.64X boost in dynamic scenarios. The code and data for PGTuner are available online at https://github.com/hao-duan/PGTuner.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Generalized Out-of-Distribution Detection: A SurveyInternational Journal of Computer Vision, 2024
- Overview of the SISAP 2023 Indexing Challenge2023
- Talisman: Targeted Active Learning for Object Detection with Rare Classes and Slices Using Submodular Mutual Information2022
- To Actively Initialize Active LearningPattern Recognition, 2022
- MOOD: Multi-level Out-of-distribution Detection2021
- EI-LSH: An early-termination driven I/O efficient incremental c-approximate nearest neighbor searchThe VLDB Journal, 2020
- Towards Proximity Graph Auto-configuration: An Approach Based on Meta-learning2020
- Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision2020
- Full Text Search Engine as Scalable k-Nearest Neighbor Recommendation System2010
- A survey of content-based image retrieval with high-level semanticsPattern Recognition, 2007