A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking The Privacy-Utility Trade-off
Preprint
Abstract
The application of Differential Privacy to Natural Language Processing techniques has emerged in relevance in recent years, with an increasing number of studies published in established NLP outlets. In particular, the adaptation of Differential Privacy for use in NLP tasks has first focused on the $\textit{word-level}$, where calibrated noise is added to word embedding vectors to achieve "noisy" representations. To this end, several implementations have appeared in the literature, each presenting an alternative method of achieving word-level Differential Privacy. Although each of these includes its own evaluation, no comparative analysis has been performed to investigate the performance of such methods relative to each other. In this work, we conduct such an analysis, comparing seven different algorithms on two NLP tasks with varying hyperparameters, including the $\textit{epsilon ($\varepsilon$)}$ parameter, or privacy budget. In addition, we provide an in-depth analysis of the results with a focus on the privacy-utility trade-off, as well as open-source our implementation code for further reproduction. As a result of our analysis, we give insight into the benefits and challenges of word-level Differential Privacy, and accordingly, we suggest concrete steps forward for the research field.
Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Privacy and Utility Trade-Off for Textual Analysis via Calibrated Multivariate Perturbations2020
- Investigating the Impact of Pre-trained Word Embeddings on Memorization in Neural Networks2020
- Privacy Risks of General-Purpose Language Models2020
- Leveraging Hierarchical Representations for Preserving Privacy and Utility in Text2019
- Generalised Differential Privacy for Text Document Processing2019
- Invited Paper: Local Differential Privacy on Metric Spaces: Optimizing the Trade-Off with Utility2018
- Membership Inference Attacks Against Machine Learning Models2017
- Broadening the Scope of Differential Privacy Using Metrics2013
- Mechanism Design via Differential Privacy2007
- Differential Privacy2006