DP-KFC: Data-Free Preconditioning for Privacy-Preserving Deep Learning

Authors:
  • MBMarc Molina Van Den Bosch
    Marc Molina Van Den Bosch
  • RTRiccardo Taiello
    Riccardo Taiello
  • AAAlbert Sund Aillet
    Albert Sund Aillet
  • APAndrea Protani
    Andrea Protani
  • MBMiguel Angel Gonzalez Ballester
    Miguel Angel Gonzalez Ballester
  • LSLuigi Serio
    Luigi Serio

Abstract
Differentially private optimization suffers from a fundamental geometric mismatch: deep networks have highly anisotropic loss landscapes, yet DP-SGD injects isotropic noise. Second-order preconditioning can resolve this, but estimating curvature typically requires private data (consuming privacy budget) or public data (introducing distribution shift). We show that the Fisher Information Matrix decouples into architectural sensitivity, recoverable via synthetic noise, and input correlations, approximable from modality-specific frequency statistics. We propose DP-KFC, which constructs KFAC preconditioners by probing networks with structured synthetic noise, requiring neither private nor public data. Empirically, DP-KFC consistently outperforms DP-SGD and adaptive baselines across diverse modalities in strong privacy regimes ($\varepsilon \leq 3$). DP-KFC matches private-data preconditioners while public-data variants degrade by up to $4.8\%$, showing that curvature can be estimated without consuming privacy budget or introducing distribution shift. This enables privacy-preserving learning in specialized domains (e.g., medical applications) where regulatory constraints make data scarce.
All Related Versions