Unlocking the Power of Differentially Private Zeroth-order Optimization for Fine-tuning LLMs
No ratings
Presented at USENIX Security 2025 by
Differentially private zeroth-order optimization (DPZO in short) has shown promise in fine-tuning large language models (LLMs) while protecting record-level privacy. Compared with classical first-order methods, such as DPSGD, the main difference is that DPZO replaces the exact first-order gradients that are computed via back-propagation with its random zeroth-order approximations that are computed via querying the model's losses. However, DPZO still lags in the resulting model utility compared to existing methods, indicating that further work is needed to fully realize its potential.