Differentially Private Image Classification by Learning Priors from Random Processes

Xinyu Tang; Ashwinee Panda; Vikash Sehwag; Prateek Mittal

Differentially Private Image Classification by Learning Priors from Random Processes

Xinyu Tang, Ashwinee Panda, Vikash Sehwag, Prateek Mittal

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST and ImageNet for a range of privacy budgets ε ∈ [1, 8]. In particular, we improve the previous best reported accuracy on CIFAR10 from 60.6% to 72.3% for ε = 1. Our code is available at https://github.com/inspire-group/DP-RandP.

Original language	American English
Title of host publication	Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023
Editors	A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, S. Levine
Publisher	Neural information processing systems foundation
ISBN (Electronic)	9781713899921
State	Published - 2023
Event	37th Conference on Neural Information Processing Systems, NeurIPS 2023 - New Orleans, United States Duration: Dec 10 2023 → Dec 16 2023

Publication series

Name	Advances in Neural Information Processing Systems
Volume	36

Conference

Conference	37th Conference on Neural Information Processing Systems, NeurIPS 2023
Country/Territory	United States
City	New Orleans
Period	12/10/23 → 12/16/23

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

Tang, X., Panda, A., Sehwag, V., & Mittal, P. (2023). Differentially Private Image Classification by Learning Priors from Random Processes. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023 (Advances in Neural Information Processing Systems; Vol. 36). Neural information processing systems foundation.

Tang, Xinyu ; Panda, Ashwinee ; Sehwag, Vikash et al. / Differentially Private Image Classification by Learning Priors from Random Processes. Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. editor / A. Oh ; T. Neumann ; A. Globerson ; K. Saenko ; M. Hardt ; S. Levine. Neural information processing systems foundation, 2023. (Advances in Neural Information Processing Systems).

@inproceedings{8795a1e0806f4824b1bc1ae8632a30a7,

title = "Differentially Private Image Classification by Learning Priors from Random Processes",

abstract = "In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST and ImageNet for a range of privacy budgets ε ∈ [1, 8]. In particular, we improve the previous best reported accuracy on CIFAR10 from 60.6% to 72.3% for ε = 1. Our code is available at https://github.com/inspire-group/DP-RandP.",

author = "Xinyu Tang and Ashwinee Panda and Vikash Sehwag and Prateek Mittal",

note = "Publisher Copyright: {\textcopyright} 2023 Neural information processing systems foundation. All rights reserved.; 37th Conference on Neural Information Processing Systems, NeurIPS 2023 ; Conference date: 10-12-2023 Through 16-12-2023",

year = "2023",

language = "American English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

editor = "A. Oh and T. Neumann and A. Globerson and K. Saenko and M. Hardt and S. Levine",

booktitle = "Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023",

}

Tang, X, Panda, A, Sehwag, V & Mittal, P 2023, Differentially Private Image Classification by Learning Priors from Random Processes. in A Oh, T Neumann, A Globerson, K Saenko, M Hardt & S Levine (eds), Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. Advances in Neural Information Processing Systems, vol. 36, Neural information processing systems foundation, 37th Conference on Neural Information Processing Systems, NeurIPS 2023, New Orleans, United States, 12/10/23.

Differentially Private Image Classification by Learning Priors from Random Processes. / Tang, Xinyu; Panda, Ashwinee; Sehwag, Vikash et al.
Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. ed. / A. Oh; T. Neumann; A. Globerson; K. Saenko; M. Hardt; S. Levine. Neural information processing systems foundation, 2023. (Advances in Neural Information Processing Systems; Vol. 36).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Differentially Private Image Classification by Learning Priors from Random Processes

AU - Tang, Xinyu

AU - Panda, Ashwinee

AU - Sehwag, Vikash

AU - Mittal, Prateek

PY - 2023

Y1 - 2023

N2 - In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST and ImageNet for a range of privacy budgets ε ∈ [1, 8]. In particular, we improve the previous best reported accuracy on CIFAR10 from 60.6% to 72.3% for ε = 1. Our code is available at https://github.com/inspire-group/DP-RandP.

AB - In privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) performs worse than SGD due to per-sample gradient clipping and noise addition. A recent focus in private learning research is improving the performance of DP-SGD on private data by incorporating priors that are learned on real-world public data. In this work, we explore how we can improve the privacy-utility tradeoff of DP-SGD by learning priors from images generated by random processes and transferring these priors to private data. We propose DP-RandP, a three-phase approach. We attain new state-of-the-art accuracy when training from scratch on CIFAR10, CIFAR100, MedMNIST and ImageNet for a range of privacy budgets ε ∈ [1, 8]. In particular, we improve the previous best reported accuracy on CIFAR10 from 60.6% to 72.3% for ε = 1. Our code is available at https://github.com/inspire-group/DP-RandP.

UR - http://www.scopus.com/inward/record.url?scp=85191195037&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85191195037&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

A2 - Oh, A.

A2 - Neumann, T.

A2 - Globerson, A.

A2 - Saenko, K.

A2 - Hardt, M.

A2 - Levine, S.

PB - Neural information processing systems foundation

T2 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023

Y2 - 10 December 2023 through 16 December 2023

ER -

Tang X, Panda A, Sehwag V, Mittal P. Differentially Private Image Classification by Learning Priors from Random Processes. In Oh A, Neumann T, Globerson A, Saenko K, Hardt M, Levine S, editors, Advances in Neural Information Processing Systems 36 - 37th Conference on Neural Information Processing Systems, NeurIPS 2023. Neural information processing systems foundation. 2023. (Advances in Neural Information Processing Systems).

Differentially Private Image Classification by Learning Priors from Random Processes

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Cite this