FedCEO: "Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off" (ICML 2025, CCF A, CORE A*)

📣 06/09/25: Check out the intro to our paper on Twitter! Likes and Reposts are welcome!

📣 02/08/25: Update the Slide and Video in ICML 2025!

📣 01/05/25: This paper has been accepted to ICML 2025!

The official implementation of our paper:

Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off (FedCEO)

[ArXiv] [OpenReview] [Slide&Video] [X]

Abstract

To defend against privacy leakage of user data, differential privacy is widely used in federated learning, but it is not free. The addition of noise randomly disrupts the semantic integrity of the model and this disturbance accumulates with increased communication rounds. In this paper, we introduce a novel federated learning framework with rigorous privacy guarantees, named FedCEO, designed to strike a trade-off between model utility and user privacy by letting clients "Collaborate with Each Other". Specifically, we perform efficient tensor low-rank proximal optimization on stacked local model parameters at the server, demonstrating its capability to flexibly truncate high-frequency components in spectral space. This capability implies that our FedCEO can effectively recover the disrupted semantic information by smoothing the global semantic space for different privacy settings and continuous training processes. Moreover, we improve the SOTA utility-privacy trade-off bound by order of \sqrt{d}, where d is the input dimension. We illustrate our theoretical results with experiments on representative datasets and observe significant performance improvements and strict privacy guarantees under different privacy settings.

Lay Summary

Protecting user privacy in collaborative AI training (federated learning) requires adding carefully designed noise. However, this noise can unevenly disrupt different parts of each device's learned knowledge over time – like obscuring facial features in one device's animal recognition model while blurring limb details in another's.We introduce FedCEO, a new approach where devices "Collaborate with Each Other" under server coordination. FedCEO intelligently combines the complementary knowledge from all devices. When one device's understanding of a concept is disrupted by privacy protection, others help fill those gaps.This CEO-like coordination gradually enhances semantic smoothness across devices as training progresses. The server blends the partial understandings into a coherent whole, allowing the global model to recover disrupted patterns while maintaining privacy. The result is significantly improved AI performance across diverse privacy settings and extended training periods.

Dependence

To install the dependencies:

pip install -r requirements.txt

Data

The EMNIST and CIFAR10 datasets are downloaded automatically by the torchvision package.

Usage

We provide scripts that have been tested to produce the results stated in our paper (utility experiments and privacy experiments). Please find them in the file: train.sh.

For example,

# CIFAR-10
nohup python -u FedCEO.py --privacy True --noise_multiplier 2.0 --flag True --dataset "cifar10" --model "cnn" --lamb 0.6 --r 1.04 --interval 10 > ./logs/log_fedceo_noise=2.0_cifar10_LeNet.log 2>&1 &

Flags

• FL related

  • args.epochs: The number of communication rounds.
  • args.num_users: The number of total clients, denoted by $N$.
  • args.frac: The sampling rate of clients, denoted by $p$.
  • args.lr: The learning rate of local round on the clients, denoted by $\eta$.
  • args.privacy: Adopt the DP Gaussian mechanism or not.
  • args.noise_multiplier: The ratio of the standard deviation of the Gaussian noise to the L2-sensitivity of the function to which the noise is added.
  • args.flag: Using our low-rank processing or not.

• FedCEO related

  • args.lamb: The weight of regularization term, denoted by $\lambda$.
  • args.interval: The smoothing interval to adopt, denoted by $I$.
  • args.flag: The common ratio of the geometric series, denoted by $\vartheta$.

• Model related

  • args.model: MLP or LeNet.

• Experiment setting related

  • args.dataset: cifar10 or emnist.
  • args.index: The index for leaking images on Dataset.

🧪 Experimental Results and Commands

We have conducted extensive experiments to validate the utility improvement, privacy protection, and utility-privacy trade-off of FedCEO. Below we provide partial results and the commands to reproduce them.

Explore more experiments in the Appendices! [🔗Paper]

1. Utility Experiments

We compare FedCEO with baseline methods under different privacy settings (controlled by σ_g). FedCEO consistently achieves the highest test accuracy.

Table: Test Accuracy (%) on EMNIST and CIFAR-10

Dataset	Model	σ_g	UDP-FedAvg	PPSGD	CENTAUR	FedCEO (ϑ=1)	FedCEO (ϑ>1)
		1.0	76.59%	77.01%	77.26%	77.14%	78.05%
EMNIST	MLP-2-Layers	1.5	69.91%	70.78%	71.86%	71.56%	72.44%
		2.0	60.32%	61.51%	62.12%	63.38%	64.20%
		1.0	43.87%	49.24%	50.14%	50.09%	54.16%
CIFAR-10	LeNet-5	1.5	34.34%	47.56%	46.90%	48.89%	50.00%
		2.0	26.88%	34.61%	36.70%	37.39%	45.35%

Command to Run Utility Experiment:

# CIFAR-10
nohup python -u FedCEO.py --privacy True --noise_multiplier 2.0 --flag True --dataset "cifar10" --model "cnn" --lamb 0.6 --r 1.04 --interval 10 > ./logs/log_fedceo_noise=2.0_cifar10_LeNet.log 2>&1 &

---

2. Privacy Experiments

We use the DLG attack to evaluate privacy leakage. Lower PSNR indicates better privacy protection.

Figure: Privacy Attack Results on CIFAR-10 (PSNR in dB, lower is better)

Command to Run Privacy Experiment:

# privacy exps
nohup python -u attack_FedCEO.py --privacy True --noise_multiplier 2.0 --flag True --dataset "cifar10" --model "cnn" --index 100 --gpu "" > ./logs/log_attack_fedceo_noise=1.0_cifar10_LeNet.log 2>&1 &

---

3. Utility-Privacy Trade-off Experiments

We visualize the trade-off between utility (test accuracy) and privacy (ε_p) on CIFAR-10. FedCEO achieves the best balance.

Citation

@article{li2024clients,
  title={Clients collaborate: Flexible differentially private federated learning with guaranteed improvement of utility-privacy trade-off},
  author={Li, Yuecheng and Fu, Lele and Wang, Tong and Lou, Jian and Chen, Bin and Yang, Lei and Shen, Jian and Zheng, Zibin and Chen, Chuan},
  journal={arXiv preprint arXiv:2402.07002},
  year={2024}
}