The official implementation of paper "Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors" accepted at ICML 2024
Conventional Federated Learning (FL) involves collaborative training of a global model while maintaining user data privacy. One of its branches, decentralized FL, is a serverless network that allows clients to own and optimize different local models separately, which results in saving management and communication resources. Despite the promising advancements in decentralized FL, it may reduce model generalizability due to lacking a global model. In this scenario, managing data and model heterogeneity among clients becomes a crucial problem, which poses a unique challenge that must be overcome: How can every client's local model learn generalizable representation in a decentralized manner? To address this challenge, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Based on the theory of domain adaptation and Knowledge Distillation (KD), we theoretically and empirically show that synthesizing global anchors based on raw data distribution facilitates mutual knowledge transfer. We further design two effective regularization terms for local training: 1) REG loss that regularizes the distribution of the client's latent embedding with the anchors and 2) KD loss that enables clients to learn from others. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of DeSA in enhancing both inter- and intra-domain accuracy of each client.
We recommend using conda to install the environment. Please use environment.txt to set up the conda environment.
Please download the digits data here and put it under digit_data folder.
python iterative_desab.py --dataset=digits --ipc=50 --model_hetero=True
Please download the preprocessed data and put them into data folder. OFFICE CIFAR10C
Please use the following scripts to run the experiments from scratch
python iterative_desab.py --dataset=office --ipc=10 --model_hetero=True --pretrain=True --generate_image=True --KD=True
python iterative_desab.py --dataset=cifar10c --ipc=50 --model_hetero=True --pretrain=True --generate_image=True --KD=True --client_ratio=0.1
python iterative_desab.py --dataset=cifar10c --ipc=10 --model_hetero=True --pretrain=True --generate_image=True --KD=True --client_ratio=0.2
Currently, DeSA only supports DIGITS experiment.
Please use the following scripts to run the experiment from scratch
python iterative_desab.py --dataset=digits --ipc=10 --model_hetero=True --pretrain=True --generate_image=True --KD=True --DP=True
For our MIA experiment, we follow the metric from Carlini et al..
If you find this work helpful, please cite our paper as follows:
@article{huang2024overcoming,
title={Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchors},
author={Huang, Chun-Yin and Srinivas, Kartik and Zhang, Xin and Li, Xiaoxiao},
journal={arXiv preprint arXiv:2405.11525},
year={2024}
}