IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation"

Introduction

Abstract

The divergence between labeled training data and unlabeled testing data is a significant challenge for recent deep learning models. Unsupervised domain adaptation (UDA) attempts to solve such a problem. Recent works show that self-training is a powerful approach to UDA. However, existing methods have difficulty in balancing scalability and performance. In this paper, we propose an instance adaptive self-training framework for UDA on the task of semantic segmentation. To effectively improve the quality of pseudo-labels, we develop a novel pseudo-label generation strategy with an instance adaptive selector. Besides, we propose the region-guided regularization to smooth the pseudo-label region and sharpen the non-pseudo-label region. Our method is so concise and efficient that it is easy to be generalized to other unsupervised domain adaptation methods. Experiments on 'GTA5 to Cityscapes' and 'SYNTHIA to Cityscapes' demonstrate the superior performance of our approach compared with the state-of-the-art methods.

IAST Overview

Result

source	target	device	GPU memory	mIoU-19	mIoU-16	mIoU-13	model
GTA5	Cityscapes	Tesla V100-32GB	18.5 GB	51.88	-	-	download
GTA5	Cityscapes	Tesla T4	6.3 GB	51.20	-	-	download
SYNTHIA	Cityscapes	Tesla V100-32GB	18.5 GB	-	51.54	57.81	download
SYNTHIA	Cityscapes	Tesla T4	9.8 GB	-	51.24	57.70	download

Setup

1) Envs

• Pytorch >= 1.0
• Python >= 3.6
• cuda >= 9.0

Install python packages

$ pip install -r  requirements.txt

apex : Tools for easy mixed precision and distributed training in Pytorch

git clone https://github.com/NVIDIA/apex
cd apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

2) Download Dataset

Please download the datasets from these links:

• GTA5
• SYNTHIA
• Cityscapes

Dataset directory should have this structure:

${ROOT_DIR}/data/GTA5/
${ROOT_DIR}/data/GTA5/images
${ROOT_DIR}/data/GTA5/labels

${ROOT_DIR}/data/SYNTHIA_RAND_CITYSCAPES/RAND_CITYSCAPES
${ROOT_DIR}/data/SYNTHIA_RAND_CITYSCAPES/RAND_CITYSCAPES/RGB
${ROOT_DIR}/data/SYNTHIA_RAND_CITYSCAPES/RAND_CITYSCAPES/GT

${ROOT_DIR}/data/cityscapes
${ROOT_DIR}/data/cityscapes/leftImg8bit
${ROOT_DIR}/data/cityscapes/gtFine

3) Download Pretrained Models

We provide pre-trained models. We recommend that you download them and put them in pretrained_models/, which will save a lot of time for training and ensure consistent results.

V100 models

• GTA5 to Cityscapes: G_gtav_at_warmup_v100.pth and M_gtav_at_warmup_v100.pth
• SYNTHIA to Cityscapes: G_syn_at_warmup_v100.pth and M_syn_at_warmup_v100.pth

T4 models

• GTA5 to Cityscapes: G_gtav_at_warmup_t4.pth and M_gtav_at_warmup_t4.pth
• SYNTHIA to Cityscapes: G_syn_at_warmup_t4.pth and M_syn_at_warmup_t4.pth

(Optional) Of course, if you have plenty of time, you can skip this step and start training from scratch. We also provide these scripts.

Training

Our original experiments are all carried out on Tesla-V100, and there will be a large number of GPU memory usage (batch_size=8). For low GPU memory devices, we also trained on Tesla-T4 to ensure that most people can reproduce the results (batch_size=2).

Start self-training (download the pre-trained models first)

cd code

# GTA5 to Cityscapes (V100)
sh ../scripts/self_training_only/run_gtav2cityscapes_self_traing_only_v100.sh
# GTA5 to Cityscapes (T4)
sh ../scripts/self_training_only/run_gtav2cityscapes_self_traing_only_t4.sh

# SYNTHIA to Cityscapes (V100)
sh ../scripts/self_training_only/run_syn2cityscapes_self_traing_only_v100.sh
# SYNTHIA to Cityscapes (T4)
sh ../scripts/self_training_only/run_syn2cityscapes_self_traing_only_t4.sh

(Optional) Training from scratch

cd code

# GTA5 to Cityscapes (V100)
sh ../scripts/from_scratch/run_gtav2cityscapes_self_traing_v100.sh
# GTA5 to Cityscapes (T4)
sh ../scripts/from_scratch/run_gtav2cityscapes_self_traing_t4.sh

# SYNTHIA to Cityscapes (V100)
sh ../scripts/from_scratch/run_syn2cityscapes_self_traing_v100.sh
# SYNTHIA to Cityscapes (T4)
sh ../scripts/from_scratch/run_syn2cityscapes_self_traing_t4.sh

Evaluation

cd code
python eval.py --config_file <path_to_config_file> --resume_from <path_to_*.pth_file>

Support multi-scale testing and flip testing.

# Modify the following parameters in the config file

TEST:
  RESIZE_SIZE: [[1024, 512], [1280, 640], [1536, 768], [1800, 900], [2048, 1024]] 
  USE_FLIP: False 

Citation

Please cite this paper in your publications if it helps your research:

@article{mei2020instance,
  title={Instance Adaptive Self-Training for Unsupervised Domain Adaptation},
  author={Mei, Ke and Zhu, Chuang and Zou, Jiaqi and Zhang, Shanghang},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2020}
}

Author

Ke Mei

• email: [email protected]
• wechat: meikekekeke

If you have any questions, you can contact me directly.