FixRes

FixRes is a simple method for fixing the train-test resolution discrepancy. It can improve the performance of any convolutional neural network architecture.

The method is described in "Fixing the train-test resolution discrepancy" (Links: arXiv,NeurIPS).

BibTeX reference to cite, if you use it:

@inproceedings{touvron2019FixRes,
       author = {Touvron, Hugo and Vedaldi, Andrea and Douze, Matthijs and J{\'e}gou, Herv{\'e}},
       title = {Fixing the train-test resolution discrepancy},
       booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
       year = {2019},
}

Please notice that our models depend on previous trained models, see References to other models

Installation

The FixRes code requires

• Python 3.6 or higher
• PyTorch 1.0 or higher

and the requirements highlighted in requirements.txt (for Anaconda)

Cluster settings

Ours codes were executed on a cluster with several GPUs. As configurations are different from one cluster to another, we provide a generic implementation. You must run the code on each GPU by specifying job-id, local-rank, global-rank, and num-tasks which is not very convenient. Therefore, we strongly recommend to adapt our code according to the configuration of your cluster.

Using the code

The configurations given in the examples provide the results of the Pretrained Networks table (Table 2 in the article). The training and fine-tuning codes record the learned model in a checkpoint.pth file.

Extracting features with pre-trained networks

Pre-trained networks

We provide pre-trained networks with different trunks, we report in the table validation resolution, Top-1 and Top-5 accuracy on ImageNet validation set:

Models	Resolution	#Parameters	Top-1 / Top-5	Weights
ResNet-50 Baseline	224	25.6M	77.0 / 93.4	FixResNet50_no_adaptation.pth
FixResNet-50	384	25.6M	79.0 / 94.6	FixResNet50.pth
FixResNet-50 (*)	384	25.6M	79.1 / 94.6	FixResNet50_v2.pth
FixResNet-50 CutMix	320	25.6M	79.7 / 94.9	FixResNet50CutMix.pth
FixResNet-50 CutMix (*)	320	25.6M	79.8 / 94.9	FixResNet50CutMix_v2.pth
FixPNASNet-5	480	86.1M	83.7 / 96.8	FixPNASNet.pth
FixResNeXt-101 32x48d	320	829M	86.3 / 97.9	FixResNeXt101_32x48d.pth
FixResNeXt-101 32x48d (*)	320	829M	86.4 / 98.0	FixResNeXt101_32x48d_v2.pth
FixEfficientNet-B0 (+)	320	5.3M	80.2 / 95.4	FixEfficientNet
FixEfficientNet-L2 (+)	600	480M	88.5 / 98.7	FixEfficientNet

(*) We use Horizontal flip, shifted Center Crop and color jittering for fine-tuning (described in transforms_v2.py)

(+) We report different results with our FixEfficientNet (see FixEfficientNet for more details)

To load a network, use the following PyTorch code:

import torch
from .resnext_wsl import resnext101_32x48d_wsl

model=resnext101_32x48d_wsl(progress=True) # example with the ResNeXt-101 32x48d 

pretrained_dict=torch.load('ResNeXt101_32x48d.pth',map_location='cpu')['model']

model_dict = model.state_dict()
for k in model_dict.keys():
    if(('module.'+k) in pretrained_dict.keys()):
        model_dict[k]=pretrained_dict.get(('module.'+k))
model.load_state_dict(model_dict)

The network takes images in any resolution. A normalization pre-processing step is used, with mean [0.485, 0.456, 0.406]. and standard deviation [0.229, 0.224, 0.225] for ResNet-50 and ResNeXt-101 32x48d, use mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5] with PNASNet. You can find the code in transforms.py.

Features extracted from the ImageNet validation set

We provide the probabilities, embedding and labels of each image in the ImageNet validation so that the results can be reproduced easily.

Embedding files are matrixes of size 50000 by 2048 for all models except for PNASNet where the size is 50000 by 4320, embeddings are extracted after the last spatial pooling. The softmax are matrixes of sizes 50000 by 1000 it representing the probability of each class for each image.

Model	Softmax	Embedding
FixResNet-50	FixResNet50_Softmax.npy	FixResNet50Embedding.npy
FixResNet-50 (*)	FixResNet50_Softmax_v2.npy	FixResNet50Embedding_v2.npy
FixResNet-50 CutMix	FixResNet50_CutMix_Softmax.npy	FixResNet50_CutMix_Embedding.npy
FixResNet-50 CutMix (*)	FixResNet50_CutMix_Softmax_v2.npy	FixResNet50_CutMix_Embedding_v2.npy
FixPNASNet-5	FixPNASNet_Softmax.npy	FixPNASNet_Embedding.npy
FixResNeXt-101 32x48d	FixResNeXt101_32x48d_Softmax.npy	FixResNeXt101_32x48d_Embedding.npy
FixResNeXt-101 32x48d (*)	FixResNeXt101_32x48d_Softmax_v2.npy	FixResNeXt101_32x48d_Embedding_v2.npy

(*) We use Horizontal flip, shifted Center Crop and color jittering for fine-tuning (described in transforms_v2.py)

Evaluation of the networks

See help (-h flag) for detailed parameter list of each script before executing the code.

Classification results

main_evaluate_imnet.py evaluates the network on standard benchmarks.

main_evaluate_softmax.py evaluates the network on ImageNet-val with already extracted softmax output. (Much faster to execute)

Example evaluation procedure

# FixResNeXt-101 32x48d
python main_evaluate_imnet.py --input-size 320 --architecture 'IGAM_Resnext101_32x48d' --weight-path 'ResNext101_32x48d.pth'
# FixResNet-50
python main_evaluate_imnet.py --input-size 384 --architecture 'ResNet50' --weight-path 'ResNet50.pth'

#FixPNASNet-5
python main_evaluate_imnet.py --input-size 480 --architecture 'PNASNet' --weight-path 'PNASNet.pth'

The following code give results that corresponds to table 2 in the paper :

# FixResNeXt-101 32x48d
python main_evaluate_softmax.py --architecture 'IGAM_Resnext101_32x48d' --save-path 'where_softmax_and_labels_are_saved'

# FixPNASNet-5
python main_evaluate_softmax.py --architecture 'PNASNet' --save-path 'where_softmax_and_labels_are_saved'

# FixResNet50
python main_evaluate_softmax.py --architecture 'ResNet50' --save-path 'where_softmax_and_labels_are_saved'

Features extraction

main_extract.py extract embedding, labels and probability with the networks.

Example extraction procedure

# FixResNeXt-101 32x48d
python main_extract.py --input-size 320 --architecture 'IGAM_Resnext101_32x48d' --weight-path 'ResNeXt101_32x48d.pth' --save-path 'where_output_will_be_save'
# FixResNet-50
python main_extract.py --input-size 384 --architecture 'ResNet50' --weight-path 'ResNet50.pth' --save-path 'where_output_will_be_save'

# FixPNASNet-5
python main_extract.py --input-size 480 --architecture 'PNASNet' --weight-path 'PNASNet.pth' --save-path 'where_output_will_be_save'

Fine-tuning existing network with our Method

See help (-h flag) for detailed parameter list of each script before executing the code.

Classifier and Batch-norm fine-tuning

main_finetune.py fine-tune the network on standard benchmarks.

Example fine-tuning procedure

# FixResNeXt-101 32x48d
python main_finetune.py --input-size 320 --architecture 'IGAM_Resnext101_32x48d' --epochs 1 --batch 8 --num-tasks 32 --learning-rate 1e-3

# FixResNet-50
python main_finetune.py --input-size 384 --architecture 'ResNet50' --epochs 56 --batch 64 --num-tasks 8 --learning-rate 1e-3

# FixPNASNet-5
python main_finetune.py --input-size 480 --architecture 'PNASNet' --epochs 1 --batch 64 --num-tasks 8 --learning-rate 1e-4

Using transforms_v2 for fine-tuning

To reproduce our best results we must use the data-augmentation of transforms_v2 and use almost the same parameters as for the classic data augmentation, the only changes are the learning rate which must be 1e-4 and the number of epochs which must be 11. For FixResNet-50 fine-tune you have to use 31 epochs and a learning rate of 1e-3 and for FixResNet-50 CutMix you have to use 11 epochs and a learning rate of 1e-3. Here is how to use transforms_v2 :

from torchvision import datasets
from .transforms_v2 import get_transforms

transform = get_transforms(input_size=Train_size,test_size=Test_size, kind='full', crop=True, need=('train', 'val'), backbone=None)
train_set = datasets.ImageFolder(train_path,transform=transform['val_train'])
test_set = datasets.ImageFolder(val_path,transform=transform['val_test'])

Training

See help (-h flag) for detailed parameter list of each script before executing the code.

Train ResNet-50 from scratch

main_resnet50_scratch.py Train ResNet-50 on standard benchmarks.

Example training procedure

# ResNet50
python main_resnet50_scratch.py --batch 64 --num-tasks 8 --learning-rate 2e-2

Contributing

See the CONTRIBUTING file for how to help out.

References to other models

Model definition scripts are based on https://github.com/pytorch/vision/ and https://github.com/Cadene/pretrained-models.pytorch.

The Training from scratch implementation is based on https://github.com/facebookresearch/multigrain.

Our FixResNet-50 CutMix is fine-tune from the weights of the GitHub page : https://github.com/clovaai/CutMix-PyTorch. The corresponding paper is

@inproceedings{2019arXivCutMix,
       author = {Sangdoo Yun and Dongyoon Han and Seong Joon Oh and Sanghyuk Chun and Junsuk Choe and Youngjoon Yoo,
       title = "{CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features}",
       journal = {arXiv e-prints},
       year = "2019"}

Our FixResNeXt-101 32x48d is fine-tuned from the weights of the Pytorch Hub page : https://pytorch.org/hub/facebookresearch_WSL-Images_resnext/

The corresponding paper is

@inproceedings{mahajan2018exploring,
       author = {Mahajan, Dhruv and Girshick, Ross and Ramanathan, Vignesh and He, Kaiming and Paluri, Manohar and Li, Yixuan  and Bharambe, Ashwin and van der Maaten, Laurens,
       title = "{Exploring the limits of weakly supervised pretraining}",
       journal = {European Conference on Computer Vision},
       year = "2018"}

For FixEfficientNet we used model definition scripts and pretrained weights from https://github.com/rwightman/pytorch-image-models.

The corresponding papers are:

For models with extra-training data :

@misc{xie2019selftraining,
	author={Qizhe Xie and Minh-Thang Luong and Eduard Hovy and Quoc V. Le,
    title="{Self-training with Noisy Student improves ImageNet classification}",
    journal = {arXiv e-prints},
    year=2019}
}

For models without extra-training data :

@misc{xie2019adversarial,
	author={Cihang Xie and Mingxing Tan and Boqing Gong and Jiang Wang and Alan Yuille and Quoc V. Le,
    title="{Adversarial Examples Improve Image Recognition}",
    journal = {arXiv e-prints},
    year="2019"}
}

License

FixRes is CC BY-NC 4.0 licensed, as found in the LICENSE file.