Cutmix

[dandelin]

clovaai/CutMix-PyTorch: Official Pytorch implementation of CutMix regularizer GitHub: https://github.com/clovaai/CutMix-PyTorch

Hi, I saw that you’ve been implementing mixup as an additional feature.
However, if the model trained with mixup used as the backbone of object detector, it seems the performance of the detector degenerates.

Could you please consider cutmix in addition to mixup?

thanks!

[rwightman]

@dandelin looks like it should be pretty straightforward to add that, will take a look, tx

[pichuang1984]

@rwightman did you have any success with training efficientnet using cutmix? I tried training couple efficientnet models with and without cutmix and I usually see > 0.5% accuracy degradation with the model trained with cutmix. I also followed the implementation from https://github.com/clovaai/CutMix-PyTorch but is getting similar result.

[rwightman]

@pichuang1984 I've tried mixup and I've tried cutout (random erasing) with efficientnets without definitive improvements (for imagenet training), so I have not bothered to try cutmix...

[pichuang1984]

Thanks for the confirmation! This matches my observation as well

[pichuang1984]

@rwightman I am revisiting mixup again since a colleague is getting good result from it. Notice that when using your implementation, the following line (https://github.com/rwightman/pytorch-image-models/blob/aa4354f4668ee7cf349f24de60e3a23dee43a6ef/timm/data/mixup.py#L37) seems to cause the following error:

AttributeError: 'Tensor' object has no attribute 'astype'

[rwightman]

@pichuang1984 the FastCollate variant is expecting the inputs to the collate fn to be numpy arrays. It does the mixup while performing the collate and is intended to be used with the PrefetchLoader and the transforms that don't include normalization and conversion to tensors (that is done in prefetcher).

The variation of mixup compatible with a more standard PyTorch loader setup that uses the default collate and doesn't use the NVIDIA PrefetchLoader is integrated into the train script here: https://github.com/rwightman/pytorch-image-models/blob/master/train.py#L438

[hellbell]

@rwightman @dandelin @pichuang1984
Thanks for having interests in CutMix.
I attached CutMix to EfficientNet-{b0,b1,b2} and I got some performance improvements, so here I share some results.

Arch	top-1 acc.
EfficientNet-b0	77.4
EfficientNet-b0+CutMix	77.4 (+0.0)
EfficientNet-b1	79.4
EfficientNet-b1+CutMix	79.6 (+0.2)
EfficientNet-b2	80.2
EfficientNet-b2+CutMix	80.5 (+0.3)

• The difference from the original paper's ResNet setting, the CutMix probability and alpha were set to 0.2 instead of 1.0.
• Performance on b0 didn't show improvement, but b1 and b2 can be improved by CutMix augmentation.
• The implementation of CutMix has simply followed the repo https://github.com/clovaai/CutMix-PyTorch.

Hyperparameter argments (4GPUs setting)

Baseline EfficientNet-b0

--model efficientnet_b0 --weight-decay 1.00E-05 --drop 0.2 --drop-connect 0.2 \
--lr 0.032 --batch-size 128 --epochs 450 --decay-epochs 2.4 --decay-rate 0.97  \
--opt rmsproptf --opt-eps 0.001 --workers 8 --warmup-lr 1.00E-06 --model-ema \
--model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp \
--bn-momentum 0.1 --cutmix-alpha 0.0 --cutmix-prob 0.0

CutMix+EfficientNet-B0

--model efficientnet_b0 --weight-decay 1.00E-05 --drop 0.2 --drop-connect 0.2 \
--lr 0.032 --batch-size 128 --epochs 450 --decay-epochs 2.4 --decay-rate 0.97 \
--opt rmsproptf --opt-eps 0.001 --workers 8 --warmup-lr 1.00E-06 --model-ema \
--model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp \
--bn-momentum 0.1 --cutmix-alpha 0.2 --cutmix-prob 0.2

Baseline EfficientNet-b1

--model efficientnet_b1 --weight-decay 1.00E-05 --drop 0.2 --drop-connect 0.2  \
--lr 0.032 --batch-size 128 --epochs 450 --decay-epochs 2.4 --decay-rate 0.97 \
--opt rmsproptf --opt-eps 0.001 --workers 8 --warmup-lr 1.00E-06 --model-ema \ 
--model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp \
--bn-momentum 0.1 --cutmix-alpha 0.0 --cutmix-prob 0.0

CutMix+EfficientNet-B1

--model efficientnet_b1 --weight-decay 1.00E-05 --drop 0.2 --drop-connect 0.2 \
--lr 0.032 --batch-size 128 --epochs 450 --decay-epochs 2.4 --decay-rate 0.97 \
--opt rmsproptf --opt-eps 0.001 --workers 8 --warmup-lr 1.00E-06 --model-ema \
--model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp \
--bn-momentum 0.1 --cutmix-alpha 0.2 --cutmix-prob 0.2

Baseline EfficientNet-b2

--model efficientnet_b2 --weight-decay 1.00E-05 --drop 0.2 --drop-connect 0.3 \
--lr 0.032 --batch-size 128 --epochs 450 --decay-epochs 2.4 --decay-rate 0.97 \
--opt rmsproptf --opt-eps 0.001 --workers 8 --warmup-lr 1.00E-06 --model-ema \
--model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp \
--bn-momentum 0.1 --cutmix-alpha 0.0 --cutmix-prob 0.0

CutMix+EfficientNet-B2

--model efficientnet_b2 --weight-decay 1.00E-05 --drop 0.2 --drop-connect 0.2 \
--lr 0.032 --batch-size 128 --epochs 450 --decay-epochs 2.4 --decay-rate 0.97 \
--opt rmsproptf --opt-eps 0.001 --workers 8 --warmup-lr 1.00E-06 --model-ema \
--model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp \
--bn-momentum 0.1 --cutmix-alpha 0.2 --cutmix-prob 0.2

• Implementation: Modified https://github.com/rwightman/pytorch-image-models/blob/master/train.py#L508-L510

        if args.cutmix and np.random.rand(1) < args.cutmix_prob:
            lam = np.random.beta(args.cutmix_alpha, args.cutmix_alpha)
            rand_index = torch.randperm(input.size()[0]).cuda()
            target_a = target
            target_b = target[rand_index]
            bbx1, bby1, bbx2, bby2, lam = rand_bbox(input.size(), lam)
            input[:, :, bbx1:bbx2, bby1:bby2] = input[rand_index, :, bbx1:bbx2, bby1:bby2]

            # compute output
            output = model(input)
            loss = loss_fn(output, target_a) * lam + loss_fn(output, target_b) * (1. - lam)
        else:
            output = model(input)
            loss = loss_fn(output, target)

[rwightman]

@hellbell Thanks for the detailed update, those results are certainly interesting, perhaps it's finally time to take a closer look and integrate this

[pichuang1984]

@hellbell just 1 minor point, in the clovaai repo the rand_bbox doesn't return a lam, so need to modify the following
bbx1, bby1, bbx2, bby2, lam = rand_bbox(input.size(), lam)
to be
bbx1, bby1, bbx2, bby2 = rand_bbox(input.size(), lam)

[hellbell]

@pichuang1984 Thanks for pointing out :) Hope it works well!

[rwightman]

finally have this underway, basically integrating with my two mixup modes (part of prefetched loader + collate or stand alone in the train loop) ... looking good so far, testing some variations like per sample random switch between mixup/cutmix per in the loader version

one comment re that reference impl ... they've mixed up the dimension names ... PyTorch layout is NCHW ... so they've switched W and H but at least in a consistent way.. however names all wrong

[rwightman]

@hellbell something I noticed while testing this, on one of the datasets I've been experimenting with (CUB2011, fine grained classification), implementing the lambda correction significantly reduces the accuracy of the end result... has this been observed on other datasets? if correction isn't applied cutmix > mixup > baseline by 1-2% abs top-1 in each step ... with correction applied cutmix was way below mixup and the baseline