Are conv2d_transpose layers shift variant, too?

[JuheonYi]

Hi, thanks for sharing the code!
I really enjoyed reading your paper.

As I understand, you have mainly considered downsampling layers (pool, strided convolution)
does the same shift variance issue exist for conv2d_transpose layers as well?
If it does, could you share your insights on how to replace the layer?

Thanks very much!

[richzhang]

Thanks for the question. Upsampling convolution is shift-invariant, but if you do it naively, it is aliased and leads to stippling patterns.

If you do bilinear upsampling (convolve with a [1 2 1] filter after doing the naive upsampling), then you are better antialiased. The StyleGAN paper does this, for example.

[JuheonYi]

@richzhang
Thanks for the answer (and also the reference to StyleGAN example)!

Can I ask one more question?

In case of the strided convolutions, does the performance vary depending on whether the BlurPool is put before or after the convolution?
(i.e., Conv (stride 1) -> ReLU -> BlurPool (stride 2) vs. BlurPool (stride 2) -> Conv (stride 1) -> ReLU)
It seems that putting BlurPool before Conv could save computation, but I wonder how the performance would be.

Could you share your thoughts or experiences on this?

Thank you!

[richzhang]

Yes, it makes a difference. The second (BlurPool-->Conv-->ReLU) works worse, because it destroys information as the first step.

[iperov]

but subpixel upscale layer is not antialiased?

[mrgloom]

In paper there is not much information on antialiased upsampling(or I just don't get it):

We mirror the downsampling by applying the same filter after upsampling. Note that applying the Rect-2 and Tri-3 filters while upsampling correspond to “nearest” and “bilinear” upsampling, respectively

Is it should be done via torch.nn.ConvTranspose2d with groups= in_channels and with same filter as in downsampling code?

Also seems there is not groups parameter in TensorFlow for conv_transpose, is there any workaround? i.e. use some layer that can do x2 upscale(bilinear resize, conv_traspose, pixel_shuffle, etc) and then add 'blur' layer(avr pool with stride 1 and kernel size 2x2) on top?

[richzhang]

Thanks for the question!

Yes, upsample with ConvTranspose2d with groups=in_channels, and then use [1, 1] or [1, 2, 1] as your weights.

For TensorFlow, perhaps you can refer to the StyleGan2 code/paper, which does smoothed upsampling as well.

[mrgloom]

Looking at StyleGan2 code, seems they are doing it like:

blur(upsample(x)):
https://github.com/NVlabs/stylegan2/blob/7d3145d23013607b987db30736f89fb1d3e10fad/training/networks_stylegan.py#L519

downsample(blur(x)):
https://github.com/NVlabs/stylegan2/blob/7d3145d23013607b987db30736f89fb1d3e10fad/training/networks_stylegan.py#L607

I wonder is this approach(i.e. via 2 steps) is different from resampling-by-convolution(i.e. via 1 step) ? https://clouard.users.greyc.fr/Pantheon/experiments/rescaling/index-en.html

[richzhang]

It's the same. The two steps can be folded into one. We don't actually blur the whole map at every spatial location and then subsample by factor of 2, since many of the calculations are thrown away. We simply evaluate the blur at every other location.