Fix dtype casting in swinv2 and swinv2sr to allow non-FP32 inference

[aliencaocao]

What does this PR do?

The current implementation uses .float() in

transformers/src/transformers/models/swin2sr/modeling_swin2sr.py

Lines 286 to 287 in 0f67ba1

	relative_coords_h = torch.arange(-(self.window_size[0] - 1), self.window_size[0], dtype=torch.int64).float()
	relative_coords_w = torch.arange(-(self.window_size[1] - 1), self.window_size[1], dtype=torch.int64).float()

which causes subsequent relative_coords_table to be always in torch.float32, not respecting whatever precision the other weights might be, e.g. torch.float16.

This PR adds a cast to the same dtype as the continuous_position_bias_mlp layer since relative_coords_table is being passed directly into the layer at

transformers/src/transformers/models/swin2sr/modeling_swin2sr.py

Line 349 in 0f67ba1

relative_position_bias_table = self.continuous_position_bias_mlp(self.relative_coords_table).view(

Same issue & fix for swinv2

Prerequisite for #31342 image to image pipeline FP16 test to pass.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@amyeroberts

[amyeroberts]

Thanks for fixing!

Could you also add some small tests like this one for vit?

[aliencaocao]

Added them, they pass locally

[amyeroberts]

@aliencaocao Great! Could you push an empty commit with the message: [run_slow] swin2sr, swinv2. I trust the tests are passing locally, but because of differences that can creep in because of hardware and env set-up, the logits can still be slightly different. So let's make sure the numbers match what's going to be running on the CI : )

[amyeroberts]

Thanks for fixing this!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[aliencaocao]

@amyeroberts need your approval for slow tests

[aliencaocao]

ugh it seems the specific gpu used indeed has numerical difference...
I ran the tests and got logit using RTX 3080Ti, torch2.3.1+cu121, nvidia 555.95 driver, Windows 10

[amyeroberts]

@aliencaocao As the change is simple, and look OK, updating the tests to use the results from the CI runs should be OK

[aliencaocao]

How do I get the CI outputs?
Do I have to print in CI?

[amyeroberts]

@aliencaocao Good question! Indeed, they're not part of the console output. Let me see if I can ssh in

[amyeroberts]

@aliencaocao Running on the runners, I get the following logits

swin2sr

tensor([[0.5454, 0.5542, 0.5640],
        [0.5518, 0.5562, 0.5649],
        [0.5391, 0.5425, 0.5620]], device='cuda:0', dtype=torch.float16)

swinv2

tensor([-0.3938, -0.4290,  0.0020], device='cuda:0', dtype=torch.float16)

[aliencaocao]

Thanks, triggered again

[amyeroberts]

@aliencaocao Thanks! All looks good - we can merge 🤗