Fix donut image processor

[amyeroberts]

What does this PR do?

This PR addresses failing integration tests for the Donut image processor which involves four main changes:

• Resolve bug where size wasn't passed to do_align_axis
• Remove a bug in the get_resize_output_image_size function which wouldn't take account of max_size (inherited from previous resize without fixing)
• Update logic for getting output size in thumbnail method - ensuring the image dimensions are never increased.
• Update test values to reflect changes in resizing logic for thumbnail creation - see notes below.

Changing resizing logic for thumbnail method

The DonutFeatueExtractor used the Pillow thumbnail functionality to resize images which was replaced with reusing resize in the image_transforms library. This was done primarily as image.thumbnail modifies in place and uses Pillow's resize with some additional logic for calculating the output size. Unlike resize which will resize an image to the requested (height, width), thumbnail will produce an image which is no larger than the original image or requested size i.e. it will scale down an image preserving the aspect ratio c.f. Pillow docs.

This is a similar behaviour to torchvision when resizing:

• the shortest image edge is resized to size (int for torchvision, min(requested_heigh, requsted_width) for Pillow)
• the other edge is resized to preserve the aspect ratio
• if the longest edge > max_size, the longest edge is resized to max_size and the shortest edge resized to preserve the aspect ratio.

The calculation of the other dimension to preserve the aspect ratio is slightly different between the libraries. In pytorch the length of the edge is found using int to round, whereas Pillow rounds to the value which produces an aspect ratio closest to the original image. The torchvision resizing logic is replicated in our image transforms library here.

In the test tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::DonutModelIntegrationTest::test_inference_docvqa, the input image to the thumbnail method has dimension (3713, 1920). The requested size is (2560, 1920). image.thumbnail will resize to (2560, 1373) and our resizing logic (matching torchvision) will resize to (2560, 1374).

As using torchvision resizing logic is more consistent with the rest of the library; Donut is the only model in the library that used the Pillow thumbnail functionality, and is more experimental than other models; I considered this to be an acceptable change.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for the fix!

[ydshieh]

Thanks @amyeroberts . LGTM, but I don't see any change related to

Resolve bug where size wasn't passed to do_align_axis

Do I miss anything?

[amyeroberts]

Thanks @amyeroberts . LGTM, but I don't see any change related to

Resolve bug where size wasn't passed to do_align_axis

Do I miss anything?

Nope - I've pushed it now :)

[amyeroberts]

@sgugger @ydshieh This also uncovered another sneaky bug when resizing:

• When resizing, the image is coverted to PIL.Image.Image from numpy. The channel dimension format of the input image, before resizing inferred is also inferred.
• When the image is converted back to numpy the image is always in "ChannelDimension.LAST" format
• A final to_channel_dimension_format call is made to make sure the output resized image is in the same channel dimension format as the input.
• In to_channel_dimension_format the input image (resized in this case) channel dimension format is inferred and compared to the requested format.
• If the height dimensions is of size 3 or 1, then format is incorrectly inferred as ChannelDimension.FIRST
• This resulted in images in the incorrect format being returned after resizing

For practical purposes, this doesn't cause an issue as it's very unlikely an image has a height dimension of 3. However, it results in flaky tests and is a bug.

I've added an optional input_channel_dimension argument to to_channel_dimension_format which resolves this and additional tests for our resize functionality which previously failed and now pass with this update.

[sgugger]

Works for me, thanks for the deep cleaning!

[ydshieh]

Thanks again

[ydshieh]

let's clean this line before merge :-)

[amyeroberts]

Done :)