Add Image Processor Fast Deformable DETR

[yonigozlan]

What does this PR do?

Adds a fast image processor for Deformable DETR. Follows issue #33810.
This image processor is a result of this work on comparing different image processing method.

The diffs look bad but this PR is almost exclusively made up of # Copied from based on the fast image processor for DETR!

Implementation

See #34063

Usage

Except for the fact that it only returns torch tensors, this fast processor is fully compatible with the current one.
It can be instantiated through AutoImageProcessor with use_fast=True, or through the Class directly:

from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("SenseTime/deformable-detr", use_fast=True)

from transformers import DeformableDetrImageProcessorFast

processor = DeformableDetrImageProcessorFast.from_pretrained("SenseTime/deformable-detr")

Usage is the same as the current processor, except for the device kwarg:

from torchvision.io import read_image
images = torchvision.io.read_image(image_path)
processor = DeformableDetrImageProcessorFast.from_pretrained("SenseTime/deformable-detr")
images_processed = processor(images , return_tensors="pt", device="cuda")

If device is not specified:

• If the input images are tensors, the processing will be done on the device of the images.
• If the inputs are PIL or Numpy images, the processing is done on CPU.

Performance gains

• Average over 100 runs on the same 480x640 image. No padding needed, as "all" the images have the same size.

---

• Average over 10% of the COCO 2017 validation dataset, with batch_size=8. Forcing padding to 1333x1333 (="longest_edge"), as otherwise torch.compile needs to recompile if the different batches have different max sizes.

---

• Average over 10% of the COCO 2017 validation dataset, with batch_size=1. Forcing padding to 1333x1333.
  

---

Tests

• The new image processor is tested on all the tests of the current processor.
• I have also added two consistency tests (panoptic and detection) for processing on GPU vs CPU.

Who can review?

@ArthurZucker Pinging you directly as there is almost no "new" code here.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Thanks, same comment as for the other PR mostly! 🤗

[yonigozlan]

Will make the modifications once this PR #34354 is merged, as most of them will be copied from :)

[ArthurZucker]

One thing I don't understand: literally everything is copied from. Why not directy map to use the detr class?

[yonigozlan]

One thing I don't understand: literally everything is copied from. Why not directy map to use the detr class?

All pre-processing functions are copied from image_processing_detr_fast, but the post-processing function are copied from image_processing_deformable_detr. I guess the post-processing functions are also the reasons why there is a base DeformableDetrImageProcessorFast in the first place, as all the base pre-processing functions are also copied from image_processing_detr

[ArthurZucker]

Got it, thanks! Let's work to make it simpler to add these, with maybe a bit of abstraction on the FastImageProcessor class!