Add Image Processor Fast RT-DETR

[yonigozlan]

What does this PR do?

Adds a fast image processor for RT-DETR. Follows issue #33810.
This image processor is a result of this work on comparing different image processing method.

The diffs look bad but this PR is almost exclusively made up of # Copied from based on the fast image processor for DETR!

Implementation

See #34063

Usage

Except for the fact that it only returns torch tensors, this fast processor is fully compatible with the current one.
It can be instantiated through AutoImageProcessor with use_fast=True, or through the Class directly:

from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("PekingU/rtdetr_r50vd", use_fast=True)

from transformers import RTDetrImageProcessorFast

processor = RTDetrImageProcessorFast.from_pretrained("PekingU/rtdetr_r50vd")

Usage is the same as the current processor, except for the device kwarg:

from torchvision.io import read_image
images = torchvision.io.read_image(image_path)
processor = RTDetrImageProcessorFast.from_pretrained("PekingU/rtdetr_r50vd")
images_processed = processor(images , return_tensors="pt", device="cuda")

If device is not specified:

• If the input images are tensors, the processing will be done on the device of the images.
• If the inputs are PIL or Numpy images, the processing is done on CPU.

Performance gains

• Average over 10% of the COCO 2017 validation dataset, with batch_size=1.

---

• Average over 10% of the COCO 2017 validation dataset, with batch_size=8.

---

Tests

• The new image processor is tested on all the tests of the current processor.
• I have also added a consistency test for processing on GPU vs CPU.

Who can review?

@ArthurZucker Pinging you directly as there is almost no "new" code here.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[molbap]

Thanks! Just a few nits - saw you were removing the kwargs validation as well, can we also do that for detr?

[molbap]

return_segmentation_masks is unused here, could either reuse the one from detr_fast so we have a # Copied from statement here?

[yonigozlan]

Yes it's a bit weird, return_segmentation_masks is present in several places, but rt-detr does not support segmentation. I added a copied from here, with an Ignore copy for the segmentation part (as otherwise we would need to import/copy a function that would never be used.

EDIT: actually the Ignore copy makes the CI crash, not sure why, so I left it as is for now...

[ArthurZucker]

# Ignore copy does not work like this 😉 forget about it in this case!

[ArthurZucker]

Thanks for working on this!
A few more nits, overall good, IMO all your graph should be placed in the documentation as well and not just on the PR description!

[ArthurZucker]

# Ignore copy does not work like this 😉 forget about it in this case!

[yonigozlan]

@ArthurZucker Refactored DETR and RT-DETR image processor fast to loop as few times as possible over annotations and images, and added some docs!

[ArthurZucker]

Does this improve the perf you saw? 😉

[ArthurZucker]

Thanks for iterating with me! 🤗

[yonigozlan]

Thanks for iterating with me! 🤗

Thank you!

Does this improve the perf you saw? 😉

Hmm hard to tell, maybe very slightly when on GPU, as on CPU the potential gains are overshadowed by the processing time. But at least it's cleaner that way! :)