Add DetrImageProcessorFast

[yonigozlan]

What does this PR do?

Adds a fast image processors for DETR. Follows issue #33810.
This image processor is a result of this work on comparing different image processing method.

The processing methods use only torchvision transforms (either v1 or v2, depending on the torchvision version) and torch tensors.
Just like the current DETR image processor, this processor can also process object detection or segmentation annotations. This processing also uses only torch tensors and torchvision transforms.
The post-processing methods have not been modified from the original image processor.

Implementation

A previous fast image processor implementation for VIT (link) uses torchvision transform classes and Compose to create a one step processing class. However this poses two problems:

• The torchvision v2 Transforms are only torch compile/scripting compatible in their functional forms and not in their Class form (source).
• A one step processing class is not possible when the processing depends on the input, like it's the case for DETR for resizing and padding.

So this implementation uses the functional forms of torchvision transforms, and it's structure is very similar to the current DETR image processor.

All the numpy/PIL operations have been converted to torch or torchvision operations, and like the VIT fast image processor, this processor only accept return_tensors = "pt"

The processor call function accept a device kwarg, as processing can be performed on both CPU and GPU, but is much faster on GPU.
I wanted to add device as an init argument, but that would make the signatures of fast and slow processors different, which make some tests fails.

Usage

Except for the fact that it only returns torch tensors, this fast processor is fully compatible with the current one.
It can be instantiated through AutoImageProcessor with use_fast=True, or through the Class directly:

from transformers import AutoImageProcessor

processor = AutoImageProcessor.from_pretrained("facebook/detr-resnet-50", use_fast=True)

from transformers import DetrImageProcessorFast

processor = DetrImageProcessorFast.from_pretrained("facebook/detr-resnet-50")

Usage is the same as the current processor, except for the device kwarg:

from torchvision.io import read_image
images = torchvision.io.read_image(image_path)
processor = DetrImageProcessorFast.from_pretrained("facebook/detr-resnet-50")
images_processed = processor(images , return_tensors="pt", device="cuda")

If device is not specified:

• If the input images are tensors, the processing will be done on the device of the images.
• If the inputs are PIL or Numpy images, the processing is done on CPU.

Performance gains

Main Takeaways

Processing speedup

• ~60x faster processing on GPU (single image)
• ~80x faster processing on GPU (batch_size=8)
• ~5x faster processing on CPU (single image)
• ~2.6x faster processing on CPU (batch_size=8)

Inference pass speedup (GPU)

• ~2.2x speedup on whole model inference pass (single image, eager)
• ~3.2x speedup on whole model inference pass (single image, compiled)
• ~2.4x speedup on whole model inference pass (batch_size=8, eager)

---

• Average over 100 runs on the same 480x640 image. No padding needed, as "all" the images have the same size.

---

• Average over 10% of the COCO 2017 validation dataset, with batch_size=8. Padding needed, as the images have different sizes, and the DETR processor resize them using "shortest_edge"/"longest_edge", resulting in different sized resized images.

---

• Average over 10% of the COCO 2017 validation dataset, with batch_size=8. Forcing padding to 1333x1333 (="longest_edge"), as otherwise torch.compile needs to recompile if the different batches have different max sizes.
  (I'm not sure what is going wrong when using the compiled model with the current processor)

---

• Average over 10% of the COCO 2017 validation dataset, with batch_size=1. Forcing padding to 1333x1333 for comparison with batched inputs

---

Tests

• The new image processor is tested on all the tests of the current processor.
• I have also added two consistency tests (panoptic and detection) for processing on GPU vs CPU.

---

Looking forward to your feedback!
I was also wondering if we should adopt a more modular approach to the fast image processors, as there is quite a lot of repetition with the "slow" processor for now. It looks like something like this was done for Fast tokenizers? If someone that worked on Fast tokenizers has any advice on that I'll gladly hear them 🤗.
There will also be the question of how to advertise this "use_fast" option to users, and if we want to make it default eventually when torchvision is available?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[molbap]

Very nice work! I added a couple comments, in addition to one also on the page you linked rel to performance of .to(device).to(dtype) VS .to(device, dtype), finding the second one to be always faster when moving to a cuda device, fwiw

[molbap]

This method doesn't have any tensor <-> array change right?

Suggested change

	# Copied from transformers.models.detr.image_processing_detr.get_size_with_aspect_ratio

[yonigozlan]

Yes indeed. I'm not sure if I should import those functions directly from image_processing_detr.py instead of using # Copied from at this point, wdyt?

[molbap]

We typically don't import from another file, but it'd be nice to do it yes - in general we should be able to import from an ...utils file (say processing_utils.py, image_processing_utils.py, image_processing_utils_fast.py ) more generic methods instead of redefining them for each model. cc @ArthurZucker for this pattern type as we discussed it before

[ArthurZucker]

Yep I think it's good to think about how we can refactor a bit!

[molbap]

and btw (for later) these methods should be slightly 🧹 🧹 rel to variable naming for oh + ow

[molbap]

we can use the deprecate_kwarg util here

[yonigozlan]

I agree but using it causes the test_image_processor_preprocess_arguments to fail. I'm guessing this tests does not work well with the way decorators calls the function and the arguments

[molbap]

Dove a bit here, nice refactor and improvement - I think if we can move to a more modular way of building processors, as we try to do with models, it'll reduce loc count greatly and improve readability. A modular_image_processing_detr_fast that inherits from the necessary funcs and then build automatically the image_processing_detr_fast would be amazing for instance, cc @ArthurZucker for visibility (not a requirement on this PR imo).

[yonigozlan]

Yes I was thinking about this too, because the loc count is huge compare to the actual changes. I might be wrong but it looks like something similar was done to reduce loc count for fast tokenizers? Not sure who to ping here for clarifications

[ArthurZucker]

Yeah! Kind of like we have a PreTrainedModel we can have a ImageProcessorMixin that would be a bit more powerfull!

For fast tokenizers, we just rely on the tokenizers library directly, so it's a bit different

[molbap]

✅

[molbap]

It's interesting the git diff is so large here - unless I'm wrong we are generalizing the tests to integrate a list of processors, fast and slow, but the tests do not change?

[yonigozlan]

Yes I'm guessing it's because of the change in indentation, but all I changed is testing a list of processors instead of only one

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

LGTM in general, but having 1600 LOC for an image processor is quite intense!
Great performance gains tho

[ArthurZucker]

Yeah! Kind of like we have a PreTrainedModel we can have a ImageProcessorMixin that would be a bit more powerfull!

For fast tokenizers, we just rely on the tokenizers library directly, so it's a bit different

[ArthurZucker]

Yep I think it's good to think about how we can refactor a bit!

[ArthurZucker]

This is kind of void of copied from, which I am a bit suprised about!

[yonigozlan]

Most methods are very similar to the ones in image_processing_detr, but slightly changed to work with tensors instead of numpy arrays. I haven't changed the post-processing methods at all (as they are not as much of a speed bottleneck), so they are all added with copied from (end of the file)

[ArthurZucker]

ah okay, got it!

[SangbumChoi]

@yonigozlan Hi, super excited to see this. just left two questions

1. Figure says it is RTDetr but this is Detr right?
2. I think this can be also adapted to all vision processor?
3. Can you also elaborate the computing device?

[yonigozlan]

Hi @SangbumChoi !

1. Oops, thanks for pointing this out, yep it's a typo, this is indeed Detr (but I'm also working on a fast image processor for RTDetr :) ).
2. Yes absolutely this is the plan. There are quite a few other image processors based on the Detr one so their fast image processor should be out soon.
3. This was on a A10 GPU!

[SangbumChoi]

@yonigozlan thanks! I would love support this task if you need some help. Also if you can share the comparison script for compile + visualize plot etc... it would be very helpful!