Modernize CLIP modeling code

[molbap]

What does this PR do?

As per title. Adds the nice recent modeling utils to CLIP. Was motivated to be able to use it in the looong-standing #33962 .

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[molbap]

cc @zucchini-nlp if you want to take a look!

[zucchini-nlp]

Amazing, happy to see the models being standardized 🚀 Let's run slow tests with CLIP and LLaVA-CLIP before merging, just to be sure :)

[molbap]

run-slow: clip, metaclip_2

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/clip', 'models/metaclip_2']
quantizations: [] ...

[molbap]

run-slow: clip, metaclip_2

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: clip, metaclip_2

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/clip', 'models/metaclip_2']
quantizations: [] ...

[ArthurZucker]

Nine ty

[yukiu00]

Hi there,

We recently encountered an issue with transformers v5.0.0rc1 that seems to be related to the changes in this PR.

Specifically, CLIPVisionModel.forward() now returns hidden_states=None even when output_hidden_states=True is passed. It appears that the output_hidden_states argument was removed from ClipEncoder.forward (or is not being utilized from TransformersKwargs correctly).

Comparison:

• v5: modeling_clip.py#L485
• v4: modeling_clip.py#L506

We suspect this is a regression. Could you please check if this was intended?

Related

• [v5 regression] CLIPVisionModel.forward returns hidden_states=None even when output_hidden_states=True #43761
• Fix CLIPVisionModel hidden_states issue in Llava convergence test for transformers v5 linkedin/Liger-Kernel#1061

[ArthurZucker]

Thanks for the report, having a look and yes it should work with check_model_inputs