Use CLIP model config to set some kwargs for components

[ydshieh]

What does this PR do?

In CLIPModel, set output_attentions and output_hidden_states using CLIPModel.config if these values are specified in the configuration + not specified in the arguments.

(currently, these operations are done in its vision & text components separately, and cause a WIP CLIP PT/TF equivalence test failing - #16557)

Details

Currently, CLIPModel uses its 2 components' (vision_model and text_model) configurations to perform things like

(here self is CLIPVisionTransformer or CLIPTextTransformer)

output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions

If output_attentions/output_hidden_states are not passed to CLIPModel.forward at this line

transformers/src/transformers/models/clip/modeling_clip.py

Lines 966 to 967 in 9fd5e6b

	output_attentions: Optional[bool] = None,
	output_hidden_states: Optional[bool] = None,

but CLIPModel.config has these values set, CLIPModel.config.output_attentions and CLIPModel.config.output_hidden_states won't have any effect. This case happens here

transformers/tests/test_modeling_tf_common.py

Lines 544 to 547 in 9fd5e6b

	# Output all for aggressive testing
	config.output_hidden_states = True
	if self.has_attentions:
	config.output_attentions = True

Therefore, CLIP PT/TF equivalence test won't returns hidden_states/attentions for the PT model.

In TF,

transformers/src/transformers/modeling_tf_utils.py

Line 393 in b33ab4e

def input_processing(func, config, input_ids, **kwargs):

will use config to set the kwargs at the CLIPModel level. These kwargs are passed to the 2 components, and CLIP PT/TF equivalence test returns hidden_states/attentions for the TF model.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ydshieh]

cc @gante (just for information) since he is recently working on unpack_inputs & input_processing in TF

[sgugger]

Looks okay to me but will defer to @patil-suraj on this :-)
Thanks for your PR!

[patil-suraj]

LGTM, thank you for fixing this!