Clean-up composite configs

[zucchini-nlp]

What does this PR do?

Based on #34410, I realized that BLIP models were the only ones failing a certain common test and the reason is that we manually set the tie_word_embedding from text_config. I think some config attributes, including vocab_size (successfully deprecated in VLMs) or tie_word_embeddings should be only attributes of text config. We should access them via config.get_text_config() whenever needed

Having an option to access through general composite config and through text config creates two sources of truth, and we might easily end up with situations like in the check_config_arguments_init. If one sets the attribute through setattr, it will not be different from the attribute in text config.

So we expect users to set the attr in the respective config if they want it to be used correctly with transformers code. We could restrict setting these common attr in correct sub-config by attribute_map prob, but I thought it was too much

Plus, clean up some attn_implementation=config._attn_implementation that were left, because it should be passed already as part of respective config

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Thanks!
I would make sure all tests are run as for tie weight embedding here the changes would only work for the decoder (decoder=True) !

A commit with run-all is enough! 🤗

[zucchini-nlp]

Tests are passing, the run-all last time worked as well with a few new VLMs that needed a fix. Merging when CI turns green again