Address changes in transformers VLM architecture

[githubnemo]

transformers PR #37033 re-arranges the way visual language models are built by moving the LM head from the language model to the top-level VLM (among other things).

This breaks the following code (from test_vision_models):

peft_config = PrefixTuningConfig(task_type="CAUSAL_LM", num_virtual_tokens=20)
model.language_model = get_peft_model(model.language_model, peft_config)

with

AttributeError: 'LlamaModel' object has no attribute 'prepare_inputs_for_generation'

Reason being that model.language_model is not a *CausalLM anymore but a base model (e.g., LlamaModel).
PEFT assumes that the model is specific and all soft-prompting methods need a task type since each task type has specific handling of the soft prompt (e.g., padding the labels according to the number of virtual tokens for causal LM). We also can't simply use task_type='FEATURE_EXTRACTION' as a workaround because this would not deal with labels either.

Luckily the newly structured VLM is almost behaving like a LM (e.g., get_input_embeddings just refers to the underlying LM), therefore we can target the VLM itself and need to have the soft prompt methods detect if we're fine-tuning a VLM so that we take the respective config variables from the base_model.text_config instead of base_model directly.

Backward compatibility

Old models trained with PEFT should still work since VLM models like llava use the lm_head and language_model from the state dict and re-arrange them accordingly. However, old code that uses model.language_model as a target will fail with AttributeError: 'LlamaModel' object has no attribute 'prepare_inputs_for_generation'. I fear that we have no way of mitigating this since we don't have access to the 'top-level' model, the only thing we could do is to raise a more helpful error message but for reliably detecting the case that we're targeting a VLM's language model we'd need the VLM's model config (e.g., to check for the text_config key) or have access to the VLM itself. We cannot do much here, sadly.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for fixing this, the solution LGTM. Failing MacOS tests are unrelated.

However, old code that uses model.language_model as a target will fail with AttributeError: 'LlamaModel' object has no attribute 'prepare_inputs_for_generation'. I fear that we have no way of mitigating this since we don't have access to the 'top-level' model, the only thing we could do is to raise a more helpful error message but for reliably detecting the case that we're targeting a VLM's language model we'd need the VLM's model config (e.g., to check for the text_config key) or have access to the VLM itself. We cannot do much here, sadly.

I agree with your conclusion here. My suggestion is to take a note that in the next release text, we should mention this possible backwards incompatibility and provide an example how to migrate the code.