TypeError: LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position'

[Naozumi520]

System Info

• transformers version: 4.39.0.dev0
• Platform: Windows-10-10.0.22621-SP0
• Python version: 3.11.8
• Huggingface_hub version: 0.21.3
• Safetensors version: 0.4.2
• Accelerate version: 0.27.2
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.1+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker @amyeroberts

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1. Run llava using llava.serve.cli and libraries in my versions.
2. Ask model some questions.
3. Return error:

<|im_start|>user
: hi
<|im_start|>assistant
: Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Naozu\Desktop\AI\2.Models\Nazuna20240304\LLaVA\llava\serve\cli.py", line 128, in <module>
    main(args)
  File "C:\Users\Naozu\Desktop\AI\2.Models\Nazuna20240304\LLaVA\llava\serve\cli.py", line 98, in main
    output_ids = model.generate(
                 ^^^^^^^^^^^^^^^
  File "C:\Users\Naozu\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Naozu\Desktop\AI\2.Models\Nazuna20240304\LLaVA\llava\model\language_model\llava_llama.py", line 137, in generate
    return super().generate(
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\Naozu\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Naozu\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\generation\utils.py", line 1597, in generate
    result = self.sample(
             ^^^^^^^^^^^^
  File "C:\Users\Naozu\AppData\Local\Programs\Python\Python311\Lib\site-packages\transformers\generation\utils.py", line 2711, in sample
    outputs = self(
              ^^^^^
  File "C:\Users\Naozu\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Naozu\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: LlavaLlamaForCausalLM.forward() got an unexpected keyword argument 'cache_position'

Expected behavior

Everything should work just fine.

[ArthurZucker]

Hey! I think you should open this issue on the LLaVA repo maybe? or just update your version of transformers!

[Naozumi520]

Hm that's weird, cuz I install transformers by forking the repo, it should be the latest one...

[ArthurZucker]

Could you provide a transformers reproducer? Here it is wrapped around the \LLaVA\llava\model\language_model\llava_llama.py and I am not sure what is in there!

[YujieLu10]

a workaround here, add cache_position=None to the forward() method in Class LlavaLlamaForCausalLM

[LsTam91]

I get the same error when I Fine-Tune my llama type models since I updated transformer library

[ArthurZucker]

Again I need a reproducer to be able to push a fix. Also now the cache_positions should no be required by the model and is intialized if not passed, see:

transformers/src/transformers/models/llama/modeling_llama.py

Lines 980 to 985 in 76a33a1

	if cache_position is None:
	if isinstance(past_key_values, StaticCache):
	raise ValueError("cache_position is a required argument when using StaticCache.")
	cache_position = torch.arange(
	past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
	)

.

[amankumarhal]

I had the same issue when I tried with transformers version 4.38.x and 4.39.x. I installed transformers==4.37.2 and it worked.

[ArthurZucker]

Thanks @amankumarhal , but again, without reproducers I am having a really hard time fixing this

[2015aroras]

This repros for me with a different model with the following code. I imagine it repros with many models.

from transformers import TextGenerationPipeline
from transformers.models.auto import AutoModelForCausalLM, AutoTokenizer

model_path = "allenai/OLMo-1B"
model = AutoModelForCausalLM.from_pretrained(model_path, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
pipeline = TextGenerationPipeline(model=model, tokenizer=tokenizer)
output = pipeline("question: who wrote romeo and juliet? answer: ", max_new_tokens=30)

I believe the issue is caused by 23db187 at src/transformers/generation/utils.py line 1940. This introduced "cache_position" into model_kwargs (as I saw when I went over the above repro with my debugger).

[ArthurZucker]

Oh this is using a custom code (remote code). The issue should be opened and fixed on the OlMO repo. Otherwise just wait for #29885 to be merged, which will have native support!

[vanitech]

+1 encountered this error when trying to use olmo

[ChenRan2000]

+1 encountered this error.

[ArthurZucker]

Olmo is not part of transformers will be with #29885

[vanitech]

Workaround that got rid of the error for me was an older version of transformers
%pip install transformers==4.38

[ArthurZucker]

Olmo is on transformers now

[MaximilianKr]

Got the same error for OLMo-1B with Transformers 4.39.3

OLMoForCausalLM.forward() got an unexpected keyword argument 'cache_position'

edit: I was using conda, Transformers 4.40.0 is not yet available. Reinstalling most recent Transformers 4.40.0 via pip led to the following error mesage when importing hf_olmo:

ValueError: 'olmo' is already used by a Transformers config, pick another name.

[2015aroras]

Please use -hf versions of OLMo (e.g. https://huggingface.co/allenai/OLMo-1.7-7B-hf) for Transformer 4.40.0 onwards (OLMo just got integrated into transformers!).

[segalinc]

I also added an issue in the LLAva repo haotian-liu/LLaVA#1448

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Keyan-Hu]

Transformers 4.37.2 OK！

[ChL-eng]

Transformers 4.37.2 OK！

it works!

[ArthurZucker]

Closing then! Thanks 🤗

[KevinXu-01]

a workaround here, add cache_position=None to the forward() method in Class LlavaLlamaForCausalLM

It works for me. Thanks!!!