Dead code, cache_kwargs

[freckletonj]

System Info

This is rather low priority. I found it confusing chasing down the occurrences of cache_kwargs to try to understand how sin and cos are getting used in the Cache, presumably for RoPE, but it appears they are only used in the SinkCache which, as far as I can tell, isn't commonly made use of by any transformer. cache_positions appears to be used, but not sin and cos.

This dead bit of code shows up in 3 popular models, but also many others, and I don't think it has any bearing on anything at all:

Llama: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L339
Mistral: https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L255
Qwen: https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2/modeling_qwen2.py#L278

Per your tagging request: @ArthurZucker, and I saw @zucchini-nlp and @zhenglongjiepheonix in git-blame
And sorry if this is just noise, I'll shut up about small stuff if you like.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

nothing to reproduce, just dead code

Expected behavior

just dead code

[freckletonj]

Ah, forgive me, it appears that these params are in place should someone choose to use the SinkCache, even if it is not the default.

If this is correct, maybe the issue is just one of documentation, but either way, please feel free to close this issue.

[ArthurZucker]

Yep this is correct, if you want to update the documentation feel free to do it!

[freckletonj]

I appreciate the confirmation!