Patch Mistral config

[juliendenize]

Purpose

This PR does the following:

• rope parameters are now casted to the type expected by Transformers v5. I believe it has no effect on vLLM computations but please correct me if I'm wrong. This silences warnings raised by Transformers
• ignore warnings from Transformers relative to apply_yarn_scaling parameter not found that is an argument stored by Mistral config but unknown to Transformers.
• infer dtype directly in the Mistral config instead of later in the code. This way errors mentioning the model is not a safetensors repo that is raised when infering dtype in multiple places are removed by only one change instead of multiple ones.

Test Plan

Checked by serving a Mistral model

Test Result

serving worked

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces several patches for Mistral model configuration handling. It silences some warnings from the Transformers library, improves data type casting for RoPE parameters, and refactors the data type inference to be more robust. The changes are generally good, but I've found a critical thread-safety issue with how global constants are being modified. My review includes a suggestion to fix this potential race condition.

[gemini-code-assist]

The modification of global constants in huggingface_hub.constants is not thread-safe. In a scenario where multiple models are loaded concurrently in different threads (e.g., one Mistral model and one standard Hugging Face model), this monkey-patching can create a race condition. One thread might be expecting the default constant values while another has temporarily changed them, potentially leading to FileNotFoundError or other unpredictable behavior during model loading. To prevent this, the critical section where constants are modified should be protected by a lock.

Please also add import threading at the top of the file.

_mistral_patch_lock = threading.Lock()


@contextmanager
def _mistral_patch_hf_hub_constants() -> Iterator[None]:
    with _mistral_patch_lock:
        hf_safetensors_single_file = constants.SAFETENSORS_SINGLE_FILE
        hf_safetensors_index_file = constants.SAFETENSORS_INDEX_FILE
        constants.SAFETENSORS_SINGLE_FILE = "consolidated.safetensors"
        constants.SAFETENSORS_INDEX_FILE = "consolidated.safetensors.index.json"
        try:
            yield
        finally:
            constants.SAFETENSORS_SINGLE_FILE = hf_safetensors_single_file
            constants.SAFETENSORS_INDEX_FILE = hf_safetensors_index_file

[hmellor]

To clarify, are these warnings that are appearing in v4 or v5?

Also, would it not be better to add apply_yarn_scaling to the list of expected keys in Transformers so that this issue is fixed for you everywhere, not just in vLLM?

[juliendenize]

@hmellor it only occurs for v5.

Also, would it not be better to add apply_yarn_scaling to the list of expected keys in Transformers so that this issue is fixed for you everywhere, not just in vLLM?

vLLM use arguments slightly differently than HF that requires adding this key which is not needed in HF so not sure it would make sense. Either way for now it's not in hf so silencing the warnings would be cool as a temporary fix if that's ok to you.

[hmellor]

Ok, would a better soltuion be to add apply_yarn_scaling to optional_keys in https://github.com/huggingface/transformers/blob/1af62a6046ae574b719f0418fdd1d5895080d72e/src/transformers/modeling_rope_utils.py#L758-L767?

[juliendenize]

Yeah we can do it if you prefer we do it like that. So for this current pr do you want me to discard only this filter warning but keep the casting ?

[hmellor]

Yeah I think that's best. It hardens the conversion in vLLM and ensures that you don't get the unexpected key warning anywhere that Mistral configs are loaded with Transformers.

[juliendenize]

Made the Transformers PR here :)
huggingface/transformers#44747

[hmellor]

Could we instead pass apply_yarn_scaling to validate_rope in

vllm/vllm/transformers_utils/config.py

Line 370 in bf9a185

config.validate_rope()

(I wasn't aware that this was an option before).

Something like:

ignore_keys = {}
if config_format == "mistral" and config.rope_parameters.type == "yarn":
    ignore_keys.add("apply_yarn_scaling")
config.validate_rope(ignore_keys=ignore_keys)

[juliendenize]

Would have been better indeed i'll do a followup pr once i have time

[hmellor]

I've implemented this in #37292