[BugFix] Correct max_model_len derivation from config.json for Mistral format

[princepride]

FIX #17747

@tjohnson31415

Thank you for your patience and suggestions.

• I've reverted the code to its original version first before applying modifications.

• I have removed all logic related to max_seq_len.

• The subsequent logic (to fetch from standard Hugging Face config) will only be triggered if max_position_embeddings cannot be obtained from params.json.

• I am referencing the configuration loaded from config.json as it's done for ConfigFormat.HF (i.e., using AutoConfig.from_pretrained).

• Something went wrong, and I accidentally caused issues in the original branch, so I've opened a new PR to merge the code.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[princepride]

@DarkLight1337 Please excuse my being a newbie. I accidentally messed up the original branch, so I closed this pull request: #17777. Therefore, after modifying the code, I submitted it again. Could you please re-assign @tjohnson31415 as the reviewer for this bug fix? Thank you.

[DarkLight1337]

cc @tjohnson31415

[tjohnson31415]

Another round of simplification suggestions 😅
I think we should be able to get it down to something like this mock code:

    # If max_position_embeddings is not in params.json
    if config_dict.get("max_position_embeddings") is None:
        # try to get it from the HF config, but default to 128_000
        max_position_embeddings = 128_000
        try:
            hf_config = get_config(
                    model,
                    revision=revision,
                    config_format=ConfigFormat.HF,
                    <<other args>>,
            )
            if hf_value := hf_config.get_text_config().max_position_embeddings:
                max_position_embeddings = hf_value
        except Exception as e:
            logger.warning('Useful warning message...')
        config_dict["max_position_embeddings"] = max_position_embeddings

[princepride]

@tjohnson31415 Thanks for your patience; I've already changed the code as you directed.

[tjohnson31415]

Thanks for the quick iteration!
I had a couple more thoughts, but do let me know if you disagree with the suggestions!

[tjohnson31415]

Using AutoConfig directly instead of calling back through get_config means that we don't get the check for the "custom model class" in the registry, i.e. this code.

If we can use get_config() here, it would ensure that this fallback uses the same configuration that would be used with --config-format=hf, e.g. for models that are supported in vLLM but not transformers.

[princepride]

Thanks for the suggestion. Unfortunately, we can't use get_config() here as it would lead to a circular call.Perhaps I could refer to the code block from

vllm/vllm/transformers_utils/config.py

Lines 308 to 349 in 0189a65

	config_dict, _ = PretrainedConfig.get_config_dict(
	model,
	revision=revision,
	code_revision=code_revision,
	token=HF_TOKEN,
	**kwargs,
	)
	# Use custom model class if it's in our registry
	model_type = config_dict.get("model_type")
	if model_type in _CONFIG_REGISTRY:
	config_class = _CONFIG_REGISTRY[model_type]
	config = config_class.from_pretrained(
	model,
	revision=revision,
	code_revision=code_revision,
	token=HF_TOKEN,
	**kwargs,
	)
	else:
	try:
	config = AutoConfig.from_pretrained(
	model,
	trust_remote_code=trust_remote_code,
	revision=revision,
	code_revision=code_revision,
	token=HF_TOKEN,
	**kwargs,
	)
	except ValueError as e:
	if (not trust_remote_code
	and "requires you to execute the configuration file"
	in str(e)):
	err_msg = (
	"Failed to load the model config. If the model "
	"is a custom model not yet available in the "
	"HuggingFace transformers library, consider setting "
	"`trust_remote_code=True` in LLM or using the "
	"`--trust-remote-code` flag in the CLI.")
	raise RuntimeError(err_msg) from e
	else:
	raise e

to initialize hf_config. This approach could help ensure better configuration support for models that are supported in vLLM but not in Transformers, aligning with the goal of having the fallback use a consistent configuration.

[tjohnson31415]

We can set an explicit config_format=ConfigFormat.HF to avoid it being circular. I think that should work to get the functionality without having to copy the code.

[princepride]

Thank you for your suggestions. I followed your advice, and it successfully ran while also making the code simpler. As a newcomer, I cannot express in words my gratitude for your patience.

[tjohnson31415]

LGTM
Thanks for picking up this bugfix @princepride!