[Bugfix] Pass hf_token through config loading paths for gated model support

[javierdejesusda]

Purpose

Fixes #31894

The hf_token argument provided to the LLM instance was not forwarded to HuggingFace API calls during config loading, causing gated models to fail authentication when hf_token is passed explicitly (rather than via environment variable).

Three gaps existed:

1. maybe_override_with_speculators() called PretrainedConfig.get_config_dict() without the token, so speculators auto-detection failed for gated models.
2. ModelConfig.__post_init__() called get_config() without forwarding self.hf_token, so the main config loading path (PretrainedConfig.get_config_dict, AutoConfig.from_pretrained, config_class.from_pretrained) also lacked authentication.
3. try_get_generation_config() called GenerationConfig.from_pretrained() without the token, so generation config loading failed for gated models.

This fix threads hf_token through all three paths, using token= at the HuggingFace API boundary (matching the convention at config.py L1031).

Note: PR #31974 attempted a similar fix but was closed with 0 commits.

Test Plan

# Non-gated model (no regression)
python -c "from vllm.transformers_utils.config import maybe_override_with_speculators; \
  print(maybe_override_with_speculators('gpt2', None, False))"

# Gated model with explicit token (speculators path)
python -c "from vllm.transformers_utils.config import maybe_override_with_speculators; \
  print(maybe_override_with_speculators('meta-llama/Llama-2-7b-hf', None, False, hf_token='hf_...'))"

Test Result

Non-gated models work as before. With hf_token=None (default), behavior is unchanged — HuggingFace Hub falls back to environment-based token lookup internally.

---

• The purpose of the PR
• The test plan
• The test results

[gemini-code-assist]

Code Review

The pull request introduces support for passing a HuggingFace token (hf_token) through the vLLM engine configuration. This token is now accepted as an argument in create_engine_config and maybe_override_with_speculators, and is subsequently used when calling PretrainedConfig.get_config_dict to enable authenticated access for fetching model configurations from HuggingFace.

[yewentao256]

Please merge from main to fix the pre-commit issue

[yewentao256]

LGTM, thanks for the work!