Remove all_special_tokens_extended from tokenizer code

[hmellor]

This attribute is:

• equivalent to all_special_tokens
• not actually used anywhere in vLLM
• deleted in Transformers v5 by rm slow tokenizers huggingface/transformers#40936 (so refraining from accessing it enables forward compatibility with Transformers v5)

Should resolve the failures in https://buildkite.com/vllm/ci/builds/41020/steps/canvas?sid=019ac943-dca1-43d6-ad64-ca0187f645bc

[chatgpt-codex-connector]

💡 Codex Review

https://github.com/vllm-project/vllm/blob/44491395c266dc41b892fd8a98f25fa50b83d7ee/vllm/transformers_utils/config.py#L460-L461
Handle legacy rope_theta on transformers>=5

In patch_rope_parameters the transformers>=5 branch now no-ops, so legacy configs that still expose config.rope_theta (common for custom transformers v4 models) are no longer converted into rope_parameters or validated. When running with transformers v5, such configs keep rope_parameters unset, so later consumers (e.g., prepare_hf_config and max length scaling) drop the provided RoPE scaling/θ information and may initialize RoPE with incorrect defaults. Please keep the standardization/validation here for v5 the same way the v4 branch still does.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[DarkLight1337]

I was planning on removing it as well since it's unused in vLLM, thanks for doing it!

[hmellor]

I've enabled the Transformers nightly tests to verify that the errors relating to this attr are gone https://buildkite.com/vllm/ci/builds/41093/steps/canvas?sid=019acb27-ad2f-4514-ab1e-68141500585c

[gemini-code-assist]

Code Review

This pull request correctly removes the all_special_tokens_extended attribute, which simplifies the tokenizer-related code and improves forward compatibility with Transformers v5. The changes are applied consistently across tests and implementations. However, the pull request also includes undocumented changes to RoPE parameter handling in vllm/transformers_utils/config.py and vllm/transformers_utils/configs/qwen3_next.py. Specifically, a block of code for Transformers v5 compatibility has been commented out. While this might be intentional, leaving commented-out code reduces maintainability. I've added a suggestion to remove it. It would be beneficial to update the pull request description to cover all changes for better clarity.