[Bugfix] Fix transformers 5.x compat issues in online TTS serving

[linyueqian]

Summary

• Remove fix_mistral_regex=True from AutoTokenizer.from_pretrained (parameter removed in transformers 5.x)
• Add fallback for 'default' rope_type missing from ROPE_INIT_FUNCTIONS in transformers 5.x (inline standard sinusoidal RoPE)
• Clamp num_cached_tokens to max(0, ...) in OmniGenerationScheduler to prevent negative value crash

These fixes are required for online TTS serving to work with the current environment (transformers 5.2.0, pinned via uv.lock).

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 339b3ddb2b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[hsliuustc0106]

Summary

This PR fixes three compatibility issues with transformers 5.x that were breaking online TTS serving. The changes are minimal, focused, and address real breaking changes in the transformers library.

Pros:

• Addresses actual breaking changes in transformers 5.x
• Small, focused fixes (21 additions, 4 deletions)
• Good inline documentation explaining the 'default' rope_type fallback
• Defensive programming with the max(0, ...) clamp
• Clear error message for unsupported rope types

Cons:

• No test coverage for the new fallback logic
• The num_cached_tokens negative value issue suggests a deeper problem upstream

Recommendation: Approve with suggestions for follow-up investigation.

[hsliuustc0106]

Good: Well-documented fallback

The inline implementation of 'default' RoPE is well-documented and correct. The comment clearly explains why this is needed (transformers 5.x removed 'default' from ROPE_INIT_FUNCTIONS).

Suggestion: Consider adding a reference to the transformers version where this changed:

# transformers>=5.0 removed 'default' from ROPE_INIT_FUNCTIONS (see transformers PR #xxxxx)

[hsliuustc0106]

Good: Clear error message

The error message provides helpful context about what rope types are supported. This will make debugging easier if an unsupported type is encountered.

[hsliuustc0106]

Issue: Symptom fix, not root cause

Clamping num_cached_tokens to max(0, ...) prevents the crash, but it's treating the symptom rather than the root cause. A negative num_cached_tokens suggests:

1. There's a bug upstream where request.num_cached_tokens is being set to a negative value
2. Or there's a logic error in how cached tokens are being counted

Recommendation:

• Add a warning log when clamping occurs to help track down the root cause:

num_cached = request.num_cached_tokens
if num_cached < 0:
    logger.warning(f"Negative num_cached_tokens ({num_cached}) detected for request {request.request_id}, clamping to 0")
    num_cached = 0
num_cached_tokens=num_cached,

• File a follow-up issue to investigate why num_cached_tokens can be negative

This defensive fix is fine for now, but understanding the root cause would prevent potential issues elsewhere.

[hsliuustc0106]

Same issue here

Same recommendation as above - consider adding logging to track when this clamping occurs.

[lishunyang12]

Nit: _default_rope_init doesn't close over anything — pull it out to module level so you're not creating a new function object per instance.

[lishunyang12]

+1 to adding a logger.warning when clamping fires. Silent clamps on negative values will mask whatever upstream bug is producing them.

[lishunyang12]

Left a couple minor comments. The fixes look correct overall.

[linyueqian]

@hsliuustc0106 check this again?

[linyueqian]

Added two more commits to this PR:

1. Fix MRotaryEmbedding import and code predictor dtype/sampling issues (ea3ae58)

• gpu_model_runner.py: Import OmniMRotaryEmbedding instead of upstream MRotaryEmbedding. After the vLLM 0.16.0 rebase, upstream renamed get_input_positions_tensor to get_next_input_positions_tensor, but OmniMRotaryEmbedding has custom omni-specific position logic in the old method. This only affects the else branch (non-mrope models).
• qwen3_tts_code_predictor_vllm.py: Replace hardcoded torch.bfloat16 cast in prefill_logits with the model's actual weight dtype. On GPUs without bfloat16 support (compute capability < 8.0), vLLM auto-casts weights to float16 but the hardcoded bfloat16 input causes a dtype mismatch in rms_norm.
• qwen3_tts_code_predictor_vllm.py: Cast logits to float32 before softmax/multinomial sampling to prevent NaN/Inf from lower-precision dtypes crashing torch.multinomial.

2. Preserve fix_mistral_regex compat for transformers 4.x (622b5df)

• The previous commit removed fix_mistral_regex=True entirely, but users on transformers 4.x still need it. Now conditionally passes it based on transformers.__version__.

[linyueqian]

@hsliuustc0106 Ready for another review when you get a chance. Two new commits added addressing bugs found during Qwen3-TTS benchmarking on cc=7.5 GPUs.

[hsliuustc0106]

do we need to point out the version dependency in examples?

[linyueqian]

No version dependency to document — the fixes are all backward-compatible. The fix_mistral_regex kwarg is now conditionally passed based on transformers.__version__, so it works on both 4.x and 5.x without any user action. The other fixes (RoPE fallback, num_cached_tokens clamp, MRotaryEmbedding import, code predictor dtype/sampling) are transparent to users.

[hsliuustc0106]

Code review

Found 1 issue:

1. Missing regression test for recurring num_cached_tokens bug (conventions.md says "Bug fixes: Regression test? Root cause understood?")

This is the second PR fixing the same num_cached_tokens negative value crash:

• PR [Bugfix] Set num_cached_tokens = num_computed_tokens if unset #1471 (same author) added defensive check in schedule() method (line 130-131)
• PR [Bugfix] Fix transformers 5.x compat issues in online TTS serving #1536 now adds another check in update_from_output() method (lines 450-453, 488-493)

The fact that a second fix is needed proves the first fix was incomplete. Without a regression test, there's no way to verify all code paths are covered. This bug causes a Prometheus counter crash (ValueError: Counters can only be incremented by non-negative amounts) in production.

Recommend adding a test that:

1. Verifies num_cached_tokens is never negative after scheduling
2. Covers both the schedule() and update_from_output() code paths

vllm-omni/vllm_omni/core/sched/omni_generation_scheduler.py

Lines 448 to 454 in f128308

	if new_token_ids or pooler_output is not None or kv_transfer_params or stopped:
	# Add EngineCoreOutput for this Request.
	num_cached = request.num_cached_tokens
	if num_cached < 0:
	logger.warning("Negative num_cached_tokens (%d) for request %s, clamping to 0", num_cached, req_id)
	num_cached = 0
	outputs[request.client_index].append(

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

[linyueqian]

This is a defensive fix for a negative value coming from upstream vLLM's generation scheduler. Adding a regression test here would mean mocking the full scheduler pipeline, which is brittle across vLLM rebases. The warning log helps us track occurrences and investigate the root cause upstream.

[Gaohan123]

LGTM