[https://nvbugs/6160085][fix] At tensorrt_llm/tokenizer/tokenizer.py import time, re-export `bytes_to_unicod

[tensorrt-cicd]

Summary

• Root cause: Kimi-K2's trust_remote_code tokenization_kimi.py imports bytes_to_unicode from transformers.models.gpt2.tokenization_gpt2, but transformers 5.x moved that symbol to transformers.convert_slow_tokenizer, so the import fails and TRT-LLM falls back to tokenizer=None.
• Fix: At tensorrt_llm/tokenizer/tokenizer.py import time, re-export bytes_to_unicode onto transformers.models.gpt2.tokenization_gpt2 from transformers.convert_slow_tokenizer if missing; remove the two matching nvbugs/6160085 waivers. Verified with the full test (passed in 1421s).
• Automated fix generated by repair-bot

Test plan

• Verify fix on the same GPU type as the original failure
• Check for regressions in related tests

Links

• Bug: https://nvbugs/6160085

Summary by CodeRabbit

• Chores

  • Added compatibility support for Transformers 5.x to ensure seamless tokenizer integration with newer versions of the library.

• Tests

  • Removed waivers for previously failing test cases, indicating resolution of underlying issues.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds a compatibility shim to handle Transformers 5.x moving the bytes_to_unicode symbol, and removes two test waivers for long-sequence MoE tests that are now passing.

Changes

Transformers 5.x Compatibility and Test Cleanup

Layer / File(s)	Summary
Transformers 5.x bytes_to_unicode compatibility shim tensorrt_llm/tokenizer/tokenizer.py	Adds a try/except block that re-exports bytes_to_unicode from the new Transformers 5.x location into the legacy transformers.models.gpt2.tokenization_gpt2 module path, ensuring custom and remote tokenizers importing from the old location continue to work.
Test waiver cleanup tests/integration/test_lists/waives.txt	Removes two SKIP waivers for TestKimiK2 long-sequence TRTLLM MoE async and stress tests that no longer require exemption.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Suggested reviewers

• jieli-matrix
• pcastonguay
• yihwang-nv

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: a compatibility shim re-exporting bytes_to_unicode at tokenizer.py import time for Transformers 5.x compatibility.
Description check	✅ Passed	The description provides root cause, solution, verification results, and references the bug ticket; all key information is present despite abbreviated format.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/tokenizer/tokenizer.py`:
- Around line 18-24: The compatibility shim is importing bytes_to_unicode from
the wrong module; update the fallback import so we pull bytes_to_unicode from a
model-specific tokenizer module (e.g., from
transformers.models.clvp.tokenization_clvp or
transformers.models.gpt2.tokenization_gpt2) instead of
transformers.convert_slow_tokenizer — i.e., in tokenizer.py, inside the try
block where _gpt2_mod is loaded and you check hasattr(_gpt2_mod,
"bytes_to_unicode"), attempt to import bytes_to_unicode from known
model-specific modules (try clvp.tokenization_clvp then gpt2.tokenization_gpt2)
and assign it to _gpt2_mod.bytes_to_unicode so the monkey‑patch succeeds.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4dd320ce-dd84-4ab0-aff3-90c60e840b0b

📥 Commits

Reviewing files that changed from the base of the PR and between 553b8de and 96ec4d6.

📒 Files selected for processing (2)

• tensorrt_llm/tokenizer/tokenizer.py
• tests/integration/test_lists/waives.txt

💤 Files with no reviewable changes (1)

• tests/integration/test_lists/waives.txt

[ziyixiong-nv]

/bot run

[tensorrt-cicd]

PR_Github #50482 [ run ] triggered by Bot. Commit: a2bc16f Link to invocation

[tensorrt-cicd]

PR_Github #50482 [ run ] completed with state SUCCESS. Commit: a2bc16f
/LLM/main/L0_MergeRequest_PR pipeline #39991 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[Superjomn]

LGTM

[ziyixiong-nv]

/bot run

[tensorrt-cicd]

PR_Github #50683 [ run ] triggered by Bot. Commit: a2bc16f Link to invocation

[tensorrt-cicd]

PR_Github #50683 [ run ] completed with state SUCCESS. Commit: a2bc16f
/LLM/main/L0_MergeRequest_PR pipeline #40171 completed with status: 'SUCCESS'

CI Report

Link to invocation