fix: un-blocklist deepseek_v3 in transformers tokenizer class set by ZhiyuLi-Nvidia · Pull Request #3116 · NVIDIA-NeMo/RL

ZhiyuLi-Nvidia · 2026-07-08T05:57:27Z

Summary

#2764
Two coordinated fixes so Moonlight-16B-A3B (and any future deepseek_v3 model) can load its tokenizer offline.

Bug 1 — transformers blocks the tokenizer

transformers 5.4–5.11 lists "deepseek_v3" in two internal registries that together force the fast tokenizer backend and suppress trust_remote_code:

Registry	Effect
`MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS` (set)	Suppresses the model's `tokenizer_class` / `auto_map`
`TOKENIZER_MAPPING_NAMES` (dict)	Pins `deepseek_v3` to `TokenizersBackend` (fast)

Under HF_HUB_OFFLINE=1 the fast backend can't enumerate the repo to find tiktoken.model and fails. Moonlight ships no tokenizer.json, only a slow TikTokenTokenizer reachable via auto_map — so the online fallback also can't help.

Fix: nemo_rl/__init__.py removes both entries at import time. discard / pop-with-default are no-ops if the entries are absent, so the patch is safe on any transformers version.

Bug 2 — dead code in Megatron tokenizer setup

nemo_rl/models/megatron/setup.py:setup_model_and_optimizer used to mutate hf_tokenizer_kwargs["trust_remote_code"] = True on the Megatron TokenizerConfig. This was dead code:

Field	Snapshot at `__post_init__` (empty dict)	Original code's intent	Actual runtime	This PR
`trust_remote_code`	`dict.get("trust_remote_code", False)` = False	True (dead)	False — dead	True (attribute set directly)
`tokenizer_hf_no_use_fast`	`not dict.get("use_fast", True)` = False (i.e. `use_fast=True`)	True (dead)	`use_fast=True` via default	Unchanged, `use_fast=True`

Root cause of the deadness: TokenizerConfig.__post_init__ (see Megatron-Bridge/src/megatron/bridge/training/tokenizers/config.py) snapshots hf_tokenizer_kwargs into plain attributes once at construction time; from then on build_tokenizer reads the attribute, not the dict.

Fix: assign the attribute directly — megatron_cfg.tokenizer.trust_remote_code = True. This is the API Megatron-Bridge's own deprecation notice recommends.

Blast radius of the `trust_remote_code=True` flip

Historically every model on this Megatron path ran with trust_remote_code=False (the dead-code effect above). Making it actually True:

Standard-tokenizer models (Llama / Qwen / GLM / gpt-oss / …) — no behavior change; transformers only consults the flag when the model has auto_map / tokenizer_class pointing at remote code.
Remote-code tokenizer models (Moonlight-16B-A3B, DeepSeek-V3, …) — bug fixed.

Test plan

llm_grpo_moonlight_16b_automodel_1n8g_ep8 (1n8g automodel) — 10+ training steps offline, KL 0.0003, no NaN, no backend-tokenizer errors on driver + 8 workers.
llm_grpo_moonlight_16ba3b_4n8g_megatron (4n8g Megatron) — 17+ training steps offline, KL 0.0003 stable across steps, zero tokenizer errors on driver + 32 workers.
Sanity probe: import nemo_rl; "deepseek_v3" not in MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS and "deepseek_v3" not in TOKENIZER_MAPPING_NAMES → True.
Sanity probe: offline AutoTokenizer.from_pretrained("moonshotai/Moonlight-16B-A3B-Instruct", trust_remote_code=True) returns TikTokenTokenizer with byte-identical input_ids to the worker-side NeMoAutoTokenizerWithBosEosEnforced under add_special_tokens=False.
llm_grpo_moonlight_16ba3b_4n8g_megatron_tq_simple and _fp8_e2e — variants of the base 4n8g recipe; expected to pass since the fix is at tokenizer construction (pre-quantization).

copy-pr-bot · 2026-07-08T05:57:31Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

ZhiyuLi-Nvidia · 2026-07-08T17:54:10Z

/ok to test 02e7e91

ZhiyuLi-Nvidia · 2026-07-08T20:20:58Z

/ok to c82d9a0

yuki-97

@ZhiyuLi-Nvidia thanks for the fix! left some minor comments.

Fixes #2764. transformers 5.4-5.11 lists "deepseek_v3" in two internal registries -- MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS (a set) and TOKENIZER_MAPPING_NAMES (a dict pinning it to "TokenizersBackend"). Together they force the fast tokenizer backend and suppress trust_remote_code, so AutoTokenizer can only load via a local tokenizer.json. Models like Moonlight-16B-A3B ship no tokenizer.json (only tiktoken.model + a remote-code TikTokenTokenizer), so offline loading fails with "Couldn't instantiate the backend tokenizer". Two coordinated patches: 1. nemo_rl/__init__.py: discard "deepseek_v3" from both registries at nemo_rl import time. Adds a version assertion that fires if transformers is bumped past 5.12 (upstream fix), noting Megatron-Bridge is the current blocker for that bump. 2. nemo_rl/models/megatron/setup.py: set megatron_cfg.tokenizer.trust_remote_code attribute directly. The pre-existing hf_tokenizer_kwargs["trust_remote_code"] dict mutation was dead code because Megatron-Bridge's TokenizerConfig snapshots hf_tokenizer_kwargs into plain attributes at __post_init__ and never re-reads the dict afterwards. Both patches are held together by the same MBridge-caps-transformers-below-5.12 dependency chain and can be dropped together once that pin is relaxed. Verified end-to-end: llm_grpo_moonlight_16b_automodel_1n8g_ep8 trains 10+ steps offline; llm_grpo_moonlight_16ba3b_4n8g_megatron trains 17+ steps offline with KL 0.0003 stable, no NaN, zero backend-tokenizer errors on driver or 32 workers. Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

…sert String comparison "5.5.0" < "5.12.0" is False in Python (lexicographic '5' > '1' at the minor position), so the assertion would fire and raise AssertionError on every import for every currently-pinned transformers version (>=5.5.0,<5.9.0). Switch to packaging.version.Version. Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

…op packaging dep) Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

…y canonical pattern Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

ZhiyuLi-Nvidia · 2026-07-09T09:10:07Z

/ok to test ca6a042

The lint CI's editable install pipeline (uv sync + setuptools) imports nemo_rl to statically read __version__ before deps are guaranteed to be present. Without a guard, 'import transformers' at nemo_rl load time crashes the build with 'nemo_rl has no attribute __version__'. Wrap the top-level 'import transformers' in try/except so the patch becomes a no-op under build isolation. Runtime venv always has the package, so the assertion + discards still fire in real use. Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

…ead of try/except Same rationale as the existing _check_container_fingerprint guard on line 99: setuptools' uv-based build isolation imports nemo_rl to read __version__, but transformers isn't in the isolated build env. Match the existing pattern -- skip the patch at the call site under build isolation, keep the function body clean (no exception handling for control flow). Runtime venv is never a build isolation, so the patch still fires in real use. Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

…emo_rl.models.policy.__init__ The patch is only needed by policy code paths that eventually call AutoTokenizer.from_pretrained -- driver's get_tokenizer(), Megatron worker's build_tokenizer, DTensor worker's tokenizer reconstruction. All three share nemo_rl.models.policy as an import ancestor. Placing it in nemo_rl.models.policy.__init__.py fires exactly once per process when policy code is first imported. Non-policy consumers (tests, utilities, environments) no longer mutate the global transformers registries as a side effect of importing nemo_rl. Bonus: setuptools build isolation reads nemo_rl.__version__ from nemo_rl.__init__.py but never cascades into models/policy, so the _is_build_isolation() guard is no longer needed at the call site. Verified locally with 'from nemo_rl.models.policy import PolicyConfig': deepseek_v3 removed from MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS and TOKENIZER_MAPPING_NAMES as expected. Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

ZhiyuLi-Nvidia · 2026-07-09T10:18:51Z

/ok to test 0405109

yuki-97

LGTM, thanks @ZhiyuLi-Nvidia @ahmadki

ZhiyuLi-Nvidia requested a review from a team as a code owner July 8, 2026 05:57

ZhiyuLi-Nvidia added the r0.7.0 label Jul 8, 2026

ZhiyuLi-Nvidia requested a review from a team as a code owner July 8, 2026 07:53

ZhiyuLi-Nvidia force-pushed the fix/moonlight-tokenizer-offline-2764 branch 2 times, most recently from 1b10a01 to 02e7e91 Compare July 8, 2026 17:52

ZhiyuLi-Nvidia added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Jul 8, 2026

ZhiyuLi-Nvidia requested review from ahmadki and yuki-97 July 8, 2026 18:00

yuki-97 reviewed Jul 9, 2026

View reviewed changes

Comment thread nemo_rl/__init__.py Outdated

Comment thread nemo_rl/__init__.py Outdated

Comment thread nemo_rl/models/megatron/setup.py Outdated

ZhiyuLi-Nvidia added 3 commits July 9, 2026 01:51

docs: add explicit TODO(#2764) marker above transformers version assert

41c8e08

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

docs: switch TODO format to canonical style (TODO: + issue URL)

a74e966

Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

ZhiyuLi-Nvidia force-pushed the fix/moonlight-tokenizer-offline-2764 branch from f421f4b to a74e966 Compare July 9, 2026 08:57

ZhiyuLi-Nvidia added 3 commits July 9, 2026 01:57

refactor: use stdlib tuple compare for transformers version check (dr…

a1d1a68

…op packaging dep) Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

refactor: use PkgVersion alias matching nemo_rl/utils/flops_tracker.p…

ca6a042

…y canonical pattern Signed-off-by: Zhiyu Li <zhiyul@NVIDIA.com>

ZhiyuLi-Nvidia added 3 commits July 9, 2026 02:42

ZhiyuLi-Nvidia requested a review from a team as a code owner July 9, 2026 10:17

yuki-97 approved these changes Jul 9, 2026

View reviewed changes

yuki-97 enabled auto-merge (squash) July 9, 2026 11:04

yuki-97 merged commit f18ca21 into main Jul 9, 2026
80 checks passed

yuki-97 deleted the fix/moonlight-tokenizer-offline-2764 branch July 9, 2026 11:29

yuki-97 mentioned this pull request Jul 10, 2026

fix: fix dapo-deepseek-v3-64n8g.v2 and dsv3 tokenizer #3154

Merged

Navigation Menu

Uh oh!

fix: un-blocklist deepseek_v3 in transformers tokenizer class set - #3116

fix: un-blocklist deepseek_v3 in transformers tokenizer class set#3116
yuki-97 merged 9 commits into
mainfrom
fix/moonlight-tokenizer-offline-2764

ZhiyuLi-Nvidia commented Jul 8, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

ZhiyuLi-Nvidia commented Jul 8, 2026

Uh oh!

ZhiyuLi-Nvidia commented Jul 8, 2026

Uh oh!

yuki-97 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZhiyuLi-Nvidia commented Jul 9, 2026

Uh oh!

ZhiyuLi-Nvidia commented Jul 9, 2026

Uh oh!

yuki-97 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ZhiyuLi-Nvidia commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Bug 1 — transformers blocks the tokenizer

Bug 2 — dead code in Megatron tokenizer setup

Blast radius of the trust_remote_code=True flip

Test plan

Uh oh!

copy-pr-bot Bot commented Jul 8, 2026

Uh oh!

ZhiyuLi-Nvidia commented Jul 8, 2026

Uh oh!

ZhiyuLi-Nvidia commented Jul 8, 2026

Uh oh!

yuki-97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZhiyuLi-Nvidia commented Jul 9, 2026

Uh oh!

ZhiyuLi-Nvidia commented Jul 9, 2026

Uh oh!

yuki-97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZhiyuLi-Nvidia commented Jul 8, 2026 •

edited

Loading

Blast radius of the `trust_remote_code=True` flip