feat(frontend): make tokenizer fallback configurable

[xianlubird]

Summary

Basetenkenizer support landed in #12376 alongside the existing fastokens backend. Both alternate tokenizer backends currently prioritize availability: when the selected implementation cannot load a model's tokenizer.json, the frontend logs a warning and transparently uses HuggingFace instead.

That default is useful for general serving, but it can hide a configuration mismatch when an operator explicitly requires an alternate backend. For example, a performance run can complete successfully while measuring HuggingFace after an unnoticed fallback, and a production deployment can start without the tokenizer implementation it was configured to use.

This change makes that fallback policy configurable while keeping existing deployments compatible:

• Add the negatable --tokenizer-fallback / --no-tokenizer-fallback frontend option, with DYN_TOKENIZER_FALLBACK as the equivalent environment variable.
• Keep fallback enabled by default, preserving the existing behavior for all current deployments.
• Pass the parsed frontend value explicitly through EntrypointArgs, ModelRuntimeConfig, and every ModelWatcher path so dynamically discovered model cards receive the same policy before tokenizer construction.
• Validate environment values strictly. Supported boolean spellings are true/false, 1/0, on/off, and yes/no; invalid values now fail with a configuration error instead of silently restoring the default.
• When fallback is disabled, return the original fastokens or basetenkenizer loading error with context instead of constructing the HuggingFace fallback.
• Apply the same fail-fast behavior to non-UTF-8 tokenizer paths, which alternate backends cannot consume.
• Document the option in the tokenizer guide, frontend configuration reference, and Fastokens usage guide.

Example:

python -m dynamo.frontend \
  --tokenizer basetenkenizer \
  --no-tokenizer-fallback

Validation

• cargo test -p dynamo-llm --no-default-features tokenizer_fallback --lib --tests (3 focused tests passed)
• Configuration utility tests (30 passed)
• Added Rust coverage for explicit configuration precedence, strict environment parsing, serialization compatibility, fail-fast loading behavior, and applying an explicit false override to a dynamically discovered model card
• cargo fmt --all -- --check
• python3 -m black --check components/src/dynamo/common/configuration/utils.py components/src/dynamo/common/tests/configuration/test_utils.py components/src/dynamo/frontend/frontend_args.py
• python3 docs/fern/scripts/check_asset_paths.py
• python3 docs/fern/scripts/gen_python_api.py --check (Python 3.13, matching CI)
• python3 -m py_compile components/src/dynamo/common/configuration/utils.py components/src/dynamo/common/tests/configuration/test_utils.py components/src/dynamo/frontend/frontend_args.py

Related Issues

None.

Reviewer Guide

Start with the negatable frontend option in components/src/dynamo/frontend/frontend_args.py and the explicit Python-to-Rust handoff in components/src/dynamo/frontend/main.py and lib/bindings/python/rust/llm/entrypoint.rs. Then review strict fallback precedence in lib/llm/src/local_model/runtime_config.rs, propagation into dynamic model discovery in lib/llm/src/discovery/watcher.rs, and the fail-fast branches in lib/llm/src/model_card.rs.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[dynamo-ops]

/ok to test 58097b8

[devin-ai-integration]

Devin Review found 1 potential issue.

[coderabbitai]

Walkthrough

Changes

The frontend now configures tokenizer fallback through CLI and environment settings. The model card preserves fallback by default and supports strict errors for alternate tokenizer failures. Tests and documentation cover both modes.

Tokenizer fallback control

Layer / File(s)	Summary
Frontend configuration and environment wiring components/src/dynamo/frontend/frontend_args.py, components/src/dynamo/frontend/main.py	Adds the tokenizer_fallback setting, negatable CLI options, and DYN_TOKENIZER_FALLBACK propagation.
Tokenizer loading behavior lib/llm/src/model_card.rs, lib/llm/tests/model_card.rs	Parses the fallback setting. FastTokenizer and BasetenTokenizer failures either fall back to HuggingFace or return errors. Tests cover both behaviors.
Configuration and behavior documentation docs/fern/pages/developer-guide/..., docs/fern/pages/reference/..., docs/fern/pages/use-cases/...	Documents configuration, default behavior, strict startup failures, examples, compatibility, and troubleshooting.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: configurable tokenizer fallback behavior.
Description check	✅ Passed	The description covers the change, implementation details, reviewer starting points, validation, and related-issue status.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/fern/pages/use-cases/fastokens-tokenizer/overview.md`:
- Line 120: Update the Mermaid flow chart near the fastokens loading path to
branch on whether tokenizer fallback is disabled: failed fastokens loading
should stop model initialization when `--no-tokenizer-fallback` is set, while
retaining the existing HuggingFace fallback and request-serving path when
fallback remains enabled.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 621a7633-3dc0-4d64-959b-acf83abb9127

📥 Commits

Reviewing files that changed from the base of the PR and between a3a3c24 and 58097b8.

📒 Files selected for processing (7)

• components/src/dynamo/frontend/frontend_args.py
• components/src/dynamo/frontend/main.py
• docs/fern/pages/developer-guide/knowledge-base/modular-components/frontend/tokenizer.md
• docs/fern/pages/reference/components/frontend-configuration.mdx
• docs/fern/pages/use-cases/fastokens-tokenizer/overview.md
• lib/llm/src/model_card.rs
• lib/llm/tests/model_card.rs

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019ff8ba-75d5-76ca-bf0b-e0712093a1fe.docs.buildwithfern.com/dynamo/dev

[dynamo-ops]

/ok to test 82d9f0b

[dynamo-ops]

/ok to test 01159fc

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 36.36%
• Overall Coverage: 46.15% (-6.63%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 87d75bf | Docs | Datadog PR Page | Give us feedback!}

[jthomson04]

The explicit fallback setting is lost in the normal dynamic frontend path, and the boolean parsing behavior is inconsistent between the Python and Rust entry points. Please keep the fix focused and add one regression test that applies an explicit false override to a discovered model card.

[dynamo-ops]

/ok to test b7b4a6e

[dynamo-ops]

/ok to test f342a71

[dynamo-ops]

/ok to test ad47f56

[jthomson04]

The earlier propagation and strict parsing issues are fixed. One non-blocking CLI help and documentation note remains.

[dynamo-ops]

/ok to test fe28206

[harryskim]

Docs-only review (the 4 hand-written pages; the three docs/fern/pages/reference/api/python/*.mdx files are regenerated output and look consistent with _core.pyi).

Overall this reads well and the option is documented in all three places a reader would look. I checked the "in dynamic mode, discovery retries the load while the frontend continues running" claim against the discovery controller's retry/backoff path and it holds. The ParamField added to frontend-configuration.mdx matches the file's existing convention for negatable booleans, and the mermaid branch split in the fastokens guide is correct.

One blocking issue (wrong valid values in the tokenizer guide table) plus a couple of nits inline.

[dynamo-ops]

/ok to test 9b1a49d

[dynamo-ops]

/ok to test 6d24456

[harryskim]

LGTM

[dynamo-ops]

/ok to test 87d75bf

[xianlubird]

Looks like this CI failure is unrelated to the code changes in this PR. The first lychee pass hit a connection failure on an Istio docs URL. The retry did run and that link recovered, but a different CNCF Slack redirect then failed with a connection reset by peer. All PR-specific checks passed, so this appears to be an external network flake.