Skip to content

chore(deps): bump dynamo-tokenizers to 1.8.0 - #12707

Merged
jthomson04 merged 1 commit into
mainfrom
jthomson04/bump-dynamo-tokenizers-1.8.0
Aug 6, 2026
Merged

chore(deps): bump dynamo-tokenizers to 1.8.0#12707
jthomson04 merged 1 commit into
mainfrom
jthomson04/bump-dynamo-tokenizers-1.8.0

Conversation

@jthomson04

@jthomson04 jthomson04 commented Aug 5, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

  • bump the exact dynamo-tokenizers workspace pin from 1.7.0 to 1.8.0
  • update the root, Python bindings, and KVBM lockfiles, including fastokens 0.3.1
  • document segmented encoding support for the fastokens backend

Validation

  • cargo metadata --locked --format-version 1 --no-deps in the root workspace, lib/bindings/python, and lib/bindings/kvbm
  • cargo test --locked -p dynamo-llm --test model_card --test tokenizers
  • cargo clippy --locked --no-deps -p dynamo-llm --all-targets -- -D warnings
  • cargo fmt --all -- --check in the root workspace, lib/bindings/python, and lib/bindings/kvbm
  • fern check
  • fern docs broken-links
  • git diff --check
Open in Devin Review

Summary by CodeRabbit

  • Documentation

    • Clarified that basetenizer supports native encoding and decoding.
    • Added guidance for supported tokenizer.json models, including compatible Kimi tokenizer artifacts.

  • Improvements

    • Updated the tokenizer component to version 1.8.0 for improved compatibility.

@jthomson04
jthomson04 requested review from a team as code owners August 5, 2026 18:15
@github-actions github-actions Bot added chore documentation Improvements or additions to documentation labels Aug 5, 2026
devin-ai-integration[bot]
devin-ai-integration Bot reviewed Aug 5, 2026
View reviewed changes

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Open in Devin Review

Comment thread Cargo.toml
Comment thread docs/fern/pages/developer-guide/knowledge-base/modular-components/frontend/tokenizer.md
@github-actions

github-actions Bot commented Aug 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🌿 Fern Docs Preview: https://nvidia-preview-019fd3fb-1129-756b-bdb0-4c447c266173.docs.buildwithfern.com/dynamo/dev

@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 7bd45fda-bdb7-49ad-8937-ce892d489b70

📥 Commits

Reviewing files that changed from the base of the PR and between e322c8d and f695bd8.

⛔ Files ignored due to path filters (3)
  • Cargo.lock is excluded by !**/*.lock
  • lib/bindings/kvbm/Cargo.lock is excluded by !**/*.lock
  • lib/bindings/python/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • Cargo.toml
  • docs/fern/pages/developer-guide/knowledge-base/modular-components/frontend/tokenizer.md

Walkthrough

The workspace dependency changed from dynamo-tokenizers 1.7.0 to 1.8.0. The tokenizer documentation now describes native encoding and decoding, segmented encoding, performance guidance, and support for compatible Kimi tokenizer artifacts.

Changes

Tokenizer update

Layer / File(s) Summary
Tokenizer dependency upgrade
Cargo.toml		 The workspace now requires dynamo-tokenizers version 1.8.0.
Tokenizer backend documentation
docs/fern/pages/developer-guide/knowledge-base/modular-components/frontend/tokenizer.md		 The documentation adds segmented-encoding and performance guidance. It states that basetenkenizer performs native encoding and decoding and supports compatible Kimi tokenizer artifacts.

Estimated code review effort: 2 (Simple) | ~5 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description summarizes the dependency and documentation changes and lists validation, but it omits the required template sections, including Related Issues. Add the required Overview, Details, reviewer-start, and Related Issues sections, and confirm whether the pull request has a linked issue.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the dependency version update, which is the primary change.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@datadog-official

This comment has been minimized.

Sign in to view
@jthomson04
chore(deps): bump dynamo-tokenizers to 1.8.0
fa640ee 
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
@jthomson04
jthomson04 force-pushed the jthomson04/bump-dynamo-tokenizers-1.8.0 branch from f695bd8 to fa640ee Compare August 5, 2026 21:31
@copy-pr-bot

copy-pr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@jthomson04

jthomson04 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test fa640ee

@jthomson04
jthomson04 merged commit 80865c9 into main Aug 6, 2026
117 checks passed
@jthomson04
jthomson04 deleted the jthomson04/bump-dynamo-tokenizers-1.8.0 branch August 6, 2026 04:51
hhzhang16 added a commit that referenced this pull request Aug 6, 2026
@hhzhang16
Merge branch 'main' of https://github.com/ai-dynamo/dynamo into hannahz/
614e1c1 
dyn-3691-extract-shared-target-pid-cuda-customstorage-operation-layer

* 'main' of https://github.com/ai-dynamo/dynamo: (65 commits)
  fix(frontend): emit SGLang stream role once (#12741)
  docs(fern): promote v1.3.1 to current release (#12752)
  fix(docs): remove duplicate unscoped community-rail CSS rules (#12615)
  feat(operator): migrate CRD storage to v1beta1 (#11904)
  fix: synchronize self-benchmark capacity across DP ranks (#12021)
  chore(deps): bump dynamo-tokenizers to 1.8.0 (#12707)
  fix(frontend): preserve split UTF-8 characters (#12688)
  docs: align Kubernetes build selector with CLI (#12729)
  fix(frontend): preserve completion backend error status (#12706)
  fix(operator): replace snapshot pods after GMS restart (#11286)
  refactor(media): rename installer module, drop --packages per review
  fix(media): harden installer against three pre-redesign review findings
  fix(media): verify installs in a fresh interpreter; teach --pip-args= form
  test(serve): install test-time decoders at the validated bounds
  feat(media): explicit installer for additional media decoders
  docs(spica): correct kv_load_ratio support guidance (#12714)
  feat(operator): add experimental grove.forceScalingGroup for single-node components (#11772)
  fix(vllm): declare entry-stage engine_input_source in GLM-Image NIXL config (#12709)
  chore: bump trtllm to v1.3.0rc23 (#12532)
  perf: remove trtllm postprocessing workers from the args as post processing workers are not effective in dynamo (#12592)
  ...

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>
@dagil-nvidia dagil-nvidia mentioned this pull request Aug 14, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@devin-ai-integration devin-ai-integration[bot] devin-ai-integration[bot] left review comments
@PeaBrane PeaBrane PeaBrane approved these changes
@ishandhanani ishandhanani ishandhanani approved these changes
@nv-tusharma nv-tusharma nv-tusharma approved these changes
+1 more reviewer
@dynamo-review-agent dynamo-review-agent[bot] dynamo-review-agent[bot] approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

chore documentation Improvements or additions to documentation size/XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jthomson04 @PeaBrane @ishandhanani @nv-tusharma