fix(frontend): preserve split UTF-8 characters by xianlubird · Pull Request #12688 · ai-dynamo/dynamo

xianlubird · 2026-08-05T06:10:59Z

Summary

The SGLang frontend previously restarted incremental decoding from a fixed token lookback. With byte-fallback tokenizers, that lookback could begin inside a multi-token UTF-8 character, causing replacement characters (U+FFFD) to be emitted as output.

This change:

tracks committed decode context separately from pending generated token IDs
retains the complete pending region when decoding produces no text or ends in U+FFFD
commits pending IDs only after reaching a clean decode boundary
initializes decode context from the prompt tail to preserve tokenizer cleanup behavior
flushes pending output when generation finishes, including a legitimate terminal U+FFFD
passes prompt token IDs through both inline and preprocessing-pool paths
continues to use the tokenizer's standard decode() API without requiring DecodeStream

Validation

pytest components/src/dynamo/frontend/tests/test_sglang_processor_unit.py — 197 passed
verified randomized prompt/output chunking against full decoding for Qwen3, GPT-2, and TinyLlama — 300 cases per tokenizer
added a real TinyLlama byte-fallback regression covering the eight-plus-one token split for 🙂🙂
added coverage for short split-byte sequences and legitimate terminal U+FFFD
git diff --check

Summary by CodeRabbit

Bug Fixes
- Improved streaming text output for multibyte UTF-8 characters split across token chunks.
- Prevented incomplete replacement characters from appearing prematurely in intermediate responses.
- Preserved valid replacement characters when generation completes, including responses without final token IDs.
Tests
- Added coverage for split UTF-8 sequences and final replacement-character handling.

copy-pr-bot · 2026-08-05T06:11:03Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

dynamo-ops · 2026-08-05T06:20:27Z

/ok to test 8ca8b96

coderabbitai · 2026-08-05T06:23:34Z

Walkthrough

The SGLang processor now buffers incomplete UTF-8 output during incremental decoding, suppresses intermediate replacement characters, and flushes valid trailing U+FFFD output when generation finishes. Unit tests cover split multibyte characters and final-only flushing.

Changes

Incremental decoding

Layer / File(s)	Summary
Decode buffering and final flush `components/src/dynamo/frontend/sglang_prepost.py`	`_incremental_decode` distinguishes intermediate decoding from final flushing. `process_output` triggers final decoding when a finish reason exists, including responses without token IDs.
UTF-8 and replacement-character tests `components/src/dynamo/frontend/tests/test_sglang_processor_unit.py`	Tests add a byte-oriented tokenizer and verify split UTF-8 characters and buffered legitimate U+FFFD output.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the implementation and validation, but it omits the required template sections and the required Related Issues declaration.	Add the Overview, Details, reviewer-start guidance, and Related Issues sections; select either a linked issue or confirm that no issue is related.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the primary change: preserving UTF-8 characters split across streaming chunks.

_{Comment @coderabbitai help to get the list of available commands.}

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Signed-off-by: xianlubird <xianlubird@gmail.com>

dynamo-ops · 2026-08-06T02:03:32Z

/ok to test 03ed922

dyn-3691-extract-shared-target-pid-cuda-customstorage-operation-layer * 'main' of https://github.com/ai-dynamo/dynamo: (65 commits) fix(frontend): emit SGLang stream role once (#12741) docs(fern): promote v1.3.1 to current release (#12752) fix(docs): remove duplicate unscoped community-rail CSS rules (#12615) feat(operator): migrate CRD storage to v1beta1 (#11904) fix: synchronize self-benchmark capacity across DP ranks (#12021) chore(deps): bump dynamo-tokenizers to 1.8.0 (#12707) fix(frontend): preserve split UTF-8 characters (#12688) docs: align Kubernetes build selector with CLI (#12729) fix(frontend): preserve completion backend error status (#12706) fix(operator): replace snapshot pods after GMS restart (#11286) refactor(media): rename installer module, drop --packages per review fix(media): harden installer against three pre-redesign review findings fix(media): verify installs in a fresh interpreter; teach --pip-args= form test(serve): install test-time decoders at the validated bounds feat(media): explicit installer for additional media decoders docs(spica): correct kv_load_ratio support guidance (#12714) feat(operator): add experimental grove.forceScalingGroup for single-node components (#11772) fix(vllm): declare entry-stage engine_input_source in GLM-Image NIXL config (#12709) chore: bump trtllm to v1.3.0rc23 (#12532) perf: remove trtllm postprocessing workers from the args as post processing workers are not effective in dynamo (#12592) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

pull-request-size Bot added the size/M label Aug 5, 2026

xianlubird temporarily deployed to external_collaborator August 5, 2026 06:11 — with GitHub Actions Inactive

github-actions Bot added fix external-contribution Pull request is from an external contributor trusted-contributor Org-External user who is trusted to run CI without Org-member approval frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels Aug 5, 2026

xianlubird force-pushed the bug/sglang-utf8-streaming branch from 0390618 to 8ca8b96 Compare August 5, 2026 06:20

xianlubird temporarily deployed to external_collaborator August 5, 2026 06:20 — with GitHub Actions Inactive

xianlubird marked this pull request as ready for review August 5, 2026 06:20

xianlubird requested a review from a team as a code owner August 5, 2026 06:20

devin-ai-integration Bot reviewed Aug 5, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

rmccorm4 reviewed Aug 6, 2026

View reviewed changes

Comment thread components/src/dynamo/frontend/sglang_prepost.py

fix(frontend): preserve split UTF-8 characters

03ed922

Signed-off-by: xianlubird <xianlubird@gmail.com>

xianlubird force-pushed the bug/sglang-utf8-streaming branch from 8ca8b96 to 03ed922 Compare August 6, 2026 02:03

pull-request-size Bot removed the size/M label Aug 6, 2026

xianlubird temporarily deployed to external_collaborator August 6, 2026 02:03 — with GitHub Actions Inactive

pull-request-size Bot added the size/L label Aug 6, 2026

rmccorm4 enabled auto-merge (squash) August 6, 2026 02:38

rmccorm4 approved these changes Aug 6, 2026

View reviewed changes

rmccorm4 merged commit 3de5663 into ai-dynamo:main Aug 6, 2026
105 checks passed

rmccorm4 mentioned this pull request Aug 7, 2026

[Bug] streaming detokenizer emits U+FFFD and drops completed characters on multi-byte UTF-8 (CJK) at chunk boundaries #12630

Open

jain-ria mentioned this pull request Aug 11, 2026

fix(frontend): backport SGLang chat logprobs to 1.4.0 #12988

Merged

Navigation Menu

Uh oh!

fix(frontend): preserve split UTF-8 characters - #12688

fix(frontend): preserve split UTF-8 characters#12688
rmccorm4 merged 1 commit into
ai-dynamo:mainfrom
xianlubird:bug/sglang-utf8-streaming

xianlubird commented Aug 5, 2026 •

edited by rmccorm4

Loading

Uh oh!

copy-pr-bot Bot commented Aug 5, 2026

Uh oh!

dynamo-ops commented Aug 5, 2026

Uh oh!

coderabbitai Bot commented Aug 5, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

This comment has been minimized.

Uh oh!

dynamo-ops commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

xianlubird commented Aug 5, 2026 • edited by rmccorm4 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented Aug 5, 2026

Uh oh!

dynamo-ops commented Aug 5, 2026

Uh oh!

coderabbitai Bot commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

❌ Failed checks (1 warning)

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

This comment has been minimized.

Uh oh!

dynamo-ops commented Aug 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xianlubird commented Aug 5, 2026 •

edited by rmccorm4

Loading

coderabbitai Bot commented Aug 5, 2026 •

edited

Loading