[kano-zheng]

Motivation

parallel_prompt_encode (enabled by default via SGLANG_PARALLEL_PROMPT_ENCODE) only covers the Jinja chat-template
path in OpenAIServingChat. The three native encoders render the whole conversation to one string and then call
self.tokenizer_manager.tokenizer.encode(real_input) in a single call, on the HTTP event loop. With DeepSeek-V4.1-Flash
serving coding agents (Claude Code / Codex / opencode), prompts routinely reach several hundred thousand tokens, so that
one call is the largest single cost before the request reaches the scheduler, and it delays every other request handled
by that frontend process (first tokens and streamed chunks alike).

Measured in the 2026-10-20 nightly image (lmsysorg/sglang nightly-dev-20261020-ab580b2e) with the
deepseek-ai/DeepSeek-V4.1-Flash-1017 tokenizer, on a real agent-shaped prompt rendered by encoding_dsv41
(system + tools + 990 tool-call / tool-result turns):

prompt	tokenizer.encode	parallel_prompt_encode	ids identical
4.14 M chars / 947,250 tokens	1.92 s	0.21 s	yes

parallel_prompt_encode's plan accepts the DeepSeek V4 tokenizer (no lstrip / rstrip / single_word added tokens, no
template post-processor), so the cuts at added tokens apply unchanged; its intra-gap cuts are only enabled for the
GLM / Llama-3 Split regex, so a single huge message without added tokens still takes the single-call path, which is the
existing, correct fallback.

In production we run the same approach as an out-of-tree patch; its in-process shadow check (re-encoding one request
in ten with the original single call and comparing ids) recorded about 3,100 comparisons on two serving nodes over
roughly eight hours of agent traffic, with no mismatch.

Modifications

• serving_chat.py: the dsv4, dsv41 and dsv32 branches of _process_messages call
  parallel_prompt_encode(self.tokenizer_manager.tokenizer, real_input) instead of
  self.tokenizer_manager.tokenizer.encode(real_input). No encode_kwargs are passed, matching the previous call.
  Below SGLANG_PARALLEL_PROMPT_ENCODE_MIN_CHARS (32,768 chars) the helper calls tokenizer.encode exactly as before.
• test_serving_chat.py: test_native_encoders_tokenize_through_parallel_prompt_encode pins that all three native
  encoders route through the helper and use its ids.

This is complementary to the open event-loop work (#39716, #41031, #44502): those move prompt preparation off the loop,
this makes the tokenization itself ~9x cheaper wherever it runs. #42187 (segment cache) would further skip unchanged
segments; it also targets the Jinja path.

Accuracy Tests

The helper's contract (ids identical to the single encode) is already pinned by test_parallel_prompt_encode.py.
For this tokenizer specifically: 0 mismatches over 5,906 comparisons (~1.1 G chars) of encoding_dsv41-rendered agent
conversations and adversarial text (CJK, emoji, partial added-token strings, whitespace next to added tokens), with cut
points forced every ~40 chars; plus the production shadow check above (~3,100 comparisons, 0 mismatches).

Benchmarking and Profiling

See the table above (single prompt, CPU only, same machine and process).

Checklist

• test_serving_chat.py: 160 passed (applied on nightly-dev-20261020-ab580b2e; the diff is against main eb4b167
  and applies with line offsets only).
• The new test fails without the change (all three specs) and passes with it.

🤖 Generated with Claude Code

---

CI States

Latest PR Test (Base): ❌ Run #39450261392
Latest PR Test (Extra): ❌ Run #39450261171
Latest PR Test (AMD ROCm 10): ❌ Run #39450261263

[lunarwhite]

Hi @kano-zheng, may I know if there is a specific reason why this PR got closed? Thanks