[Render] Add reasoning/tool parsing to /derender + fix byte-fallback FFFD

[aoshen02]

Summary

Merges the reasoning/tool-call parsing from PR #45045 and fixes the byte-fallback U+FFFD regression in derender logprobs.

Reasoning & tool call parsing (from #45045)

• New build_chat_message() in serve/utils/chat_message_builder.py — single source of truth shared by coupled chat path and /derender
• derender_chat_response() accepts chat_request for parser-aware parsing (reasoning + tool calls)
• chat_completion/serving.py delegates ~268 lines of inline logic to build_chat_message()
• OpenAIServingRender.__init__ initializes parser_cls, tool_call_id_type

Byte-fallback FFFD fix

• resolve_token_id_placeholder uses tokenizer.decode() instead of convert_ids_to_tokens + convert_tokens_to_string
• _resolve_logprobs tracks context token IDs and applies correct_decoded_token when U+FFFD is detected, mirroring v1/engine/logprobs.py

Not duplicating an existing PR

PR #45045 is open but stale (based on older codebase, conflicts with current main). This PR adapts its changes to the current codebase and adds the FFFD fix + solid CI.

Test plan

• test_derender_byte_fallback.py — 17 unit tests: baseline DOES produce U+FFFD, fix does NOT (emoji, CJK, mixed, consecutive)
• test_render_derender_e2e.py — render→derender roundtrip tests: plain roundtrip, reasoning extraction, tool call extraction, token-identity, JSON boolean drift, non-ASCII roundtrip
• test_derender.py — updated from [Render] Add reasoning and tool call parsing to /derender #45045 with parity tests
• Pre-commit hooks pass (ruff, mypy, signoff)

python -m pytest tests/entrypoints/serve/render/test_derender_byte_fallback.py -v
python -m pytest tests/entrypoints/serve/render/test_render_derender_e2e.py -v
python -m pytest tests/entrypoints/serve/render/test_derender.py -v

AI assistance was used (Claude). Co-authored with @hickeyma (PR #45045 original author).

🤖 Generated with Claude Code

[cjackal]

I've checked that 70a8a12 fixes the regression, thanks for a quick fix!

[aoshen02]

I've checked that 70a8a12 fixes the regression, thanks for a quick fix!

Will add corresponding CI to prevent this from happening next time.

[aoshen02]

Did you use the /derender endpoint?

[cjackal]

Did you use the /derender endpoint?

Oh I didn't, my issue was about chat completions endpoint to be clear.

[aoshen02]

Wait, how does pr 43606 affect chat completions.

[aoshen02]

I think 43606 is only about add two independent endpoint?

[cjackal]

I haven't read the PR through and report, but a git bisect on my end points to that PR. Is there other places where the resolve_token_id_placeholder method is called in normal chat completion path?

[aoshen02]

weird, you sure? I mean, this is only called by /derender endpoint related stuff. It matters a lot to me as I want to make sure that pr introduced no regression. Could you provide more context?

[cjackal]

It seems I should have posted a proper issue body. The context is:

We are testing glm-5.1 online server on near-weekly basis, the previous commit that have no issue was 7852e50 (dated 2026.06.11) and the next commit that we got trouble was b997071. So I have bisected the culprit commit with the following test script and arrived at 43f0e02.

# full server launch script
vllm serve zai-org/GLM-5.1-FP8 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --max-model-len 202752 \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --hf-overrides.index_topk_freq 4


# client-side test script - try to make the model spit some hangul(non-ascii)

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model": "zai-org/GLM-5.1-FP8", "messages": [{"role": "user", "content": "삼성전자 주주총회는 어디에서 열려?"}], "stream": true}'


# then the model response contains a U+FFFD in the middle of the response content for about 1-in-3 chance
# an example case is like: ...영�구에 있는...

[aoshen02]

May I reach out to you via slack, let's keep in touch and I will try to reproduce it tomorrow. aoshen524@gmail.com/
Ao Shen I think e0871ad — [Refactor] Chat Completions Streaming Harmony Refactor (#45104) and f712fd0 — [Refactor] Chat Completions Harmony Refactor, non-streaming (#45171) are more suspicious.

[cjackal]

May I reach out to you via slack, let's keep in touch and I will try to reproduce it tomorrow. aoshen524@gmail.com/ Ao Shen I think e0871ad — [Refactor] Chat Completions Streaming Harmony Refactor (#45104) and f712fd0 — [Refactor] Chat Completions Harmony Refactor, non-streaming (#45171) are more suspicious.

Thanks for your time and effort, I am on slack by the profile Juhyun Kim. Come to think of it, my bisect logic relies on probabilistic sampling (I ran a pre-existing set of ~12 requests and searched for the existence of U+FFFD per each commit) so it may point to a wrong commit, I'll try to find a more stable reproducer and do the bisect again as well.

[sfeng33]

Giving a yes because the parser wiring looks generally correct, assuming you have done the testing on the expected /derender behvaiour

[mergify]

Hi @aoshen02, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

[cjackal]

Hey @cjackal, thanks for the repro script! I don't think this PR is related to the bug you mentioned though - it's on different code path, yours is on U+FFFD in streaming chat completion content text, while this PR is for U+FFFD in logprob token strings on the /derender endpoint.

Indeed, I and @aoshen02 had discussed on slack and reached a confirmation that the derender PR is not the root cause. I couldn't narrowed down the root cause (mostly due to reproducibility) but it seems @bbrowning's analysis sounds valid - in the sense that the commit ranges that I have bisected are in the middle of unified parser refactoring.

[aoshen02]

Hey @cjackal, thanks for the repro script! I don't think this PR is related to the bug you mentioned though - it's on different code path, yours is on U+FFFD in streaming chat completion content text, while this PR is for U+FFFD in logprob token strings on the /derender endpoint.

Indeed, I and @aoshen02 had discussed on slack and reached a confirmation that the derender PR is not the root cause. I couldn't narrowed down the root cause (mostly due to reproducibility) but it seems @bbrowning's analysis sounds valid - in the sense that the commit ranges that I have bisected are in the middle of unified parser refactoring.

Please do try this, [Bugfix][Parser] Fix U+FFFD leak at reasoning-to-content transition in engine parsers by bbrowning · Pull Request #46159 · vllm-project/vllm

[sfeng33]

Are you sure this is equivalent?

[aoshen02]

Yes, please check create_engine_config() (arg_utils.py:2200-2201), but init_render_app_state never calls create_engine_config() (no engine needed), so it stays empty. args.reasoning_parser always has a value, regardless of whether the engine is initialized.

[sfeng33]

Is this check necessary

[aoshen02]

Harmony parser like GptOssReasoningParser does not need chat_template_kwargs to get any values while most parsers like qwen3 parser need chat_template_kwargs to

: self.thinking_enabled = chat_kwargs.get("enable_thinking").

We can also remove the check, what do you think.

[hickeyma]

Thanks @aoshen02 for taking the parsing on board and driving this PR. Sorry but I was OOTO and missed the fun! :)