Fix gpt‑oss Harmony token leaks in tool names and streaming content #32587

[baonudesifeizhai]

Purpose

#32587

Defined sanitize_harmony_tool_name and strip_harmony_control_tokens in harmony_utils.py.

Used the sanitizer when reconstructing Harmony messages from chat/response history and when converting Harmony outputs back into API responses in harmony_utils.py.

Applied the sanitizer in tool extraction for chat completions (non‑stream) and stripped control tokens from final/commentary content in openai_tool_parser.py.

Applied the sanitizer in streaming deltas for chat completions and stripped control tokens from streamed content/reasoning deltas in stream_harmony.py.

Applied the sanitizer for function call events and stripped control tokens from response streaming deltas and done events in serving.py.

Test Plan

python -m vllm.entrypoints.openai.api_server \
  --model openai/gpt-oss-20b \
  --served-model-name gpt-oss-20b \
  --host 0.0.0.0 \
  --port 8000 \
  --gpu-memory-utilization 0.93 \
  --tensor-parallel-size 1 \
  --max-num-batched-tokens 8192 \
  --disable-log-requests \
  --tool-call-parser openai \
  --reasoning-parser openai_gptoss \
  --enable-auto-tool-choice

Test Result

main branch : https://paste.ubuntu.com/p/YthB2VTPQ8/

this branch

pytest tests/entrypoints/openai/parser/test_harmony_utils.py -v -- passed

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces sanitization functions to prevent leaking Harmony control tokens in tool names and content. The changes are applied across several files to ensure that user-facing output is clean. The implementation of the sanitization logic is sound. I've pointed out a couple of places where the tool name parsing logic is inconsistent with the rest of the codebase, which could lead to incorrect behavior if tool names contain dots. Fixing this would improve the robustness of the code.

[gemini-code-assist]

The logic to extract the function name using recipient.split(".")[-1] can be incorrect if the function name itself contains dots. For consistency with other parts of the codebase (e.g., serving.py), it's better to slice the string after the functions. prefix. This will correctly handle function names that might contain dots.

Suggested change

	function_name = sanitize_harmony_tool_name(recipient.split(".")[-1])
	function_name = sanitize_harmony_tool_name(recipient[len("functions.") :])

[gemini-code-assist]

Similar to another comment, using current_recipient.split(".")[-1] to extract the function name can be problematic if the function name contains dots. To ensure correctness and consistency, it's safer to slice the string after the functions. prefix.

Suggested change

	function_name = sanitize_harmony_tool_name(current_recipient.split(".")[-1])
	function_name = sanitize_harmony_tool_name(current_recipient[len("functions.") :])

[chaunceyjiang]

Could you provide a reproduction example?
I’d like to reproduce it locally.

[baonudesifeizhai]

 
cat > repro_32587_loop.py <<'PY'
import json
import os
import time
import requests

BASE = "http://127.0.0.1:8000/v1/chat/completions"
MODEL = "gpt-oss-20b"

tools = [
    {
        "type": "function",
        "function": {
            "name": "ls",
            "description": "List files in a directory",
            "parameters": {
                "type": "object",
                "properties": {"path": {"type": "string"}},
                "required": ["path"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "write_file",
            "description": "Write content to a file",
            "parameters": {
                "type": "object",
                "properties": {"path": {"type": "string"}, "content": {"type": "string"}},
                "required": ["path", "content"],
            },
        },
    },
]

def bad(s: str) -> bool:
    return s is not None and "<|channel|>" in s

def run_tool(name: str, args: dict):
    if name == "ls":
        p = args.get("path", ".")
        # 只允许相对路径，避免乱写
        if p.startswith("/"):
            p = "."
        return {"path": p, "entries": sorted(os.listdir(p))[:200]}
    if name == "write_file":
        path = args.get("path", "hello.txt")
        content = args.get("content", "")
        # 防止写到奇怪路径
        if "/" in path or path.startswith("."):
            path = "hello.txt"
        with open(path, "w", encoding="utf-8") as f:
            f.write(content)
        return {"ok": True, "path": path, "bytes": len(content.encode("utf-8"))}
    return {"error": f"unknown tool: {name}"}

for i in range(1, 2001):
    messages = [
        {"role": "system", "content": "You are a coding agent. Use tools when needed."},
        {"role": "user", "content": "You MUST call tool ls with path='.' first, then call write_file to create hello.txt containing exactly: hello\\n. Do not answer without tools."},
    ]

    for step in range(1, 6):  # 最多 5 次 tool/assistant 循环
        payload = {
            "model": MODEL,
            "messages": messages,
            "tools": tools,
            "tool_choice": "auto",
            "temperature": 0.7,
            "max_tokens": 256,
            # 你启动用了 openai_gptoss reasoning/parser，这里保持请求干净
        }

        r = requests.post(BASE, json=payload, timeout=120)
        if r.status_code != 200:
            print("HTTP", r.status_code, r.text[:2000])
            raise SystemExit(1)

        data = r.json()
        msg = (data.get("choices") or [{}])[0].get("message") or {}
        content = msg.get("content") or ""
        tool_calls = msg.get("tool_calls") or []

        names = [((tc.get("function") or {}).get("name")) for tc in tool_calls]
        print(f"iter {i} step {step}: tool_calls={names} content_head={content[:80]!r}")

        # 检测污染：tool name / content
        for n in names:
            if bad(n):
                print("HIT: corrupted tool name")
                print(json.dumps(data, indent=2, ensure_ascii=False)[:5000])
                raise SystemExit(0)
        if bad(content):
            print("HIT: leaked special tokens in content")
            print(json.dumps(data, indent=2, ensure_ascii=False)[:5000])
            raise SystemExit(0)

        # 没工具调用：说明模型给了最终回复
        if not tool_calls:
            # 也扫一下整个 message 的序列化（有些会藏在别的字段）
            if bad(json.dumps(msg, ensure_ascii=False)):
                print("HIT: leaked special tokens somewhere in message")
                print(json.dumps(data, indent=2, ensure_ascii=False)[:5000])
                raise SystemExit(0)
            break

        # 执行 tool，并把结果回传给模型
        for tc in tool_calls:
            fn = (tc.get("function") or {}).get("name")
            arg_str = (tc.get("function") or {}).get("arguments") or "{}"
            try:
                args = json.loads(arg_str)
            except Exception:
                args = {"_raw": arg_str}

            result = run_tool(fn, args)
            # 把 assistant 的 tool_call 消息也塞回 history
            messages.append({"role": "assistant", "tool_calls": [tc], "content": ""})
            messages.append(
                {
                    "role": "tool",
                    "tool_call_id": tc.get("id"),
                    "name": fn,
                    "content": json.dumps(result, ensure_ascii=False),
                }
            )

    time.sleep(0.05)

print("No hit within loop. If issue is probabilistic, increase iterations or change prompt.")
PY
python repro_32587_loop.py
main branch:
iter 1 step 1: tool_calls=['ls'] content_head=''
iter 1 step 2: tool_calls=['write_file'] content_head=''
iter 1 step 3: tool_calls=[] content_head='Done.'
iter 2 step 1: tool_calls=['ls'] content_head=''
iter 2 step 2: tool_calls=['write_file'] content_head=''
iter 2 step 3: tool_calls=[] content_head='Here is the `hello.txt` file with the requested content.'
iter 3 step 1: tool_calls=['ls'] content_head=''
iter 3 step 2: tool_calls=['write_file'] content_head=''
iter 3 step 3: tool_calls=[] content_head='The file `hello.txt` has been created with the specified content.'
iter 4 step 1: tool_calls=['ls'] content_head=''
iter 4 step 2: tool_calls=['write_file'] content_head=''
iter 4 step 3: tool_calls=[] content_head='Done.'
iter 5 step 1: tool_calls=['ls'] content_head=''
iter 5 step 2: tool_calls=['write_file'] content_head=''
iter 5 step 3: tool_calls=[] content_head='Done.'
iter 6 step 1: tool_calls=['ls'] content_head=''
iter 6 step 2: tool_calls=['write_file'] content_head=''
iter 6 step 3: tool_calls=[] content_head='Done. The file `hello.txt` has been created with the content `hello\\n`.'
iter 7 step 1: tool_calls=['ls'] content_head=''
iter 7 step 2: tool_calls=['write_file'] content_head=''
iter 7 step 3: tool_calls=[] content_head='Created `hello.txt` with the requested content.'
iter 8 step 1: tool_calls=['ls'] content_head=''
iter 8 step 2: tool_calls=['write_file<|channel|>commentary'] content_head=''
HIT: corrupted tool name
{
  "id": "chatcmpl-834b7d15e2e3d827",
  "object": "chat.completion",
  "created": 1768875413,
  "model": "gpt-oss-20b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "tool_calls": [
          {
            "id": "chatcmpl-tool-b53363dc1ff9f57e",
            "type": "function",
            "function": {
              "name": "write_file<|channel|>commentary",
              "arguments": "{\"path\": \"hello.txt\", \"content\": \"hello\\n\"}"
            }
          }
        ],
        "reasoning": null,
        "reasoning_content": null
      },
      "logprobs": null,
      "finish_reason": "tool_calls",
      "stop_reason": 200012,
      "token_ids": null
    }
  ],
  "service_tier": null,
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 682,
    "total_tokens": 707,
    "completion_tokens": 25,
    "prompt_tokens_details": null
  },
  "prompt_logprobs": null,
  "prompt_token_ids": null,
  "kv_transfer_params": null
}

> Could you provide a reproduction example? I’d like to reproduce it locally.

[chaunceyjiang]

Okay, I’ll try to reproduce it locally.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @baonudesifeizhai.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork