[Parser][Bugfix] Ensure tool call or other special tokens don't leak in non-streaming tool parsing

[bbrowning]

Purpose

In the new parser engine, non-streaming tool parsing had several paths where tokens that should be consumed or suppressed leaked into response content.

Auto-drop special tokens

Special tokens (turn markers, channel delimiters, etc.) were only being filtered via token IDs, so they leaked into response content on the non-streaming text path of engine-based parsers (skip_special_tokens=False). The per-model workaround — hand-maintained drop_tokens sets and adjust_request overrides toggling skip_special_tokens — was fragile and required config for every new model.

Replace this with automatic discovery: build __DROP__ terminals from tokenizer.all_special_tokens (minus already-configured terminals) and route them through the existing scanner/lexer/state-machine pipeline. This works for both paths — token IDs (streaming) and text (non-streaming) — with zero per-model configuration.

This effectively emulates skip_special_tokens=True behavior, but after we've handled all the special tokens needed by the parsing engine. We cannot directly set skip_special_tokens=True in adjust_request because then the reasoning and tool parsers would never see those special tokens they need to do their parsing work.

Some models (currently only Gemma 4 for models using the parser engine) need some special tokens to be preserved outside of the lower layers of the parser and exposed to the upper levels, to be used during parsing of tool call args. There's a new preserve_tokens in the ParserEngineConfig that lets a parser configure tokens to not automatically drop.

Suppress tool calls for tool_choice=none

Once we fixed the above, it exposed a bug in our tool_choice="none" and thinking disabled handling in the new parser engine. Previously tool_choice="none" set skip_tool_parsing=True, which emitted tool terminal text as content — the opposite of what callers expect. Separate this into a distinct _suppress_tool_calls mechanism that lets the state machine parse tool calls normally (consuming terminals via transitions) but filters out all tool events, so no tool markup leaks into content and no tool call objects are returned. This was partially masked for some models (like Gemma4) because they would override adjust_request and set skip_special_tokens=True in this case before, where now they rely entirely on the parser engine to strip these special tokens.

There was inconsistency here in the streaming vs non-streaming paths, so Gemma4 only had this exposed on main in the streaming path but after removing its adjust_request override it exposed this in both streaming and non-streaming.

Flush deferred whitespace in non-streaming parse

Fixing that exposed one last bug, where non-streaming scenarios that end up with whitespace-only content was deferred but never flushed within the engine. This is a small consistency fix at the engine level between streaming and non-streaming that I don't believe was actually visible from the external serving layer.

Test Plan

Unit Tests

Multiple new unit tests were added, using TDD to ensure the unit tests triggered the errors properly before the fix and that they passed after the fix.

pytest -v \
  tests/parser/engine/ \
  tests/reasoning/test_gemma4_reasoning_parser.py \
  tests/tool_parsers/test_gemma4_tool_parser.py \
  tests/tool_use/test_gemma4_responses_adjust_request.py

Manual test against live models

This uses the simple test script from https://gist.github.com/bbrowning/490740a32acebc262cc06b4d8c426955 saved as test_drop_token_leaks.py.

Run a model, such as Gemma 4 31B:

vllm serve google/gemma-4-31B-it \
  --tensor-parallel-size 2 \
  --enable-auto-tool-choice \
  --tool-call-parser gemma4 \
  --reasoning-parser gemma4 \
  --default-chat-template-kwargs '{"enable_thinking": true}' \
  --chat-template examples/tool_chat_template_gemma4.jinja \
  --port 8088

And run the manual test suite that checks various combinations of tool_choice and thinking enabled/disabled in streaming and non-streaming to detect any leaked special tokens:

python test_drop_token_leaks.py \
  --base-url http://localhost:8088/v1

BFCL testing with Gemma4

Gemma4 was the one that had the most complete tests and expectations around adjust_request, skip_special_tokens conditionally set, and dropping of tokens. And, it has a special case where some of its special tokens (string delimiter) has to survive all the way out to our parser engine arg converter layer to be properly handled. So, I centered real-world testing on this model.

OPENAI_BASE_URL="http://localhost:8088/v1" \
OPENAI_API_KEY="fake" \
bfcl generate \
  --model google/gemma-4-31B-it \
  --num-threads 4 \
  --allow-overwrite \
  --test-category multi_turn_base

OPENAI_API_KEY="fake" bfcl evaluate

Test Result

Unit Tests

3167 passed, 19 warnings

Manual test against live models

Before, exposing tool_choice="none" and thinking explicitly disabled bug:

Discovered model: google/gemma-4-31B-it

   1/12  no_tools_thinking_disabled (non-streaming) .................. PASS
   2/12  no_tools_thinking_disabled (streaming) ...................... PASS
   3/12  no_tools_thinking_enabled (non-streaming) ................... PASS
   4/12  no_tools_thinking_enabled (streaming) ....................... PASS
   5/12  tool_call_thinking_enabled (non-streaming) .................. PASS
   6/12  tool_call_thinking_enabled (streaming) ...................... PASS
   7/12  tool_call_thinking_disabled (non-streaming) ................. PASS
   8/12  tool_call_thinking_disabled (streaming) ..................... PASS
   9/12  tool_choice_none_thinking_disabled (non-streaming) .......... PASS
  10/12  tool_choice_none_thinking_disabled (streaming) .............. FAIL
        Leaked: ['<tool_call|>', '<|tool_call>']
        Content: 'call:get_weather{location:New York City}<|tool_call><tool_call|>'
  11/12  multi_turn_tool_call (non-streaming) ........................ PASS
  12/12  multi_turn_tool_call (streaming) ............................ PASS

Results: 11/12 passed, 1 failed

After, with this PR applied to the running server:

Discovered model: google/gemma-4-31B-it

   1/12  no_tools_thinking_disabled (non-streaming) .................. PASS
   2/12  no_tools_thinking_disabled (streaming) ...................... PASS
   3/12  no_tools_thinking_enabled (non-streaming) ................... PASS
   4/12  no_tools_thinking_enabled (streaming) ....................... PASS
   5/12  tool_call_thinking_enabled (non-streaming) .................. PASS
   6/12  tool_call_thinking_enabled (streaming) ...................... PASS
   7/12  tool_call_thinking_disabled (non-streaming) ................. PASS
   8/12  tool_call_thinking_disabled (streaming) ..................... PASS
   9/12  tool_choice_none_thinking_disabled (non-streaming) .......... PASS
  10/12  tool_choice_none_thinking_disabled (streaming) .............. PASS
  11/12  multi_turn_tool_call (non-streaming) ........................ PASS
  12/12  multi_turn_tool_call (streaming) ............................ PASS

Results: 12/12 passed

BFCL testing with Gemma4

Before

   Model: google_gemma-4-31B-it
🔍 Running test: multi_turn_base
✅ Test completed: multi_turn_base. 🎯 Accuracy: 81.50%

After

I see this fluctuate up and down by as much as 1-2% every run, so within 0.5% is considered identical for this model and this eval.

   Model: google_gemma-4-31B-it
🔍 Running test: multi_turn_base
✅ Test completed: multi_turn_base. 🎯 Accuracy: 81.00%

[bbrowning]

Opened as a draft while I do some more testing of this in real-world scenarios.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[bbrowning]

This took some time to clean up and handle all special cases, but this hardens our special token handling in the new parser engine by ensuring none ever leak by default, emulating what skip_special_tokens=True used to do except doing it after we've parsed the necessary reasoning / tool calls out.

That also exposed a case where we were sometimes leaking tool calls out for Gemma4 today only when tool_choice="none", enable_thinking=False, and tools provided in a request. That was closely tied to the special token dropping code paths, and cleaning that up exposed this a bit wider, so this fixes that as well by ensuring we don't returned parsed tool calls when tool_choice="none" but also don't return the unparsed tool call content.

[sfeng33]

@claude review

[claude]

Architectural changes to the parser engine (auto-drop via tokenizer discovery, new tool-call suppression path, removal of per-model drop_token lists) — this needs a human reviewer with parser-engine context to sign off.

Extended reasoning...

Overview

This PR refactors how special tokens are filtered in the new parser engine across 14 files (8 in vllm/parser/ core code + 6 test files). The main changes:

• Replaces hand-maintained drop_tokens sets with auto-discovery via tokenizer.all_special_tokens, building __DROP__ terminals routed through the existing lexer/state-machine pipeline.
• Introduces preserve_tokens in ParserEngineConfig for tokens that need to bypass auto-drop (e.g. Gemma4's <|"|> string delimiter).
• Separates tool_choice="none" handling from skip_tool_parsing into a new _suppress_tool_calls mechanism that consumes tool terminals via state-machine transitions but filters out tool events.
• Removes Gemma4's adjust_request override and the hardcoded _GEMMA4_MODEL_DROP_TOKENS set.
• Flushes deferred whitespace at end-of-stream in _single_pass_parse.
• Simplifies TokenIDScanner by removing the now-redundant drop_token_ids parameter.

Security risks

No direct security risks. The changes do not touch auth, crypto, permissions, or input validation at trust boundaries. They affect parsing of model-generated output, where the worst-case failure mode is malformed tool calls or leaked delimiter text into response content — a correctness issue, not a security one.

Level of scrutiny

High. This touches the core parser engine that handles tool calling and reasoning extraction for all engine-based parsers (Gemma4, Qwen3, Hermes, etc.). Tool calling is a primary feature; a regression here breaks downstream agentic workflows. The changes are not mechanical — they shift architectural responsibility (special-token filtering moves from per-model config into the engine's StreamingParserEngine), which deserves a maintainer who understands the lexer/state-machine layering and the implications across all parsers in registered_adapters.

Other factors

• Test coverage is substantial: new replay tests across all registered parsers verify drop tokens are stripped in both streaming and non-streaming, regression tests for tool_choice="none", and the bug-hunting system found no issues.
• The author included live-model testing against Gemma4 (12/12 manual tests passing post-fix, BFCL multi_turn_base within noise of pre-PR baseline).
• One reviewer (sfeng33) explicitly requested my review, but no human approval has been recorded yet.
• The Gemma4 adjust_request removal is load-bearing: the new test test_gemma4_keeps_skip_special_tokens_false_when_nothing_to_preserve inverts the prior expectation (was True, now False). That inversion relies entirely on the auto-drop mechanism working for every special token the tokenizer reports — a reviewer should confirm the assumption holds across the model set vLLM ships.

[bbrowning]

Merging latest main, the new tests added here catch some bugs in the Kimi parser implementation that was merged since I opened this. Debugging locally and will add another commit to fix that in a bit.

[bbrowning]

The Kimi parser was fine actually but the test expectations in the replay suite around the new drop tokens and suppression of tool calls when needed had to be tweaked a bit to handle the Kimi parser's slightly different tool call sections.