fix(parser): preserve GLM delimiters and trailing stop anchors

[logprobz]

Summary

• Keep GLM argument framing outside TOOL_ARGS, then delay each </arg_value> decision until following bytes show whether it is structural or argument data.
• Preserve literal </tool_call> and recoverable literal </arg_value> values without disabling progressive argument deltas.
• Make partial and final GLM conversion agree on truncated and malformed values so completed streamed arguments remain valid JSON.
• Bind reconstructed scanner text as a suffix, retain ordered token counts through empty TextChunk carriers, and preserve anchors when serving strips trailing DROP text but retains its token ID.
• Prevent auto-DROP aliases from replacing IDs protected by parser terminals, token-ID terminals, or preserve_tokens.
• Preserve the pre-existing positional ParserEngineConfig field layout and remove Inkling post-tool content handling made redundant by the generic single-pass flush.

Serving behavior

The scanner repair covers both matching text plus token IDs and stop-stripped text plus a retained stop ID. Both forms produce the same exact GLM tool call.

The serving regression now combines literal tool delimiter IDs 154844 and 154843 with an empty final delta retaining observation stop ID 154829. It requires one stable call, progressive exact JSON, exact reasoning, post-tool content, finish_reason="tool_calls", stop_reason=154829, and returned final token IDs [154829].

Non-streaming parser-engine calls drain deferred output after finish() and build visible content from the completed extraction result. This is a parser-engine-wide correction, not GLM-only. DROP aliases participate in strict token-ID filtering, so literal text lookalikes remain content while actual special-token IDs retain DROP behavior.

GLM boundary contract

Inside an argument value, tool delimiters are data. An </arg_value> candidate becomes structural when the next non-whitespace bytes are <arg_key> or the outer </tool_call> boundary. Other candidates remain part of the value.

Malformed wire data remains ambiguous. If an argument never closes, later tool framing and text remain argument data, but the parser finishes with valid JSON. A completed first argument plus a truly unterminated second argument is now covered in whole, character, protocol, and real token-ID modes.

The exact sequence </arg_value><arg_key> is structural even if later key framing is malformed. Recovering that malformed false-next-key case would require another delayed state and a different final fallback, while intent would remain ambiguous. This PR documents that boundary rather than adding speculative parser behavior.

This PR does not change chat templates, role EOS IDs, reasoning transitions, or progressive argument deltas. It does not install a prompt or history codec.

Compatibility cleanup

non_whitespace_transitions remains opt-in and is placed after all pre-existing ParserEngineConfig fields, preserving unknown downstream positional callers. All in-tree callers use keywords. Inkling now relies on the generic post-tool content flush, and its complete parser suite preserves the prior behavior.

Regression coverage

• tests/parser/engine: 3811 passed
• tests/tool_parsers/test_glm47_moe_tool_parser.py: 75 passed
• test_serving_chat.py -k glm47: 6 passed
• tests/parser/engine/test_inkling.py: 85 passed
• branch-wide pre-commit: passed

Validation used CPU-only, network-disabled containers with read-only source mounts. No GPU, live service, or port 8100 operation was used.

GLM coverage includes whole input, character chunks, protocol chunks, and real token-ID streaming through the production incremental detokenizer. It covers literal delimiters, malformed and truncated values, duplicate anchors, stop-stripped trailing DROP IDs, exact reasoning, and multibyte detokenizer holdback.

AI assistance

OpenAI Codex assisted with diagnosis, implementation, tests, and PR text. Claude Opus 5 performed the earlier independent branch review whose findings drove this remediation. A human must review and understand every changed line before this draft is marked ready.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[logprobz]

Superseded by #639 (parser: literal GLM delimiters inside argument values) and #640 (scanner: stop token ID whose text was stripped). Those two carry the same fixes with a smaller diff and tests that fail on the base branch.