[WIP] tool-call: experimental migration of all parsers to peg-parser infra (w/ better test coverage)#18353

Draft

ochafik wants to merge 148 commits intoggml-org:masterfrom

ochafik:peg-migration-squashed

Commits on Dec 31, 2025

peg-parser: enum-based tags, lambda mappers, and grammar improvements

ochafik
and
claude
committed
chat: migrate all parsers to modular PEG infrastructure

ochafik
and
claude
committed
test: add needle streaming tests and metatest infrastructure

ochafik
and
claude
committed
server: add --experimental-new-parsers flag for PEG migration

ochafik
and
claude
committed
test required tool calls w/ new parsers
ochafik
committed
chat-parsers: enforce no content in tool_choice=required mode

ochafik
and
claude
committed
fix typo in test_tool_call
ochafik
committed
peg-parser: add schema_or_raw_string_until helper

ochafik
and
claude
committed
test: fix tool_required_allows_content for GLM 4.6, MiniMax M2, Apertus

ochafik
and
claude
committed
fix: remove unused variables and trailing whitespace

ochafik
and
claude
committed
fix: update SPACE_RULE and test expectations for grammar changes

ochafik
and
claude
committed
fix: convert Nemotron V3 template from CRLF to LF

ochafik
and
claude
committed
fix: reject content in tool_choice=required mode, refactor tests

ochafik
and
claude
committed
test: add format detection test for templates with tools

ochafik
and
claude
committed
fix: remove README.md from Python test template list

ochafik
and
claude
committed
fix additionalProperties default back to false
ochafik
committed
manual refactoring: more uniform triggers, grammar gen & refs resolution
ochafik
committed
test-chat: make some failures easier to debug
ochafik
committed
test-chat: run needle tests first
ochafik
committed
Update firefunction-v2.cpp
ochafik
committed
Update apertus.cpp
ochafik
committed
Update command-r7b.cpp
ochafik
committed
Update deepseek-r1.cpp
ochafik
committed
Update apriel-1-5.cpp
ochafik
committed
Update generic.cpp
ochafik
committed
Update gpt-oss.cpp
ochafik
committed
Update granite.cpp
ochafik
committed
Update lfm2.cpp
ochafik
committed
Update mistral-nemo.cpp
ochafik
committed
Update nemotron-v2.cpp
ochafik
committed
Update apertus.cpp
ochafik
committed
Update minimax-m2.cpp
ochafik
committed
Update nemotron-v3.cpp
ochafik
committed
Update lfm2.cpp
ochafik
committed
Update qwen3-coder-xml.cpp
ochafik
committed
Update glm-4-5.cpp
ochafik
committed
Update functionary-v3-1-llama-3-1.cpp
ochafik
committed
Update kimi-k2.cpp
ochafik
committed
Update llama-3-x.cpp
ochafik
committed
Update llama-3-x.cpp
ochafik
committed
Update magistral.cpp
ochafik
committed
Update kimi-k2.cpp
ochafik
committed
fix foreach_parameter regression
ochafik
committed
fix lints
ochafik
committed
Update test-chat.cpp
ochafik
committed
Update nemotron-v3.cpp
ochafik
committed
Update minimax-m2.cpp
ochafik
committed
feat: add json_schema support to 4 parsers + needle test scaffolding

ochafik
and
claude
committed
Fix GPT OSS template parser and test configuration

ochafik
and
claude
committed
Fix GPT OSS tool call parser - add missing TOOL_OPEN tag

ochafik
and
claude
committed
Fix GPT OSS tool call parser and test configuration

ochafik
and
claude
committed
Fix init_delta to use params_prefix for correct parser configuration

ochafik
and
claude
committed
fix parser dispatch (json_schema cases)
ochafik
committed
drop ToolSupport enum (assume all yes)
ochafik
committed
Update deepseek-v3-1.cpp
ochafik
committed
Simplify GLM 4.5 parser and fix test injection for thinking_forced_open

ochafik
and
claude
committed
Fix DeepSeek R1 parallel tool calls and add fixed template tests

ochafik
and
claude
committed
Update firefunction-v2.cpp
ochafik
committed
build_json_args_peg_parser helper
ochafik
committed
json refactors
ochafik
committed
Update firefunction-v2.cpp
ochafik
committed
Update command-r7b.cpp
ochafik
committed
Update lfm2.cpp
ochafik
committed
Update chat-parsers-internal.h
ochafik
committed
switch json grammars to native peg parser (w/ temp hack)
ochafik
committed
remove cruft around / between needles
ochafik
committed
Update llama-3-x.cpp
ochafik
committed
tool call parser helpers refactor!
ochafik
committed
Update minimax-m2.cpp
ochafik
committed
revert change in legacy common_chat_params_init_nemotron_v3
ochafik
committed
Update chat-parsers-internal.h
ochafik
committed
Update apertus.cpp
ochafik
committed
Update qwen3-coder-xml.cpp
ochafik
committed
Update deepseek-r1.cpp
ochafik
committed
Update seed-oss.cpp
ochafik
committed
Update minimax-m2.cpp
ochafik
committed
Update glm-4-5.cpp
ochafik
committed
Update deepseek-r1.cpp
ochafik
committed
Update chat-parser.cpp
ochafik
committed
Update apriel-1-5.cpp
ochafik
committed
pass more parsers to helpers
ochafik
committed
peg-constructed: create tool call only when tool name arrives
ochafik
committed
space nits
ochafik
committed
refactor: introduce format structs for PEG parser helpers

ochafik
and
claude
committed
json helper: format.tool_call factory
ochafik
committed
json helper: migrate parsers to format.tool_call lambda pattern

ochafik
and
claude
committed
fix: add space after colon in tool_call lambdas

ochafik
and
claude
committed
fix: make PEG mappers lazy to avoid spurious tool calls during backtracking

ochafik
and
claude
committed
refactor: route Command R7B to native_mapper (uses build_json_tool_calls_peg_parser)

ochafik
and
claude
committed
fix: improve PEG mapper content handling and add stricter tag validation

ochafik
and
claude
committed
Update kimi-k2.cpp

ochafik
and
claude
committed
test-chat: unskip mimo & apriel

ochafik
and
claude
committed
fix mimo

ochafik
and
claude
committed
fix apriel

ochafik
and
claude
committed
refactor tests a bit

ochafik
and
claude
committed
nits

ochafik
and
claude
committed
fix needle tests

ochafik
and
claude
committed
nemotron v2: wire enable_thinking behaviour w/ adhoc user messages

ochafik
and
claude
committed
refactor tests some more

ochafik
and
claude
committed
end tokens for command r7b in needle tests

ochafik
and
claude
committed
fix command r7b parsing (accept id before name in common_chat_peg_native_mapper)

ochafik
and
claude
committed
tighten template detection of xiaomi mimo

ochafik
and
claude
committed
minimize diff

ochafik
and
claude
committed
fix apriel typo, drop nemv3 enum, minimize diffs

ochafik
and
claude
committed
linter nit

ochafik
and
claude
committed
test-chat: fix inputs.enable_thinking when reasoning format is none
ochafik
committed
inline build_json_tool_calls_peg_parser

ochafik
and
claude
committed
test-chat: split out parser tests to their own files

ochafik
and
claude
committed
test-chat: green test w/ lots of skipping
ochafik
committed
fix functionary v3.2 trigger
ochafik
committed
test-chat: aggregate failures, warn about skips, filter w/ TEST=name (or * for all including skipped)
ochafik
committed
Update test-chat.cpp
ochafik
committed
define all template_caps.end_tokens
ochafik
committed
peg mappers: ignore structural wrappers
ochafik
committed
peg grammar: optional of epsilon is epsilon
ochafik
committed
test-kimi-k2: switch to thinking model!
ochafik
committed
test-ministral-3: tool call do have ids
ochafik
committed
fix end_tokens typos, unskip / reskip (+ TEST=skipped syntax)
ochafik
committed
fix TEST=llama_3_x_legacy test-chat
ochafik
committed
fix TEST=gpt_oss_experimental
ochafik
committed
test-chat: better logging upon failure (dump messages)
ochafik
committed
rename: test_systematic_needle_streaming -> run_template_test_suite
ochafik
committed
have TEST match substrings
ochafik
committed
fix typo in test-nemotron-v2
ochafik
committed
pass param_ends as array of delimiters
ochafik
committed
qwen3-coder: use additional stops instead of consume_end_block()
ochafik
committed
qwen3-coder: add test case that required multiple param_ends
ochafik
committed
make TOOL_NAME and TOOL_ARG_NAME ~atomic in mappers
ochafik
committed
fix streaming diff bugs for Command R7B and Mistral Nemo

ochafik
and
claude
committed
peg-parser: add unicode-aware trie for GBNF exclusion patterns

ochafik
and
claude
committed
chat-peg-parser: fix streaming regressions for tool calls

ochafik
and
claude
committed
chat: fix magistral template detection and parser format

ochafik
and
claude
committed
chat-parsers: allow leading whitespace before tool calls

ochafik
and
claude
committed
chat-parsers: add content-only fallback when tools provided but not called

ochafik
and
claude
committed
deepseek-r1: fix tool call parsing with trailing whitespace

ochafik
and
claude
committed
nemotron-v3: fix parameter delimiter parsing

ochafik
and
claude
committed
test-kimi-k2: adjust for template's message splitting behavior

ochafik
and
claude
committed
kimi-k2: fix p.chars() character class syntax

ochafik
and
claude
committed
test: add server test exclusion list for experimental parsers

ochafik
and
claude
committed
peg-parser: handle zero-repetition as eps()

ochafik
and
claude
committed
test-lfm2: skip needle test suite for legacy parser

ochafik
and
claude
committed
generic: allow optional content field in tool call response

ochafik
and
claude
committed
rm dead code: common_chat_peg_mapper_func, content_until in qwen3-coder
ochafik
committed
test_peg_parser: pass impl parameter instead of forcing experimental

ochafik
and
claude
committed