Skip to content

Update grammar#1023

Merged
ikawrakow merged 17 commits intomainfrom
fcp/grammar_update
Nov 30, 2025
Merged

Update grammar#1023
ikawrakow merged 17 commits intomainfrom
fcp/grammar_update

Conversation

@firecoperana
Copy link
Collaborator

@firecoperana firecoperana commented Nov 29, 2025

Hopefully fixes #1019.
This PR port a few commits related to grammar from mainline. In my test, grammar is now at least enforced.
Use test from: ggml-org/llama.cpp#10612

# without grammar
./bin/llama-server -m ../models/qwen2.5-32b-instruct/ggml-model-q8_0.gguf

curl --request POST --url http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n '{ messages: [{ role: "system", content: "You are a helpful assistant." }, { role: "user", content: "How to say \"Good morning\" in Chinese?" }], "top_k": 1 }')" | jq -r .choices[0].message.content

# To say "Good morning" in Chinese, you can say:
# 
# **早晨好** (zǎo chén hǎo)
# 
# or more commonly:
# 
# **早上好** (zǎo shàng hǎo)
# 
# Both phrases are used to greet someone in the morning.
# with English-only grammar
./bin/llama-server -m ../models/qwen2.5-32b-instruct/ggml-model-q8_0.gguf --grammar-file ../grammars/english.gbnf

curl --request POST --url http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n '{ messages: [{ role: "system", content: "You are a helpful assistant." }, { role: "user", content: "How to say \"Good morning\" in Chinese?" }], "top_k": 1 }')" | jq -r .choices[0].message.content

# To say "Good morning" in Chinese, you can say " mornings" which is pronounced as "zao3 shang4" (the numbers represent the tone).

jemc and others added 16 commits November 29, 2025 07:42
Prior to this commit, using a JSON Schema containing a string
with `pattern` regular expression that uses top-level alternation
(e.g. `"pattern": "^A|B|C|D$"`) would result in invalid JSON
output from the constrained sampling grammar, because it
ended up creating a grammar rule like this for the string:

```
thing ::= "\"" "A" | "B" | "C" | "D" "\"" space
```

Note that this rule will only match a starting quote for the "A" case,
and will only match an ending quote for the "D" case,
so this rule will always produce invalid JSON when used for sampling
(that is, the JSON will always be lacking the starting quote,
the ending quote, or both).

This was fixed in a simple way by adding parentheses to the
generated rule (for all string pattern rules, to keep it simple),
such that the new generated rule looks like this (correct):

```
thing ::= "\"" ("A" | "B" | "C" | "D") "\"" space
```
Co-authored-by: Richard Lyons <frob@cloudstaff.com>
* grammar : support array references in json schema

* Update json-schema-to-grammar.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* grammar : improve regex when naming ref derived rules

* grammar : replace non-conformant definitions array with anyOf test case

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
# Conflicts:
#	tests/test-json-schema-to-grammar.cpp
* Fix DoS / integer overflow

* Remove optional, use INT64_MAX instead as placeholder value (it's technically -1, so it fits :)

* White space

* Actually, since it's unsigned, use UINT64_MAX
# Conflicts:
#	src/llama-grammar.cpp
* grammar: fix regression caused by #17381

* more readable
# Conflicts:
#	src/llama-grammar.cpp
@ikawrakow ikawrakow merged commit 52adcf1 into main Nov 30, 2025
@firecoperana firecoperana deleted the fcp/grammar_update branch December 12, 2025 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: ik_llama.cpp crashes when Kimi K2 Thinking sometimes produces invalid tool call due to not working grammar