Eval bug: "failed to parse grammar" for regex tools that use `\d'

[serjrd]

Name and Version

root@llm:/app# /app/llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from /app/libggml-cuda.so
load_backend: loaded CPU backend from /app/libggml-cpu-icelake.so
version: 6795 (ee09828)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX 5090

Models

gpt-oss-20b

Problem description & steps to reproduce

When a tool is declared with z.string().regex(/^(anon-)?\d{11,}$/) using typescript-sdk llama.cpp fails with failed to parse grammar.

When serving the same model through LM Studio same code works fine.

If I change regex to z.string().regex(/^(anon-)?[0-9]{11,}$/) llama.cpp works as well.

Here's a simple code to reproduce the issue:

package.json
test_llm.js
test_mcp.js

First Bad Commit

No response

Relevant log output

llm  | parse: error parsing grammar: unknown escape at \d"{11,}) "\"" space
llm  | test-args-testId-kv ::= "\"testId\"" space ":" space test-args-testId
llm  | test-call ::= "test"channel " <|constrain|>json"? "<|message|>" test-args
llm  | test-call0 ::= "test" " <|constrain|>json"? "<|message|>" test-args
llm  | 
llm  | 
llm  | channel ::= "<|channel|>" ( "commentary" | "analysis" )
llm  | recipient-in-channel ::= channel " to=functions." ( test-call0 )
llm  | recipient-in-role ::= "<|start|>assistant"? " to=functions." ( test-call )
llm  | root ::= recipient-in-role | recipient-in-channel
llm  | space ::= | " " | "\n"{1,2} [ \t]{0,20}
llm  | test-args ::= "{" space test-args-testId-kv "}" space
llm  | test-args-testId ::= "\"" (("anon-")? "\d"{11,}) "\"" space
llm  | test-args-testId-kv ::= "\"testId\"" space ":" space test-args-testId
llm  | test-call ::= "test"channel " <|constrain|>json"? "<|message|>" test-args
llm  | test-call0 ::= "test" " <|constrain|>json"? "<|message|>" test-args
llm  | 
llm  | llama_grammar_init_impl: failed to parse grammar
llm  | srv    send_error: task id = 365, error: Failed to parse grammar
llm  | srv  process_sing: failed to launch slot with task, id_task = 365
llm  | srv  update_slots: all slots are idle
llm  | srv  cancel_tasks: cancel task, id_task = 365
llm  | srv  update_slots: all slots are idle
llm  | srv  log_server_r: request: POST /v1/chat/completions 172.19.0.1 400

[aldehir]

The JSON schema documentation recommends a subset of the ECMA 262 dialect, which does not include shorthand character classes like \d.

I'm guessing that is why support was not added in llama.cpp.

[serjrd]

@aldehir Thanks for pointing it out.

I wonder why I have so much less trouble with the whole JSON schema when serving a model with LM Studio.
Apart from from missing the \d definition and another issue with integer definition that zod4 uses that has been fixed now, I've struggled for some time with another definition:

const SomeObject = z.object({ test: z.string() })

const testType = z.lazy(() =>
  z.object({
    and: z.union([SomeObject, testType]),
    or: z.union([SomeObject, testType]),
    not: z.union([SomeObject, testType])
  })
)

server.registerTool('test',
  {
    title: 'Get test record by ID',
    description: 'Get test record by ID',
    inputSchema: {
      test: testType
    }
  },
  ({ testId }) => {
    return {
      content: [{ type: 'text', text: 'You requested test record ID: ' + testId }]
    }
  }
)

Will result in

{"error":{"code":500,"message":"JSON schema conversion failed:\\nError resolving ref #/properties/test/properties/and/anyOf/0: 0 not in [{\\"type\\":\\"object\\",\\"properties\\":{\\"test\\":{\\"type\\":\\"string\\"}},\\"required\\":[\\"test\\"],\\"additionalProperties\\":false},{\\"$ref\\":\\"#/properties/test\\"}]\\nError resolving ref #/properties/test/properties/and/anyOf/0: 0 not in [{\\"type\\":\\"object\\",\\"properties\\":{\\"test\\":{\\"type\\":\\"string\\"}},\\"required\\":[\\"test\\"],\\"additionalProperties\\":false},{\\"$ref\\":\\"#/properties/test\\"}]","type":"server_error"}}

While if I define testType to use someObject value directly - it works just fine:

const testType = z.lazy(() =>
  z.object({
    and: z.union([z.object({ test: z.string() }), testType]),
    or: z.union([z.object({ test: z.string() }), testType]),
    not: z.union([z.object({ test: z.string() }), testType])
  })
)

And LM Studio didn't have any of these issues!

Doesn't LM Studio use llama.cpp under the hood? Why is it so much more versatile? Any way to use the same library they use?

[aldehir]

You can build with llguidance, which offers better JSON schema support. But I believe it is broken with gpt-oss because of a workaround put in place.