Skip to content

fix: broaden llama.cpp grammar error classifier to match newer builds - #51527

Closed
gonzalofrancoceballos wants to merge 2 commits into
NousResearch:mainfrom
gonzalofrancoceballos:fix/llama-cpp-grammar-classifier-broader-match
Closed

fix: broaden llama.cpp grammar error classifier to match newer builds#51527
gonzalofrancoceballos wants to merge 2 commits into
NousResearch:mainfrom
gonzalofrancoceballos:fix/llama-cpp-grammar-classifier-broader-match

Conversation

@gonzalofrancoceballos

Copy link
Copy Markdown

Problem

The llama.cpp grammar error classifier in agent/error_classifier.py only recognizes older error phrasing ("error parsing grammar"), but newer llama.cpp builds return "Failed to initialize samplers: failed to parse grammar". When the classifier fails to match, Hermes skips the grammar recovery path (stripping pattern/format keywords from tool schemas and retrying locally) and falls directly to a cloud fallback provider.

This is especially impactful for users running local inference via custom providers (llama.cpp) with MCP servers whose tool schemas include regex patterns like "\\d{4}-\\d{2}-\\d{2}".

Reproduction

# Local llama.cpp server returns HTTP 400:
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "qwen3-80b-next-q4_k_m",
  "messages": [{"role":"user","content":"test"}],
  "tools": [{"type":"function","function":{"name":"f","parameters":{"type":"object","properties":{"d":{"type":"string","pattern":"\\d{4}-\\d{2}-\\d{2}"}}}}}],
  "max_tokens": 50
}'
# → {"error":{"message":"Failed to initialize samplers: failed to parse grammar"}}

In Hermes, this error passes through the classifier unrecognized, triggering cloud fallback instead of the schema-stripping recovery.

Fix

Added "failed to parse grammar" to the match list in the classifier condition, alongside the existing "error parsing grammar" and "unable to generate parser" checks.

Changes

  • agent/error_classifier.py: Added "failed to parse grammar" in error_msg condition
  • tests/agent/test_error_classifier.py: Added test_llama_cpp_failed_to_parse_grammar test case

Testing

pytest tests/agent/test_error_classifier.py::TestClassifyApiError::test_llama_cpp_failed_to_parse_grammar
# PASSED
pytest tests/agent/test_error_classifier.py::TestClassifyApiError::test_llama_cpp_grammar_parse_error
# PASSED (existing test still passes)

@alt-glitch alt-glitch added type/bug Something isn't working comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists labels Jun 23, 2026
The error message varies across llama.cpp versions:
- Older: "error parsing grammar: unknown escape at \\d"
- Newer: "Failed to initialize samplers: failed to parse grammar"

Only the older phrasing was recognized, causing Hermes to fall back
to cloud providers instead of stripping pattern/format keywords and
retrying locally. Added "failed to parse grammar" to the match list
and updated the classifier comment to document the newer variant.

Test: added test_llama_cpp_failed_to_parse_grammar covering the new phrase.
@gonzalofrancoceballos
gonzalofrancoceballos force-pushed the fix/llama-cpp-grammar-classifier-broader-match branch from 035bce3 to 930d0f6 Compare June 30, 2026 08:31
@gonzalofrancoceballos
gonzalofrancoceballos deleted the fix/llama-cpp-grammar-classifier-broader-match branch July 1, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants