Skip to content

fix(agent): classify OpenAI regex lookaround schema rejection as recoverable - #42635

Open
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/schema-regex-lookaround-recovery
Open

fix(agent): classify OpenAI regex lookaround schema rejection as recoverable#42635
liuhao1024 wants to merge 1 commit into
NousResearch:mainfrom
liuhao1024:fix/schema-regex-lookaround-recovery

Conversation

@liuhao1024

Copy link
Copy Markdown
Contributor

What does this PR do?

Classifies HTTP 400 errors from OpenAI-compatible endpoints that reject tool schemas containing regex lookaround patterns as recoverable, routing them through the existing strip_pattern_and_format() retry path so the conversation continues instead of failing permanently.

Related Issue

Fixes #42631

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • agent/error_classifier.py: Extended the llama.cpp grammar-pattern condition to also match OpenAI-compatible "regex lookaround is not supported" errors, routing them through the same FailoverReason.llama_cpp_grammar_pattern recovery path
  • tests/agent/test_error_classifier.py: Added 2 tests — one verifying the lookaround error is classified as retryable, one verifying partial keyword matches don't trigger false positives

How to Test

  1. Run pytest tests/agent/test_error_classifier.py -q — all 157 tests pass (including the 2 new ones)
  2. Run pytest tests/tools/test_schema_sanitizer.py -q — all 40 schema sanitizer tests pass (verifies the downstream recovery path)
  3. The new test_openai_regex_lookaround_rejection test verifies the exact error message from the issue is classified as FailoverReason.llama_cpp_grammar_pattern with retryable=True

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Code Intelligence

  • Analyzed: agent/error_classifier.py (error classification logic), agent/conversation_loop.py:2304 (recovery path consumer), tools/schema_sanitizer.py:308 (strip_pattern_and_format)
  • Blast radius: LOW — only affects HTTP 400 errors with specific regex lookaround phrasing; all existing llama.cpp grammar tests unchanged
  • Related patterns: Existing llama_cpp_grammar_pattern recovery path at lines 606-621; this extends the same condition with an additional disjunct

…verable

Some OpenAI-compatible endpoints (e.g. strict GPT tool-schema validation)
reject tool schemas containing regex lookaround patterns in `pattern`
keywords with HTTP 400 and the message:
  "Invalid JSON schema: regex lookaround is not supported."

Previously this was classified as a non-retryable client error, causing
the conversation to fail permanently. Route through the existing
`strip_pattern_and_format()` recovery path (same as llama.cpp grammar
errors) so Hermes strips the incompatible `pattern`/`format` keywords
from tool schemas and retries once.

Fixes NousResearch#42631
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Jun 9, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused compatibility recovery. Current main still classifies only llama.cpp grammar-parser phrases at agent/error_classifier.py:698-713; the reported lookaround 400 therefore reaches generic non-retryable 4xx handling. The added condition maps that exact error family to the established one-shot strip_pattern_and_format(agent.tools) recovery at agent/conversation_loop.py:2940-2964.

The focused classifier tests are appropriate for the new recognition branch, and the existing recovery path preserves its no-op/fall-through guard when no compatible schema keyword exists to strip.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users labels Jul 14, 2026
@mvdbastos

Copy link
Copy Markdown

We're pulling #67349 into our fork for a related llama.cpp/LM Studio grammar-phrase gap (same error_classifier.py condition, same recovery path). Flagging so the two don't collide — this PR and #67349 both add disjuncts to the same llama_cpp_grammar_pattern branch, worth reviewing together to avoid duplicate/overlapping matches. Happy to help reconcile if that's useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:blast-contained Sweeper blast radius: contained — one narrow path / opt-in / few users sweeper:risk-caching Sweeper risk: may break/degrade prompt caching or cache-key stability (invariant) sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GPT/OpenAI-compatible endpoints reject tool schemas with regex lookaround pattern

4 participants