WIP: \tool call hardening small models - #614
Closed
michaelneale wants to merge 2 commits into
Closed
Conversation
…ctx compaction
Adds a new crate `mesh-llm-guardrails` that ports forge v0.6.0's
tool-call rescue and tiered context compaction into two transparent
`OpenAiBackend` decorators:
- `GuardrailBackend` — engaged when the host's model is single-digit
B (small tier). Rescues bare-text tool calls (JSON, rehearsal
`tool[ARGS]{}`, Qwen3-Coder XML), validates names against the
request's tool catalog, retries with a corrective nudge, and falls
through as text on budget exhaustion instead of 5xx-ing.
- `CompactingBackend` — engaged when the host's `n_ctx` is small
(default ≤ 8192). Three-phase compaction (drop nudges → drop tool
results → drop reasoning, last gated behind a flag) plus escalating
context-warning injection. Prefers `usage.total_tokens` reported by
the inner backend over a char/4 heuristic once a turn has been seen
— matches forge's ContextManager and catches dense tool-schema
bloat that char/4 underestimates.
Wired into `SkippyModelHandle::{load_with_hooks, load_stage0_config}`
so every consumer (direct `/v1`, MoA workers, virtual-LLM consults,
`/v1/responses`) gets the wrap for free without per-call-site logic.
Wrap decision is made once at construction based on (model name, n_ctx).
Also lifts the single-digit-B heuristic from `mesh-mixture-of-agents`
into `mesh_llm_routing::is_small_tier_name` so the guardrail decorator
and the MoA router can share one source of truth. (Switching the MoA
call site to delegate is a small mechanical follow-up.)
Per-request override fields `mesh_guardrails` and `mesh_compact` in
`request.extra` follow the existing `mesh_hooks` pattern.
Out of scope (deferred):
- Streaming guardrails (buffered-tool first cut; SSE passes through).
- StepEnforcer for ingress (config exposes it; defaults empty).
- `respond()` tool injection (config exposes it; default off).
- TOML / CLI config (uses WrapConfig::default for now).
- Forge validation gates / sampling defaults (separate workstream).
Verified end-to-end: goose driving `Qwen3-8B-Q4_K_M` through the mesh
ran tool calls cleanly with no false-positive interference from the
decorator (validator saw structured `tool_calls`, all names known,
Execute path).
Tests: 43 unit tests in mesh-llm-guardrails (rescue strategies,
validator, error tracker, step enforcer, nudges, respond, compaction
phases, threshold warning, backend decorators with a scripted inner
backend, usage tracking across turns) + 2 in mesh-llm-routing for the
small-tier helper. All passing. Clippy clean. cargo fmt clean.
See docs/design/FORGE_INTEGRATION.md for the full design.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc @ndizazzo this is using that forge pattern/design to rescue tool calls (a new crate) - FYI, not sure if will keep this approach but showing signs of promise, may be fun to try. I imagine we need a way to make this automatic for some small models (not sure where the cut is)