feature(tool-calling): harden tool calling for agent harnesses - #645
Conversation
i386
left a comment
There was a problem hiding this comment.
Review comments from local pass. I focused on correctness risks in the guardrail runtime/API integration and contract behavior.
|
Results from the laboratory Forge Tool Calling Feature-Only Benchmark ReportSummaryTested Results
Run Control
Progress
Benchmark Matrix And MetricsAll client traffic ran locally from the coordinator through Carrack only. Fixture: run-scoped patched copy of
Off vs Enforce Deltas
Hard-Schema Observations
Guardrail And Readiness Evidence
OpenCode Client Phase
Failures Or Risks
Commands RunRepresentative commands only: # Host build/archive phase on each host
cd ~/dev/mesh/mesh-llm
rm -rf ./target
# Coordinator readiness checks through Carrack gateway
curl -sS http://carrack.patio51.com:9337/v1/models
curl -sS http://carrack.patio51.com:3131/api/status
jq -e '.runtime.openai_guardrails.mode == "disabled"' off-control-status-carrack-before.json
jq -e '.runtime.openai_guardrails.mode == "enforce" and .runtime.openai_guardrails.max_tool_retries == 1 and .runtime.openai_guardrails.max_structured_retries == 2' enforce-status-carrack-before.json
# Coordinator raw fixture shape (repeated for 2 models x 2 modes x 2 max-token values)
python3 .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/fixture/forge_tool_call_fixture.py raw \
--base-url http://carrack.patio51.com:9337/v1 \
--model unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL \
--engine-label mesh-llm \
--branch-label feature-enforce \
--branch-name feature/forge-inspired-model-nudging \
--commit 8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d \
--case-count 50 \
--max-tokens 256 \
--mesh-guardrails true \
--out .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/feature/0.8b/raw-openai/enforce/max_tokens_256
# Cleanup on each host
/home/ndizazzo/mesh-runs/forge-tool-feature-bench-20260522/bin/mesh-llm-feature-8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d stopArtifacts
Post-Run Learning
RecommendationUse the 512-token setting for this fixture on both target models: it eliminates hard-schema failures on 4B and avoids the 256-token enforce HTTP-400 tradeoff. For 0.8B, enforce mode helps the 256-token nested-object family but does not improve 512-token aggregate accuracy. If the goal is to demonstrate feature value, focus follow-up analysis on 0.8B/256 hard-schema repair and on whether HTTP 400 enforcement should be counted separately from tool-call accuracy in dashboards. |
|
I like idea of this. Would like to see it with some other harnesses and models, not just tiny ones to see if net benefit. If so I think its own crate makes sense as this is essentially a port to rust of "forge", and we could even break it out as a crate for other rust projects to use. I'm not sure where the threshold is for applying this to models but worthwhile trying a bit more with harnesses imo. |
8d05e19 to
a9b9393
Compare
There was a problem hiding this comment.
Tested locally — found a blocking bug in enforce mode
Tested PR #645 by building locally, running mesh-llm serve --mesh-guardrails enforce with Qwen/Qwen2.5-3B-Instruct, and running typical agent-style requests.
Blocking bug: the standard agent loop breaks under enforce 🛑
Repro:
# Turn 1: model emits a tool call
chat([{role:user, content:"What is 5+5?"}], tools=[calc]) -> tool_calls: calc({x: "5+5"})
# Turn 2: agent returns tool result. Tools kept in request (OpenAI convention).
chat([..., {role:tool, content:"10"}], tools=[calc])
# -> HTTP 400 guardrail_validation_failedReproduced 3/3 with two different prompts (recipe-read, calculator). Same payload without tools on turn 2 succeeds; same payload with --mesh-guardrails disabled succeeds.
Root cause
rescue.rs::classify_response line ~96: when request_expects_guarded_contract is true (i.e. request includes tools), any plain-text model output is classified as MalformedToolText and retried. The classifier doesn't account for the legitimate case "model is answering after consuming a tool result" — which is exactly when tools is still in the request but the correct response is natural language.
request_expects_guarded_contract just checks has_real_tools() without looking at conversation state. It needs to know that if the last message is a role: tool result, a text answer satisfies the contract.
|
I think the critical one is that it shouldn't break multi turn |
Odd, this didn't turn up during my testing... Lemme investigate |
|
@ndizazzo haven't had a chance to test but this fixed the last issue with multiple turns? If so I think this is ok as it is low risk and may make it better? |
ccb4f0b to
dbd4ca6
Compare
One-Node Multi-Turn Tool-Call Regression ReportSummaryCarrack passed the one-node multi-turn tool-call regression for branch Results
Run Control
Progress
Metrics
OpenCode Client Phase
Raw OpenAI-Compatible Probe
Successive Tool-Call Failure State
Failures Or Risks
Commands Run# Remote Carrack build/startup/cleanup evidence was captured by the Carrack agent.
just build
./target/debug/mesh-llm serve ... --listen-all --mesh-guardrails enforce
curl http://127.0.0.1:3131/api/status
curl http://127.0.0.1:9337/v1/models
curl http://carrack.patio51.com:9337/v1/models
./target/debug/mesh-llm stop
# Coordinator-local client phases used the Carrack gateway.
python3 multi_turn_tool_probe.py
opencode run --pure --format json --model mesh-test/unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL --dir <run-workdir> <prompt>Artifacts
Post-Run Learning
RecommendationKeep using coordinator-local client phases through the Carrack gateway for OpenCode/raw OpenAI-compatible validation. For future small-model OpenCode smoke tests, use exact absolute fixture paths and explicitly forbid unrelated tools to avoid agent drift. |
Summary
Adds Forge-inspired OpenAI guardrails for hosted Skippy chat so small/open models can more reliably satisfy tool-call and structured-output contracts. The guardrail layer stays behind the
OpenAiBackendseam instead of changing public OpenAI route handling, mesh protocol, plugin protocol, or the Skippy ABI.What changed
mesh-llm-guardrailsand OpenAI frontend guardrail modules for:tools,tool_choice,parallel_tool_calls,response_format,mesh_guardrails)GuardedOpenAiBackendwrapping for hosted Skippy non-streaming chat./v1/responsesinherits guarded chat behavior through the existing adapter.mesh-llm serve --mesh-guardrails <disabled|metrics|enforce>mesh-llm runtime guardrails --mode <...>POST /api/runtime/mesh-guardrails/api/status.runtime.openai_guardrails.docs/design/OPENAI_GUARDRAILS.mdscripts/run-openai-guardrail-corpus.pyWhy
Agent harnesses and tool-using clients are sensitive to small formatting failures from open models: malformed tool JSON, fake tool text, sentinel leakage, or structured-output drift can break otherwise usable responses. This adds a conservative, opt-in middleware layer that can rescue or retry those cases for Skippy-backed chat while preserving existing API compatibility and keeping enforcement disabled by default until live evidence justifies broader rollout.
Testing
cargo test -p openai-frontend --lib- 124 passedcargo test -p openai-frontend --test benchy_contract- 8 passedcargo test -p skippy-server --lib- 82 passedcargo test -p mesh-llm-host-runtime --lib- 1446 passed, 6 ignoredcargo test -p mesh-llm-host-runtime runtime::survey::tests --lib- 18 passedcargo test -p mesh-llm-host-runtime telemetry_config --lib- 4 passedjust build- passedMulti-host benchmark evidence was collected in
forge-tool-bench-20260522T063000Zacross Carrack, Lemony, and Lemony29. Mesh request-level and OpenCode client phases passed on bothmainand this feature branch for Qwen3.5 0.8B and 4B GGUF models through the Carrack gateway.Raw mesh correctness was identical in aggregate between
mainandfeatureWHEN NOT ENABLED, ensuring that the default path does not introduce any regression / additional latency.Testing is currently ONGOING with new features enabled across 3 noted test hosts. This should give notable findings
Notes / caveats
Supersedes #614