Skip to content

feature(tool-calling): harden tool calling for agent harnesses - #645

Merged
ndizazzo merged 6 commits into
mainfrom
feature/forge-inspired-model-nudging
May 25, 2026
Merged

feature(tool-calling): harden tool calling for agent harnesses#645
ndizazzo merged 6 commits into
mainfrom
feature/forge-inspired-model-nudging

Conversation

@ndizazzo

Copy link
Copy Markdown
Collaborator

Summary

Adds Forge-inspired OpenAI guardrails for hosted Skippy chat so small/open models can more reliably satisfy tool-call and structured-output contracts. The guardrail layer stays behind the OpenAiBackend seam instead of changing public OpenAI route handling, mesh protocol, plugin protocol, or the Skippy ABI.

What changed

  • Added reusable guardrail primitives in mesh-llm-guardrails and OpenAI frontend guardrail modules for:
    • request-contract parsing (tools, tool_choice, parallel_tool_calls, response_format, mesh_guardrails)
    • tool-call rescue from malformed model text
    • retry nudges after completed semantic model responses
    • validated structured-output emulation
    • OpenAI-shaped guardrail errors
  • Added GuardedOpenAiBackend wrapping for hosted Skippy non-streaming chat.
    • Streaming stays pass-through.
    • /v1/responses inherits guarded chat behavior through the existing adapter.
    • Legacy completions remain outside v1 guardrails.
  • Added operator controls:
    • mesh-llm serve --mesh-guardrails <disabled|metrics|enforce>
    • mesh-llm runtime guardrails --mode <...>
    • POST /api/runtime/mesh-guardrails
  • Surfaced privacy-safe runtime status via /api/status.runtime.openai_guardrails.
  • Added bounded guardrail telemetry for decisions/outcomes without prompt, schema, endpoint, or argument leakage.
  • Added docs and benchmark scaffolding:
    • docs/design/OPENAI_GUARDRAILS.md
    • CLI/usage docs for runtime controls
    • telemetry docs
    • guardrail corpus runner: scripts/run-openai-guardrail-corpus.py

Why

Agent harnesses and tool-using clients are sensitive to small formatting failures from open models: malformed tool JSON, fake tool text, sentinel leakage, or structured-output drift can break otherwise usable responses. This adds a conservative, opt-in middleware layer that can rescue or retry those cases for Skippy-backed chat while preserving existing API compatibility and keeping enforcement disabled by default until live evidence justifies broader rollout.

Testing

  • cargo test -p openai-frontend --lib - 124 passed
  • cargo test -p openai-frontend --test benchy_contract - 8 passed
  • cargo test -p skippy-server --lib - 82 passed
  • cargo test -p mesh-llm-host-runtime --lib - 1446 passed, 6 ignored
  • cargo test -p mesh-llm-host-runtime runtime::survey::tests --lib - 18 passed
  • cargo test -p mesh-llm-host-runtime telemetry_config --lib - 4 passed
  • just build - passed

Multi-host benchmark evidence was collected in forge-tool-bench-20260522T063000Z across Carrack, Lemony, and Lemony29. Mesh request-level and OpenCode client phases passed on both main and this feature branch for Qwen3.5 0.8B and 4B GGUF models through the Carrack gateway.

Raw mesh correctness was identical in aggregate between main and feature WHEN NOT ENABLED, ensuring that the default path does not introduce any regression / additional latency.

Testing is currently ONGOING with new features enabled across 3 noted test hosts. This should give notable findings

Notes / caveats

  • Guardrails remain opt-in and disabled by default.
  • The deterministic corpus fallback produced identical off behavior to main, so default enforcement does not affect normal code paths.
  • This branch also includes sibling compaction plumbing alongside guardrails; it is wired as a backend decorator, not as a mesh/protocol change.

Supersedes #614

@ndizazzo
ndizazzo requested a review from michaelneale May 22, 2026 19:03
@ndizazzo ndizazzo self-assigned this May 22, 2026
@ndizazzo ndizazzo changed the title feature(tool-calling): harden tool calling for agent harnesses throug… feature(tool-calling): harden tool calling for agent harnesses May 22, 2026
@ndizazzo ndizazzo added the enhancement New feature or request label May 22, 2026

@i386 i386 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments from local pass. I focused on correctness risks in the guardrail runtime/API integration and contract behavior.

Comment thread crates/mesh-llm-host-runtime/src/api/routes/runtime.rs
Comment thread crates/openai-frontend/src/guardrails/engine.rs
Comment thread crates/openai-frontend/src/guardrails/mod.rs
@ndizazzo

Copy link
Copy Markdown
Collaborator Author

@michaelneale

Results from the laboratory


Forge Tool Calling Feature-Only Benchmark Report

Summary

Tested feature/forge-inspired-model-nudging only, across Carrack, Lemony, and Lemony29, using the exact required model IDs through the Carrack gateway (http://carrack.patio51.com:9337/v1). The strict selected-host phases passed: all hosts preflighted cleanly, built with just build, archived feature binaries, started off-control and enforce runtimes from the archived binaries, proved readiness/guardrail state, completed the 8 raw fixture combinations, and cleaned up run-specific processes. Result: pass with caveats. The feature run produced valid comparison data; enforce mode did not improve the already-perfect 4B/512 result, improved the 0.8B/256 correct-argument rate versus off-control, matched 0.8B/512 and 4B/512, and reduced 4B/256 correct-argument rate versus off-control because guardrail-enforced failures returned HTTP 400 rather than malformed tool JSON.

Results

Host Status Git Ref Binary Runtime Key Evidence
carrack pass feature/forge-inspired-model-nudging @ 8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d mesh-llm 0.66.0, SHA ea50acba89e229f253b653fca5d94348fa15962d10d47a9fd28c1eed404b97cd Gateway on 0.0.0.0:9337/3131; off mode disabled, enforce mode enforce; exact 0.8B and 4B IDs Build real 67.17; off/enforce readiness JSONs under remote off-control/status and enforce/status; cleanup stopped PID 1361346, post-stop ports refused
lemony pass feature/forge-inspired-model-nudging @ 8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d mesh-llm 0.66.0, SHA 1587cc4e63b678ffbbdafc4e87aa32e1ab4a1fa17809de789b89e499b3008458 Worker/model node on loopback 9337/3131; exact models; enforce status proved locally and in Carrack view Build real 268.05; cleanup archived stop rc 0, post-stop no listeners/processes
lemony29 pass feature/forge-inspired-model-nudging @ 8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d mesh-llm 0.66.0, SHA 2e0a061ca23198813b89e732a99caa3b80df032cba9b3d3d5c1acf9c55ae692d Worker/model node on loopback 9337/3131; exact models; enforce status proved locally Build real 276.82; cleanup archived stop rc 0, no remaining processes/listeners

Run Control

  • Run ID: forge-tool-feature-bench-20260522
  • Policy: strict all-node
  • Decision: continued through all planned phases; no strict-phase abort
  • Blocker trigger: none
  • Cleanup after run: completed on all hosts; no model deletion or persistent TOML overwrite

Progress

  • TODOs used: yes
  • Phase updates: coordinator progress notes at phase boundaries plus host progress/final notes in the shared log
  • Last progress note before report: cleanup completed on all three hosts; all benchmark summaries verified present

Benchmark Matrix And Metrics

All client traffic ran locally from the coordinator through Carrack only. Fixture: run-scoped patched copy of scripts/forge_tool_call_fixture.py, SHA 66ff700c083d43d408f750bc1df2553cc704db0c209184e9dbbf7463a5bd668e; fixture spec SHA 8f7b2724c29987673ff80022a83bc8f712d34941495349e64f802e898947255c. Each combination used case_count=50, producing 100 initial tool-call requests (50 auto + 50 forced variants). Off-control omitted request mesh_guardrails; enforce used --mesh-guardrails true.

Model Mode Max tokens HTTP success Structured/correct tool name Correct args Tool-result final OK Raw p50 / p95 (s) Notes
0.8B Q8 off 256 100/100 98/100 86/100 86/86 3.42 / 8.76 Nested-object-with-array failures dominate at 256
0.8B Q8 enforce 256 96/100 96/100 92/100 92/92 3.77 / 7.98 Correct args +6 vs off; four enforced HTTP 400s
0.8B Q8 off 512 100/100 100/100 96/100 96/96 3.52 / 9.05 Residual repeated array-enum and number-nested-dim mismatch
0.8B Q8 enforce 512 100/100 100/100 96/100 96/96 3.51 / 8.93 Same score as off, similar latency
4B Q4 off 256 100/100 100/100 88/100 88/88 4.24 / 12.93 Date/string/array/integer JSON truncation at 256
4B Q4 enforce 256 90/100 90/100 90/100 90/90 4.70 / 11.02 HTTP 400 replaces many malformed JSON cases; net -10 vs perfect availability, +2 correct args vs off by strict arg count
4B Q4 off 512 100/100 100/100 100/100 100/100 4.94 / 14.26 Perfect fixture pass
4B Q4 enforce 512 100/100 100/100 100/100 100/100 4.94 / 14.28 Perfect fixture pass, no material latency shift

Off vs Enforce Deltas

Model Max tokens Correct args delta HTTP success delta Tool-result final OK delta Latency p50 delta
0.8B Q8 256 +6 -4 +6 +0.34s
0.8B Q8 512 0 0 0 -0.01s
4B Q4 256 +2 by argument count, but -10 HTTP successes -10 +2 completed follow-ups, -10 total-request availability +0.46s
4B Q4 512 0 0 0 +0.00s

Hard-Schema Observations

  • 0.8B/256 off-control struggled with nested object with array number boolean (0/10 correct args) and smaller misses in array enum and number nested dimensions.
  • 0.8B/256 enforce improved the nested-object-with-array family from 0/10 to 6/10 correct args, while returning HTTP 400 for several otherwise-bad attempts.
  • 0.8B/512 reached 96/100 in both modes; remaining misses were stable across off/enforce (array enum and number nested dimensions).
  • 4B/256 off-control produced structured tool calls for every request but malformed/truncated JSON in date string array integer and one nested-object case, ending at 88/100 correct args.
  • 4B/256 enforce returned HTTP 400 for those hard cases, making the failure cleaner but lowering successful-response availability to 90/100.
  • 4B/512 was perfect in both modes.

Guardrail And Readiness Evidence

  • Off-control Carrack status before scored traffic: .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/feature/readiness/off-control-status-carrack-before.json, with .runtime.openai_guardrails.mode == "disabled".
  • Enforce Carrack status before scored traffic: .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/feature/readiness/enforce-status-carrack-before.json, with mode enforce, target skippy, streaming pass_through, retry exhaustion error, max tool retries 1, and max structured retries 2.
  • Enforce probe request/response: feature/readiness/guardrail-probe-request.json and feature/readiness/guardrail-probe-response.json; request included mesh_guardrails:true, response produced a record_event tool call for probe_guardrail_enabled.
  • Exact model readiness through Carrack: feature/readiness/carrack-models.json, feature/readiness/carrack-models-enforce.json, and feature/readiness/model-ids.txt.
  • Post-enforce status captures after fixture traffic: feature/0.8b/guardrail-status-after-enforce.json and feature/4b/guardrail-status-after-enforce.json, both still enforce.

OpenCode Client Phase

  • Runner: not run
  • Provider: not applicable; raw OpenAI-compatible JSON fixture traffic was the required/prioritized client phase
  • OpenCode version preflight: 1.15.7
  • Result: not applicable
  • Evidence: local tool preflight recorded in Phase 0; no optional OpenCode reruns were requested or needed

Failures Or Risks

  • Baseline caveat: the request did not supply a concrete existing main-baseline or metrics-mode artifact path. Placeholder reference files were written under baseline-reference/; no main/LMS/metrics baseline was rerun, per plan.
  • Mock-smoke caveat: no mock-server mode/helper was found in the fixture copy, so the mock smoke step was not run.
  • Startup flag naming: the plan mentioned --openai-guardrails enforce, while the feature branch runtime/startup used --mesh-guardrails enforce; status still exposed/verified .runtime.openai_guardrails with all required enforce settings.
  • Fixture patch: the repository fixture lacked --max-tokens; only the run-scoped fixture copy was patched and hashed. Repository files were not modified.
  • Enforce 400 behavior: enforce at 256 tokens sometimes rejected malformed tool-call output with HTTP 400. Treat this as cleaner failure handling, not a successful tool-call completion.
  • Sensitive artifacts: raw remote status/process/startup artifacts may contain private join/status tokens. Report and shared-log prose use redacted paths/summaries.

Commands Run

Representative commands only:

# Host build/archive phase on each host
cd ~/dev/mesh/mesh-llm
rm -rf ./target

# Coordinator readiness checks through Carrack gateway
curl -sS http://carrack.patio51.com:9337/v1/models
curl -sS http://carrack.patio51.com:3131/api/status
jq -e '.runtime.openai_guardrails.mode == "disabled"' off-control-status-carrack-before.json
jq -e '.runtime.openai_guardrails.mode == "enforce" and .runtime.openai_guardrails.max_tool_retries == 1 and .runtime.openai_guardrails.max_structured_retries == 2' enforce-status-carrack-before.json

# Coordinator raw fixture shape (repeated for 2 models x 2 modes x 2 max-token values)
python3 .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/fixture/forge_tool_call_fixture.py raw \
  --base-url http://carrack.patio51.com:9337/v1 \
  --model unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL \
  --engine-label mesh-llm \
  --branch-label feature-enforce \
  --branch-name feature/forge-inspired-model-nudging \
  --commit 8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d \
  --case-count 50 \
  --max-tokens 256 \
  --mesh-guardrails true \
  --out .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/feature/0.8b/raw-openai/enforce/max_tokens_256

# Cleanup on each host
/home/ndizazzo/mesh-runs/forge-tool-feature-bench-20260522/bin/mesh-llm-feature-8d05e19ec5bbf094b6df4422fa6decf0ccad4a8d stop

Artifacts

  • Local run root: .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522
  • Final report: .opencode/multi-team/opencode-runs/forge-tool-feature-bench-20260522/report/forge-tool-calling-feature-only-benchmark-report.md
  • Fixture copy and hashes: fixture/forge_tool_call_fixture.py, fixture/fixture.sha256, fixture/max-tokens-fixture-patch.txt
  • Raw benchmark summaries: feature/{0.8b,4b}/raw-openai/{off,enforce}/max_tokens_{256,512}/{summary.json,tool-result-summary.json}
  • Readiness/probe artifacts: feature/readiness/*
  • Host artifact roots: /home/ndizazzo/mesh-runs/forge-tool-feature-bench-20260522/{preflight,bin,off-control,enforce,cleanup} on each host
  • Shared conversation log: .opencode/multi-team/sessions/forge-tool-feature-bench-20260522/conversation.jsonl

Post-Run Learning

  • carrack: .opencode/multi-team/expertise/carrack-mental-model.yaml updated during enforce; cleanup found no new durable lesson.
  • lemony: .opencode/multi-team/expertise/lemony-mental-model.yaml updated during enforce and cleanup; YAML validation passed.
  • lemony29: .opencode/multi-team/expertise/lemony29-mental-model.yaml updated during build/cleanup; YAML validation passed.
  • coordinator: .opencode/multi-team/expertise/coordinator-mental-model.yaml updated with the observed feature-branch guardrail startup flag/status naming distinction; YAML validation passed.

Recommendation

Use the 512-token setting for this fixture on both target models: it eliminates hard-schema failures on 4B and avoids the 256-token enforce HTTP-400 tradeoff. For 0.8B, enforce mode helps the 256-token nested-object family but does not improve 512-token aggregate accuracy. If the goal is to demonstrate feature value, focus follow-up analysis on 0.8B/256 hard-schema repair and on whether HTTP 400 enforcement should be counted separately from tool-call accuracy in dashboards.

@michaelneale

Copy link
Copy Markdown
Collaborator

I like idea of this. Would like to see it with some other harnesses and models, not just tiny ones to see if net benefit. If so I think its own crate makes sense as this is essentially a port to rust of "forge", and we could even break it out as a crate for other rust projects to use. I'm not sure where the threshold is for applying this to models but worthwhile trying a bit more with harnesses imo.

@ndizazzo
ndizazzo force-pushed the feature/forge-inspired-model-nudging branch from 8d05e19 to a9b9393 Compare May 23, 2026 01:42

@michaelneale michaelneale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally — found a blocking bug in enforce mode

Tested PR #645 by building locally, running mesh-llm serve --mesh-guardrails enforce with Qwen/Qwen2.5-3B-Instruct, and running typical agent-style requests.

Blocking bug: the standard agent loop breaks under enforce 🛑

Repro:

# Turn 1: model emits a tool call
chat([{role:user, content:"What is 5+5?"}], tools=[calc]) -> tool_calls: calc({x: "5+5"})

# Turn 2: agent returns tool result. Tools kept in request (OpenAI convention).
chat([..., {role:tool, content:"10"}], tools=[calc])
# -> HTTP 400 guardrail_validation_failed

Reproduced 3/3 with two different prompts (recipe-read, calculator). Same payload without tools on turn 2 succeeds; same payload with --mesh-guardrails disabled succeeds.

Root cause

rescue.rs::classify_response line ~96: when request_expects_guarded_contract is true (i.e. request includes tools), any plain-text model output is classified as MalformedToolText and retried. The classifier doesn't account for the legitimate case "model is answering after consuming a tool result" — which is exactly when tools is still in the request but the correct response is natural language.

request_expects_guarded_contract just checks has_real_tools() without looking at conversation state. It needs to know that if the last message is a role: tool result, a text answer satisfies the contract.

@michaelneale

Copy link
Copy Markdown
Collaborator

I think the critical one is that it shouldn't break multi turn

@ndizazzo

Copy link
Copy Markdown
Collaborator Author

I think the critical one is that it shouldn't break multi turn

Odd, this didn't turn up during my testing... Lemme investigate

@michaelneale

Copy link
Copy Markdown
Collaborator

@ndizazzo haven't had a chance to test but this fixed the last issue with multiple turns? If so I think this is ok as it is low risk and may make it better?

@ndizazzo
ndizazzo force-pushed the feature/forge-inspired-model-nudging branch from ccb4f0b to dbd4ca6 Compare May 25, 2026 02:14
@ndizazzo

Copy link
Copy Markdown
Collaborator Author

One-Node Multi-Turn Tool-Call Regression Report

Summary

Carrack passed the one-node multi-turn tool-call regression for branch feature/forge-inspired-model-nudging at commit feb03c7938a27214ac0602e6ebaa061689838061 using model unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL. The raw OpenAI-compatible multi-turn probe and coordinator-local OpenCode smoke both completed through the Carrack gateway at http://carrack.patio51.com:9337/v1. Successive tool-call failure is not present in this run: after a tool result, the model returned final assistant text and did not emit another tool_calls array.

Results

Host Status Git Ref Binary Runtime Key Evidence
carrack PASS feature/forge-inspired-model-nudging @ feb03c7938a27214ac0602e6ebaa061689838061 mesh-llm 0.66.0 one serving node, model unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL, API 9337, console 3131, guardrails enforce just build exit 0; raw API probe passed; OpenCode attempt 2 passed; final status 58/58 successful requests; cleanup completed

Run Control

  • Run ID: 20260525T042423Z-one-node-multi-turn-tools
  • Policy: degraded-capable per-host execution; final scope narrowed to Carrack only.
  • Decision: continued with Carrack only.
  • Blocker trigger: none for Carrack.
  • Cleanup after run: completed on Carrack; no force kill, model deletion, or persistent TOML changes.

Progress

  • TODOs used: yes.
  • Phases completed: run-control classification, build, startup/readiness, raw OpenAI-compatible probe, OpenCode smoke, cleanup/report, post-run learning.
  • Last progress result: Carrack artifact/timing extraction completed at 2026-05-25T05:18:03.863Z.

Metrics

Host Metric Value
carrack build duration 8s artifact elapsed; Cargo reported 6.86s
carrack startup readiness API/web ready ~2.2s after first startup log; model/runtime ready ~6.5s after first startup log
carrack readiness after --listen-all ready on attempt 1, elapsed 0s
carrack request count 58
carrack successful requests 58
carrack retry count 0
carrack failover count 0
carrack average throughput 33.498 tokens/s
carrack completion tokens observed 14,071
carrack local service share 1.0
carrack peer count 0
carrack cleanup duration ~2s
coordinator OpenCode attempt 2 wall time 34.22s

OpenCode Client Phase

  • Runner: coordinator local.
  • Provider: mesh-test -> http://carrack.patio51.com:9337/v1.
  • Model: mesh-test/unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL.
  • OpenCode path/version: /Users/ndizazzo/.nvm/versions/node/v24.3.0/bin/opencode, version 1.15.10.
  • Result: PASS on attempt 2.
  • Evidence: .opencode/multi-team/opencode-runs/20260525T042423Z-one-node-multi-turn-tools/logs/opencode-multi-turn-attempt2.jsonl, exit file opencode-multi-turn-attempt2.exit = 0, summary artifacts/opencode-summary-attempt2.txt with json_lines=16, tool_mentions=91, final_marker=True.
  • Attempt 1 caveat: ran ~624.97s without final marker and wandered into unrelated tools; attempt 2 constrained fixture paths/tools and passed.

Raw OpenAI-Compatible Probe

  • Runner: coordinator local through Carrack gateway.
  • Result: PASS.
  • Evidence root: .opencode/multi-team/opencode-runs/20260525T042423Z-one-node-multi-turn-tools/.
  • Log marker: MULTI_TURN_TOOL_CALL_RAW_API_OK in logs/multi-turn-tool-probe.log.
  • Behavior proven: deterministic latest-message-is-tool request returned HTTP 200 with non-empty assistant text and no final tool_calls; first end-to-end turn produced one tool call; second after-tool-result request returned text/no tool calls.

Successive Tool-Call Failure State

  • Definitive state: PASS / not reproduced on Carrack for this branch, commit, model, and gateway path.
  • The specific failure mode under test was a successive tool-call loop or malformed follow-up where the assistant emits another tool_calls response after receiving a role: tool result.
  • The deterministic raw probe disproved that failure mode for this run: the after-tool-result response was HTTP 200, contained non-empty assistant text, and contained no final tool_calls.
  • The end-to-end raw sequence also passed: first turn produced exactly one tool call; second turn, after the tool result was supplied, resolved to normal assistant text with no additional tool call.
  • The OpenCode smoke independently passed on attempt 2 through the same Carrack gateway, confirming the agent-style client path could complete the multi-turn tool workflow.
  • Final Carrack runtime evidence showed 58/58 successful requests, retry_count=0, failover_count=0, and the guardrail/error scan found no guardrail, malformed, reject, retry, or error matches.
  • Conclusion: for the tested configuration, successive tool-call failures are resolved/not observed. Any future recurrence should be treated as a new regression and compared against the artifacts listed below.

Failures Or Risks

  • Carrack: no test failure. Public gateway initially required restarting the one-node serve command with --listen-all; after that, http://carrack.patio51.com:9337/v1/models was reachable.
  • Carrack: broad pgrep -fc target/debug/mesh-llm counted plugin child processes; pgrep -fc 'target/debug/mesh-llm serve' correctly showed one serving node.
  • Report path caveat: the original report.md was generated on Carrack at /home/ndizazzo/dev/mesh/mesh-local-testing/runs/20260525T042423Z-one-node-multi-turn-tools/report.md; this local copy is at /Users/ndizazzo/dev/mesh/mesh-local-testing/runs/20260525T042423Z-one-node-multi-turn-tools/report.md.

Commands Run

# Remote Carrack build/startup/cleanup evidence was captured by the Carrack agent.
just build
./target/debug/mesh-llm serve ... --listen-all --mesh-guardrails enforce
curl http://127.0.0.1:3131/api/status
curl http://127.0.0.1:9337/v1/models
curl http://carrack.patio51.com:9337/v1/models
./target/debug/mesh-llm stop

# Coordinator-local client phases used the Carrack gateway.
python3 multi_turn_tool_probe.py
opencode run --pure --format json --model mesh-test/unsloth/Qwen3.5-0.8B-GGUF:UD-Q8_K_XL --dir <run-workdir> <prompt>

Artifacts

  • Local report path: /Users/ndizazzo/dev/mesh/mesh-local-testing/runs/20260525T042423Z-one-node-multi-turn-tools/report.md.
  • Remote Carrack run root: /home/ndizazzo/dev/mesh/mesh-local-testing/runs/20260525T042423Z-one-node-multi-turn-tools.
  • Remote Carrack key artifacts: manifest.env, artifacts/git-branch.txt, artifacts/git-head.txt, artifacts/mesh-llm-version.txt, artifacts/status.json, artifacts/status-final.json, artifacts/models.json, artifacts/models-final.json, artifacts/guardrail-log-scan.txt, logs/just-build.log, logs/mesh-llm-node.log, logs/phase5-final-capture.log, logs/phase5-cleanup.log, logs/phase5-report-rewrite.log, report.md.
  • Coordinator client run root: .opencode/multi-team/opencode-runs/20260525T042423Z-one-node-multi-turn-tools/.
  • Coordinator key artifacts: logs/multi-turn-tool-probe.log, artifacts/deterministic-after-tool-result.json, artifacts/first-tool-call.json, artifacts/second-after-tool-result.json, work/opencode.json, logs/opencode-multi-turn-attempt2.jsonl, logs/opencode-multi-turn-attempt2.time, artifacts/opencode-summary-attempt2.txt.

Post-Run Learning

  • Coordinator: updated .opencode/multi-team/expertise/coordinator-mental-model.yaml with Carrack-gateway routing for OpenCode/raw probes and more reliable OpenCode prompts for small Qwen models.
  • Carrack: updated its host expertise for process-count/listen-all/report-writing lessons during the run; later read-only timing extraction found no additional durable lessons.

Recommendation

Keep using coordinator-local client phases through the Carrack gateway for OpenCode/raw OpenAI-compatible validation. For future small-model OpenCode smoke tests, use exact absolute fixture paths and explicitly forbid unrelated tools to avoid agent drift.

@ndizazzo
ndizazzo merged commit 6631cc0 into main May 25, 2026
21 checks passed
@ndizazzo
ndizazzo deleted the feature/forge-inspired-model-nudging branch May 25, 2026 05:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request waiting on review Non-author review required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants