Add KV/tool-loop stability certification harness

[IvGolovach]

Summary

Adds a standalone KV/tool-loop stability certification harness for live OpenAI-compatible mesh endpoints.

The harness exercises the specific failure shape from recent Skippy KV/tool-loop work: repeated non-streaming tool-result continuations against a long stable prefix, same-prefix/different-tail cache reuse, exact-prefix cache reuse, and optional Skippy native-log scanning for fatal KV/decode signals.

Why

The broad nightly stability harness is useful for trend and endpoint health, but it does not give a focused lab proof for the KV/cache failure modes behind repeated agent tool loops. This PR adds a narrow certification tool maintainers can run against a real direct-model Skippy endpoint before relying on Goose, Pi, OpenCode, or release-candidate agent workloads.

The target regressions are concrete: llama_decode failed, failed to find a memory slot, unexpectedly low usage.prompt_tokens_details.cached_tokens on same-prefix/different-tail prompts, proactive eviction errors in native logs, and cached responses that report good cache metrics while returning the wrong completion.

Diff scope

• Adds scripts/qa-kv-tool-loop-stability.py, a stdlib-only importable CLI probe with --print-plan support.
• Runs a growing two-tool-call conversation with pressure turns and final recall of deterministic fixture values.
• Adds same_prefix_cache and exact_prefix_cache checks using OpenAI usage cache metrics.
• Validates cache-probe completion text as well as cache metrics, so high cached_tokens cannot hide an incorrect cached answer.
• Adds optional --native-log scanning for slot exhaustion, decode failures, and proactive eviction errors appended after the certification starts.
• Preserves physical native-log file line numbers when scanning from a checkpoint offset.
• Writes repeatable evidence: manifest.json, results.jsonl, summary.json, summary.md, and sanitized transcript JSONL.
• Resets transcripts/ at run start so repeated latest runs do not mix stale attempt records into fresh evidence.
• Includes execution-affecting options in --print-plan / manifest.json: pressure_turns, timeout_seconds, suffix_prefill_limit, and native-log scan mode.
• Adds repo-local agent guidance in .agents/skills/kv-tool-loop-stability/SKILL.md.
• Documents the certification flow in docs/design/TESTING.md.

How to run

scripts/qa-kv-tool-loop-stability.py \
  --base-url http://127.0.0.1:9337/v1 \
  --models Qwen/Qwen2.5-3B-Instruct-GGUF:q4_k_m \
  --attempts 5 \
  --pressure-turns 8 \
  --timeout 180 \
  --min-cached-tokens 2048 \
  --suffix-prefill-limit 256 \
  --native-log ~/.mesh-llm/runtime/<pid>/logs/skippy-native.log \
  --output-dir target/kv-tool-loop-stability/local

--print-plan is side-effect-free and emits the exact checks, models, thresholds, runtime options, native-log scan mode, and evidence files before running a live endpoint.

Compatibility

Verification tooling and docs only. No runtime behavior, protobuf, gossip, mesh protocol, Skippy ABI, workflow, or required CI gate changes.

This intentionally avoids workflow YAML so it does not collide with active CI optimization work and remains a lab/release-confidence check rather than a new PR blocker.

Branch integrity

• Base branch: main
• Validated base: origin/main@d50a4683
• Branch state after rebase/update: 0 behind / 3 ahead
• Head commit: affa268fb3652711f0e7dbba0041d142ae006efa

Validation

• Validation tier: Tier 4 - verification tooling and testing documentation for KV/tool-loop stability certification, with post-review correctness/log-line evidence repairs; no runtime, protocol, workflow, or CI gate change.
• git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at d50a4683.
• git rebase origin/main: PASS
• git diff --check origin/main...HEAD: PASS, no output
• git diff --check: PASS, no output
• git diff --cached --check: PASS, no output
• python3 -m unittest scripts.tests.test_qa_kv_tool_loop_stability: PASS, 16 passed
• python3 -m unittest discover -s scripts/tests: PASS, 36 passed
• python3 -m py_compile scripts/qa-kv-tool-loop-stability.py scripts/tests/test_qa_kv_tool_loop_stability.py: PASS
• python3 scripts/qa-kv-tool-loop-stability.py --models auto,mesh --attempts 1 --pressure-turns 2 --timeout 30 --min-cached-tokens 128 --suffix-prefill-limit 64 --output-dir target/kv-tool-loop-stability/review-smoke --print-plan > /tmp/kv-tool-loop-plan.json && python3 -m json.tool /tmp/kv-tool-loop-plan.json >/dev/null: PASS
• Remote PR Build/Quality checks on affa268fb3652711f0e7dbba0041d142ae006efa: PASS for changes and PR Quality summary; Rust/build/live-smoke contexts were skipped by workflow path selection for this scripts/docs-only update.
• Ledger: not applicable - not required for selected validation tier/change family.
• Version: not applicable - verification tooling/docs only; no release/version sync required.
• Not run: cargo check/tests - not required for selected validation tier; no Rust/runtime code changed.
• Not run: live direct-model Skippy certification run - no local loaded direct-model endpoint was available; this PR adds and hardens the reproducible harness and local unit/plan proof.

Review status

Pending reviewer re-review after the earlier CHANGES_REQUESTED review. All known actionable review comments have code/test coverage in the updated head.

Rollback

Revert this PR.

DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.

Known residual risks

The live certification should still be run by a maintainer or lab host with the affected direct-model Skippy endpoint and native log path available. Local validation proves the harness behavior, evidence contract, side-effect-free plan output, and post-review evidence repeatability; it does not claim a specific live model endpoint is now stable.

[i386]

Very nice. Can we get an agent skill for this too please?

[i386]

Codex review comments below.

[IvGolovach]

I pushed a focused follow-up for the review notes here.

The harness now checkpoints --native-log files at the start of the certification and only scans bytes appended during that run, so old llama_decode failed / slot exhaustion lines in long-lived logs won’t poison fresh evidence. I also reset transcripts/ at run start so repeated latest runs stay auditable, and expanded --print-plan / manifest.json with the run-shaping options: pressure_turns, timeout_seconds, suffix_prefill_limit, and the native-log scan mode.

I added a repo-local agent skill for this too: .agents/skills/kv-tool-loop-stability/SKILL.md, plus targeted regression coverage for the log checkpoint behavior, transcript reset behavior, and plan/manifest fields.

Local validation passed, and the updated PR checks are green on the new head. 🙂