Make auto routing fall back to healthy models

[IvGolovach]

Summary

model=auto now avoids choosing a model whose only available targets are cooling, unavailable, or too small for the request context when another suitable model can serve the prompt. Explicit requests for a named model keep the existing availability-preserving behavior, but auto routing gets a stricter pre-selection path so agent/chat traffic can fall through to a healthier model instead of repeatedly spending turns on a known-bad top pick.

Fixes #625.

Why

#625 describes the public-mesh failure mode directly: auto could classify a prompt, pick the preferred model, and then keep routing to that model even when its only peer had just failed. The existing target-health filter intentionally preserved availability for explicit model requests, but auto routing needed model-level admission before it committed to a model.

The review pass also caught an important stickiness edge: a cached model=auto session should not abandon its current model merely because that model is cooling if every alternative is cooling too. In that case switching models does not buy a healthier route and can break agent/tool-loop KV locality.

Diff scope

• Adds a strict target-health path for auto routing while keeping the existing lenient path for explicit model routes.
• Adds an auto-route admission module that checks both request context budget and target health before a model participates in auto selection.
• Applies the same ready-model pool to both passive mesh OpenAI routing and local API ingress routing.
• Reclassifies cached auto-model sessions only when the cached model is missing, cannot satisfy media requirements, or a non-empty ready-model pool excludes it.
• Preserves cached auto-model stickiness when no ready alternate model exists.
• Adds regression coverage for single-candidate cooldown, all-cooling fallback behavior, ready-model pool selection, cached all-cooling stickiness, and switching to a ready alternate.

Behavior notes

• If at least one auto candidate has a healthy/context-compatible target, auto selects from that ready pool.
• If every candidate is cooling or otherwise ineligible, auto preserves availability by falling back to the original candidate pool instead of manufacturing an immediate empty-route failure.
• Cached auto sessions keep their model when the ready pool is empty; they switch only when another compatible model has a proven eligible target.
• Missing cached models and media-incompatible cached models are still hard reclassification reasons.
• ContextOverflow, 4xx rejections, and client disconnects still do not poison target health; only retryable timeout/unavailable outcomes cool targets.

Compatibility

No mesh protocol, Skippy ABI, package format, release metadata, or CLI contract changes. This is process-local routing behavior only; no new gossip fields or mixed-version incompatibility.

Validation

Validation tier: Tier 3 - shared OpenAI routing behavior for agent/model auto selection, refreshed on current main and updated for the cached auto-model stickiness review fix.

• git fetch --no-tags origin main:refs/remotes/origin/main: PASS, origin/main at 6d2f6b104f0ebc63f933fced1d840bacbdc15b34
• git rebase origin/main: PASS, no conflicts
• git diff --check origin/main...HEAD: PASS, no output
• git diff --check: PASS, no output
• git diff --cached --check: PASS, no output
• cargo fmt --all -- --check: PASS
• LLAMA_STAGE_BUILD_DIR=<llama-stage-build-dir> cargo test -p mesh-llm-host-runtime openai::transport --lib -- --test-threads=1: PASS, 63 passed
• LLAMA_STAGE_BUILD_DIR=<llama-stage-build-dir> cargo test -p mesh-llm-host-runtime auto_route --lib -- --test-threads=1: PASS, 2 passed
• LLAMA_STAGE_BUILD_DIR=<llama-stage-build-dir> cargo test -p mesh-llm-host-runtime target_health --lib -- --test-threads=1: PASS, 11 passed
• LLAMA_STAGE_BUILD_DIR=<llama-stage-build-dir> cargo test -p mesh-llm-host-runtime strict_eligible --lib -- --test-threads=1: PASS, 1 passed
• LLAMA_STAGE_BUILD_DIR=<llama-stage-build-dir> cargo check -p mesh-llm: PASS
• LLAMA_STAGE_BUILD_DIR=<llama-stage-build-dir> cargo-clippy clippy -p mesh-llm-host-runtime --all-targets -- -D warnings: PASS

Ledger: not applicable - not required for selected validation tier/change family.
Version: not applicable - no release/version sync required for this non-release routing behavior change.

Not run: live multi-node agent smoke. I did not have a local model/runtime mesh endpoint available; deterministic routing-health and auto-route unit coverage plus shipped-binary check/clippy cover the changed paths.

Rollback

Revert this PR.

DB downgrade: not applicable.
Data repair: not applicable.
Operational caveats: none known.

Known residual risk

This should still get normal PR CI and one real public/private mesh smoke with model=auto after opening: force one preferred model target into cooldown/unavailable state, confirm auto falls through to another compatible model, then confirm the preferred model can participate again after health recovers.

[i386]

Thanks for the routing-health work here. I think this needs one behavior fix before merge.

[P2] Preserve cached auto-model stickiness when no alternate ready model exists

cached_auto_model_reclassify_reason now forgets the cached auto model whenever model_has_eligible_remote_host(...) is false. That check runs before the later ready-pool fallback in auto_route::pool_for_ready_models, which intentionally preserves availability by falling back to the full candidate set when no model has a ready target.

Concretely, if the cached model is cooling and every other candidate is also cooling, this path forgets the session cache anyway and lets reclassification choose another cooling model. That contradicts the stated behavior of switching only when another model has a healthy target, and it can break agent/tool-loop model stickiness without gaining any healthier route.

Please defer health-based cached-model invalidation until after confirming there is at least one alternate ready model for this request, or otherwise preserve the cached model when the ready set is empty.

Validation note: I attempted cargo test -p mesh-llm-host-runtime auto_route --lib -- --test-threads=1, but this local checkout did not have the prepared llama.cpp static ABI artifacts, so skippy-ffi failed before tests with missing llama-common.