fix(buzz): missing CLI binary degrades platform instead of killing gateway (OOF-208)

[shannonsands]

Problem (OOF-208)

Five hosted Hermes Cloud instances fail gateway startup with a missing "Buzz CLI binary" (fingerprint hermes-cloud:prod:gateway-runtime-missing-buzz-cli). Two independent defects compound:

A (systemic — one optional binary kills the whole gateway): plugins/platforms/buzz/adapter.py::check_requirements() only checked BUZZ_RELAY_URL + private key, not the binary. On hosted images the registry constructed the adapter anyway → connect() hit _set_fatal_error("cli_missing", retryable=False) → with Buzz as the only connectable platform, gateway/run.py classified it into startup_nonretryable_errors → gateway_state=startup_failed, exit 78, s6 stops the restart loop. Same escalation class as NS-609/NS-541 (exit-78 on mixed startup failures, fixed in #70987) — this is the per-platform gating half.

B (packaging — the binary was never in the image): block/buzz publishes no prebuilt CLI artifacts (releases are desktop bundles only; no crates.io crate). The Dockerfile baked Photon sidecar deps but never built/copied the buzz binary.

Fix

File	Change
plugins/platforms/buzz/adapter.py	check_requirements() now gates on _resolve_cli_path() (warn-once latch — this runs on hot /api/status poll paths); explicit BUZZ_CLI_PATH takes precedence with no PATH fallback; connect()'s cli_missing fatal flipped to retryable=True as defense-in-depth against the check→connect race
gateway/run.py	No-adapter branch now writes per-platform runtime status (state=fatal, error_code=adapter_unavailable, needs_attention=true) — previously a requirements-failed platform was completely invisible in /api/status
Dockerfile	New buzz_build stage: digest-pinned rust:1.90-slim-bookworm, builds buzz-cli --locked from a 40-char commit SHA (= tag v0.5.2), binary to /usr/local/bin/buzz. Pure-rustls workspace — no OpenSSL, glibc-only, bookworm→trixie safe (same rationale as the node_source stage). Layer-cached on BUZZ_GIT_REF; ~2 min cold-cache Rust build, amd64+arm64 cross-compile clean

Fix A alone stops the outage class on old images (gateway degrades, platform parked with a visible error); Fix B actually brings Buzz up on the affected instances at next image release.

Validation

• test_buzz_adapter.py 30 passed (7 new: binary gating, BUZZ_CLI_PATH precedence, warn-once, retryable connect fatal)
• test_runner_startup_failures.py 8 passed (new: gateway stays running while parked platform surfaces adapter_unavailable)
• test_dockerfile_immutable_install.py 6 passed (contract: buzz stage exists, SHA-pinned not tag-pinned, --locked, binary on PATH)
• Adjacent suites (test_runner_fatal_adapter, test_platform_registry, test_startup_no_eager_platform_install, test_startup_connect_parallel): 46 passed
• ruff clean; hadolint --config .hadolint.yaml Dockerfile exit 0
• Build feasibility proven locally: cargo build --release -p buzz-cli --locked at the pinned SHA → 12MB self-contained binary in 1m51s

Rebased onto current main (fe8c9f1c92) before opening; all suites re-run post-rebase.

No live instances were mutated (per ticket instruction) — affected instances recover via image upgrade, or degrade gracefully immediately once a Fix-A build ships.

Linear: OOF-208

[shannonsands]

Pushed 7622d57 addressing the three review findings from the independent second-opinion review:

1. Config-aware check_requirements() — the gate only consulted BUZZ_* env vars, so a documented config.yaml-only setup (extra.cli_path / extra.credentials_file / extra.relay_url) got parked as adapter_unavailable even though BuzzAdapter(config) resolves those extras fine. The gate now reads the same YAML sources via a best-effort _config_yaml_extra() helper (lookup order matches gateway/config.py: top-level buzz: → platforms.buzz → gateway.platforms.buzz; env vars win; fail-quiet to env-only on any read error). Also bridges extra.credentials_file → BUZZ_CREDENTIALS_FILE in _apply_yaml_config — the one documented key the hook skipped.

2. Pinned Rust toolchain — the digest pin on the builder image didn't pin the compiler: the official rust images are rustup-managed and the pinned Buzz checkout ships rust-toolchain.toml (channel = "1.95.0"), so rustup silently downloaded whatever the checkout requested. Bumped the builder to rust:1.95-slim-bookworm@sha256:d748208… (matching the checkout's channel) and set RUSTUP_TOOLCHAIN=1.95.0 explicitly, so the effective compiler is exactly the baked one with no hidden network fetch — even if a future BUZZ_GIT_REF's toolchain file moves. Contract test now asserts the image version and toolchain pin agree.

3. Durable retry for missing binary — cli_missing being retryable was correct but not durable: the reconnect watcher evicted a platform from the retry queue whenever _create_adapter() returned None (check_fn failing), so an operator installing the binary while the gateway was up could never be noticed — recovery required a full restart. The watcher now applies the standard backoff (30s → 300s cap) and keeps re-probing (check_fn is cheap — pure filesystem/env checks, no network), surfacing platform_state=retrying / error_code=adapter_unavailable. Startup also queues unavailable-adapter platforms for retry instead of only marking them fatal. No noisy-loop risk: probes are passive, and the existing needs-attention escalation still flags perma-failing platforms.

Nit also taken: reworded the Dockerfile rustls comment — the dep tree includes aws-lc-rs/aws-lc-sys alongside ring, so the claim is now scoped to what's true (no OpenSSL/native-tls linkage).

Validation: 63 targeted tests pass (buzz adapter, reconnect watcher, Dockerfile contracts — 8 new), broader buzz or reconnect or startup selection 283 passed / 1 skipped, ruff clean.