diff --git a/docs/NATIVE_RUNTIME_HIVE_VALIDATION_PLAN.md b/docs/NATIVE_RUNTIME_HIVE_VALIDATION_PLAN.md new file mode 100644 index 00000000..2ad218e5 --- /dev/null +++ b/docs/NATIVE_RUNTIME_HIVE_VALIDATION_PLAN.md @@ -0,0 +1,162 @@ +# Native Runtime — Multi-Machine HIVE Validation Plan (§6 entry criteria) + +> **Status:** Plan only — this document designs the validation campaign for +> [NATIVE_RUNTIME_V2_SPEC.md §6](NATIVE_RUNTIME_V2_SPEC.md)'s default-runtime-flip entry +> criteria. Executing this plan produces the **evidence**; it does not itself authorize the +> flip. Per §6, the flip additionally requires an explicit, recorded product decision — that +> decision belongs to the maintainer, not to this plan or any green run of it. +> +> **Fleet surveyed live 2026-07-20** (SSH, `nvidia-smi`, `git rev-parse` on each box — not +> assumed). All three machines reachable over SSH with key auth; both remotes synced to +> `master` @ `aa07f41` during the survey. + +## 1. Fleet under test + +| Machine | Role | GPU / VRAM | Driver | Repo path | Notes (verified) | +|---|---|---|---|---|---| +| **NewcorePC** | Warchief + reference worker | RTX 5070 Ti, 16 GB | (reference box) | `F:\Ai\OrchestratorIDE-dev` | All Phase A–D evidence to date produced here. Test GGUF already present. | +| **HardcorePC** | Worker, low-VRAM class | RTX 3050, 6 GB | 560.94 | `F:\Ai\OrchestratorIDE-dev` | ⚠ Known unresolved native-lib regression (CF-7 gate broken since the 2026-07-04 clean rebuild; root cause never found). ~4 TB free. | +| **HardcoreLaptopMSI** | Worker, mobile class | RTX 4060 Laptop, 8 GB | 581.80 | `C:\Ai\OrchestratorIDE-dev` | CUDA-toolkit redist DLLs not present as system libs — the `OrchestratorIDE.NativeRuntime.csproj` bundling path (build-box `CUDA_PATH`) must be verified here or binaries deployed from NewcorePC. Only ~18 GB free on C:. | + +The 16 / 8 / 6 GB VRAM spread and the driver spread (560.94 vs 581.80) are deliberate assets: +§6 asks for "representative hardware," and this fleet genuinely represents the desktop-large / +laptop-mid / desktop-small classes TheOrc targets. Record GPU model + driver + VRAM in every +evidence artifact. + +**Pinned model fixture (distributed during the survey):** +`Dolphin3.0-Llama3.2-3B-Q4_K_M.gguf`, +SHA-256 `5d6d02eeefa1ab5dbf23f97afdf5c2c95ad3d946dc3b6e9ab72e6c1637d54177`. +Locations: NewcorePC `F:\Ai\GarfChat\checkpoints\android-test-models\`; HardcorePC +`F:\Ai\models\`; HardcoreLaptopMSI `C:\Ai\models\`. Hash must be re-verified on each box +before any phase runs (a corrupted copy invalidates cross-machine comparisons). + +## 2. Criteria → phases map + +Each §6 bullet maps to exactly one campaign phase below; a phase is DONE only with retained +evidence (same discipline as the Phase D E2E lane's `.orc/native-e2e-lane/` artifacts). + +| §6 entry criterion | Phase | +|---|---| +| Successful native workloads across the intended machine roles | HV-1 | +| Correct capability- and resource-aware scheduling across machines | HV-2 | +| Verified model and adapter lifecycle behavior across machines | HV-3 | +| Failure, cancellation, disconnect, and recovery exercises across machines | HV-4 | +| Consistent telemetry and diagnosability across machines | HV-5 | +| No silent fallback across runtime boundaries | HV-4 + HV-5 (asserted in every phase) | +| Repeatable end-to-end evidence on representative hardware | HV-6 | +| Explicit product decision | **Out of scope for this plan** — maintainer's call, recorded separately | + +## 3. Phases + +### HV-0 — Fleet readiness (preconditions; no §6 credit) + +1. **Merge gate:** PR #81 (E2E lane + evidence store) and #82 (docs sync) merged; all three + boxes on the same `master` commit. (Repos synced already; re-sync after merges.) +2. **Model gate:** pinned GGUF hash verified on all three boxes. +3. **Build gate:** `dotnet build` green on each box, and the CUDA backend actually selected on + each — `NativeBackendBootstrap`'s backend report per box, not assumed. On HardcoreLaptopMSI + this specifically verifies the CUDA redist bundling; if the local build lacks `CUDA_PATH`, + deploy binaries built on NewcorePC instead and record which path was taken. +4. **HardcorePC regression gate (the known blocker):** the unresolved native-lib regression + must be root-caused and fixed — or explicitly waived with the box excluded — before HV-1. + Diagnosing it IS the campaign's first real "diagnosability" exercise: use + `THEORC_KVCACHE_DIAGNOSTICS=1`, the backend report, and the Phase D lane run locally on + that box. HardcorePC has Claude CLI installed — prefer running diagnosis there directly + over command-by-command SSH. +5. **Single-box lane gate:** the PR #81 E2E lane green on each box with retained evidence — + three per-box artifacts recording tok/s, TTFT, measured VRAM. This is the per-box floor + under every multi-machine phase. + +### HV-1 — Native workloads across machine roles + +Warchief (NewcorePC) dispatches real native-role campaign jobs to both workers through the +existing HIVE campaign contracts (leasing, persistence, verification — already in code) using +the fleet's `hive-peers.json` (use the auto-resync tool, not hand edits). + +- Each worker executes real native generations for jobs it did not originate. +- Evidence per job: runtime name (`NativeRoleRuntime`), machine identity, binding, output, + stats — and **no fallback marker anywhere**. Native campaign jobs already fail closed by + design; this proves it holds when dispatched remotely. +- Pass: N≥5 jobs per worker, all native, zero fallback, evidence retained per job. + +### HV-2 — Capability/resource-aware scheduling + +Exploit the VRAM spread deliberately: submit jobs whose context-aware footprint (PR #76 +estimator, large `ContextLength`) fits 16 GB and 8 GB but must be **denied** on 6 GB. + +- HardcorePC must deny with a real `RuntimeAdmissionDeniedException` (correct numbers in the + reason), observable in its telemetry (`RejectedAdmissionCount`, `LastRejectionReason`). +- The Warchief must respect capability/placement — the denied job either lands on a box that + fits or fails visibly; it must never silently reroute to Ollama. +- Also run the inverse: a small job admitted on all three, proving denial is footprint-driven, + not box-driven. + +### HV-3 — Model/adapter lifecycle across machines + +- Sequential load → generate → dispose cycles per worker; residency (`ActiveCount`) returns to + baseline between jobs; reservation behavior matches the documented decoupling (reservation + persists with the loaded model; residency does not). +- Concurrent second role on the same worker — folds in the deferred Phase D "second concurrent + role" increment: cross-role admission accounting proven inside one evidence-bearing run. +- Rebind/recycle: force a role recycle (the PR #79 `MarkRoleDegraded` path) remotely and prove + the next job on that role gets a fresh, working executor. + +### HV-4 — Failure, cancellation, disconnect, recovery + +All on real jobs mid-flight, all asserting fail-closed (no Ollama substitution) and clean +recovery: + +1. **Remote cancellation** mid-generation → `OperationCanceledException` surfaces to the + dispatcher; worker role reusable afterward (PR #79's fix, now proven cross-machine). +2. **Worker process kill** mid-job → Warchief detects the death, job fails **visibly**, + lease/queue behavior correct; worker restarts and rejoins (auto-resync), then serves a new + job. +3. **Network disconnect** (temporarily block the HIVE port / drop the link) mid-campaign → + same visibility + recovery expectations; no half-spliced outputs. +4. **Ollama-absent worker**: with Ollama stopped on a worker, a native-routed job still runs + natively; a deliberately broken native config on that worker fails closed with an explicit + native error — CF-6's "Ollama-absence death test" precedent, now fleet-wide. + +### HV-5 — Telemetry consistency + no-silent-fallback sweep + +- Same evidence JSON schema from all three boxes for one shared campaign; per-box + reservation/residency/measured-VRAM snapshots collected centrally. +- Log sweep across all three boxes: zero silent-fallback markers, plus the standing + `NoKvSlot` grep before trusting any numbers. +- Diagnosability drill: for one induced failure per box, the retained diagnostics + (`error_type`, KV diagnostics, backend report) must be sufficient to identify the cause + without interactive debugging — measured by actually doing it. + +### HV-6 — Repeatability + fleet report + +- The full HV-1→HV-5 campaign, run **3× back-to-back**, all green, no manual intervention + between runs. +- Aggregate fleet report (per-box + campaign-level JSON and a human summary) retained as the + §6 evidence bundle. The report explicitly does NOT claim the flip — it presents the + evidence for the maintainer's §6 decision. + +## 4. Harness shape (implementation guidance, not code) + +- **Driver:** a `Tools/` PowerShell orchestration script on the Warchief (SSH for box-level + actions: build, kill, restart; HIVE contracts for job dispatch) — multi-machine flows do not + fit NUnit. Per-box, the existing gated NUnit lanes remain the single-box floor (HV-0.5). +- **Evidence:** extend the `NativeE2ELaneEvidenceStore` schema pattern with + `machine`, `gpu`, `driver_version` fields (a sibling fleet store, keeping `schema_version` + discipline); central collection via scp back to the Warchief. +- **Small, reviewed PRs as always:** evidence-store extension, then driver script, then + per-phase additions — not one mega-PR. + +## 5. Known blockers & risks (honest, current as of 2026-07-20) + +1. **HardcorePC native-lib regression** — unresolved since 2026-07-04; hard precondition + (HV-0.4). Until fixed, the fleet is effectively two machines and §6's "across machines" + evidence is weakened. +2. **PR #81/#82 unmerged** — the harness builds on #81's lane and evidence store. +3. **HardcoreLaptopMSI disk headroom** (~18 GB free): one pinned model is fine; do not stage + multi-model suites there without cleanup. +4. **CUDA redist on the laptop** — bundling depends on the build box's `CUDA_PATH`; a wrong + build silently lands on the CPU backend at ~1.7 tok/s, which HV-0.3's backend report check + exists to catch. +5. **Driver spread** (560.94 vs 581.80) — treated as representative, but if HardcorePC's + regression turns out driver-related, a driver update becomes part of HV-0.4 and must be + recorded in the evidence.