From 611dfb5e6da3985314942e54e66d8bf5ad53046a Mon Sep 17 00:00:00 2001 From: hardcoreerik Date: Sun, 19 Jul 2026 21:32:16 -0700 Subject: [PATCH 1/2] docs: sync ROADMAP/CURRENT_STATE with Native Runtime v2.0 Phase B addendum + D Both files still described Phase B's cost-estimate half as deliberately deferred and Phase D as untouched -- stale since PR #76 (cost estimate), #77 (load-time measurement), #78 (negative fail-closed test), and #79 (cancellation-corruption fix) all landed since the last sync. Updates the Native Runtime v2.0 direction table in ROADMAP.md and the native_runtime note in CURRENT_STATE.yaml to reflect current reality: Phase B fully landed (estimate + measurement), Phase D essentially complete pending the E2E evidence-lane PR (#81, open). Co-Authored-By: Claude Sonnet 5 --- docs/CURRENT_STATE.yaml | 42 ++++++++++++++++++++++++----------------- docs/ROADMAP.md | 22 +++++++++++++-------- 2 files changed, 39 insertions(+), 25 deletions(-) diff --git a/docs/CURRENT_STATE.yaml b/docs/CURRENT_STATE.yaml index 848bc9aa..8ec5764e 100644 --- a/docs/CURRENT_STATE.yaml +++ b/docs/CURRENT_STATE.yaml @@ -84,24 +84,32 @@ subsystems: current/default/fallback picture per workload, including the real NativeWithFallbackRuntime mechanics and the one open gap it documents (benchmark reports record RuntimeName as a static label, not per-call - fallback counts). As of 2026-07-19, docs/NATIVE_RUNTIME_V2_SPEC.md's - foundation-hardening Phases A-C are landed, with one deliberate gap: - VRAM admission is a single authoritative, fail-closed boundary - (AdapterManager's conversation-minting is internal, reachable only - through RuntimeOrchestrator's gate; a missing budget now denies rather - than silently loading unadmitted); the admission budget's TOTAL/USED - read is now live via nvidia-smi, not a static snapshot -- but the - per-role VRAM cost ESTIMATE (OrcScheduler.EstimateRequiredBytes) is - still GGUF-file-size-only, not KV/rs-cache-aware; that half of Phase B - was deliberately deferred (PR #72) rather than shipping an ungrounded - generalization, and remains open alongside Phase D. RuntimeOrchestrator/ + fallback counts). As of 2026-07-20, docs/NATIVE_RUNTIME_V2_SPEC.md's + foundation-hardening Phases A-C are landed: VRAM admission is a single + authoritative, fail-closed boundary (AdapterManager's conversation- + minting is internal, reachable only through RuntimeOrchestrator's gate; + a missing budget now denies rather than silently loading unadmitted); + the admission budget's TOTAL/USED read is live via nvidia-smi; the + per-role VRAM cost ESTIMATE (OrcScheduler.EstimateRequiredBytes) is now + KV/rs-cache-aware (a byte-exact GGUF-header formula, arch-gated for + recurrent architectures) rather than file-size-only -- the deferred half + of Phase B landed too (PR #76), plus a real load-time VRAM MEASUREMENT + parsed from llama.cpp's own log lines (PR #77), replacing the WDDM-dead + nvidia-smi per-process read Phase C originally shipped. RuntimeOrchestrator/ AdapterManager telemetry (reservation state, adapter residency, - rejected-admission counts, measured per-process VRAM) is real and - partially surfaced in the Activity Log. This is foundation hardening, - not a default-runtime change -- Phase D (real-model native-path proof - lane), the deferred cost-estimate work, and the default-runtime flip - all remain open, the last one separately gated - future milestone per that spec's §6. + rejected-admission counts, measured VRAM) is real and partially surfaced + in the Activity Log. Phase D (real-model native-path proof lane) is + essentially complete: a negative fail-closed test (no silent Ollama + substitution on admission denial, PR #78), a cancellation-corruption bug + found and fixed (cancelling mid-generation could permanently break a + role's executor; fixed by extending the existing MarkForRecycle + mitigation to any failed generation, not just NoKvSlot, PR #79), and a + real-model E2E lane with a retained JSON evidence artifact (discovery + through real admission through inference through telemetry; + 25.7 tok/s, 284ms TTFT, 2.5GB measured VRAM on the reference box) are + built and verified on real hardware. This is foundation hardening, not + a default-runtime change -- the default-runtime flip remains a + separately gated future milestone per that spec's Section 6. llamacpp_server: status: opt-in diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 49445c21..71645249 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1,8 +1,11 @@ # TheOrc — Roadmap -> Last updated: 2026-07-19 (v1.13.0 — Context Fabric complete; since then, Native Runtime v2.0 -> foundation-hardening Phases A-C (admission boundary, live VRAM budget, real telemetry) landed -> as three separate PRs ahead of any new tagged release. See the Native Runtime section below.). +> Last updated: 2026-07-20 (v1.13.0 — Context Fabric complete; since then, Native Runtime v2.0 +> foundation-hardening Phases A-C (admission boundary, live VRAM budget, real telemetry) plus +> the Phase B addendum (KV/rs-cache-aware cost estimate + real load-time measurement) and most +> of Phase D (real-model proof: fail-closed negative test, a cancellation-corruption bug found +> and fixed, and an E2E lane with retained evidence) have landed as separate PRs ahead of any +> new tagged release. See the Native Runtime section below.). > This document is updated after every GitHub release, and incrementally between releases when a > section goes materially stale. It reflects actual code state, not aspirations — features marked > Shipped have been verified in the running app. @@ -651,11 +654,14 @@ and run as a service before the GUI-as-client changeover happens. > Contracts/design spec: [`docs/RUNTIME_PHASE0_SPEC.md`](RUNTIME_PHASE0_SPEC.md). Status changed after ORC ACADEMY v3 completed: early runtime groundwork has landed, but native runtime is still pre-production and not the default. > > Current foundation-hardening plan (the specific work the Phase 3/4 rows below flag as -> "remaining"): [`docs/NATIVE_RUNTIME_V2_SPEC.md`](NATIVE_RUNTIME_V2_SPEC.md). As of 2026-07-19, +> "remaining"): [`docs/NATIVE_RUNTIME_V2_SPEC.md`](NATIVE_RUNTIME_V2_SPEC.md). As of 2026-07-20, > Phases A (fail-closed admission boundary, closing the AdapterManager bypass), B (live VRAM -> budget), and C (real telemetry) are landed. Phase D (real-model native-path proof lane) is -> still open. None of this changes the default runtime or Ollama's role as fallback — see that -> spec's explicit out-of-scope list. +> budget, **and** the cost-estimate and load-measurement halves — both landed, not deferred +> anymore), and C (real telemetry) are landed. Phase D (real-model native-path proof lane) is +> essentially complete: the negative fail-closed test and the cancellation leg (which caught and +> fixed a real executor-corruption bug) are landed; the real-model E2E lane with a retained +> evidence artifact is built, verified on real hardware, and pending merge. None of this changes +> the default runtime or Ollama's role as fallback — see that spec's explicit out-of-scope list. **What it is:** an orchestration / swarm-aware layer *on top of* LLamaSharp (llama.cpp bindings) — **not** a from-scratch inference engine. llama.cpp owns the kernels; TheOrc owns scheduling, session management, adapter hot-swap, VRAM-aware dispatch, and direct Avalonia streaming. The moat is making the warband feel like one cohesive mind on the GPU instead of a series of independent HTTP calls. @@ -667,7 +673,7 @@ and run as a service before the GUI-as-client changeover happens. | **1** | `LlamaCppServerRuntime` — wraps the **existing** `LlamaServerManager` + `InferenceBackend.LlamaCpp`. | ✅ Landed — server lifecycle and HTTP routing exist behind the runtime interface. | Low | | **2** | `LLamaSharpRuntime` — in-process GGUF streaming, embedded-template probing, stats, shared text tool-call parsing. | ⚠️ Prototype landed — useful for validation, not production/default. LoRA hot-swap, backend install flow, and full runtime selection are not complete. | Med | | **2.5** | Close abstraction leaks from the first migration. | ✅ Closed for `HiveWorkerAgent` and reviewer inference — both use `IModelRuntime`; SwarmSession's Ollama-specific eviction escape hatch and remote HIVE task-queue/node HTTP remain separate follow-up plumbing. | Med | -| **3** | `ModelDepot` (local registry first; downloader later), `SessionManager` (persistent base model), `AdapterManager` (boss/worker/reviewer LoRAs), telemetry. | 🔶 Live opt-in proof path landed — ModelDepot, SessionManager, AdapterManager (per-role persistent executors, adapter attached once at creation per the §7 verdict), and `RuntimeOrchestrator` all landed; `IRoleRuntime`/`NativeRoleRuntime` now stream that role stack, `THEORC_TEST_GGUF` has an opt-in role-runtime smoke lane, Avalonia Settings exposes manual native smoke/fallback/evidence capture, and `HiveWorkerAgent` can opt into native role execution with logged fallback to the configured model runtime. Telemetry surfacing and the OrcScheduler/AdapterManager wiring below are now landed (NATIVE_RUNTIME_V2_SPEC.md Phases A/C). Remaining: keep proving the real-model path (Phase D); the per-role VRAM cost estimate is still GGUF-file-size-only, not KV/rs-cache-aware (Phase B's estimate half was deliberately deferred, not landed — see that PR); do not make native the main chat/swarm default yet. | Med-High | +| **3** | `ModelDepot` (local registry first; downloader later), `SessionManager` (persistent base model), `AdapterManager` (boss/worker/reviewer LoRAs), telemetry. | 🔶 Live opt-in proof path landed — ModelDepot, SessionManager, AdapterManager (per-role persistent executors, adapter attached once at creation per the §7 verdict), and `RuntimeOrchestrator` all landed; `IRoleRuntime`/`NativeRoleRuntime` now stream that role stack, `THEORC_TEST_GGUF` has an opt-in role-runtime smoke lane, Avalonia Settings exposes manual native smoke/fallback/evidence capture, and `HiveWorkerAgent` can opt into native role execution with logged fallback to the configured model runtime. Telemetry surfacing and the OrcScheduler/AdapterManager wiring below are now landed (NATIVE_RUNTIME_V2_SPEC.md Phases A/C). The per-role VRAM cost estimate is now KV/rs-cache-aware (a byte-exact GGUF-header formula, arch-gated for recurrent architectures) with a real load-time measurement on top parsed from llama.cpp's own log lines, replacing the WDDM-dead nvidia-smi per-process read — both landed (Phase B addendum). Real-model proof (Phase D) is essentially complete: negative fail-closed test, cancellation-corruption fix, and an E2E lane with a retained evidence artifact (25.7 tok/s, 284ms TTFT, 2.5GB measured VRAM on the reference box) are built and verified; do not make native the main chat/swarm default yet. | Med-High | | **4** | `OrcScheduler` — capability + VRAM + lane-aware dispatch; pipeline boss→workers. | 🔶 Started — interface + data model + a real VRAM-budget admission check landed; `RuntimeOrchestrator` now tracks active per-role reservations (generation-tagged, serialized through one admission gate) instead of a static zero-reserved snapshot, closing the over-admission gap the static check left open. NATIVE_RUNTIME_V2_SPEC.md Phase A closed the "not wired into AdapterManager" gap this row used to flag: `AdapterManager`'s conversation-minting methods are now `internal`, reachable only through `RuntimeOrchestrator`'s admission gate, and admission fails closed (not open) when no budget is configured. Phase B added a live nvidia-smi VRAM read in place of the static zero-reserved snapshot. Still no live GPU dispatch or pipeline queueing. | High | | **5** | *(Research, non-blocking)* prefix KV cache for the shared warband prompt; multi-LoRA cache experiments. | ✅ Research closed — `Conversation.Fork()`/`MemorySequenceCopy` is a real, cheap shared-prefix mechanism, confirmed via LLamaSharp's own XML docs; blocked for cross-role sharing because `SetLoraAdapters` is context-scoped, not per-sequence — same-role prefix forking remains a viable future win, see `.grok/PREFIX_KV_CACHE_RESEARCH.md`. | Research | From 6f86a27c867f9793faab0e90283354c761a4bd04 Mon Sep 17 00:00:00 2001 From: hardcoreerik Date: Mon, 20 Jul 2026 07:23:18 -0700 Subject: [PATCH 2/2] docs: mark E2E evidence lane landed (PR #81 merged), not pending CodeRabbit finding: ROADMAP.md's banner still said the E2E lane was "built, verified on real hardware, and pending merge" while CURRENT_STATE.yaml's parallel note omitted that qualifier entirely -- inconsistent, and now stale either way since PR #81 actually merged. Updated both to say landed. Co-Authored-By: Claude Sonnet 5 --- docs/CURRENT_STATE.yaml | 4 ++-- docs/ROADMAP.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/CURRENT_STATE.yaml b/docs/CURRENT_STATE.yaml index 8ec5764e..f7e44bd4 100644 --- a/docs/CURRENT_STATE.yaml +++ b/docs/CURRENT_STATE.yaml @@ -106,8 +106,8 @@ subsystems: mitigation to any failed generation, not just NoKvSlot, PR #79), and a real-model E2E lane with a retained JSON evidence artifact (discovery through real admission through inference through telemetry; - 25.7 tok/s, 284ms TTFT, 2.5GB measured VRAM on the reference box) are - built and verified on real hardware. This is foundation hardening, not + 25.7 tok/s, 284ms TTFT, 2.5GB measured VRAM on the reference box, PR + #81) has landed too, verified on real hardware. This is foundation hardening, not a default-runtime change -- the default-runtime flip remains a separately gated future milestone per that spec's Section 6. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 71645249..b6fdae84 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -660,7 +660,7 @@ and run as a service before the GUI-as-client changeover happens. > anymore), and C (real telemetry) are landed. Phase D (real-model native-path proof lane) is > essentially complete: the negative fail-closed test and the cancellation leg (which caught and > fixed a real executor-corruption bug) are landed; the real-model E2E lane with a retained -> evidence artifact is built, verified on real hardware, and pending merge. None of this changes +> evidence artifact (PR #81) has landed too, verified on real hardware. None of this changes > the default runtime or Ollama's role as fallback — see that spec's explicit out-of-scope list. **What it is:** an orchestration / swarm-aware layer *on top of* LLamaSharp (llama.cpp bindings) — **not** a from-scratch inference engine. llama.cpp owns the kernels; TheOrc owns scheduling, session management, adapter hot-swap, VRAM-aware dispatch, and direct Avalonia streaming. The moat is making the warband feel like one cohesive mind on the GPU instead of a series of independent HTTP calls.