feat(runtime): Phase C - real runtime telemetry

[hardcoreerik]

What this is

Implementation PR for Phase C of docs/NATIVE_RUNTIME_V2_SPEC.md §2/Phase C — surfaces production-meaningful telemetry from real SessionManager/AdapterManager state instead of placeholders. Follows Phase A (#70) and Phase B (#72), both merged.

Will get an independent review (CodeRabbit) before merge, per the standing rule for Native Runtime PRs.

What ships

• SessionManager: renamed HasPendingAdapter → HasBoundAdapter, corrected the class doc and load message. SessionManager never touches adapter state (that's AdapterManager's job) — the old "pending AdapterManager support" wording was stale; AdapterManager has shipped and applied adapters since Phase 3.
• AdapterManager.GetResidencySnapshot() — new, read-only, non-blocking, per-role view of ActiveCount/ConversationsCreated/ForceRecycle plus a computed AdapterRoleResidencyStatus (Healthy/RecyclePending/Degraded/AtHardLimit). _entries is now a ConcurrentDictionary (was Dictionary) so this synchronous telemetry read can never see a torn write from the async minting pipeline — the async _gate still exclusively serializes the higher-level mutation flow, unchanged.
• RuntimeOrchestrator: new GetResidencySnapshot() forwarding to AdapterManager; RuntimeReservationSnapshot gained RejectedAdmissionCount/LastRejectionReason — a lifetime tally of every admission denial, guarded by the existing _telemetryGate pattern.
• IRoleRuntime: two new interface members with default implementations (null/[]) — NativeRoleRuntime overrides both to forward to its orchestrator; other implementers (fakes, scripted test runtimes used across several other test files) silently inherit the honest default, no breakage.
• EstimatedVramBytes is now a real measurement: NativeVramProbe grew TryQueryCurrentProcessVramBytes() (nvidia-smi per-process accounting, filtered to this process's PID), replacing the old base+adapter file-size guess. Factored the shared bounded-subprocess logic out of Phase B's live-budget query into one RunNvidiaSmi helper so the deadlock fix CodeRabbit caught on feat(runtime): Phase B - live VRAM budget for native admission #72 isn't duplicated in a second copy.
• MainWindow: logs the initial VRAM admission snapshot via the existing Activity Log pattern once the native runtime is constructed — the first real UI-visible use of what Phases A/B/C track. Residency isn't logged there since it's always empty at construction time (no role has streamed yet); there's no natural hook point for a mid-flight log without touching the chat-streaming call sites, which is out of scope for a read-only accessor phase.

A real regression caught before it shipped

An existing, non-gated test (NativeRoleRuntime_SchedulerDenial_Returns_ClearFailure_Before_ModelLoad) asserted EstimatedVramBytes was always non-null, based on the old file-size guess. With the real-measurement change, a test process that never actually loaded a model correctly reports null — I updated the assertion to match the new, honest behavior rather than preserving the old dishonest one. Caught by re-running the full suite before committing, not by review — flagging it here so it's visible in review too.

What's deliberately deferred (and why)

Phase C's own /verify description ("drive a native role; read the reservation + residency snapshot mid-flight... and after disposal") needs a real GGUF via THEORC_TEST_GGUF, which is unavailable in this environment — same honest boundary the existing gated native-smoke tests already draw. What I verified instead:

• Every new accessor's empty/baseline/denied-state behavior — real code paths, no model needed (e.g. a real scheduler denial's snapshot shows RejectedAdmissionCount: 1 and confirms no phantom reservation was committed).
• The live VRAM queries genuinely re-verified against this machine's real GPU (RTX 5070 Ti) via a throwaway scratch program (not committed) — both TryQueryLiveNvidiaBudget() and the new TryQueryCurrentProcessVramBytes() return correct real values.

Verification

• All 7 consumer projects build clean — 0 warnings, 0 errors.
• Full OrchestratorIDE.UnitTests suite: 599 passed, 0 failed, 4 skipped (same pre-existing THEORC_TEST_GGUF-gated skips).
• Manually re-read every production-file diff line-by-line against my own design intent before committing (learned from a real bug CodeRabbit caught on feat(runtime): Phase B - live VRAM budget for native admission #72 where a comment described behavior the code didn't actually have) — no discrepancies found this time.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added live VRAM and per-process GPU usage reporting.
  • Added runtime snapshots for adapter residency, active conversations, and health status.
  • Added admission-denial counts and reasons to runtime reservation details.
  • Session status now clearly indicates when a role has a configured adapter.

• Bug Fixes

  • Improved handling of adapter entry recycling during concurrent activity.
  • VRAM estimates now reflect actual process usage when available.
  • Clarified failure reporting when admission is denied before model loading.

[coderabbitai]

Warning

Review limit reached

@hardcoreerik, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 52 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: beeb0671-dd75-4c2a-a95d-651d67ce119f

📥 Commits

Reviewing files that changed from the base of the PR and between 948956b and ec3e448.

📒 Files selected for processing (1)

• OrchestratorIDE/Core/Runtime/AdapterManager.cs

📝 Walkthrough

Walkthrough

The runtime now exposes native VRAM and adapter residency telemetry, records admission denials in reservation snapshots, updates adapter-state reporting, logs VRAM admission details in the UI, and adjusts tests for the new telemetry behavior.

Changes

Runtime telemetry and admission state

Layer / File(s)	Summary
Native VRAM measurement and runtime stats OrchestratorIDE/Core/Runtime/NativeVramProbe.cs, OrchestratorIDE/Core/Runtime/IRoleRuntime.cs	Native probing supports live total and per-process VRAM queries, and success or failure statistics use the native process measurement.
Admission denial telemetry and snapshots OrchestratorIDE/Core/Runtime/RuntimeOrchestrator.cs	Admission rejections are counted with their latest reason, reservation snapshots include that telemetry, and adapter residency snapshots are forwarded.
Concurrent adapter residency tracking OrchestratorIDE/Core/Runtime/AdapterManager.cs	Role entries use concurrent storage, recycle removal uses TryRemove, and per-role residency statuses are exposed through new snapshot contracts.
Runtime integration and validation OrchestratorIDE.Avalonia/MainWindow.axaml.cs, OrchestratorIDE/Core/Runtime/SessionManager.cs, OrchestratorIDE.UnitTests/*	The UI logs reservation values, session snapshots report bound adapters, and tests cover residency, admission denial, native VRAM stats, and updated adapter handling.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant NativeRoleRuntime
  participant RuntimeOrchestrator
  participant NativeVramProbe
  participant AdapterManager
  NativeRoleRuntime->>RuntimeOrchestrator: GetReservationSnapshot()
  RuntimeOrchestrator-->>NativeRoleRuntime: Reservation and rejection telemetry
  NativeRoleRuntime->>AdapterManager: GetResidencySnapshot()
  AdapterManager-->>NativeRoleRuntime: Per-role residency entries
  NativeRoleRuntime->>NativeVramProbe: TryQueryCurrentProcessVramBytes()
  NativeVramProbe-->>NativeRoleRuntime: Current process VRAM bytes or null

Loading

Possibly related PRs

• hardcoreerik/TheOrc#40: Both changes modify adapter recycle and rebuild handling in AdapterManager.
• hardcoreerik/TheOrc#70: Both changes modify RuntimeOrchestrator admission-denial handling.
• hardcoreerik/TheOrc#72: Both changes modify native VRAM probing and runtime admission plumbing.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 38.10% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: Phase C runtime telemetry replacement with real runtime data.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/native-runtime-telemetry

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

OrchestratorIDE/Core/Runtime/IRoleRuntime.cs (1)

458-475: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Probe VRAM before acquiring _telemetryGate.

Line 475 can hold the telemetry lock for the probe’s three-second timeout, blocking GetHealth, GetStats, and other failure updates. Capture the measurement before entering the lock.

Proposed fix

 private void RecordFailure(RuntimeRole role, RuntimeRoleBinding binding, string? runtimeMessage)
 {
+    var estimatedVramBytes = NativeVramProbe.TryQueryCurrentProcessVramBytes();
+
     lock (_telemetryGate)
     {
         ...
         _lastStatsByRole[role] = new RuntimeStats(
             RuntimeName,
             ActiveModel: FormatActiveModel(binding),
-            EstimatedVramBytes: NativeVramProbe.TryQueryCurrentProcessVramBytes());
+            EstimatedVramBytes: estimatedVramBytes);

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Core/Runtime/IRoleRuntime.cs` around lines 458 - 475, Update
RecordFailure to call NativeVramProbe.TryQueryCurrentProcessVramBytes before
acquiring _telemetryGate, store the result locally, and use that captured value
when constructing RuntimeStats inside the lock. Preserve the existing health and
stats updates.

OrchestratorIDE/Core/Runtime/RuntimeOrchestrator.cs (1)

365-366: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Expose fail-closed rejections instead of discarding their snapshot.

Line 266 records missing scheduler/budget as a rejection, but these lines make that count and reason permanently inaccessible. Return a zero-budget snapshot when admission fails closed; reserve null for the explicit unbudgeted opt-out.

Proposed fix

-        if (_scheduler is null || _budgetProvider is null)
-            return null;
+        if (_scheduler is null || _budgetProvider is null)
+        {
+            if (_allowUnbudgetedExecution)
+                return null;
+
+            lock (_telemetryGate)
+                return new RuntimeReservationSnapshot(
+                    [],
+                    TotalBytes: 0,
+                    ReservedBytes: 0,
+                    AvailableBytes: 0,
+                    RejectedAdmissionCount: _rejectedAdmissionCount,
+                    LastRejectionReason: _lastRejectionReason);
+        }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@OrchestratorIDE/Core/Runtime/RuntimeOrchestrator.cs` around lines 365 - 366,
Update the fail-closed guard for _scheduler and _budgetProvider so admission
rejection returns a zero-budget snapshot rather than null, preserving the
recorded rejection count and reason. Keep null reserved exclusively for the
explicit unbudgeted opt-out path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@OrchestratorIDE/Core/Runtime/AdapterManager.cs`:
- Around line 352-358: Update the AdapterRoleResidency projection to capture the
relevant counter values from kv.Value once into locals, then populate
ActiveCount and ConversationsCreated from those captured values. Derive Status
using the same captured counters rather than calling ComputeResidencyStatus with
kv.Value, including the equivalent projection at the additional occurrence.

---

Outside diff comments:
In `@OrchestratorIDE/Core/Runtime/IRoleRuntime.cs`:
- Around line 458-475: Update RecordFailure to call
NativeVramProbe.TryQueryCurrentProcessVramBytes before acquiring _telemetryGate,
store the result locally, and use that captured value when constructing
RuntimeStats inside the lock. Preserve the existing health and stats updates.

In `@OrchestratorIDE/Core/Runtime/RuntimeOrchestrator.cs`:
- Around line 365-366: Update the fail-closed guard for _scheduler and
_budgetProvider so admission rejection returns a zero-budget snapshot rather
than null, preserving the recorded rejection count and reason. Keep null
reserved exclusively for the explicit unbudgeted opt-out path.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 8fd773e2-e952-417f-82cf-3453016cc5b3

📥 Commits

Reviewing files that changed from the base of the PR and between 45f044a and 948956b.

📒 Files selected for processing (10)

• OrchestratorIDE.Avalonia/MainWindow.axaml.cs
• OrchestratorIDE.UnitTests/AdapterManagerTests.cs
• OrchestratorIDE.UnitTests/NativeRuntimeTestSupportTests.cs
• OrchestratorIDE.UnitTests/RuntimeOrchestratorTests.cs
• OrchestratorIDE.UnitTests/SessionManagerTests.cs
• OrchestratorIDE/Core/Runtime/AdapterManager.cs
• OrchestratorIDE/Core/Runtime/IRoleRuntime.cs
• OrchestratorIDE/Core/Runtime/NativeVramProbe.cs
• OrchestratorIDE/Core/Runtime/RuntimeOrchestrator.cs
• OrchestratorIDE/Core/Runtime/SessionManager.cs