DeepSeek-V4 HCA C1: single accounting authority for CSA/HCA device state groups (B6)

[justinchuby]

DeepSeek‑V4 HCA "C1" runtime slice — single accounting authority for CSA/HCA device state groups (B6)

Draft. Do not merge — opened for independent review.

This is the first, GPU‑verified increment of the DeepSeek‑V4 HCA (Hierarchical Compressed Attention) "C1" runtime slice. It closes design blocker B6 from deckard-deepseek-v4-csa-hca-cuda-slice.md §7 ("onnx-genai-kv/shared accounting is not the authority; a second unaccounted alloc_raw path exists for CSA device buffers") by introducing a single property‑typed accounting authority for CSA (CompressedSparseAttention) / HCA device state groups in the CUDA EP.

What landed

New module crates/onnx-runtime-ep-cuda/src/kernels/csa_state_group.rs:

• CsaStateGroupDescriptor — a property gate. Parsed from the node's declared attributes + state input edges (compression_ratio, cache_format, num_heads/head_dim/qk_rope_head_dim, index_head_dim, device_count, presence of the compressed(6)/carry(7) state edges). Not a model‑name/shape allowlist. Carries the exact reason on refusal via the typed CsaStateGroupRefusal enum (UnsupportedRatio, UnknownCacheFormat, Ratio4RequiresFp8Cache, Ratio4RequiresIndexHeadDim128, Ratio128RejectsFp4, InvalidHeadGeometry, RopeExceedsHeadDim, MultiDeviceAmbiguity, MissingStateEdge, OutOfMemory, UnsupportedC1Ratio).
  • validate() accepts the op's full ratio‑4/128 range (op‑level).
  • validate_c1_runtime() is the stricter C1 admission gate: ratio‑128 (HCA) only — ratio‑4/MTP are op‑supported but out of C1 scope and are typed‑refused, never silently threaded or dense‑fallback'd.
• CsaStateGroupLedger — the sole, mandatory accountant. Owns bytes, never cursors; charges per (request, device) for isolation; try_charge is a check‑then‑charge under one mutex that fails closed BEFORE any physical allocation (managed limit; 0 = unlimited, the steady‑state default); RAII CsaStateGroupCharge releases on Drop so teardown returns to baseline. It is instance state threaded through CsaMetrics (§8) — not a process‑wide global, and it is not an allocator/cache/page‑manager: it accounts the bytes the existing physical path (runtime.alloc_raw) reserves.

No optional/no‑ledger escape hatch. Per the no‑backward‑compatibility during development directive, the ledger is a mandatory Arc<CsaStateGroupLedger> on the reservation path (not an Option), so the type system alone forbids an unaccounted reservation — the exact B6 hazard. The earlier Option<ledger> seam was replaced outright rather than wrapped in a compat shim, and every in‑repo caller/fixture/test was updated atomically. "Disarmed" is therefore an unlimited limit (0 sentinel), not a structural bypass: accounting still runs, it simply never refuses, so byte‑identity is preserved without leaving a second unaccounted path.

Wiring:

• CsaDeviceBufferManager::reserve unconditionally charges the ledger before alloc_raw and holds the RAII charge for the buffer lifetime (Drop frees buffers, then releases the charge).
• The CUDA CSA factory create() builds the descriptor, validate()s it (typed refusal), takes a per‑runner request id, and threads the (mandatory) ledger + (request, device) key.
• CsaMetrics gained the Arc<CsaStateGroupLedger> plus pub observability accessors (resident/peak/compressed/dense‑ring bytes, charge failures, active group count, per‑(request,device) residency, get/set limit).

Invariants honored

• Default‑off / byte‑identical. The charge happens once at runner construction, never per token and never on the captured/replay stream; release is host‑only. With the ledger unlimited (default) the reservation is byte‑identical to the pre‑accounting path. No device op or host sync is added to steady‑state replay.
• Fail‑closed. Over‑limit groups are refused with a typed OutOfMemory before any device memory is reserved (no leak).
• Isolation. Residency is tracked per (request, device); teardown returns to baseline.
• Property‑based refusals (not model‑name); cursors stay backend‑owned; PMM/VMM/governor remains the mapping/accounting authority for physical device memory (this ledger is an accounting seam in front of it, not a competing manager).
• No PagedAttention / Mobius / QMoE / BQMoE / residency / feat(cuda-ep): boundary-time route-telemetry consumer wiring into #1854 coarse residency (#1810 Slice 7B) #1971 files touched.

Tests & measurement

• 11 CPU‑only descriptor/ledger tests: accept/refuse matrix, fail‑closed OOM without mutation, per‑(request,device) isolation, baseline/peak, monotonic ids, C1 ratio gate, limit round‑trip.
• 6 GPU tests on a real CsaDeviceBufferManager (A100‑80GB, GPU 2): charge == sum(reserved), teardown → baseline, fail‑closed‑before‑alloc with no leak, per‑request isolation, stable workspace addresses, and an unlimited‑ledger reservation that accounts without ever refusing (charge_failures == 0, teardown → baseline — replacing the old no‑ledger case now that the ledger is mandatory).
• Regression: the existing CSA op‑parity GPU suite — 26 passed / 1 ignored (the ignored one needs external Mobius artifacts) — stays green, covering eager + capture/replay + speculative rollback + observability. This is the byte‑identical‑when‑disarmed proof.
• cargo fmt clean; cargo clippy -D warnings clean on lib and tests.
• Overhead: by construction there is no steady‑state kernel/host‑sync cost (accounting is one‑time at runner construction, off the captured stream), so there is no per‑token GPU delta to measure for this slice; the op‑parity capture/replay suite is the byte‑identity control.
• Independent code‑review (excluding the original author and myself): no substantive findings.

Honestly scoped next steps (tracked, NOT in this slice)

1. Physical‑allocator unification: route CsaStateGroupLedger::try_charge to delegate to the adopted MemoryGovernor reserve(Tier::Device, bytes, MemoryRole::Workspace) / onnx-genai-kv so the physical bytes become governor‑visible (final B6 unification). The ledger's try_charge is already shaped as that adapter seam. This is an architecture migration (not a compat shim), deliberately deferred.
2. Native‑decode threading: the smallest production caller that threads the ratio‑128 present_* → past_* compressed/carry state at the existing ensure_capacity safe boundary and calls validate_c1_runtime, with captured ≥16‑decode CPU/CUDA bit‑parity and fallbacks == 0.

CPU op‑level present→past carry/cursor progression across the decode boundary is already covered by the CPU oracle test ratio128_stateful_carry_matches_full_recompute_across_decode_boundary.

_{Co-authored-by: Copilot}

[justinchuby]

Force-pushed d25b4d13a → b104ce73b to apply the no-backward-compatibility during development directive.

Change: the Option<Arc<CsaStateGroupLedger>> seam on CsaDeviceBufferManager::reserve (and the CSA factory create() call site) is replaced by a mandatory Arc<CsaStateGroupLedger>. There is now no optional/no-ledger path, so the type system alone forbids an unaccounted reservation — directly strengthening B6 (single accountant). The earlier compat seam was removed outright, not wrapped, and every in-repo caller/fixture/test was updated atomically.

Byte-identity preserved: "disarmed" is now an unlimited limit (0 sentinel) rather than a structural bypass — accounting still runs but never refuses, and nothing lands on the captured/replay stream.

Test delta: the obsolete reserve_without_ledger_holds_no_charge (passed None) is replaced by unlimited_ledger_accounts_without_refusing (asserts the default unlimited ledger accounts, charge_failures == 0, teardown → baseline).

Re-verified green after the change: cargo fmt clean; cargo clippy -D warnings clean (lib + tests); 17 lib unit tests pass (11 CPU descriptor/ledger + 6 GPU on A100 GPU 2); CSA op-parity regression 26 passed / 1 ignored (byte-identical capture/replay control).

Scope unchanged: physical-allocator unification onto the governor / onnx-genai-kv and native-decode present_*→past_* threading with captured ≥16-decode CPU/CUDA bit-parity remain honestly-scoped tracked next steps. Still draft — do not merge; awaiting independent review.

_{Co-authored-by: Copilot}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.93%. Comparing base (e17fc1b) to head (b104ce7).
⚠️ Report is 32 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2008      +/-   ##
==========================================
+ Coverage   80.42%   80.93%   +0.51%     
==========================================
  Files         424      425       +1     
  Lines      207556   208150     +594     
  Branches   207556   208150     +594     
==========================================
+ Hits       166924   168472    +1548     
+ Misses      34994    34035     -959     
- Partials     5638     5643       +5     

Flag	Coverage Δ
cli-ort-linux	72.51% <ø> (ø)
cli-ort-windows	72.01% <ø> (ø)
mlas	85.20% <ø> (ø)
offline	81.09% <ø> (+0.53%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 18 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[justinchuby]

APPROVE B6 ACCOUNTING SLICE FOR MERGE

Independent, read-only review at exact HEAD b104ce73b1424f78b17194a94d9a3539911cddaf (draft). Reviewed the B6 accounting-authority slice only — a single property-typed CsaStateGroupLedger for CSA/HCA device state groups. Not native-decode state threading, not physical-allocator unification.

Verified gates

Single accounting authority — no unaccounted route.

• CsaDeviceBufferManager::reserve is the only constructor of the manager (no struct-literal anywhere); its sole production caller is the CSA factory at compressed_sparse_attention.rs. reserve now takes a mandatory Arc<CsaStateGroupLedger> + (request, device) charge key — no Option, default, null, or alternate factory path, including tests/fixtures.
• The ledger is shared once via Arc<CsaMetrics> (factory.metrics.clone() into each runner → state_group_ledger()), so all runners charge one authority. next_request_id() gives each create() an isolated (request_id, device_ordinal) key.

Charges before allocation, fail-closed.

• reserve calls ledger.try_charge(...) before any alloc_raw. If admission is refused (limit exhausted) no physical device memory is touched, so a rejected group cannot leak.
• try_charge runs the whole check-then-charge under the charges mutex (all atomic-counter mutations happen under that lock → no lost update / no race past the limit), and mutates no state on refusal (no partial reservation); it increments charge_failures and returns a typed OutOfMemory refusal.

Exact per-class/total bytes, checked/saturating overflow.

• CsaStateGroupBytes::from_layout is byte-identical to what reserve physically allocates: scratch = Σ workspace.max(1) (matches alloc_raw(size.max(1))), and the five class fields sum to exactly the seven fixed buffer sizes. total() uses saturating_add; residency uses saturating add/sub; peak is a monotonic high-water mark via compare_exchange_weak.

RAII release exactly once on every exit path.

• The charge is held in a local across the allocation loop; any early return (alloc failure → rollback() frees partial buffers) drops the local → single RAII release. On success it is moved into the manager's charge field. Drop frees workspaces + buffers first, then the charge field drops last → release after physical free. No manual release method → no double-release.

Isolation, unlimited sentinel, observability.

• resident_for(request, device) / active_group_count() prove per-(request, device) isolation; a GPU test asserts two requests don't cross-account.
• Unlimited 0 limit still records residency/peak/per-class/active-count and only disables refusal — accounting shim, not a structural bypass.
• Refusals are property-typed (CsaStateGroupRefusal) from node geometry/dtype/cache-format — no model-name allowlist.

Byte-identity / op-parity preserved.

• The ledger touches state only at runner construction (reserve) and teardown (Drop) — never in compute/capture/replay/record_device_metrics. With the default unlimited ledger, no device op is added and the reservation is byte-identical.
• The factory now runs descriptor.validate()? as a new gate on the existing op path. Confirmed empirically it does not refuse any previously-accepted config: op-parity suite 26 passed / 1 ignored (the ignored one needs external Mobius/MTP artifacts), including all byte-identity cases (ratio4_capture_replay_decode_is_byte_identical_to_eager_and_cpu, ratio128_fp8_decode_capture_replay_matches_cpu, ratio4_greedy_token_bit_identity_across_draft_verify, both speculative_rollback_*_bit_exact). Factory correctly uses op-level validate() (ratio 4 and 128), not the stricter validate_c1_runtime reserved for the future native-decode slice.

Truthful scope (verified honest).

• CsaDeviceBufferManager still uses runtime.alloc_raw; the PR does not claim sole physical-allocator / onnx-genai-kv authority. Physical-allocator (governor) unification and native-decode ratio-128 threading (calling validate_c1_runtime) are explicitly deferred next steps.

Independent test/lint runs (idle A100, GPU 2, serial)

• cargo fmt -p onnx-runtime-ep-cuda --check → clean
• cargo clippy -p onnx-runtime-ep-cuda --features cuda,cuda-13000,gpu-tests --lib --tests -- -D warnings → clean
• kernels::csa_state_group:: CPU unit tests → 11/11 pass
• kernels::csa_device_state:: GPU tests → 6/6 pass (real CUDA: charge/release, fail-closed-no-leak, per-request isolation, stable workspace addresses, unlimited-accounts)
• --test compressed_sparse_attention_gpu op-parity → 26 passed / 1 ignored

Non-blocking

• Branch is 20 commits behind main; needs a rebase before merge. The only conflict is a trivial kernels/mod.rs pub mod adjacency against merged feat(cuda-ep): native PagedAttention LATENT kernel for GLM-5.2 dense MLA (slice 3A.2) #1978 (pub mod paged_attention; vs this PR's pub mod csa_state_group;) — resolve by keeping both. No functional overlap with PagedAttention / BQMoE / residency.

Next honestly-scoped step

Physical-allocator/governor unification (delegate alloc_raw behind the governor) + native-decode ratio-128 state threading that calls validate_c1_runtime.

Do not merge (draft, and pending rebase onto latest main). Verdict: APPROVE B6 ACCOUNTING SLICE FOR MERGE.

— Independent reviewer, requested by Justin Chu