fix(OMN-15642): remediation — unwire attach-readiness fold from the gated /health status

[jonahgabriel]

OMN-15642 remediation round

This is a follow-up to #2618 (already merged as 363120f4c), fixing 6 verifier-found defects in that PR on the SAME branch. #2618's own head branch (jonah/omn-15642-build-lane) was auto-deleted by GitHub after merge, so this reopens as a new PR from the recreated branch (see "Branch history" below) rather than reusing PR #2618, which stays MERGED.

What was wrong with #2618

An independent adversarial verifier found 6 defects. This PR fixes the safety-critical ones (3-6) directly; 1-2 are addressed honestly, not falsely closed (see "Honest scope").

1. AC2 has no implementing code (correlation regression bisection) — not closed by this PR either; see "Honest scope."
2. AC3 proves the wrong artifact (RED/GREEN was for the health-status fold, not the correlation regression) — same.
3. Unassessed blast radius, already merged: /health's top-level status is a hard, no-DEGRADED-tolerance gate for four real automated consumers — .github/workflows/reusable-runtime-boot.yml (jq -e '.status=="healthy" ...' || exit 1, 6 call sites), scripts/deploy-agent/deploy_agent/executor.py's _runtime_health_passed, and scripts/runtime_build/verify_stability_refresh.py / verify_dev_refresh.py. None were audited before fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618 merged.
4. Deploy-wedge risk: ModelRuntimeAttachReadiness's own docstring says attach-readiness "is not a source of truth for contract lifecycle," and OMN-13237 deliberately made a NOT_READY/DEGRADED contract non-fatal (recorded, skipped, never restart/redeploy-triggering). fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618 folded that DEGRADED state into /health's gated status anyway — the first ordinary NOT_READY skip (e.g. one unprovisioned topic, a documented live condition on onex-dev per OMN-15330) would fail every one of the four consumers above on every future boot/deploy.
5. The fix was unwired for the surface it named: deploy-onex-staging.yml never reads /health or /health/detailed at all (verified: grep -nEi 'health|readyz|/ready' shows only k8s /ready, omnidash's /api/health/data-sources, and consumer-group health) — so fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618's stated purpose (closing deploy-onex-staging's blind spot) was never true for either endpoint, and its actual effect was entirely in the four consumers in (3)/(4), unaudited.
6. Dead branch: elif status == "degraded": http_status = 200 on /health/detailed could never change anything (every pre-fold path reaching "degraded" already set http_status=200, and the fold never returns "degraded" from a pre-fold "unhealthy").

The fix

• _handle_health (/health, the gated liveness endpoint): fold_attach_readiness_into_status call removed entirely — for both DEGRADED and the currently-production-unreachable FAILED/core_gap case, so there's no partial/inconsistent behavior depending on severity. /health's status is pinned back to its pre-OMN-15642 behavior: attach readiness never changes it. The aggregate stays fully visible via the pre-existing (OMN-15512) details.components.runtime_wiring block — just not gated.
• _handle_health_detailed: keeps the fold (both DEGRADED and FAILED/core_gap) — verified unread by any of the four hard-gate consumers above, and by deploy-onex-staging.yml, so it's safe and still closes the original human/dashboard visibility gap this ticket's incident motivated.
• Removed the dead elif status == "degraded": http_status = 200 branch on /health/detailed, with a comment explaining why it was always a no-op.
• Updated docstrings (module + fold_attach_readiness_into_status) to state the caller-scoping decision explicitly, so a future caller doesn't re-wire this into /health without re-reading the wedge-risk analysis.

Seam definition

• fold_attach_readiness_into_status(payload_status, readiness_state) -> status (unchanged signature/behavior) — now called by ServiceHealth._handle_health_detailed only.
• /health JSON status field: no longer reads self._attach_readiness at all (reverted to pre-OMN-15642 shape). details.components.runtime_wiring (unchanged, OMN-15512) remains the only place /health surfaces the aggregate.
• /health/detailed JSON status/http_status: unchanged from fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618 (DEGRADED→"degraded"/200, FAILED→"unhealthy"/503), minus the dead branch.

Acceptance-criteria mapping (this round)

Defect	Test	Proves
3+4 (blast radius / deploy-wedge)	tests/unit/runtime/test_service_health_attach_readiness.py::TestAttachReadinessJoin::test_degraded_attach_readiness_does_not_degrade_gated_health_status, ::test_core_gap_does_not_flip_gated_health_status_either	/health's gated status stays "healthy" under both DEGRADED and FAILED attach-readiness states — the four consumers' status=="healthy" assertion can no longer be broken by this mechanism
5 (unwired-for-its-stated-surface)	PR body + grep output above (no test possible — proving a negative about a different repo's workflow)	Documented, not silently left implied
6 (dead branch)	Removed; existing test_detailed_endpoint_flips_http_status_on_core_gap / test_detailed_endpoint_also_folds_attach_readiness still cover the two real (non-dead) http_status transitions on /health/detailed	No behavior change, confirmed by full green rerun

RED-before/GREEN-after (mutation-proven on .200): stashed only the two source files (kept the rewritten tests), ran against the merged 363120f4c source — 2 of 53 tests in the touched files fail (assert 'degraded' == 'healthy', assert 'unhealthy' == 'healthy' — exists-but-wrong, not merely absent), restored, reran — 53/53 pass. 167/167 across the full focused set (health/attach-readiness/wiring/serialization). ruff format+check clean, mypy clean, pre-commit run --files <changed> 100% Passed/Skipped, governed pre-push selector escalated to the full suite (shared module) — 23036 passed, 40 skipped, 0 failed in 454s.

Honest scope (defects 1-2, NOT closed here)

This PR does not close AC2 ("bisect the correlation seam on the live plane") or AC3 ("a test that fails at 72d27ef and passes at the fix") for the correlation_id regression itself. Delegation-projection and system-event-stream rows are still absent by correlation_id on onex-dev as of this PR.

Live access attempt (repeated from #2618, same result): aws sts get-caller-identity — live (account 272493677981). kubectl --kubeconfig ~/.kube/omninode-mvp1 get pods -n omninode-k3s-dev-system (corrected namespace, per omni_home/docs/runbooks/DEPLOY_ENVIRONMENTS.md — not the onex-dev namespace I first guessed) — empty, and kubectl config get-contexts shows only a placeholder default/default/default context with no live cluster (cluster-info hangs to timeout). Per memory reference_k3s_cloud_deploy_surface ("SSM-not-kubectl"), direct kubectl access needs an SSM tunnel this lane did not have time to stand up. Same infeasibility the previous lane hit, confirmed independently.

Additional bisection done without live access (static, code-level, clearly INFERENCE not proof — omnibase_infra only, the repo in scope for this ticket):

• node_projection_delegation (the delegation projection) and the unified system-event-stream writer are not implemented in this repo (grep -rln "DelegationProjection\|node_projection_delegation" src/ → zero hits outside the vendored migration SQL from OMN-15503). They're owned by omnimarket. This repo's role is limited to (a) the runtime engine that wires omnimarket's contracts' Kafka subscriptions at boot, and (b) whatever ingress code sets correlation_id on outbound envelopes.
• Re-examined cc5ec9253 (OMN-15546, the ticket's prime suspect) independently: it modifies validate_runtime_local_ingress_payload in runtime_local_ingress.py, which is the local-ingress path (used by e.g. onex delegate CLI dispatch), not the gateway-driven POST /v1/workflows path the business-proof gate actually drives. Concur with fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618's ruling-out, on independent re-reading.
• New finding, not on the ticket's suspect list: in subscribe_wired_contract_topics (handler_wiring.py), a contract with zero registered dispatchers (if not result.dispatchers_registered: ... continue, an OMN-15474-adjacent path) is filtered out of eligible before _interleave_contract runs — such contracts never appear in attach_results/attach_results_out and therefore never appear in ModelRuntimeAttachReadiness.results at all. They are invisible to the DEGRADED signal, not merely counted as a NOT_READY/FAILED entry within it. If the delegation-projection or system-event-stream consumer's contract lost its dispatcher registration entirely in this window (as opposed to attaching-then-failing), the symptom would show with no visible DEGRADED signal anywhere, including in components.runtime_wiring — which would also mean fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618's fix (even before this revert) could never have caught it. I could not confirm this against a live boot log or manifest within this lane's time; flagging as a genuinely different, deeper hypothesis than fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618's.
• Also separately confirmed: _require_registered_contract_dispatcher_scope (the function fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618's body attributed the DEGRADED mechanism to) is called outside any try/except in _interleave_contract — if it ever raised there, the exception would propagate through the unguarded asyncio.gather in subscribe_wired_contract_topics, not degrade gracefully. In practice this never fires because the zero-dispatcher pre-filter above removes every contract that would trigger its empty-scope check before it's reached. Noted for whoever picks up AC2/AC3 next — this function's failure mode is NOT what fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618 described.

Recommend as a genuine follow-up (not absorbed here, and not by this repo alone): live cluster access (via SSM, not kubectl directly) to read the actual boot-time wiring report / dispatcher registration for the delegation-projection and system-event-stream contracts on the corrected tree, cross-repo with omnimarket if the owning contract's dispatcher registration is what changed.

Branch history

jonah/omn-15642-build-lane's original head (5de9e9a79) was squash-merged as 363120f4c (PR #2618, 2026-08-02T00:01:31Z) and GitHub auto-deleted the branch. This PR's branch was recreated from origin/dev (which already contains 363120f4c) plus this remediation commit on top — confirmed via git diff 5de9e9a792 origin/dev -- <the 4 touched files> returning empty before making any new edits, i.e. the new commit's parent tree for these files is byte-identical to what #2618 already landed.

Gates

Run on stickybeatz-studio (.200) via patch-transfer + sha256 verification (all 4 changed files matched local before any gate ran, and again after the RED/GREEN stash round-trip):

• ruff format --check / ruff check — clean
• mypy on both changed source files — clean
• Focused suite (167 tests, same set as fix(OMN-15642): fold boot attach-readiness into /health status (close false-green consumer-drop gap) #2618) — all green
• pre-commit run --files <changed> (via zsh -lc for the interactive PATH / uv) — 100% Passed/Skipped
• git push pre-push hook chain: mypy, architecture validation, governed impacted-test selector escalated to the full suite (shared module — health_checker.py is broadly imported) — 23036 passed, 40 skipped, 0 failed in 454s, deploy-scope DoD parity NOTICE (OCC companion not yet merged, expected pre-merge)

OMN-15642

Evidence-Source: 09c5dd565e7ded968640151d0c968366f2485abf
Evidence-Ticket: OMN-15642

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 587873fc-9925-41f6-953c-1cbed19a57b5

📥 Commits

Reviewing files that changed from the base of the PR and between 94a3a11 and 1d6ab43.

📒 Files selected for processing (4)

• src/omnibase_infra/runtime/health/runtime_health_block.py
• src/omnibase_infra/services/health_checker.py
• tests/unit/runtime/test_health_detailed.py
• tests/unit/runtime/test_service_health_attach_readiness.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8001 ] unreachable — preflight short-circuit (no models available)

---

Gate semantics (pilot phase)

Verdict	Meaning	Blocks merge?
passed	No critical findings	No
blocked	CRITICAL findings found	Yes
degraded	All models unavailable (infra)	No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)