Skip to content

fix(OMN-15642): remediation — unwire attach-readiness fold from the gated /health status #2619

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jonahgabriel merged 1 commit into from
Aug 2, 2026
Merged

fix(OMN-15642): remediation — unwire attach-readiness fold from the gated /health status #2619

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 26 additions & 14 deletions src/omnibase_infra/runtime/health/runtime_health_block.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,20 +154,32 @@ def fold_attach_readiness_into_status(
the ``ServiceRuntimeHealthMonitor`` verdict (see the module docstring): a
runtime can boot fully, with EVERY rollout/digest/dashboard/staleness gate
green, while one Kafka consumer contract silently never attaches its
subscription (``_require_contract_dispatcher_scope`` et al. raise inside
``subscribe_wired_contract_topics``, caught per-contract and downgraded to
a non-fatal ``ModelContractAttachResult`` — see
``omnibase_infra.event_bus.model_runtime_attach_readiness``). A projection
or reducer that stops consuming produces zero new rows for whatever it
writes, invisible to every liveness/rollout check that never queries the
wiring detail. OMN-15642 observed exactly this shape live on onex-dev:
steps 30-36 of ``deploy-onex-staging`` (rollout, digest triple-match,
dashboard health, post-deploy verification, staleness) all passed while a
correlation-keyed projection and the unified system-event stream stayed
silently empty, surfacing only ~60s later at the terminal business-proof
gate. Folding the aggregate into ``status`` here means the SAME class of
silent drop is visible at ``/health`` immediately, not only after a
downstream consumer happens to probe by correlation_id.
subscription (raises inside ``subscribe_wired_contract_topics``, caught
per-contract and downgraded to a non-fatal ``ModelContractAttachResult`` —
see ``omnibase_infra.event_bus.model_runtime_attach_readiness``). A
projection or reducer that stops consuming produces zero new rows for
whatever it writes, invisible to every liveness/rollout check that never
queries the wiring detail.

Caller scoping (remediation, read before wiring this into a new caller):
this fold is used by ``ServiceHealth._handle_health_detailed`` ONLY, not
``_handle_health``. ``ModelRuntimeAttachReadiness``'s own docstring states
"the readiness endpoint reports attach status ONLY — it is not a source of
truth for contract lifecycle", and OMN-13237 deliberately designed a
NOT_READY/DEGRADED contract to be recorded and skipped, never fatal, never
a restart/redeploy trigger. ``/health`` (unlike ``/health/detailed``) is a
hard, no-tolerance boot/deploy gate for four real automated consumers —
``.github/workflows/reusable-runtime-boot.yml``,
``scripts/deploy-agent/deploy_agent/executor.py``,
``scripts/runtime_build/verify_stability_refresh.py`` /
``verify_dev_refresh.py`` — that assert ``status == "healthy"`` with no
DEGRADED tolerance. Folding a documented-non-fatal DEGRADED into
``/health``'s gated status would convert every ordinary NOT_READY skip
(e.g. one unprovisioned topic, a documented live condition on onex-dev —
OMN-15330) into a hard boot/deploy failure. Route new visibility needs
through ``/health/detailed`` or the existing
``details.components.runtime_wiring`` block, not through this fold on
``/health``.

Args:
payload_status: Status derived from ``RuntimeHostProcess.health_check()``,
Expand Down
59 changes: 39 additions & 20 deletions src/omnibase_infra/services/health_checker.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1170,20 +1170,29 @@ async def _handle_health(self, request: web.Request) -> web.Response:
status = fold_runtime_verdict_into_status(
status, verdict.status if verdict is not None else None
)
# OMN-15642: fold the boot attach-readiness aggregate the SAME way
# OMN-15217 folded the runtime-health-monitor verdict above. Before
# this, a DEGRADED/FAILED attach_readiness reached only
# details.components.runtime_wiring (below) -- nothing upstream
# reads that nested detail, so a consumer contract that silently
# failed to attach its Kafka subscription left `status` reporting
# "healthy" with HTTP 200. See fold_attach_readiness_into_status's
# docstring for the live incident this closes.
status = fold_attach_readiness_into_status(
status,
self._attach_readiness.state
if self._attach_readiness is not None
else None,
)
# OMN-15642 (remediation): boot attach-readiness is DELIBERATELY NOT
# folded into this endpoint's gated `status`, unlike the runtime-verdict
# fold above. `/health` is the liveness probe that four real automated
# gates hard-fail on with no DEGRADED tolerance --
# .github/workflows/reusable-runtime-boot.yml (`jq -e '.status ==
# "healthy" ...' || exit 1`), scripts/deploy-agent/deploy_agent/executor.py
# (`_runtime_health_passed`), and the deploy-readiness checks in
# scripts/runtime_build/verify_stability_refresh.py and
# verify_dev_refresh.py. `ModelRuntimeAttachReadiness`'s own docstring
# (omnibase_infra.event_bus.model_runtime_attach_readiness) states "The
# readiness endpoint reports attach status ONLY -- it is not a source of
# truth for contract lifecycle", and OMN-13237 deliberately designed a
# NOT-READY/DEGRADED contract to be recorded and skipped, never fatal,
# never a restart/redeploy trigger (service_kernel.py's boot-walk
# comment: "it never recycles the process"). A single unprovisioned
# topic -- a documented live condition on onex-dev (OMN-15330) -- would
# therefore have turned a deliberately-non-fatal per-contract skip into a
# hard boot/deploy failure on every future boot. The aggregate is still
# fully visible without that risk: enriched_details["components"]
# ["runtime_wiring"] below (pre-existing, OMN-15512) and
# /health/detailed's own `status` (fold_attach_readiness_into_status,
# below in _handle_health_detailed -- that endpoint is not a probe
# target for any of the four consumers above).

self._log_health_transition(
status=status,
Expand Down Expand Up @@ -1580,20 +1589,30 @@ async def _handle_health_detailed(self, request: web.Request) -> web.Response:
status = "unhealthy"
http_status = 503

# OMN-15642: same fold as /health -- see fold_attach_readiness_into_status.
# Without this, /health/detailed's own status/http_status pair stayed
# green while attach_readiness.state != READY was visible only inside
# the nested components.runtime_wiring detail below.
# OMN-15642: fold the boot attach-readiness aggregate into THIS
# endpoint's status -- deliberately NOT into /health's (see the long
# comment in _handle_health for why). /health/detailed is not a probe
# target for any automated boot/deploy gate (verified: no k8s manifest,
# CI workflow, or deploy script under this repo or omninode_infra reads
# this path), so it is safe here and matches this endpoint's own
# documented contract (503 for "unhealthy"). Without this fold,
# /health/detailed's own status/http_status pair stayed green while
# attach_readiness.state != READY was visible only inside the nested
# components.runtime_wiring detail below.
status = fold_attach_readiness_into_status(
status,
self._attach_readiness.state
if self._attach_readiness is not None
else None,
)
# Only "unhealthy" can change http_status here: fold_attach_readiness_
# into_status() early-returns unchanged for an already-"unhealthy"
# payload_status, and every pre-fold path that yields "healthy" or
# "degraded" already set http_status=200 above -- so a post-fold
# "degraded" never needs (or gets) a different http_status than it
# already had.
if status == "unhealthy":
http_status = 503
elif status == "degraded":
http_status = 200

checked_at = datetime.now(tz=UTC).isoformat()
components = build_component_health(
Expand Down
21 changes: 15 additions & 6 deletions tests/unit/runtime/test_health_detailed.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -639,12 +639,21 @@ async def test_degraded_wiring_under_a_green_endpoint(self) -> None:

half-fixed. Until OMN-15642, ``data["status"]`` stayed "healthy" here
— only the nested ``components.runtime_wiring`` said otherwise, which
nothing upstream (including deploy-onex-staging's own health-gate
steps) reads. ``fold_attach_readiness_into_status`` now folds the
aggregate into the top-level ``status`` too, the same way OMN-15217
already folds the runtime-health-monitor verdict. The HTTP status
code stays 200 — a runtime is designed to stay live with NOT_READY
contracts (OMN-13237) — but `status` itself is now honest.
nothing read either. ``fold_attach_readiness_into_status`` now folds
the aggregate into this endpoint's top-level ``status`` too, the same
way OMN-15217 already folds the runtime-health-monitor verdict. The
HTTP status code stays 200 — a runtime is designed to stay live with
NOT_READY contracts (OMN-13237) — but `status` itself is now honest.

NOTE (remediation): this fold lives on ``/health/detailed`` only, not
``/health`` — neither endpoint is read by ``deploy-onex-staging``
(verified: that workflow probes k8s `/ready`, omnidash's
`/api/health/data-sources`, and consumer-group health, never
`/health` or `/health/detailed`), and ``/health``'s gated `status` is
a hard, no-DEGRADED-tolerance boot/deploy gate for four OTHER
automated consumers (see
``tests/unit/runtime/test_service_health_attach_readiness.py``'s
module docstring) that this fold must not touch.
"""
server = ServiceHealth(runtime=_healthy_runtime(["db", "http"]), version="1.0")
server.attach_readiness(_readiness(attached=3, not_ready=2))
Expand Down
90 changes: 59 additions & 31 deletions tests/unit/runtime/test_service_health_attach_readiness.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# SPDX-FileCopyrightText: 2025 OmniNode.ai Inc.
# SPDX-License-Identifier: MIT
"""Unit tests for the /health <- boot attach-readiness join (OMN-15642).
"""Unit tests for the /health(/detailed) <- boot attach-readiness join (OMN-15642).

OMN-15642 observed a live regression on onex-dev (deploy-onex-staging runs
30720296789 and, on n=2 reproduction, 30721670043): the corrected tree booted
Expand All @@ -14,14 +14,34 @@
``ModelRuntimeAttachReadiness`` -- OMN-15512 wired that aggregate onto
``ServiceHealth`` -- but pre-fix it reached only
``details.components.runtime_wiring``, a nested payload detail nothing
upstream (deploy-onex-staging's own health-gate steps included) reads. The
top-level ``status`` field stayed "healthy" no matter how degraded the boot
attach was.

This is the exact class of mask OMN-15217 already closed for the
``ServiceRuntimeHealthMonitor`` verdict (see
``test_service_health_runtime_verdict.py``, which this file mirrors
line-for-line). These tests pin the SAME join for attach readiness.
upstream reads. The top-level ``status`` field stayed "healthy" no matter how
degraded the boot attach was.

REMEDIATION (same ticket, second round): an adversarial verifier found that
folding this aggregate into ``/health``'s gated ``status`` -- the first
version of this fix -- is a live deploy-wedge risk, not a safe visibility fix.
``ModelRuntimeAttachReadiness``'s own docstring says "the readiness endpoint
reports attach status ONLY -- it is not a source of truth for contract
lifecycle", and OMN-13237 deliberately made a NOT_READY/DEGRADED contract
non-fatal (recorded and skipped, the process is never recycled for it). But
four real automated consumers hard-gate boot/deploy readiness on
``/health``'s ``status`` with no DEGRADED tolerance --
``.github/workflows/reusable-runtime-boot.yml``,
``scripts/deploy-agent/deploy_agent/executor.py``, and
``scripts/runtime_build/verify_stability_refresh.py`` /
``verify_dev_refresh.py``. A single unprovisioned topic (a documented live
condition on onex-dev, OMN-15330) would have turned every ordinary NOT_READY
skip into a hard boot/deploy failure. The fold now lands ONLY on
``/health/detailed`` (verified: no k8s manifest, CI workflow, or deploy script
in this repo or ``omninode_infra`` reads that path), which still closes the
original visibility gap for a human or dashboard checking after steps 30-36 --
just not for the four automated hard gates. ``/health`` itself is pinned back
to its pre-OMN-15642 behavior: attach readiness never changes its ``status``.

This module still mirrors ``test_service_health_runtime_verdict.py``'s
``/health``-side shape for the "no aggregate yet" and "fully attached" cases,
which ARE endpoint-symmetric; only the DEGRADED/FAILED cases diverge per the
scoping decision above.

Related Tickets:
- OMN-15642: onex-dev correlation-keyed projection/system-event regression
Expand Down Expand Up @@ -138,33 +158,38 @@ async def test_without_an_aggregate_the_payload_cannot_see_the_gap(self) -> None
assert body["status"] == "healthy"

@pytest.mark.asyncio
async def test_degraded_attach_readiness_degrades_the_reported_status(
async def test_degraded_attach_readiness_does_not_degrade_gated_health_status(
self,
) -> None:
"""THE FIX. A DEGRADED attach aggregate is now visible in `status`.

RED against pre-OMN-15642 code: this attach_readiness() call is a
real, already-existing seam (OMN-15512) -- what was missing was
``fold_attach_readiness_into_status`` folding it into ``status``, so
pre-fix this assertion reads "healthy", not "degraded" (proven by
stashing the runtime_health_block.py / health_checker.py hunks of
this change and re-running: fails on this exact line).
"""REMEDIATION (RED against the first OMN-15642 fix, GREEN after).

A DEGRADED attach aggregate must NOT flip ``/health``'s gated
``status`` -- see the module docstring's "REMEDIATION" section for the
deploy-wedge this closes. RED against the merged-then-reverted
version of this fix (commit 363120f4c, dev): that version folded
``fold_attach_readiness_into_status`` into ``_handle_health`` too, so
this assertion read "degraded", not "healthy" (proven by re-adding
that fold locally and re-running: fails on this exact line). The
aggregate itself is still fully visible, just not through the gated
field -- via ``details.degraded`` (a pre-existing, non-gated flag) and
the nested ``components.runtime_wiring`` detail (OMN-15512).
"""
server = ServiceHealth(runtime=_live_runtime(), version="1.0.0")
server.attach_readiness(_readiness(attached=40, required=41))

status, body = await _get_health(server)

assert body["status"] == "degraded"
assert body["details"]["degraded"] is True
assert body["status"] == "healthy"
assert status == 200
# The aggregate did attach and IS visible -- just not gated.
assert body["details"]["components"]["runtime_wiring"]["status"] == "degraded"

# The HTTP status code is deliberately unchanged on /health: it is
# also the liveness probe autoheal/k8s watches, and a restart-immune
# degradation (a dispatcher-scope guard that will raise again on
# restart) must not become a restart loop -- same rationale as
# OMN-15217's identical carve-out for the runtime-verdict join.
assert status == 200
# /health/detailed, unlike /health, is not a boot/deploy probe target
# for any of the four hard-gate consumers -- the fold stays wired
# there, so the SAME aggregate is honestly visible in its status.
detailed_status, detailed_body = await _get_health_detailed(server)
assert detailed_body["status"] == "degraded"
assert detailed_status == 200

@pytest.mark.asyncio
async def test_ready_attach_aggregate_leaves_a_healthy_runtime_healthy(
Expand All @@ -180,12 +205,15 @@ async def test_ready_attach_aggregate_leaves_a_healthy_runtime_healthy(
assert body["details"]["components"]["runtime_wiring"]["status"] == "healthy"

@pytest.mark.asyncio
async def test_core_gap_reports_unhealthy_without_flipping_liveness(
async def test_core_gap_does_not_flip_gated_health_status_either(
self,
) -> None:
"""A FAILED aggregate (core control-plane contract gap) still only
degrades the payload body, never the liveness HTTP status -- pinned
symmetrically with the DEGRADED case above.
"""A FAILED aggregate (core control-plane contract gap) ALSO does not
touch /health's gated status -- the fold is fully unwired from
``_handle_health`` for both readiness states, not just DEGRADED, so
there is no partial/inconsistent behavior depending on severity.
``/health/detailed`` still carries the fold (pinned symmetrically
below and in ``test_detailed_endpoint_flips_http_status_on_core_gap``).
"""
readiness = ModelRuntimeAttachReadiness(
state=EnumRuntimeReadinessState.FAILED,
Expand All @@ -203,7 +231,7 @@ async def test_core_gap_reports_unhealthy_without_flipping_liveness(

status, body = await _get_health(server)

assert body["status"] == "unhealthy"
assert body["status"] == "healthy"
assert status == 200

@pytest.mark.asyncio
Expand Down
Loading