Skip to content

fix(OMN-14968): dev-lane runtime-worker replicas resolve fail-closed per lane (silent-zero blocked every dev deploy at RT-6) - #2536

Merged
jonahgabriel merged 1 commit into
devfrom
jonah/omn-14968-dev-worker-replicas-fail-closed
Jul 29, 2026
Merged

jonahgabriel merged 1 commit into
devfrom
jonah/omn-14968-dev-worker-replicas-fail-closed

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Jul 29, 2026 •

Copy link
Copy Markdown
Collaborator

Evidence-Ticket: OMN-14968
Evidence-Source: OCC#5417

Ticket: OMN-14968 (bare reference — deliberately NO closing keyword; this is a
live-readback-class ticket whose Done requires a real dev-lane deploy certifying
past RT-6, which this PR enables but does not itself prove).

Defect

docker/docker-compose.infra.yml resolved runtime-worker's deploy.replicas
from a bare ${WORKER_REPLICAS:-0}. No surface in this repo exports that
name (grep across runtime-policy.env, deploy-runtime.sh, ~/.omnibase/.env
on .201 → zero hits), so it always took the silent 0 branch — and
docker-compose.infra.yml with no overlay is the dev lane.

Consequence chain: docker compose up -d --no-deps runtime-worker exits 0
creating nothing → runtime-worker is in deploy-runtime.sh's RUNTIME_SERVICES,
which feeds the RT-6 deploy-readback scope → every dev-lane deploy aborts with
could not resolve a running container for in-scope service 'runtime-worker'
and auto-restores. Sole remaining blocker for dev-lane deploy certification.

Design resolution (three surfaces disagreed)

The compose comment claimed scale-to-0 was deliberate; the policy contract said
DEV_WORKER_REPLICAS=1; RUNTIME_SERVICES required the service present.
Resolved for the worker running, from five agreeing surfaces:

  1. contracts/services/runtime_policy.contract.yaml:92-96 — the dev profile's
    worker declares replicas: 1, explicitly, with an OMN-12990 comment. Same for
    all four lanes.
  2. docker/runtime-policy.env:33 — DEV_WORKER_REPLICAS=1, rendered from (1).
  3. tests/ci/test_runtime_policy_contract.py already asserts
    env["DEV_WORKER_REPLICAS"] == "1" — the dev pin is a pre-existing tested
    invariant, not an inference.
  4. The same service block in the same file already resolves its sibling vars
    lane-prefixed and fail-closed
    (ONEX_RUNTIME_CAPABILITIES: ${DEV_RUNTIME_WORKER_CAPABILITIES:?...}, plus both
    secret-resolver vars). Only the replicas line kept the bare name + silent
    default. docker/catalog/services/runtime-worker.yaml likewise uses
    ${DEV_RUNTIME_WORKER_CAPABILITIES:?...} and declares no replicas override.
  5. deploy/lane-census/lane-manifest.yaml:44-47 — "A service whose replicas
    resolve to 0 is itself drift (the WORKER_REPLICAS silent-zero regression):
    declaring replicas: 0 here is forbidden by the parity test."

The stray comment is stale, and it says so itself. OMN-7569 (PR #1162) wrote
"scale to 0 until the kernel supports profile-based subscription filtering."
That condition is met: each lane process now carries a disjoint capability set
(DEV_RUNTIME_WORKER_CAPABILITIES=workflow.dispatch,contract.update,runtime.worker
vs main's market.skill-proof,workflow.orchestration,runtime.main), enforced
fail-closed. Empirically, stability-test and prod both run a worker off this same
base file today with replicas: 1 pinned.

The counter-hypothesis (dev is designed worker-less, so remove it from RT-6
scope) has no support beyond that comment. The lane-census dev block lists only
2 services, but it also omits projection-api, postgres, redpanda and valkey —
services that unquestionably run in dev — so it is an under-specified optional:
census, not a statement of lane composition.

Seam declaration — which surfaces changed and how they now agree

Surface Before After
docker/docker-compose.infra.yml runtime-worker.deploy.replicas ${WORKER_REPLICAS:-0} (bare, silent) ${DEV_WORKER_REPLICAS:?runtime policy contract must set DEV_WORKER_REPLICAS (source docker/runtime-policy.env)}
docker/runtime-policy.env (contract-rendered) DEV_WORKER_REPLICAS=1 unchanged — now actually consumed
contracts/services/runtime_policy.contract.yaml dev worker replicas: 1 unchanged — now actually reaches compose
deploy-runtime.sh RUNTIME_SERVICES / RT-6 scope requires runtime-worker unchanged — now satisfiable
OMN-7569 compose comment claimed scale-to-0 is deliberate corrected: superseded, with the reason its own "until" condition is met
docker/README.md, docker/env-example-full.txt, docs/env-example-full.txt documented the bare WORKER_REPLICAS=N knob now document DEV_WORKER_REPLICAS, noting it is fail-closed

The var name is identical to what the renderer emits ({PROFILE}_WORKER_REPLICAS
in scripts/render_runtime_policy_env.py:121), and matches the prod/stability
overlays' message form character-for-character. No new key, type or nesting is
introduced — this seam consumes an existing ledgered value that nothing read.

deploy/lane-census/lane-manifest.yaml is deliberately not touched: it is
attestation-gated (OMN-13034 requires a fresh live verified: line on any edit)
and the dev block's under-specification is a separate concern.

Executed evidence

All renders executed on .201 (the only host here with Docker), read-only
docker compose config — no lane was mutated.

RED (base, ${WORKER_REPLICAS:-0})

dev: runtime-worker present=True  replicas=0   n_services=20
stability-test: replicas=1 | prod: replicas=1 | judge: runtime-worker not in render

GREEN (this branch)

dev: runtime-worker present=True  replicas=1   n_services=20

Fail-closed proof (the value cannot silently default)

Rendering with DEV_WORKER_REPLICAS stripped from the policy env:

exit=1  stdout_bytes=0
error while interpolating services.runtime-worker.deploy.replicas: required
variable DEV_WORKER_REPLICAS is missing a value: runtime policy contract must set
DEV_WORKER_REPLICAS (source docker/runtime-policy.env)

The same env against the pre-change tree: exit=0, rendered replicas=0 —
i.e. the old form failed OPEN into the zero-container state, silently.

policy env present policy env missing the var
before exit 0, replicas 0 exit 0, replicas 0 (silent)
after exit 0, replicas 1 exit 1, explicit error

Blast-radius / render parity — the other three lanes

docker-compose.infra.yml is shared by all four lanes, so every lane was
re-rendered pre and post with identical env and diffed (paths normalized, since
the two trees live at different scratch roots):

dev:            DIFF -> 1340c1340  <  replicas: 0   --- >  replicas: 1     (ONE line)
stability-test: BYTE-IDENTICAL  sha256=8a2eb6eb0f1eaee4daaa04da8f35492c
prod:           BYTE-IDENTICAL  sha256=1e3a2da8fd52e2104fd4e67477d119e3
judge:          BYTE-IDENTICAL  sha256=4dbf30f7fabf181f2a13c114023e4751

The dev diff is exactly one line in the whole 60KB render. stability-test and
prod are unaffected because their overlays already override deploy.replicas
with their own lane-prefixed :? value; judge is unaffected because
runtime-worker carries profiles: ["runtime","full"] and the judge lane renders
under --profile judge, so the service is filtered out of that render entirely.
All three still interpolate the new var successfully because every lane render
already sources docker/runtime-policy.env (as they must — the sibling
DEV_RUNTIME_WORKER_* :? vars in this same file already require it).

Committed ratchets (this repo's existing harness, no new pattern invented)

tests/integration/infra/test_dev_runtime_compose_render.py — the module OMN-15173
created for the identical defect class (DEV_REDPANDA_ADVERTISE_HOST), extended
with two executed-render tests:

  • test_dev_lane_renders_one_runtime_worker_replica — dev render must yield
    deploy.replicas == 1.
  • test_dev_worker_replicas_fails_closed_when_policy_value_unset — with the var
    stripped from the policy env file, the render must FAIL; asserts the error names
    the var and that no replicas: 0 was emitted.

Proven non-vacuous by execution — both FAIL against the unfixed base compose:

E   AssertionError: dev-lane runtime-worker must render deploy.replicas == 1
    (the ledgered DEV_WORKER_REPLICAS); got 0
E   AssertionError: docker compose config unexpectedly succeeded with
    DEV_WORKER_REPLICAS unset — the silent-zero hole is back
FAILED test_dev_lane_renders_one_runtime_worker_replica
FAILED test_dev_worker_replicas_fails_closed_when_policy_value_unset
2 failed, 2 deselected

and all 4 pass with the fix (run on .201, which has Docker):
4 passed in 0.34s.

tests/ci/test_runtime_policy_contract.py — the static (docker-free) twin now
covers the base file too: asserts ${DEV_WORKER_REPLICAS:? is present, no :-
soft form, and that replicas: ${WORKER_REPLICAS never returns.

tests/integration/docker/test_docker_integration.py — DEV_WORKER_REPLICAS
added to COMPOSE_CONFIG_RENDER_ENV, required by the OMN-15263
tests/ci/test_compose_required_env_coverage.py gate (that fixture's --env-file
is a non-literal path, so the gate cannot see its env-file coverage; the other
three fixtures inherit the var from docker/runtime-policy.env automatically).

Bonus: a ratchet that has been silently RED since 2026-06-23

scripts/deploy-agent/tests/unit/test_runtime_worker_census.py::test_stability_override_pins_worker_replicas_to_literal_one
asserted the OMN-12988 literal 1 pin, which OMN-12990 deliberately replaced
12 days later with ${STABILITY_TEST_WORKER_REPLICAS:?...}. It has been failing
ever since, invisible because scripts/deploy-agent/tests/ is a separate pytest
root the repo suite does not collect. Verified identical failure on unpatched
origin/dev. Corrected here to the current invariant (literal 1 or a :?
reference; any :- soft default rejected) and renamed accordingly.

Gates (run on .200 / stickybeatz-studio, per rule 11a)

Patch-transfer with sha256 verification both sides — the combined sha of all 15
changed files matched local exactly (7ae6c13b…) before any gate ran, so no gate
here was run against an unedited copy.

  • ruff format --check src/ tests/ scripts/ → 4770 files already formatted
  • ruff check src/ tests/ scripts/ → All checks passed!
  • mypy src/omnibase_infra/ → Success: no issues found in 2624 source files
  • Governed selector (scripts/ci/detect_test_paths.py) → escalated to the full
    suite (full_suite_reason: shared_module), so the full suite was run:
    26 failed, 26727 passed, 491 skipped, 1 xfailed, 42 errors in 550s.
    Every failure is environment-dependent (LLM SLO endpoints, Kafka, Postgres,
    docker runtime-health probes, and from_env unit tests sensitive to ambient env
    under xdist) and reproduces on a clean origin/dev worktree on the same host.
    The three from_env-class failures pass standalone on this branch
    (65 passed in 0.19s), confirming ambient-env/parallelism, not this change.
  • Focused compose/policy/census set on this branch: 322 passed, 42 skipped
    (test_runtime_policy_contract, test_compose_required_env_coverage,
    test_lane_census_manifest_parity, test_lane_census_plan, tests/unit/infra/,
    tests/unit/docker/, tests/integration/infra/, test_docker_integration).
  • deploy-agent census suite: 2 passed (was 1 failed / 1 passed on base).
  • pre-commit run --all-files: 2 hooks fail — validate-spdx-headers (two files
    this PR does not touch, 2026 vs 2025 copyright) and check-required-env-vars
    (GITHUB_TOKEN absent from the .200 host's ~/.omnibase/.env). Both fail
    identically on unpatched origin/dev on the same host; verified, not assumed.
    Notably that env-var hook does NOT flag the new DEV_WORKER_REPLICAS, because
    docker/runtime-policy.env supplies it.

dod_evidence

  • Fix: docker/docker-compose.infra.yml — replicas: ${DEV_WORKER_REPLICAS:?...}
  • Executed RED→GREEN: dev render replicas 0 → 1; unset-var render exit 0 (silent 0) → exit 1 (named error); both captured verbatim above.
  • Non-vacuity: the two new tests fail against the unfixed compose, pass with it.
  • Blast radius: stability-test / prod / judge renders byte-identical pre vs post
    (sha256 above); dev render differs by exactly one line.
  • Ratchets: 2 executed-render tests + 3 static assertions + 1 repaired census test.
  • Gates: ruff/mypy clean, full suite run with every failure proven pre-existing on
    origin/dev on the same host.

Unblocks the dev-lane deploy certification RT-6 readback, which gates OMN-15348 AC4
(reconcile-by-redeploy). Not claimed here: no dev-lane deploy has been run against
this branch — that readback is the ticket's live proof and remains open.

…per lane

The base compose resolved runtime-worker deploy.replicas from a BARE
${WORKER_REPLICAS:-0}. No surface in the repo exports that name, so it always
took the silent 0 branch — and docker-compose.infra.yml with no overlay IS the
dev lane. `docker compose up -d --no-deps runtime-worker` therefore exited 0
creating nothing, while deploy-runtime.sh RUNTIME_SERVICES (which feeds the RT-6
deploy readback scope) requires a running container, so every dev-lane deploy
aborted at the fail-closed readback and auto-restored.

Resolved to the lane-prefixed fail-closed form used by the sibling
DEV_RUNTIME_WORKER_* vars in the same service block:
  replicas: ${DEV_WORKER_REPLICAS:?runtime policy contract must set ...}

OMN-12990 already pinned worker replicas: 1 for EVERY lane in
contracts/services/runtime_policy.contract.yaml and converted the stability-test
and prod overlays; only this base line kept the bare name and silent default.

Also corrects the stale OMN-7569 scale-to-0 comment (its "until the kernel
supports profile-based subscription filtering" condition is met — per-process
capability sets are live and fail-closed), the operator docs that documented the
now-dead bare knob, and the OMN-12988 deploy-agent census ratchet that has been
RED since OMN-12990 superseded its literal-1 expectation on 2026-06-23.
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 8 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5fdad0bf-c61c-48d7-a43c-66ecbae345ff

📥 Commits

Reviewing files that changed from the base of the PR and between 5abb60b and 7ab3cd4.

📒 Files selected for processing (15)
  • docker/README.md
  • docker/docker-compose.infra.yml
  • docker/docker-compose.prod.yml
  • docker/docker-compose.stability-test.yml
  • docker/env-example-full.txt
  • docs/env-example-full.txt
  • docs/runbooks/stability-test-runtime-lane.md
  • scripts/deploy-agent/tests/unit/test_runtime_worker_census.py
  • scripts/render_runtime_policy_env.py
  • src/omnibase_infra/runtime/models/model_runtime_process_policy.py
  • tests/ci/test_runtime_policy_contract.py
  • tests/integration/docker/test_docker_integration.py
  • tests/integration/infra/test_dev_runtime_compose_render.py
  • tests/integration/infra/test_stability_test_runtime_compose_render.py
  • tests/unit/infra/test_stability_test_runtime_lane.py

Comment @coderabbitai help to get the list of available commands.

jonahgabriel added a commit to OmniNode-ai/onex_change_control that referenced this pull request Jul 29, 2026
#5417)

* evidence(OMN-14968): OCC companion for OmniNode-ai/omnibase_infra#2536

* evidence(OMN-14968): parity evidence check names the executed render command

* evidence(OMN-14968): PASS receipts for all five dod_evidence items

* evidence: make OMN-14968 OCC companion hosted-stable
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Hostile Reviewer — DEGRADED (informational)

Blocking findings (critical): 0
Total findings: 0
Models succeeded: none

Note: All reviewer models failed or were unavailable. Degraded results are informational during the pilot phase (OMN-8468/OMN-8524) and do not block merge. Error: all review endpoints [192.168.86.201:8000 192.168.86.201:8001 ] unreachable — preflight short-circuit (no models available)


Gate semantics (pilot phase)

Verdict Meaning Blocks merge?
passed No critical findings No
blocked CRITICAL findings found Yes
degraded All models unavailable (infra) No (pilot)

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)

@jonahgabriel
jonahgabriel merged commit 11b460d into dev Jul 29, 2026
170 of 184 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-14968-dev-worker-replicas-fail-closed branch July 29, 2026 03:50
jonahgabriel added a commit that referenced this pull request Jul 29, 2026
…lected roots (#2542)

* fix(OMN-15378): wire scripts/deploy-agent/tests into CI + guard uncollected roots

scripts/deploy-agent/tests/ is a standalone uv sub-project (own pyproject.toml
+ uv.lock) that no CI job ever invoked. It sat uncollected for ~5 weeks; a RED
test inside it (asserting a literal OMN-12990 superseded) went unnoticed the
entire time and was only fixed by hand, incidentally, in PR #2536 (OMN-14968).

- New standalone workflow (.github/workflows/deploy-agent-tests.yml) runs
  `uv run --project scripts/deploy-agent --extra dev pytest scripts/deploy-agent/tests`
  on every PR/push touching scripts/deploy-agent/**. Folding the tests into
  the main tests/ tree was rejected: both scripts/deploy-agent/tests/ and
  tests/ declare a top-level `tests` package (their own __init__.py), so
  collecting them in one pytest session raises ImportPathMismatchError
  (verified locally). scripts/deploy-agent also has its own isolated
  dependency set (deploy-agent host runtime), which the standalone
  `--project` invocation preserves instead of masking against the heavier
  root venv.

- RED found while wiring: scripts/deploy-agent/tests/unit/test_executor_lane_selection.py
  had 2 tests asserting the dev lane is single-compose-file. OMN-15379 (merged
  to dev as bcbaa4e, the tip this branch forked from) gave dev/lab its own
  migration-fence overlay (docker-compose.dev-lane.yml) — a real, intentional
  behavior change the uncollected tests never saw. Fixed both assertions to
  match; RED-proof captured before the fix (2 failed, 191 passed -> 193
  passed after).

- Guard (scripts/validation/validate_test_root_collection.py +
  tests/ci/test_validate_test_root_collection.py, wired as a pre-commit hook
  and riding the existing required full-suite/smart-selection pytest job):
  fails closed on any tests/ directory with no wired pytest invocation behind
  it — under tests/, a registered STANDALONE_PROJECT_ROOTS entry with a real
  pyproject.toml + real CI workflow, or an explicitly named
  KNOWN_UNCOLLECTED_DEBT entry. Proven RED-first (removing the new workflow
  file makes the guard fail) and on a synthetic stray dir.

RESIDUALS (out of scope for this PR, flagged not fixed): the guard's sweep
found 4 PRE-EXISTING uncollected roots of the identical class —
scripts/ci/tests/, scripts/tests/, scripts/runtime_build/tests/, and
src/omnibase_infra/services/observability/agent_actions/tests/. Named
explicitly in KNOWN_UNCOLLECTED_DEBT so the guard passes today without
silently forgiving them; each needs its own follow-up ticket.

Checked test_selection_adjacency.yaml impact: no change needed. Deploy-agent
tests run via the new standalone workflow (path-triggered on
scripts/deploy-agent/**), parallel to the governed selector rather than
through it — same pattern as compose-config-lint.yml, prod-promotion-lineage.yml,
and other standalone path-triggered test workflows already in this repo.

OMN-15378

* fix(OMN-15378): add missing DEV_REDPANDA_ADVERTISE_HOST placeholder for real-docker prod tests

CI run 30480905969 (this PR's first push) surfaced a second uncollected-class
RED the moment TestProdComposeImageSlotRealDocker actually got a docker
daemon: docker/docker-compose.infra.yml (merged as the base file for every
lane, including prod) hard-requires DEV_REDPANDA_ADVERTISE_HOST with no
default (OMN-15173's deliberate no-silent-default design). The test's
_DUMMY_BOOTSTRAP_ENV placeholder dict enumerates every other required env var
from both compose files but missed this one -- its 2026-07-26 enumeration
comment says it was validated "against a clean shell (no ~/.omnibase/.env
sourced)" on a machine where the var was apparently already ambient. These
tests skip on both this Mac and .200 (docker unreachable in either), so this
was invisible locally too; the self-hosted CI runner's real, reachable docker
daemon was the first environment ever to execute them for real. Same root
cause as this ticket: an uncollected root hides drift until something finally
runs it.

Fixed by adding DEV_REDPANDA_ADVERTISE_HOST to the placeholder dict, mirroring
the existing PROD_REDPANDA_ADVERTISE_HOST entry.

OMN-15378

* fix(OMN-15378): declare deploy-agent-tests as a MISSING direct gate in the enforcement parity manifest

Adversarial re-verification found that wiring scripts/deploy-agent/tests
into CI (deploy-agent-tests.yml) makes the RED test class VISIBLE but not
BLOCKING: the workflow is its own file with its own run_id, so
ci_summary_gate.py's poll of actions/runs/${RUN_ID}/jobs (scoped to
ci.yml's own run) structurally cannot observe it, and it is not itself a
required status check on omnibase_infra dev (required contexts =
["CI Summary"] only, confirmed live). A red deploy-agent-tests run today
would leave CI Summary green and merge would proceed -- the same
test-exists-but-does-not-gate failure mode this ticket was filed to close,
just shifted from invisible to visible-but-non-blocking.

This repo already has a declarative mechanism for exactly this class of gap
(scripts/enforcement_parity_manifest.yaml + the report-only parity ratchet
in scripts/audit_required_context_parity_cli.py), precedented by the
identical node-migration-sync/OMN-14556 entry already in this file. Adding
a matching deploy-agent-tests entry (coverage: direct) makes the audit
report it MISSING from required_status_checks until the branch-protection
PUT lands, instead of being silently invisible to any gate-completeness
check the way it is today.

Verified report-only, live: `uv run python
scripts/audit_required_context_parity_cli.py report --json` now emits a
MISSING finding for gate=deploy-agent-tests, repo=omnibase_infra --
confirms the manifest wiring is picked up correctly. Mutates no branch
protection.

Re-verified: guard (scripts/validation/validate_test_root_collection.py)
still passes; YAML parses; no test hardcodes gate counts against this
manifest.

OMN-15378

---------

Co-authored-by: jonahgabriel <jonahgabriel@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant