Repository navigation
fix(OMN-14968): dev-lane runtime-worker replicas resolve fail-closed per lane (silent-zero blocked every dev deploy at RT-6) - #2536
Conversation
…per lane
The base compose resolved runtime-worker deploy.replicas from a BARE
${WORKER_REPLICAS:-0}. No surface in the repo exports that name, so it always
took the silent 0 branch — and docker-compose.infra.yml with no overlay IS the
dev lane. `docker compose up -d --no-deps runtime-worker` therefore exited 0
creating nothing, while deploy-runtime.sh RUNTIME_SERVICES (which feeds the RT-6
deploy readback scope) requires a running container, so every dev-lane deploy
aborted at the fail-closed readback and auto-restored.
Resolved to the lane-prefixed fail-closed form used by the sibling
DEV_RUNTIME_WORKER_* vars in the same service block:
replicas: ${DEV_WORKER_REPLICAS:?runtime policy contract must set ...}
OMN-12990 already pinned worker replicas: 1 for EVERY lane in
contracts/services/runtime_policy.contract.yaml and converted the stability-test
and prod overlays; only this base line kept the bare name and silent default.
Also corrects the stale OMN-7569 scale-to-0 comment (its "until the kernel
supports profile-based subscription filtering" condition is met — per-process
capability sets are live and fail-closed), the operator docs that documented the
now-dead bare knob, and the OMN-12988 deploy-agent census ratchet that has been
RED since OMN-12990 superseded its literal-1 expectation on 2026-06-23.
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 8 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (15)
Comment |
#5417) * evidence(OMN-14968): OCC companion for OmniNode-ai/omnibase_infra#2536 * evidence(OMN-14968): parity evidence check names the executed render command * evidence(OMN-14968): PASS receipts for all five dod_evidence items * evidence: make OMN-14968 OCC companion hosted-stable
|
| Verdict | Meaning | Blocks merge? |
|---|---|---|
passed |
No critical findings | No |
blocked |
CRITICAL findings found | Yes |
degraded |
All models unavailable (infra) | No (pilot) |
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)
…lected roots (#2542) * fix(OMN-15378): wire scripts/deploy-agent/tests into CI + guard uncollected roots scripts/deploy-agent/tests/ is a standalone uv sub-project (own pyproject.toml + uv.lock) that no CI job ever invoked. It sat uncollected for ~5 weeks; a RED test inside it (asserting a literal OMN-12990 superseded) went unnoticed the entire time and was only fixed by hand, incidentally, in PR #2536 (OMN-14968). - New standalone workflow (.github/workflows/deploy-agent-tests.yml) runs `uv run --project scripts/deploy-agent --extra dev pytest scripts/deploy-agent/tests` on every PR/push touching scripts/deploy-agent/**. Folding the tests into the main tests/ tree was rejected: both scripts/deploy-agent/tests/ and tests/ declare a top-level `tests` package (their own __init__.py), so collecting them in one pytest session raises ImportPathMismatchError (verified locally). scripts/deploy-agent also has its own isolated dependency set (deploy-agent host runtime), which the standalone `--project` invocation preserves instead of masking against the heavier root venv. - RED found while wiring: scripts/deploy-agent/tests/unit/test_executor_lane_selection.py had 2 tests asserting the dev lane is single-compose-file. OMN-15379 (merged to dev as bcbaa4e, the tip this branch forked from) gave dev/lab its own migration-fence overlay (docker-compose.dev-lane.yml) — a real, intentional behavior change the uncollected tests never saw. Fixed both assertions to match; RED-proof captured before the fix (2 failed, 191 passed -> 193 passed after). - Guard (scripts/validation/validate_test_root_collection.py + tests/ci/test_validate_test_root_collection.py, wired as a pre-commit hook and riding the existing required full-suite/smart-selection pytest job): fails closed on any tests/ directory with no wired pytest invocation behind it — under tests/, a registered STANDALONE_PROJECT_ROOTS entry with a real pyproject.toml + real CI workflow, or an explicitly named KNOWN_UNCOLLECTED_DEBT entry. Proven RED-first (removing the new workflow file makes the guard fail) and on a synthetic stray dir. RESIDUALS (out of scope for this PR, flagged not fixed): the guard's sweep found 4 PRE-EXISTING uncollected roots of the identical class — scripts/ci/tests/, scripts/tests/, scripts/runtime_build/tests/, and src/omnibase_infra/services/observability/agent_actions/tests/. Named explicitly in KNOWN_UNCOLLECTED_DEBT so the guard passes today without silently forgiving them; each needs its own follow-up ticket. Checked test_selection_adjacency.yaml impact: no change needed. Deploy-agent tests run via the new standalone workflow (path-triggered on scripts/deploy-agent/**), parallel to the governed selector rather than through it — same pattern as compose-config-lint.yml, prod-promotion-lineage.yml, and other standalone path-triggered test workflows already in this repo. OMN-15378 * fix(OMN-15378): add missing DEV_REDPANDA_ADVERTISE_HOST placeholder for real-docker prod tests CI run 30480905969 (this PR's first push) surfaced a second uncollected-class RED the moment TestProdComposeImageSlotRealDocker actually got a docker daemon: docker/docker-compose.infra.yml (merged as the base file for every lane, including prod) hard-requires DEV_REDPANDA_ADVERTISE_HOST with no default (OMN-15173's deliberate no-silent-default design). The test's _DUMMY_BOOTSTRAP_ENV placeholder dict enumerates every other required env var from both compose files but missed this one -- its 2026-07-26 enumeration comment says it was validated "against a clean shell (no ~/.omnibase/.env sourced)" on a machine where the var was apparently already ambient. These tests skip on both this Mac and .200 (docker unreachable in either), so this was invisible locally too; the self-hosted CI runner's real, reachable docker daemon was the first environment ever to execute them for real. Same root cause as this ticket: an uncollected root hides drift until something finally runs it. Fixed by adding DEV_REDPANDA_ADVERTISE_HOST to the placeholder dict, mirroring the existing PROD_REDPANDA_ADVERTISE_HOST entry. OMN-15378 * fix(OMN-15378): declare deploy-agent-tests as a MISSING direct gate in the enforcement parity manifest Adversarial re-verification found that wiring scripts/deploy-agent/tests into CI (deploy-agent-tests.yml) makes the RED test class VISIBLE but not BLOCKING: the workflow is its own file with its own run_id, so ci_summary_gate.py's poll of actions/runs/${RUN_ID}/jobs (scoped to ci.yml's own run) structurally cannot observe it, and it is not itself a required status check on omnibase_infra dev (required contexts = ["CI Summary"] only, confirmed live). A red deploy-agent-tests run today would leave CI Summary green and merge would proceed -- the same test-exists-but-does-not-gate failure mode this ticket was filed to close, just shifted from invisible to visible-but-non-blocking. This repo already has a declarative mechanism for exactly this class of gap (scripts/enforcement_parity_manifest.yaml + the report-only parity ratchet in scripts/audit_required_context_parity_cli.py), precedented by the identical node-migration-sync/OMN-14556 entry already in this file. Adding a matching deploy-agent-tests entry (coverage: direct) makes the audit report it MISSING from required_status_checks until the branch-protection PUT lands, instead of being silently invisible to any gate-completeness check the way it is today. Verified report-only, live: `uv run python scripts/audit_required_context_parity_cli.py report --json` now emits a MISSING finding for gate=deploy-agent-tests, repo=omnibase_infra -- confirms the manifest wiring is picked up correctly. Mutates no branch protection. Re-verified: guard (scripts/validation/validate_test_root_collection.py) still passes; YAML parses; no test hardcodes gate counts against this manifest. OMN-15378 --------- Co-authored-by: jonahgabriel <jonahgabriel@users.noreply.github.com>
Evidence-Ticket: OMN-14968
Evidence-Source: OCC#5417
Ticket: OMN-14968 (bare reference — deliberately NO closing keyword; this is a
live-readback-class ticket whose Done requires a real dev-lane deploy certifying
past RT-6, which this PR enables but does not itself prove).
Defect
docker/docker-compose.infra.ymlresolvedruntime-worker'sdeploy.replicasfrom a bare
${WORKER_REPLICAS:-0}. No surface in this repo exports thatname (
grepacrossruntime-policy.env,deploy-runtime.sh,~/.omnibase/.envon
.201→ zero hits), so it always took the silent0branch — anddocker-compose.infra.ymlwith no overlay is the dev lane.Consequence chain:
docker compose up -d --no-deps runtime-workerexits 0creating nothing →
runtime-workeris indeploy-runtime.sh'sRUNTIME_SERVICES,which feeds the RT-6 deploy-readback scope → every dev-lane deploy aborts with
could not resolve a running container for in-scope service 'runtime-worker'and auto-restores. Sole remaining blocker for dev-lane deploy certification.
Design resolution (three surfaces disagreed)
The compose comment claimed scale-to-0 was deliberate; the policy contract said
DEV_WORKER_REPLICAS=1;RUNTIME_SERVICESrequired the service present.Resolved for the worker running, from five agreeing surfaces:
contracts/services/runtime_policy.contract.yaml:92-96— the dev profile'sworker declares
replicas: 1, explicitly, with an OMN-12990 comment. Same forall four lanes.
docker/runtime-policy.env:33—DEV_WORKER_REPLICAS=1, rendered from (1).tests/ci/test_runtime_policy_contract.pyalready assertsenv["DEV_WORKER_REPLICAS"] == "1"— the dev pin is a pre-existing testedinvariant, not an inference.
lane-prefixed and fail-closed
(
ONEX_RUNTIME_CAPABILITIES: ${DEV_RUNTIME_WORKER_CAPABILITIES:?...}, plus bothsecret-resolver vars). Only the replicas line kept the bare name + silent
default.
docker/catalog/services/runtime-worker.yamllikewise uses${DEV_RUNTIME_WORKER_CAPABILITIES:?...}and declares no replicas override.deploy/lane-census/lane-manifest.yaml:44-47— "A service whose replicasresolve to 0 is itself drift (the WORKER_REPLICAS silent-zero regression):
declaring
replicas: 0here is forbidden by the parity test."The stray comment is stale, and it says so itself. OMN-7569 (PR #1162) wrote
"scale to 0 until the kernel supports profile-based subscription filtering."
That condition is met: each lane process now carries a disjoint capability set
(
DEV_RUNTIME_WORKER_CAPABILITIES=workflow.dispatch,contract.update,runtime.workervs main's
market.skill-proof,workflow.orchestration,runtime.main), enforcedfail-closed. Empirically, stability-test and prod both run a worker off this same
base file today with
replicas: 1pinned.The counter-hypothesis (dev is designed worker-less, so remove it from RT-6
scope) has no support beyond that comment. The lane-census
devblock lists only2 services, but it also omits
projection-api, postgres, redpanda and valkey —services that unquestionably run in dev — so it is an under-specified
optional:census, not a statement of lane composition.
Seam declaration — which surfaces changed and how they now agree
docker/docker-compose.infra.ymlruntime-worker.deploy.replicas${WORKER_REPLICAS:-0}(bare, silent)${DEV_WORKER_REPLICAS:?runtime policy contract must set DEV_WORKER_REPLICAS (source docker/runtime-policy.env)}docker/runtime-policy.env(contract-rendered)DEV_WORKER_REPLICAS=1contracts/services/runtime_policy.contract.yamlreplicas: 1deploy-runtime.shRUNTIME_SERVICES/ RT-6 scoperuntime-workerdocker/README.md,docker/env-example-full.txt,docs/env-example-full.txtWORKER_REPLICAS=NknobDEV_WORKER_REPLICAS, noting it is fail-closedThe var name is identical to what the renderer emits (
{PROFILE}_WORKER_REPLICASin
scripts/render_runtime_policy_env.py:121), and matches the prod/stabilityoverlays' message form character-for-character. No new key, type or nesting is
introduced — this seam consumes an existing ledgered value that nothing read.
deploy/lane-census/lane-manifest.yamlis deliberately not touched: it isattestation-gated (OMN-13034 requires a fresh live
verified:line on any edit)and the dev block's under-specification is a separate concern.
Executed evidence
All renders executed on
.201(the only host here with Docker), read-onlydocker compose config— no lane was mutated.RED (base,
${WORKER_REPLICAS:-0})GREEN (this branch)
Fail-closed proof (the value cannot silently default)
Rendering with
DEV_WORKER_REPLICASstripped from the policy env:The same env against the pre-change tree:
exit=0, renderedreplicas=0—i.e. the old form failed OPEN into the zero-container state, silently.
Blast-radius / render parity — the other three lanes
docker-compose.infra.ymlis shared by all four lanes, so every lane wasre-rendered pre and post with identical env and diffed (paths normalized, since
the two trees live at different scratch roots):
The dev diff is exactly one line in the whole 60KB render. stability-test and
prod are unaffected because their overlays already override
deploy.replicaswith their own lane-prefixed
:?value; judge is unaffected becauseruntime-workercarriesprofiles: ["runtime","full"]and the judge lane rendersunder
--profile judge, so the service is filtered out of that render entirely.All three still interpolate the new var successfully because every lane render
already sources
docker/runtime-policy.env(as they must — the siblingDEV_RUNTIME_WORKER_*:?vars in this same file already require it).Committed ratchets (this repo's existing harness, no new pattern invented)
tests/integration/infra/test_dev_runtime_compose_render.py— the module OMN-15173created for the identical defect class (
DEV_REDPANDA_ADVERTISE_HOST), extendedwith two executed-render tests:
test_dev_lane_renders_one_runtime_worker_replica— dev render must yielddeploy.replicas == 1.test_dev_worker_replicas_fails_closed_when_policy_value_unset— with the varstripped from the policy env file, the render must FAIL; asserts the error names
the var and that no
replicas: 0was emitted.Proven non-vacuous by execution — both FAIL against the unfixed base compose:
and all 4 pass with the fix (run on
.201, which has Docker):4 passed in 0.34s.tests/ci/test_runtime_policy_contract.py— the static (docker-free) twin nowcovers the base file too: asserts
${DEV_WORKER_REPLICAS:?is present, no:-soft form, and that
replicas: ${WORKER_REPLICASnever returns.tests/integration/docker/test_docker_integration.py—DEV_WORKER_REPLICASadded to
COMPOSE_CONFIG_RENDER_ENV, required by the OMN-15263tests/ci/test_compose_required_env_coverage.pygate (that fixture's--env-fileis a non-literal path, so the gate cannot see its env-file coverage; the other
three fixtures inherit the var from
docker/runtime-policy.envautomatically).Bonus: a ratchet that has been silently RED since 2026-06-23
scripts/deploy-agent/tests/unit/test_runtime_worker_census.py::test_stability_override_pins_worker_replicas_to_literal_oneasserted the OMN-12988 literal
1pin, which OMN-12990 deliberately replaced12 days later with
${STABILITY_TEST_WORKER_REPLICAS:?...}. It has been failingever since, invisible because
scripts/deploy-agent/tests/is a separate pytestroot the repo suite does not collect. Verified identical failure on unpatched
origin/dev. Corrected here to the current invariant (literal1or a:?reference; any
:-soft default rejected) and renamed accordingly.Gates (run on
.200/ stickybeatz-studio, per rule 11a)Patch-transfer with sha256 verification both sides — the combined sha of all 15
changed files matched local exactly (
7ae6c13b…) before any gate ran, so no gatehere was run against an unedited copy.
ruff format --check src/ tests/ scripts/→4770 files already formattedruff check src/ tests/ scripts/→All checks passed!mypy src/omnibase_infra/→Success: no issues found in 2624 source filesscripts/ci/detect_test_paths.py) → escalated to the fullsuite (
full_suite_reason: shared_module), so the full suite was run:26 failed, 26727 passed, 491 skipped, 1 xfailed, 42 errors in 550s.Every failure is environment-dependent (LLM SLO endpoints, Kafka, Postgres,
docker runtime-health probes, and
from_envunit tests sensitive to ambient envunder xdist) and reproduces on a clean
origin/devworktree on the same host.The three
from_env-class failures pass standalone on this branch(
65 passed in 0.19s), confirming ambient-env/parallelism, not this change.(
test_runtime_policy_contract,test_compose_required_env_coverage,test_lane_census_manifest_parity,test_lane_census_plan,tests/unit/infra/,tests/unit/docker/,tests/integration/infra/,test_docker_integration).2 passed(was 1 failed / 1 passed on base).pre-commit run --all-files: 2 hooks fail —validate-spdx-headers(two filesthis PR does not touch,
2026vs2025copyright) andcheck-required-env-vars(
GITHUB_TOKENabsent from the.200host's~/.omnibase/.env). Both failidentically on unpatched
origin/devon the same host; verified, not assumed.Notably that env-var hook does NOT flag the new
DEV_WORKER_REPLICAS, becausedocker/runtime-policy.envsupplies it.dod_evidence
docker/docker-compose.infra.yml—replicas: ${DEV_WORKER_REPLICAS:?...}replicas 0 → 1; unset-var renderexit 0 (silent 0) → exit 1 (named error); both captured verbatim above.(sha256 above); dev render differs by exactly one line.
origin/devon the same host.Unblocks the dev-lane deploy certification RT-6 readback, which gates OMN-15348 AC4
(reconcile-by-redeploy). Not claimed here: no dev-lane deploy has been run against
this branch — that readback is the ticket's live proof and remains open.