Repository navigation
fix(OMN-15378): wire scripts/deploy-agent/tests into CI + guard uncollected roots - #2542
Merged
Merged
Conversation
…lected roots scripts/deploy-agent/tests/ is a standalone uv sub-project (own pyproject.toml + uv.lock) that no CI job ever invoked. It sat uncollected for ~5 weeks; a RED test inside it (asserting a literal OMN-12990 superseded) went unnoticed the entire time and was only fixed by hand, incidentally, in PR #2536 (OMN-14968). - New standalone workflow (.github/workflows/deploy-agent-tests.yml) runs `uv run --project scripts/deploy-agent --extra dev pytest scripts/deploy-agent/tests` on every PR/push touching scripts/deploy-agent/**. Folding the tests into the main tests/ tree was rejected: both scripts/deploy-agent/tests/ and tests/ declare a top-level `tests` package (their own __init__.py), so collecting them in one pytest session raises ImportPathMismatchError (verified locally). scripts/deploy-agent also has its own isolated dependency set (deploy-agent host runtime), which the standalone `--project` invocation preserves instead of masking against the heavier root venv. - RED found while wiring: scripts/deploy-agent/tests/unit/test_executor_lane_selection.py had 2 tests asserting the dev lane is single-compose-file. OMN-15379 (merged to dev as bcbaa4e, the tip this branch forked from) gave dev/lab its own migration-fence overlay (docker-compose.dev-lane.yml) — a real, intentional behavior change the uncollected tests never saw. Fixed both assertions to match; RED-proof captured before the fix (2 failed, 191 passed -> 193 passed after). - Guard (scripts/validation/validate_test_root_collection.py + tests/ci/test_validate_test_root_collection.py, wired as a pre-commit hook and riding the existing required full-suite/smart-selection pytest job): fails closed on any tests/ directory with no wired pytest invocation behind it — under tests/, a registered STANDALONE_PROJECT_ROOTS entry with a real pyproject.toml + real CI workflow, or an explicitly named KNOWN_UNCOLLECTED_DEBT entry. Proven RED-first (removing the new workflow file makes the guard fail) and on a synthetic stray dir. RESIDUALS (out of scope for this PR, flagged not fixed): the guard's sweep found 4 PRE-EXISTING uncollected roots of the identical class — scripts/ci/tests/, scripts/tests/, scripts/runtime_build/tests/, and src/omnibase_infra/services/observability/agent_actions/tests/. Named explicitly in KNOWN_UNCOLLECTED_DEBT so the guard passes today without silently forgiving them; each needs its own follow-up ticket. Checked test_selection_adjacency.yaml impact: no change needed. Deploy-agent tests run via the new standalone workflow (path-triggered on scripts/deploy-agent/**), parallel to the governed selector rather than through it — same pattern as compose-config-lint.yml, prod-promotion-lineage.yml, and other standalone path-triggered test workflows already in this repo. OMN-15378
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 21 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (7)
Comment |
…or real-docker prod tests CI run 30480905969 (this PR's first push) surfaced a second uncollected-class RED the moment TestProdComposeImageSlotRealDocker actually got a docker daemon: docker/docker-compose.infra.yml (merged as the base file for every lane, including prod) hard-requires DEV_REDPANDA_ADVERTISE_HOST with no default (OMN-15173's deliberate no-silent-default design). The test's _DUMMY_BOOTSTRAP_ENV placeholder dict enumerates every other required env var from both compose files but missed this one -- its 2026-07-26 enumeration comment says it was validated "against a clean shell (no ~/.omnibase/.env sourced)" on a machine where the var was apparently already ambient. These tests skip on both this Mac and .200 (docker unreachable in either), so this was invisible locally too; the self-hosted CI runner's real, reachable docker daemon was the first environment ever to execute them for real. Same root cause as this ticket: an uncollected root hides drift until something finally runs it. Fixed by adding DEV_REDPANDA_ADVERTISE_HOST to the placeholder dict, mirroring the existing PROD_REDPANDA_ADVERTISE_HOST entry. OMN-15378
Contributor
|
| Verdict | Meaning | Blocks merge? |
|---|---|---|
passed |
No critical findings | No |
blocked |
CRITICAL findings found | Yes |
degraded |
All models unavailable (infra) | No (pilot) |
Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8468/OMN-8524)
…n the enforcement parity manifest
Adversarial re-verification found that wiring scripts/deploy-agent/tests
into CI (deploy-agent-tests.yml) makes the RED test class VISIBLE but not
BLOCKING: the workflow is its own file with its own run_id, so
ci_summary_gate.py's poll of actions/runs/${RUN_ID}/jobs (scoped to
ci.yml's own run) structurally cannot observe it, and it is not itself a
required status check on omnibase_infra dev (required contexts =
["CI Summary"] only, confirmed live). A red deploy-agent-tests run today
would leave CI Summary green and merge would proceed -- the same
test-exists-but-does-not-gate failure mode this ticket was filed to close,
just shifted from invisible to visible-but-non-blocking.
This repo already has a declarative mechanism for exactly this class of gap
(scripts/enforcement_parity_manifest.yaml + the report-only parity ratchet
in scripts/audit_required_context_parity_cli.py), precedented by the
identical node-migration-sync/OMN-14556 entry already in this file. Adding
a matching deploy-agent-tests entry (coverage: direct) makes the audit
report it MISSING from required_status_checks until the branch-protection
PUT lands, instead of being silently invisible to any gate-completeness
check the way it is today.
Verified report-only, live: `uv run python
scripts/audit_required_context_parity_cli.py report --json` now emits a
MISSING finding for gate=deploy-agent-tests, repo=omnibase_infra --
confirms the manifest wiring is picked up correctly. Mutates no branch
protection.
Re-verified: guard (scripts/validation/validate_test_root_collection.py)
still passes; YAML parses; no test hardcodes gate counts against this
manifest.
OMN-15378
jonahgabriel
added a commit
to OmniNode-ai/onex_change_control
that referenced
this pull request
Jul 29, 2026
#5480) * evidence: OCC companion pass 1 for OmniNode-ai/omnibase_infra#2542 * evidence: OCC companion self-bind for #5480 --------- Co-authored-by: node-occ-companion-effect <occ-companion-effect@omninode.ai>
jonahgabriel
added a commit
that referenced
this pull request
Jul 30, 2026
…ink meta-guard baseline to zero (#2555) * fix(OMN-15410): collect the 4 remaining uncollected pytest roots, shrink meta-guard baseline to zero OMN-15378 (#2542) wired scripts/deploy-agent/tests into CI and added a meta-guard that inventoried the test roots CI never collects. It recorded four more roots of the identical class as KNOWN_UNCOLLECTED_DEBT. This collects all four and closes the two seams that let the class exist. Roots now collected (366 tests that had never run in CI): scripts/ci/tests, scripts/tests, scripts/runtime_build/tests, src/omnibase_infra/services/observability/agent_actions/tests Rot found by running them for the first time: 4 tests in the agent_actions root have been RED since 2026-04-02. OMN-7227 (#1127) made ConfigAgentActionsConsumer.kafka_bootstrap_servers required; these four constructions never supplied it and raised ValidationError for ~17 weeks with zero signal. Fixed test-side, supplying the same literal the already-passing _make_health_consumer helper in the same module uses, not by relaxing the field, since fail-fast on missing Kafka wiring is what OMN-7227 bought. Mechanism, not just a baseline edit: * pyproject.toml testpaths is now the single source of truth for what CI collects, and ci.yml full-suite step passes NO positional path so it inherits that list verbatim. It previously hardcoded "pytest tests/", which made the workflow, not pyproject, the real definition of the suite: a root added to testpaths would still never have run. * scripts/ci/detect_test_paths.py gains COLLOCATED_TEST_ROOTS so a NARROWED smart-selection run can reach the collocated roots from a diff touching their code, rather than only via an unrelated full-suite escalation. * ModelTestSelection TestPath widened to admit a collocated tests/ dir; the final path component must still be "tests", so the selector still cannot emit a source directory. * The guard now derives collected roots from testpaths, rejects a testpaths entry with no directory behind it (pytest exit 5), fails closed on an empty testpaths, asserts ci.yml passes no positional path, and asserts testpaths vs COLLOCATED_TEST_ROOTS parity in both directions. * KNOWN_UNCOLLECTED_DEBT is now EMPTY: the guard has converged to zero known uncollected roots, its stated OMN-15378 goal. docker/runners/runner-image.lock.json regenerated: runner_image_identity binds pyproject.toml bytes, so the testpaths edit re-binds the identity (digest-only change, image_version unchanged at v6). Caught by tests/ci/test_runner_image_identity.py, which is exactly the gate working. Refs: OMN-15410, OMN-15378, OMN-14880 * chore(OMN-15410): retrigger CI after autobind added Evidence-Source: OCC#5556 --------- Co-authored-by: jonahgabriel <jonahgabriel@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/deploy-agent/tests/is a standalone uv sub-project (ownpyproject.toml+uv.lock, deployed independently onto the.201host) that no CI job has ever invoked. It sat completely uncollected for ~5 weeks: a RED test inside it (test_stability_override_pins_worker_replicas_to_literal_one, asserting the OMN-12988 literal that OMN-12990 superseded) was invisible the entire time and was only fixed by hand, incidentally, in an unrelated PR (#2536 / OMN-14968). This PR closes that class per OMN-15378.What changed
Wired into CI: new standalone workflow
.github/workflows/deploy-agent-tests.ymlrunsuv run --project scripts/deploy-agent --extra dev pytest scripts/deploy-agent/tests -von every PR/push touchingscripts/deploy-agent/**.tests/? Bothscripts/deploy-agent/tests/and the roottests/declare a top-leveltestspackage (each has its own__init__.py). Collecting them in one pytest session raisesImportPathMismatchError(verified locally — reproduced and documented in the guard's module docstring).scripts/deploy-agentalso has its own isolated dependency set for its.201host runtime; testing it through the standalone--projectinvocation preserves that instead of masking a real test/deploy dependency mismatch behind the heavier root venv.test_selection_adjacency.yamlimpact: no change needed. This runs via its own path-triggered workflow, parallel to (not through) the governed selector — same pattern ascompose-config-lint.yml,prod-promotion-lineage.yml, and other standalone test workflows already in this repo.RED found while wiring, fixed (in scope, small):
scripts/deploy-agent/tests/unit/test_executor_lane_selection.pyhad 2 tests asserting the dev lane uses a single compose file. OMN-15379 (merged todevasbcbaa4ea, the exact tip this branch forked from) gave dev/lab its own migration-fence overlay (docker-compose.dev-lane.yml, per operator ruling 15) — a real, intentional behavior change the uncollected tests never saw, so they went stale silently. Fixed both assertions to match the new (correct) behavior.uv run --project scripts/deploy-agent --extra dev pytest scripts/deploy-agent/tests -qagainst the pre-fix code:2 failed, 191 passed, 8 skipped. After the fix:193 passed, 8 skipped. Reproduced independently on.200's clean environment.Guard (
scripts/validation/validate_test_root_collection.py+tests/ci/test_validate_test_root_collection.py): fails closed on anytests/directory with no wired pytest invocation behind it. Atests/dir is legitimate only if it is (a) under the root-collectedtests/tree, (b) a registeredSTANDALONE_PROJECT_ROOTSentry with a realpyproject.tomland a real, live CI workflow file, or (c) explicitly named inKNOWN_UNCOLLECTED_DEBT(pre-existing debt, not silently forgiven).onex-validate-test-root-collection,always_run: true) and as a normal test undertests/ci/, so it rides the existing required full-suite / smart-selection pytest job — no branch-protection mutation needed.scripts/deploy-agent/tests/: registered ... but the wiring workflow ... does not exist) — verified locally before restoring it.test_synthetic_stray_tests_dir_is_a_violationbuilds a throwawayscripts/widget/tests/test_widget.pyin atmp_pathrepo and asserts the guard flags it.RESIDUALS (out of scope here, flagged not fixed)
The guard's sweep of the live repo found 4 pre-existing uncollected
tests/roots of the identical class, named explicitly inKNOWN_UNCOLLECTED_DEBTso the guard passes today without silently forgiving them:scripts/ci/tests/(test_reject_node_implementations.py,test_sweep_clis.py)scripts/tests/(Keycloak seed/contract tests)scripts/runtime_build/tests/(test_verify_dev_refresh.py,test_verify_stability_refresh.py)src/omnibase_infra/services/observability/agent_actions/tests/(routing-decision/writer/hardening tests)None of these have their own standalone project (unlike
scripts/deploy-agent), so unlike this PR's fix, closing them means either folding them intotests/(with real import-path work) or standing up dedicated workflows for each. Requesting follow-up ticket(s) for these — same false-green risk class as this ticket, unverified as RED/GREEN today since they've never run.Test plan
scripts/deploy-agent/tests/collected + passing on.200's clean env:193 passed, 8 skipped— will also execute live on this PR's CI run (link once green).test_dev_lane_matches_legacy_module_constants,test_dev_lane_compose_up_uses_base_file_and_project), captured above.tests/ci/test_validate_test_root_collection.py::test_synthetic_stray_tests_dir_is_a_violation) and on a missing-workflow standalone registration (test_synthetic_unregistered_standalone_project_fails_closed).ruff format --check,ruff check,mypy,pre-commit run --files <changed>— all green on.200.pre-commit run --files ...full hook set green on.200(SPDX header, duplication sweep, arch validation, etc.)OMN-15378
Evidence-Ticket: OMN-15378
Evidence-Source: OCC#5480