Skip to content

ci(OMN-20073): add repo-owned evidence caller for omniclaude, start S5 shadow count - #2588

Merged
jonahgabriel merged 5 commits into
devfrom
jonah/omn-20073-omniclaude-repo-evidence-caller
Oct 8, 2026
Merged

jonahgabriel merged 5 commits into
devfrom
jonah/omn-20073-omniclaude-repo-evidence-caller

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Pull request omniclaude#2588 addresses ticket OMN-20073 (OCC retirement OR.5, epic OMN-20068). This PR adds a repo-owned evidence caller for omniclaude and initiates the S5 shadow count. The omnimarket canary cut over (omnimarket#3398 merged), and omnibase_core#1900 starts the same S5 shadow count there.

What changes

  • .github/workflows/call-repo-evidence-gate.yml now passes shadow "true" beside compare-with-occ "true", evidence-source caller, and verifier-version 0.4.294. The comment explains the shadow shape.
  • The reusable pin is now omnibase_core#1901's squash commit on dev, 033db282217c224f8063b94966a7f3fdd26b7894.
  • contracts/OMN-20073.yaml has four acceptance criteria (AC1-AC4) and four evidence requirements, each backed by a test_passes item. Its dod_evidence items carry no binds_ac, because the validator that contract-validation runs (onex_change_control main, through the dev pin 47342e562a51) refuses that key as extra_forbidden. The binding test checks only contracts that declare binds_ac, the shape omnibase_infra merged for this ticket.
  • tests/ci/test_repo_evidence_caller_workflow.py gains 4 sweep tests and a test proving the binding check still fails on a contract that declares binds_ac with a missing criterion or a missing test file (8 tests in the file).
  • The shadow behaviour is in the shared reusable, so it was changed once for every caller: omnibase_core#1901 adds a string input shadow (default "false") to .github/workflows/receipt-gate.yml. With evidence-source caller and shadow "true", verify runs as a reported no-op (success, not skipped), every dod-verify step except Summarise has step-level continue-on-error, steps after a refused same-repository check or ticket resolution are gated so a fork head is never checked out, and Summarise writes one line "Shadow verdict: head= new_path=<verified|refused|exempt> refused_step=<step id|none> occ_difference=<outcome/reason_code|not_compared/unavailable|not_run>" to the job summary and a notice annotation. The existing "OCC difference:" summary line is unchanged.

Why shadow mode and not a sweep exclusion

Earlier state of this PR: its caller would make omniclaude's CI Summary L5 default-deny sweep (scripts/ci/ci_summary_gate.py, SWEEP_GOOD_CONCLUSIONS = {"success"}) go red on every pull request, because repo-evidence / verify concluded skipped and repo-evidence / dod-verify concludes failure on any refusal. A first draft asked for two EXTERNAL_SWEEP_EXCLUSIONS entries. The orchestrator ruled that out as allowlist widening and set the shape: in shadow mode no repo-evidence job may produce a non-success check-run; each job records its verdict as data and concludes success.

What does not change

  • No EXTERNAL_SWEEP_EXCLUSIONS entry.
  • No branch protection or ruleset change.
  • repo-evidence / dod-verify is not a required context.
  • Every OCC context stays required.
  • receipt-gate.yml is not edited in this repository.
  • .github/workflows/contract-validation.yml is identical to dev. A bump of its validate-contract pin to ce43a1c4f95e was reverted: that action checks out its validators at github.action_ref, which resolved to v4 inside the composite, and the checkout failed.

Verification

  • uv run pytest tests/ci/test_repo_evidence_caller_workflow.py: 8 passed.
  • pre-commit on the three changed files passed.
  • omnibase_core#1901: 212 passed across the receipt-gate and bot-exempt tests.
  • Sweep proof (evaluate_external_sweep, with exclusions={} and conditional_exclusions={}): rows repo-evidence / verify success and repo-evidence / dod-verify success give no failure and both names are in the swept list; a dod-verify success row carrying a planted OCC disagreement (occ_difference=unclassified_difference/none in its output) gives no failure; rows verify skipped and dod-verify failure still give two failures; neither name is in EXTERNAL_SWEEP_EXCLUSIONS or CONDITIONAL_SWEEP_EXCLUSIONS. The recording of a planted disagreement by the reusable is tested in omnibase_core#1901 (test_a_planted_disagreement_is_recorded_and_the_job_still_passes).
  • uv run validate-yaml contracts/OMN-20073.yaml from onex_change_control main reports OK; the previous revision with binds_ac reports extra_forbidden.

Status

  • omnibase_core#1901 merged as 033db282 and the pin is its squash commit.
  • The caller does not run on this PR itself because pull_request_target reads the base branch.
  • The PR is handed to the landing controller.

Evidence-Ticket: OMN-20073
Evidence-Source: OCC#13230

…5 shadow count

Ticket OMN-20073 (OCC retirement OR.5, epic OMN-20068). The omnimarket canary cut over in omnimarket#3398; omnibase_core#1900 started its S5 shadow count with the same caller.

Adds .github/workflows/call-repo-evidence-gate.yml, copied from omnimarket's caller shape: trigger pull_request_target on dev and main (types opened, synchronize, reopened, edited, ready_for_review), permissions contents read and pull-requests read, one job repo-evidence calling OmniNode-ai/omnibase_core/.github/workflows/receipt-gate.yml pinned at full sha 39ad8e5c88f607aed6a035a3241c96ed36d547a0 (omnibase_core#1886), evidence-source caller, verifier-version 0.4.294, compare-with-occ "true".

compare-with-occ "true" is the S5 shadow: each run records the new-path verdict beside OCC's occ-preflight / eligibility verdict on the same head. The context repo-evidence / dod-verify is not required; no OCC context is removed.

Adds contracts/OMN-20073.yaml with acceptance criteria AC1, AC2 and AC3, each bound by binds_ac to a test_passes item.

Adds tests/ci/test_repo_evidence_caller_workflow.py with 3 tests: caller workflow shape, compare-with-occ for the S5 shadow count, and every repo contract binds every criterion.
…ary's sweep stays green

The caller passes shadow "true" to the receipt-gate reusable (omnibase_core#1901), so repo-evidence / verify and repo-evidence / dod-verify conclude success and record their verdict as data; no EXTERNAL_SWEEP_EXCLUSIONS entry is added. Adds AC4 and four evaluate_external_sweep tests. The pin is the head of omnibase_core#1901 until it merges.
Bump the reusable receipt-gate.yml in
.github/workflows/call-repo-evidence-gate.yml for omniclaude.
Pin moves from 6d791925396535b44a5e72de2c3d4718a9ca72ed (pre-merge head of omnibase_core#1901) to
033db282217c224f8063b94966a7f3fdd26b7894 (squash commit of
omnibase_core#1901 on dev, adds the shadow input).
The PLACEHOLDER pin comment is removed.
Caller stays shadow "true" and compare-with-occ "true".
Repo-evidence caller test and receipt-gate pin test pass.
@jonahgabriel
jonahgabriel marked this pull request as ready for review October 7, 2026 20:23
@jonahgabriel

Copy link
Copy Markdown
Contributor Author

omnibase_core#1901 merged on dev as 033db282217c224f8063b94966a7f3fdd26b7894. This pull request now pins the reusable receipt-gate.yml to that commit; the PLACEHOLDER pin comment is gone. The "draft" and "PLACEHOLDER pin" statements in the PR description are out of date; the PR is now ready for review. The caller stays in shadow mode (shadow "true", compare-with-occ "true"); no required context changes.

jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Oct 7, 2026
)

* evidence(OMN-20073): author OCC companion for OmniNode-ai/omniclaude#2588

OCC companion by node_pr_lifecycle_fix_effect (OMN-13317 F1 / OMN-13990 / OMN-14285). Product PR head 8f6da66a850c96090778c6678dc4347ff3806aa2.

* evidence(OMN-20073): self-bind OCC#13219 + rebind contract_sha256

---------

Co-authored-by: omnimarket-bot <bot@omninode.ai>
@jonahgabriel
jonahgabriel enabled auto-merge (squash) October 7, 2026 20:36
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

✅ Hostile Reviewer — PASSED

Blocking findings (critical/error): 0
Total findings: 12
Models succeeded: qwen3-review,gpt-oss-review

Findings

Severity Location Finding
error .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:qwen3-review:security pull_request_target with fork PRs enables arbitrary code execution | The workflow uses pull_request_target, which runs in the base branch context with the repository's secrets and permissions. The comment claims 'The pull request's own code is only ever checked out as data by the reusable, which refuses fork heads.' However, the reusable workflow receipt-gate.yml is pinned to a commit on the dev branch of omnibase_core, not a release tag or verified immutable artifact. If an attacker can push to omnibase_co
warning tests/ci/test_repo_evidence_caller_workflow.py:1 ai-reviewer:qwen3-review:correctness Test for contract binding does not verify binds_ac items have valid check_type | In test_every_repo_contract_binds_every_criterion, the loop iterates over checks and skips any that do not have check_type == 'test_passes' or do not start with 'uv run pytest '. This means a binds_ac item could have a check with check_type 'manual' or any other value, and the test would not validate that the criterion is actually bound to runnable test evidence. The contract OMN-20073.yaml itself uses source: 'manual' for all
warning .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:qwen3-review:completeness Missing test for reusable workflow's fork-head refusal | The comment claims the reusable 'refuses fork heads', but there is no test in this PR that verifies this behavior. The test suite only checks the caller's workflow shape, inputs, and the external sweep behavior. If the reusable workflow does not actually refuse fork heads, the security model described in the comment is invalid. This is a critical security property that should be tested or at least documented with a reference to the test in omnibase_co
info .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:qwen3-review:performance Concurrency group may cause unnecessary cancellation of in-progress runs | The concurrency group is ${{ github.workflow }}-${{ github.event.pull_request.number }} with cancel-in-progress: true. This means that if a PR is edited (triggering a new run) while a previous run is still in progress, the previous run is cancelled. This is intended for 'superseded runs', but it may cause flaky results if the reusable workflow takes a long time to complete and the PR is edited frequently. Consider whether cancel-in-p
info tests/ci/test_repo_evidence_caller_workflow.py:1 ai-reviewer:qwen3-review:testing Test for external sweep uses hardcoded datetime that may become stale | The test uses _SWEEP_NOW = datetime(2026, 10, 7, 15, 0, tzinfo=UTC) as a fixed point in time. If the evaluate_external_sweep function's behavior changes with respect to time (e.g., it starts considering runs older than 24 hours as 'in flight' or 'expired'), this test may fail or pass for the wrong reasons. The test should use a relative time or mock the current time to ensure it tests the intended behavior. | Evidence: _SWEEP_NOW = date
info .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:qwen3-review:architecture Shadow mode creates a dead code path that may confuse future maintainers | The workflow is configured with shadow: 'true' and compare-with-occ: 'true', which means the repo-evidence job always concludes success and the actual verdict is only recorded in the job summary and a notice annotation. This creates a situation where the CI Summary sweep cannot detect failures in the repo-evidence path, which is intentional for the S5 shadow count but may be confusing for future maintainers who expect the sweep to ca
hint tests/ci/test_repo_evidence_caller_workflow.py:1 ai-reviewer:qwen3-review:correctness Test for workflow shape does not verify the reusable workflow's inputs are valid | The test test_caller_workflow_shape verifies that the reusable workflow is pinned by a full SHA and that evidence-source is 'caller', but it does not verify that the other inputs (verifier-version, compare-with-occ, shadow) are valid values for the reusable workflow. If the reusable workflow's input schema changes, this test will not catch it. Consider adding a test that validates the inputs against the reusable workflow's sc
warning .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:gpt-oss-review:security pull_request_target workflow may expose base repository to untrusted PR data | The new workflow is triggered on pull_request_target, which runs with the base repository's permissions while checking out the PR head as data. If the called reusable does not strictly sandbox the PR head, malicious code could be executed with read permissions, creating a supply‑chain risk. | Evidence: on: pull_request_target: types: [opened, synchronize, reopened, edited, ready_for_review] branches: [dev, main] | Fix:
warning .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:gpt-oss-review:correctness Boolean inputs passed as quoted strings may be misinterpreted by the reusable | The workflow supplies compare-with-occ and shadow as "true" strings. If the receipt‑gate reusable expects actual booleans, the string values could be treated as truthy literals or cause type errors, breaking the intended shadow mode behavior. | Evidence: compare-with-occ: "true" shadow: "true" | Fix: Pass unquoted booleans (compare-with-occ: true, shadow: true) or verify that the reusable explicitly parses string literals to boo
info tests/ci/test_repo_evidence_caller_workflow.py:1 ai-reviewer:gpt-oss-review:testing Global monkeypatch of REPO_ROOT is not restored, risking test interference | The test test_contract_binds_check_applies_to_a_contract_that_declares_binds_ac overwrites the module‑level REPO_ROOT variable via monkeypatch.setitem without resetting it after the test, which can affect subsequent tests that rely on the original path. | Evidence: monkeypatch.setitem(globals(), "REPO_ROOT", tmp_path) | Fix: Use monkeypatch.setattr with a context manager or restore the original REPO_ROOT in a teardown step.
info tests/ci/test_repo_evidence_caller_workflow.py:1 ai-reviewer:gpt-oss-review:performance Contract validation test loads all contract files on each run | test_every_repo_contract_binds_every_criterion enumerates every OMN-.yaml file in the repository for each invocation, which may become costly in large repositories and slow down the CI pipeline. | Evidence: contracts = sorted((REPO_ROOT / "contracts").glob("OMN-.yaml")) | Fix: Cache the list of contract paths or limit the test to a representative subset when the repository size exceeds a threshold.
info .github/workflows/call-repo-evidence-gate.yml:1 ai-reviewer:gpt-oss-review:architecture Workflow triggers on both dev and main branches may cause duplicate runs per PR | The workflow is configured to run on pull_request_target for branches dev and main. A PR targeting either branch will trigger the workflow, and if the same PR is opened against both branches (e.g., via backports), duplicate runs could consume resources unnecessarily. | Evidence: branches: [dev, main] | Fix: Restrict the workflow to a single integration branch or add conditional logic to skip execution when the PR base matches

Gate semantics (pilot phase)

Verdict Meaning Blocks merge?
passed No critical/error findings No
blocked CRITICAL or ERROR findings found Yes
degraded Reviewer ran but every model failed mid-run No (pilot)
infra_error cli_review failed to run at all (install/crash) after 2 attempts Yes
unavailable_fork_isolation Fork PR — the LAN-only model is unreachable and the fleet is off limits Yes

Powered by omniintelligence.review_pairing.cli_review — node-based adversarial review via HandlerLlmCliSubprocess (OMN-8524)

@onexbot-occ-writer

Copy link
Copy Markdown
Contributor

OCC autobind rebound this PR's evidence-source stamp line to OCC#13230.

The receipt gate's own eligibility validator, run against the change-control tree the gate pins for OCC#13230, returned eligible for head 23acb93df500b86bdcd8ff842d1a6cd0e71876f0. The body now carries exactly one stamp line. Displaced: OCC#13219.

Reported by occ_companion_emitter (OMN-18853).

… so OCC main's validator accepts the contract

Commit 23acb93 bumped the validate-contract pin to
ce43a1c4f95e, causing a checkout failure at github.action_ref
v4 in job 113026435139. This restores dev pin 47342e562a51.
The onex_change_control validator rejects binds_ac on dod_evidence
as extra_forbidden.  contracts/OMN-20073.yaml drops the four binds_ac
keys. Tests now check only contracts declaring binds_ac, matching the
omnibase_infra shape. 8 focused tests pass and validate-yaml confirms
the new contract is OK while the previous one is invalid.
@jonahgabriel

Copy link
Copy Markdown
Contributor Author

Lab: host=h202 lane=occ-driver-2330 head=58f9a7db5b1305c4660c7f51ae08979ffaaab287 command="uv run pytest tests/ci/test_repo_evidence_caller_workflow.py -q" observed="8 passed"; command="uv run validate-yaml contracts/OMN-20073.yaml" from onex_change_control main a89a6f30fa observed="[OK] ... (ticket_contract)", and the same validator on the previous revision (23acb93) observed "dod_evidence.0.binds_ac: Extra inputs are not permitted [extra_forbidden]". ruff check, ruff format --check and mypy on the test file passed; pre-commit passed on commit and push. Not runtime-affecting, no compose-dev lab pass applies.

@jonahgabriel
jonahgabriel merged commit 07a9815 into dev Oct 8, 2026
253 of 257 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-20073-omniclaude-repo-evidence-caller branch October 8, 2026 03:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant