Repository navigation
fix(OMN-16824): test_passes executes check_value and honours cwd in the hosted gate #7393
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
04e7445
fix(OMN-16824): test_passes executes check_value and honours cwd in t…
jonahgabriel 4426f19
evidence(OMN-16824): OCC self-bind for onex_change_control#7393
jonahgabriel 65aaed1
evidence(OMN-16824): PASS receipts for the two behaviour legs and the…
jonahgabriel eafa063
merge dev into OMN-16824 hosted test-passes fix
jonahgabriel 541a4ac
fix(OMN-16824): address check runner review guards
jonahgabriel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| # `check_type` reference — what each DoD check actually does | ||
|
|
||
| **Audience:** anyone authoring `dod_evidence` in `contracts/OMN-XXXX.yaml`. | ||
|
|
||
| This page exists so you never have to read a runner's source to know what your | ||
| check will do. Two different gates execute the checks you write: | ||
|
|
||
| | Runner | Where it runs | What it is | | ||
| |---|---|---| | ||
| | **Hosted** — `Contract Compliance Check` | every PR, in CI | `onex_change_control.scripts.contract_compliance_check` | | ||
| | **Local** — `node_dod_verify` | the local Done gate / autoclose evidence sweep | omnimarket `EvidenceCollector` | | ||
|
|
||
| Both read the same contract bytes. **For check types both runners execute, this | ||
| page states one shared behaviour** — that is the OMN-16824 rule, and it is | ||
| enforced by an executed case table | ||
| (`tests/fixtures/check_type_runner_semantics.yaml`) that is checked into both | ||
| repos byte-for-byte and run against both runners. Rows marked `—` are hosted-only | ||
| surfaces until the local runner implements them. | ||
|
|
||
| --- | ||
|
|
||
| ## The table | ||
|
|
||
| | `check_type` | `check_value` is | Hosted runner | Local runner | | ||
| |---|---|---|---| | ||
| | `command` | a shell command | **executes it**; exit 0 = PASS | **executes it**; exit 0 = VERIFIED | | ||
| | `test_passes` | a shell command that runs tests | **executes it**; exit 0 = PASS | **executes it**; exit 0 = VERIFIED | | ||
| | `test_exists` | a glob | globs the workspace | — | | ||
| | `file_exists` | a glob | globs the workspace | checks the path exists | | ||
| | `grep` | `{pattern, path}` | greps the workspace | — | | ||
| | `endpoint` | a URL or path | curls / stats it | — | | ||
| | `behavior_proven` | prose attestation | non-executable WARN | non-executable | | ||
| | `semantic_grading` | a receipt path | receipt-gate surface | receipt-gate surface | | ||
|
|
||
| ### `test_passes` is an executed alias of `command` | ||
|
|
||
| Write `test_passes` when the command you are running **is a test run**. It runs | ||
| exactly like `command`; the distinct name records author intent and is what the | ||
| proof-class classifier (OMN-15911) and the substance-floor tier deriver key on. | ||
|
|
||
| It does **not** mean "this PR's CI is green". Until OMN-16824 the hosted runner | ||
| read it that way — it ignored `check_value` entirely and reported the PR's own | ||
| check-run states — so an entry claiming *"the new regression test passes"* | ||
| returned PASS whenever unrelated CI was green, including when the named test did | ||
| not exist. That reading is gone. If you genuinely want to assert PR CI state, | ||
| say so in the open: | ||
|
|
||
| ```yaml | ||
| - check_type: command | ||
| check_value: "gh pr checks 1719 --repo OmniNode-ai/omnimarket" | ||
| ``` | ||
|
|
||
| That is a **merge-state** proof, not a behaviour proof, and the classifier | ||
| scores it as such. | ||
|
|
||
| --- | ||
|
|
||
| ## Where your command runs | ||
|
|
||
| * **No `cwd` declared** → the command runs in the **product checkout under | ||
| test** (the repo whose PR is being gated), which the hosted runner passes as | ||
| `--workspace`. Relative paths resolve against it. | ||
| * **`cwd` declared** → the command runs **there**. Supported template tokens: | ||
| `${OMNI_HOME}`, `${PR_NUMBER}`, `${REPO}`, `${TICKET_ID}`. A relative `cwd` | ||
| resolves against the workspace; `..` segments are refused. | ||
| * **`cwd` declared but unresolvable** → the check is **`NOT_EVALUATED`**, and | ||
| the gate says so out loud. It is **never** rerouted to the workspace: running | ||
| your command in a different tree answers a different question under your | ||
| entry's name. | ||
|
|
||
| The hosted gate checks out **one** repo. A `cwd` naming a sibling checkout | ||
| (`${OMNI_HOME}/omnimarket`) does not exist there and will be declined. That is | ||
| not a bug you should work around with a fabricated path — declare the item's | ||
| audience honestly: | ||
|
|
||
| ```yaml | ||
| - id: dod-cross-repo-behaviour | ||
| description: "…" | ||
| execution_scope: local_done_gate # OMN-15392 — hosted compliance will not evaluate this | ||
| checks: | ||
| - check_type: test_passes | ||
| check_value: "uv run pytest tests/unit/test_thing.py -q" | ||
| cwd: "${OMNI_HOME}/omnimarket" | ||
| ``` | ||
|
|
||
| `execution_scope: local_done_gate` items are reported `NOT-EVALUATED` by the | ||
| hosted gate by design, and do **not** count toward the hosted | ||
| `behavior_proving_count`. | ||
|
|
||
| --- | ||
|
|
||
| ## The shell | ||
|
|
||
| Commands execute under **`bash -o pipefail -c`** in both runners. A failing | ||
| first stage of a pipeline fails the check: | ||
|
|
||
| ```yaml | ||
| # FAILS when the gh call fails, which is what you want. | ||
| check_value: "gh api repos/OWNER/REPO/contents/x.py?ref=<sha> --jq .content | base64 -d | grep -q SYMBOL" | ||
| ``` | ||
|
|
||
| Under a plain `sh -c` this would have passed on `grep`'s exit code alone. | ||
|
|
||
| --- | ||
|
|
||
| ## Two properties every check needs | ||
|
|
||
| 1. **It must execute.** Prose, an `echo`, or a bare `true` proves nothing. | ||
| 2. **It must be able to go RED.** A check that reads only files this same | ||
| change authors (the receipt it is stamped in, the contract itself) confirms | ||
| what the author typed, not what the code does. Such checks are reported | ||
| `INERT` and demoted — see `onex_change_control.validation.evidence_admissibility` | ||
| and `admissible_evidence_guidance()`, which the runner prints on refusal. | ||
|
|
||
| Timeout: 60s per check in the hosted runner (30s in `node_dod_verify`). A check | ||
| that needs longer belongs behind `execution_scope: local_done_gate`, not behind | ||
| a bigger number. | ||
|
|
||
| --- | ||
|
|
||
| ## References | ||
|
|
||
| * OMN-16824 — one semantic for `test_passes`; `cwd` honoured or declined | ||
| * OMN-15392 — `execution_scope: local_done_gate` | ||
| * OMN-15309 / OMN-14436 — the admissibility predicate and the grandfather ratchet | ||
| * OMN-15911 — the proof-class classifier that reads these checks | ||
| * `docs/TEMPLATE_GUIDE.md` — the full contract field reference |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
drift/dod_receipts/OMN-16824/dod-16824-one-semantic-executed/test_passes.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| --- | ||
| schema_version: "1.0.0" | ||
| ticket_id: "OMN-16824" | ||
| evidence_item_id: "dod-16824-one-semantic-executed" | ||
| check_type: "test_passes" | ||
| check_value: "uv run pytest tests/test_omn16824_test_passes_semantics.py -q" | ||
| contract_sha256: "sha256:55a93b881574c32b4ebef17c45aa516779008293c805c628a6c249400f63fa39" | ||
| contract_entry_sha256: "sha256:24922885657d4925110b7f00c943eb6a8954fd2bc83c479b39771265bf65e634" | ||
| status: PASS | ||
| run_timestamp: "2026-08-28T09:31:35Z" | ||
| commit_sha: "4426f1915f62b4e4f6e03a87a40ad1fd88a3c902" | ||
| runner: "testpasses-split-lane" | ||
| verifier: "live-command-readback" | ||
| probe_command: >- | ||
| uv run pytest tests/test_omn16824_test_passes_semantics.py -q -p no:cacheprovider --no-header -p no:warnings | ||
| probe_stdout: | | ||
| .................... [100%] | ||
| 20 passed in 0.43s | ||
| actual_output: >- | ||
| PASS: 20/20 green. The suite executes the shared cross-runner case table (tests/fixtures/check_type_runner_semantics.yaml, | ||
| checked into omnimarket and run there against node_dod_verify) through the hosted runner, and pins the | ||
| AC4 regression by execution through run_compliance_check itself. Falsifiability proven by execution | ||
| rather than asserted: this same suite was RED against the pre-fix runner in this worktree before the | ||
| change landed -- 13 of 20 failed, and the gate-level case printed "[+] test_passes: All 1 CI checks | ||
| green" for a contract whose test_passes check_value was a pytest invocation naming a file that does | ||
| not exist, returning exit 0. After the fix the identical input prints "[X] test_passes: Command failed | ||
| (exit 4)" and returns exit 1, while the discriminating control with a passing check_value still returns | ||
| 0. The suite also asserts the runner makes no gh pr checks call at all, so the substituted PR-status | ||
| question cannot return by another route. | ||
| exit_code: 0 | ||
| pr_number: 7393 | ||
| branch: "jonah/omn-16824-testpasses-executes" |
30 changes: 30 additions & 0 deletions
30
drift/dod_receipts/OMN-16824/dod-16824-runner-regression-suite/test_passes.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| --- | ||
| schema_version: "1.0.0" | ||
| ticket_id: "OMN-16824" | ||
| evidence_item_id: "dod-16824-runner-regression-suite" | ||
| check_type: "test_passes" | ||
| check_value: "uv run pytest tests/test_contract_compliance_check.py -q" | ||
| contract_sha256: "sha256:55a93b881574c32b4ebef17c45aa516779008293c805c628a6c249400f63fa39" | ||
| contract_entry_sha256: "sha256:d77bac0f0fc1fc714c1f9e61a868a0e12a0e5b8347c35e8b5d0c61c56153a702" | ||
| status: PASS | ||
| run_timestamp: "2026-08-28T09:31:35Z" | ||
| commit_sha: "4426f1915f62b4e4f6e03a87a40ad1fd88a3c902" | ||
| runner: "testpasses-split-lane" | ||
| verifier: "live-command-readback" | ||
| probe_command: >- | ||
| uv run pytest tests/test_contract_compliance_check.py -q -p no:cacheprovider --no-header -p no:warnings | ||
| probe_stdout: | | ||
| ....................................................................... [100%] | ||
| 71 passed in 0.35s | ||
| actual_output: >- | ||
| PASS: 71/71 green. This is the compliance runner's own suite. Four tests that pinned the OPPOSITE semantic | ||
| (test_passes reports whether the PR's own CI is green, ignoring check_value) are replaced by two that | ||
| pin the executed reading: a failing check_value BLOCKs even when every gh call is mocked to report a | ||
| green PR, and no gh pr call is made at all. Eleven pre-existing tests that asserted the sh -c invocation | ||
| shape were updated to the bash -o pipefail -c form the runner now uses, matching node_dod_verify since | ||
| OMN-15382. Together with the OMN-16824 suite this carries the AC2 cwd proof, including the case that | ||
| asserts an unresolvable cwd does NOT execute the command in the workspace instead (the probe command | ||
| would have created a file; none exists anywhere afterwards). | ||
| exit_code: 0 | ||
| pr_number: 7393 | ||
| branch: "jonah/omn-16824-testpasses-executes" |
28 changes: 28 additions & 0 deletions
28
drift/dod_receipts/OMN-16824/occ-self-bind-pr-7393/command.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| --- | ||
| schema_version: "1.0.0" | ||
| ticket_id: "OMN-16824" | ||
| evidence_item_id: "occ-self-bind-pr-7393" | ||
| check_type: "command" | ||
| check_value: "gh pr view 7393 --repo OmniNode-ai/onex_change_control --json number,state,headRefName" | ||
| contract_sha256: "sha256:55a93b881574c32b4ebef17c45aa516779008293c805c628a6c249400f63fa39" | ||
| contract_entry_sha256: "sha256:5fc664865977b739518d05bac365d5d8aca8b0e12cde7d49ffbed2b244773e88" | ||
| status: PASS | ||
| run_timestamp: "2026-08-28T09:31:35Z" | ||
| commit_sha: "4426f1915f62b4e4f6e03a87a40ad1fd88a3c902" | ||
| runner: "testpasses-split-lane" | ||
| verifier: "live-command-readback" | ||
| probe_command: >- | ||
| gh api repos/OmniNode-ai/onex_change_control/pulls/7393/files --paginate --jq '.[].sha' | ||
| probe_stdout: | | ||
| b0b5effa3205d8937dcd73bfd9b6d43419731e3b 2f2b0b16a2c060309cc55031b83543641869974b | ||
| 92449ecc514aaaead0534781e26f5cb2690692b3 4eb5c997e4bb7b9c09f855936c18b2f450945c11 | ||
| 9abf0e8ab796816f90728f37e86c95f5dfd65c94 244ed2407964709185a6b3710362432c2471aabf | ||
| b763de14d75843ea4976e771b9f4122f89ebd84d cbe7b8dc0a90ad1a9b15f0f4a04f7fa3f36c2705 | ||
| 948658ed19a6fc2e575ff58f03f0f5e1d935628b | ||
| actual_output: >- | ||
| PASS: OCC self-bind for OMN-16824 (OCC#7393), the in-repo PR carrying the runner fix, the shared cross-runner | ||
| case table, the replaced regression tests, and the authoring documentation. The probe read the PR's | ||
| nine changed-file blob SHAs back from the live GitHub API rather than restating them. | ||
| exit_code: 0 | ||
| pr_number: 7393 | ||
| branch: "jonah/omn-16824-testpasses-executes" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.