Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions contracts/OMN-16824.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,9 @@ evidence_requirements:
- kind: "ci"
description: "PR #7393 CI checks green"
command: "gh pr checks ${PR_NUMBER} --repo ${REPO}"
- kind: "tests"
description: "Shared cross-runner semantics case table executed against the hosted runner"
command: "uv run pytest tests/test_omn16824_test_passes_semantics.py -q"
emergency_bypass:
enabled: false
justification: ""
Expand Down Expand Up @@ -39,3 +42,40 @@ dod_evidence:
checks:
- check_type: "command"
check_value: "gh pr view 7395 --repo OmniNode-ai/onex_change_control --json number,state"
- id: "dod-16824-one-semantic-executed"
description: >-
AC1/AC3/AC4. Runs the OMN-16824 suite in this repo. It executes the shared cross-runner case table
(tests/fixtures/check_type_runner_semantics.yaml, checked into omnimarket byte-for-byte and run
there against node_dod_verify) through the hosted runner, and pins the AC4 regression by execution
through run_compliance_check itself: with every gh call mocked to report a fully green PR, a contract
carrying a test_passes entry whose check_value fails returns exit 1, and the discriminating control
with a passing check_value returns exit 0. It also asserts the runner makes no gh pr checks call
at all, so the PR-status reading cannot come back by another route.
source: "manual"
status: "verified"
checks:
- check_type: "test_passes"
check_value: "uv run pytest tests/test_omn16824_test_passes_semantics.py -q"
- id: "dod-16824-runner-regression-suite"
description: >-
AC2. Runs the compliance runner's own suite, which now pins the executed reading of test_passes
(replacing four tests that pinned the opposite), the bash -o pipefail invocation, and that command
and test_passes are one runner. Together with the suite above this is the cwd proof: a declared
cwd is honoured, an unresolvable one is NOT_EVALUATED, and the command demonstrably does not run
in the workspace instead (the case asserts no file the command would have created exists anywhere).
source: "manual"
status: "verified"
checks:
- check_type: "test_passes"
check_value: "uv run pytest tests/test_contract_compliance_check.py -q"
- id: "occ-self-bind-pr-7393"
description: >-
OCC self-binding for PR #7393, the in-repo PR carrying the fix, the shared case table, and the two
behaviour-proving legs above. Binds OMN-16824 to that OCC PR by pr_number so eligibility can resolve
a PASS receipt for the PR itself (OMN-14650).
source: "manual"
status: "verified"
checks:
- check_type: "command"
check_value: >-
gh pr view 7393 --repo OmniNode-ai/onex_change_control --json number,state,headRefName
127 changes: 127 additions & 0 deletions docs/CHECK_TYPES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# `check_type` reference — what each DoD check actually does

**Audience:** anyone authoring `dod_evidence` in `contracts/OMN-XXXX.yaml`.

This page exists so you never have to read a runner's source to know what your
check will do. Two different gates execute the checks you write:

| Runner | Where it runs | What it is |
|---|---|---|
| **Hosted** — `Contract Compliance Check` | every PR, in CI | `onex_change_control.scripts.contract_compliance_check` |
| **Local** — `node_dod_verify` | the local Done gate / autoclose evidence sweep | omnimarket `EvidenceCollector` |

Both read the same contract bytes. **For check types both runners execute, this
page states one shared behaviour** — that is the OMN-16824 rule, and it is
enforced by an executed case table
(`tests/fixtures/check_type_runner_semantics.yaml`) that is checked into both
repos byte-for-byte and run against both runners. Rows marked `—` are hosted-only
surfaces until the local runner implements them.

---

## The table

| `check_type` | `check_value` is | Hosted runner | Local runner |
|---|---|---|---|
| `command` | a shell command | **executes it**; exit 0 = PASS | **executes it**; exit 0 = VERIFIED |
| `test_passes` | a shell command that runs tests | **executes it**; exit 0 = PASS | **executes it**; exit 0 = VERIFIED |
| `test_exists` | a glob | globs the workspace | — |
| `file_exists` | a glob | globs the workspace | checks the path exists |
| `grep` | `{pattern, path}` | greps the workspace | — |
| `endpoint` | a URL or path | curls / stats it | — |
| `behavior_proven` | prose attestation | non-executable WARN | non-executable |
| `semantic_grading` | a receipt path | receipt-gate surface | receipt-gate surface |

### `test_passes` is an executed alias of `command`

Write `test_passes` when the command you are running **is a test run**. It runs
exactly like `command`; the distinct name records author intent and is what the
proof-class classifier (OMN-15911) and the substance-floor tier deriver key on.

It does **not** mean "this PR's CI is green". Until OMN-16824 the hosted runner
read it that way — it ignored `check_value` entirely and reported the PR's own
check-run states — so an entry claiming *"the new regression test passes"*
returned PASS whenever unrelated CI was green, including when the named test did
not exist. That reading is gone. If you genuinely want to assert PR CI state,
say so in the open:

```yaml
- check_type: command
check_value: "gh pr checks 1719 --repo OmniNode-ai/omnimarket"
```

That is a **merge-state** proof, not a behaviour proof, and the classifier
scores it as such.

---

## Where your command runs

* **No `cwd` declared** → the command runs in the **product checkout under
test** (the repo whose PR is being gated), which the hosted runner passes as
`--workspace`. Relative paths resolve against it.
* **`cwd` declared** → the command runs **there**. Supported template tokens:
`${OMNI_HOME}`, `${PR_NUMBER}`, `${REPO}`, `${TICKET_ID}`. A relative `cwd`
resolves against the workspace; `..` segments are refused.
* **`cwd` declared but unresolvable** → the check is **`NOT_EVALUATED`**, and
the gate says so out loud. It is **never** rerouted to the workspace: running
your command in a different tree answers a different question under your
entry's name.

The hosted gate checks out **one** repo. A `cwd` naming a sibling checkout
(`${OMNI_HOME}/omnimarket`) does not exist there and will be declined. That is
not a bug you should work around with a fabricated path — declare the item's
audience honestly:

```yaml
- id: dod-cross-repo-behaviour
description: "…"
execution_scope: local_done_gate # OMN-15392 — hosted compliance will not evaluate this
checks:
- check_type: test_passes
check_value: "uv run pytest tests/unit/test_thing.py -q"
cwd: "${OMNI_HOME}/omnimarket"
```

`execution_scope: local_done_gate` items are reported `NOT-EVALUATED` by the
hosted gate by design, and do **not** count toward the hosted
`behavior_proving_count`.

---

## The shell

Commands execute under **`bash -o pipefail -c`** in both runners. A failing
first stage of a pipeline fails the check:

```yaml
# FAILS when the gh call fails, which is what you want.
check_value: "gh api repos/OWNER/REPO/contents/x.py?ref=<sha> --jq .content | base64 -d | grep -q SYMBOL"
```

Under a plain `sh -c` this would have passed on `grep`'s exit code alone.

---

## Two properties every check needs

1. **It must execute.** Prose, an `echo`, or a bare `true` proves nothing.
2. **It must be able to go RED.** A check that reads only files this same
change authors (the receipt it is stamped in, the contract itself) confirms
what the author typed, not what the code does. Such checks are reported
`INERT` and demoted — see `onex_change_control.validation.evidence_admissibility`
and `admissible_evidence_guidance()`, which the runner prints on refusal.

Timeout: 60s per check in the hosted runner (30s in `node_dod_verify`). A check
that needs longer belongs behind `execution_scope: local_done_gate`, not behind
a bigger number.

---

## References

* OMN-16824 — one semantic for `test_passes`; `cwd` honoured or declined
* OMN-15392 — `execution_scope: local_done_gate`
* OMN-15309 / OMN-14436 — the admissibility predicate and the grandfather ratchet
* OMN-15911 — the proof-class classifier that reads these checks
* `docs/TEMPLATE_GUIDE.md` — the full contract field reference
2 changes: 2 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ New to this repo? Read in this order:
2. [CLAUDE.md](../CLAUDE.md) — developer context, naming conventions, development commands.
3. [Design: Drift Control System](design/DESIGN_DRIFT_CONTROL_SYSTEM.md) — the enforcement model.
4. [Template Guide](TEMPLATE_GUIDE.md) — how to author YAML artifacts.
5. [Check Types](CHECK_TYPES.md) — what each `dod_evidence` check actually executes.

---

Expand Down Expand Up @@ -87,6 +88,7 @@ When automation creates Linear tickets from drift findings:
|-----|---------|
| [VERSIONING_POLICY.md](VERSIONING_POLICY.md) | Schema SemVer, immutability rules, breaking-change definition |
| [TEMPLATE_GUIDE.md](TEMPLATE_GUIDE.md) | Field-by-field reference for `day_close.template.yaml` and `ticket_contract.template.yaml` |
| [CHECK_TYPES.md](CHECK_TYPES.md) | What every `dod_evidence` `check_type` does in each runner, the `cwd` rules, and the shell it executes under |
| [EVAL_FRAMEWORK.md](EVAL_FRAMEWORK.md) | A/B evaluation framework: eval suites, comparator, metrics, verdicts |
| [RECEIPT_LOCATIONS.md](RECEIPT_LOCATIONS.md) | Canonical DoD receipt location (`drift/dod_receipts/`) and migration from legacy path |
| [wire-schema-contract-spec.md](wire-schema-contract-spec.md) | Wire schema contract specification |
Expand Down
3 changes: 2 additions & 1 deletion docs/TEMPLATE_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@ The Ticket Contract template (`ticket_contract.template.yaml`) is used to create

#### `dod_evidence`
- **Type**: List of objects (optional; omit if not needed)
- **Description**: Executable DoD checks. Each item's `checks[].check_value` is a shell command (`check_type: "command"`) run from the `onex_change_control` repo root by `scripts/ci/run_contract_compliance_check.py`; exit 0 = pass.
- **Description**: Executable DoD checks. For `check_type: "command"` and its executed alias `check_type: "test_passes"`, `checks[].check_value` is a shell command run by the hosted `Contract Compliance Check` **in the product checkout under test** — not the `onex_change_control` repo root — under `bash -o pipefail -c`; exit 0 = pass. A check may declare a `cwd`, which is honoured, or declined outright when it cannot be resolved. Other check types use the value shapes listed in `CHECK_TYPES.md`.
- **See [`CHECK_TYPES.md`](CHECK_TYPES.md)** for what every `check_type` does in each of the two runners, the `cwd` rules, and why `test_passes` is not a "PR CI is green" assertion (OMN-16824).
Comment thread
jonahgabriel marked this conversation as resolved.
- **`check_value` must be falsifiable — it must be able to fail if the work is actually wrong.** A command that reads the receipt file the check itself is stamped in and greps that file for `status: PASS` is **not evidence**: the receipt says PASS because the author wrote PASS, and the check confirms the author wrote PASS. It passes identically whether the code is correct or broken.

> **⚠️ DO NOT DO THIS** (measured OMN-14417 at 2,137/6,915 = 31.3% of the live corpus, and 98.4% of contracts created in the last 7 days — this exact shape, copy-pasted PR to PR):
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-16824"
evidence_item_id: "dod-16824-one-semantic-executed"
check_type: "test_passes"
check_value: "uv run pytest tests/test_omn16824_test_passes_semantics.py -q"
contract_sha256: "sha256:55a93b881574c32b4ebef17c45aa516779008293c805c628a6c249400f63fa39"
contract_entry_sha256: "sha256:24922885657d4925110b7f00c943eb6a8954fd2bc83c479b39771265bf65e634"
status: PASS
run_timestamp: "2026-08-28T09:31:35Z"
commit_sha: "4426f1915f62b4e4f6e03a87a40ad1fd88a3c902"
runner: "testpasses-split-lane"
verifier: "live-command-readback"
probe_command: >-
uv run pytest tests/test_omn16824_test_passes_semantics.py -q -p no:cacheprovider --no-header -p no:warnings
probe_stdout: |
.................... [100%]
20 passed in 0.43s
actual_output: >-
PASS: 20/20 green. The suite executes the shared cross-runner case table (tests/fixtures/check_type_runner_semantics.yaml,
checked into omnimarket and run there against node_dod_verify) through the hosted runner, and pins the
AC4 regression by execution through run_compliance_check itself. Falsifiability proven by execution
rather than asserted: this same suite was RED against the pre-fix runner in this worktree before the
change landed -- 13 of 20 failed, and the gate-level case printed "[+] test_passes: All 1 CI checks
green" for a contract whose test_passes check_value was a pytest invocation naming a file that does
not exist, returning exit 0. After the fix the identical input prints "[X] test_passes: Command failed
(exit 4)" and returns exit 1, while the discriminating control with a passing check_value still returns
0. The suite also asserts the runner makes no gh pr checks call at all, so the substituted PR-status
question cannot return by another route.
exit_code: 0
pr_number: 7393
branch: "jonah/omn-16824-testpasses-executes"
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-16824"
evidence_item_id: "dod-16824-runner-regression-suite"
check_type: "test_passes"
check_value: "uv run pytest tests/test_contract_compliance_check.py -q"
contract_sha256: "sha256:55a93b881574c32b4ebef17c45aa516779008293c805c628a6c249400f63fa39"
contract_entry_sha256: "sha256:d77bac0f0fc1fc714c1f9e61a868a0e12a0e5b8347c35e8b5d0c61c56153a702"
status: PASS
run_timestamp: "2026-08-28T09:31:35Z"
commit_sha: "4426f1915f62b4e4f6e03a87a40ad1fd88a3c902"
runner: "testpasses-split-lane"
verifier: "live-command-readback"
probe_command: >-
uv run pytest tests/test_contract_compliance_check.py -q -p no:cacheprovider --no-header -p no:warnings
probe_stdout: |
....................................................................... [100%]
71 passed in 0.35s
actual_output: >-
PASS: 71/71 green. This is the compliance runner's own suite. Four tests that pinned the OPPOSITE semantic
(test_passes reports whether the PR's own CI is green, ignoring check_value) are replaced by two that
pin the executed reading: a failing check_value BLOCKs even when every gh call is mocked to report a
green PR, and no gh pr call is made at all. Eleven pre-existing tests that asserted the sh -c invocation
shape were updated to the bash -o pipefail -c form the runner now uses, matching node_dod_verify since
OMN-15382. Together with the OMN-16824 suite this carries the AC2 cwd proof, including the case that
asserts an unresolvable cwd does NOT execute the command in the workspace instead (the probe command
would have created a file; none exists anywhere afterwards).
exit_code: 0
pr_number: 7393
branch: "jonah/omn-16824-testpasses-executes"
28 changes: 28 additions & 0 deletions drift/dod_receipts/OMN-16824/occ-self-bind-pr-7393/command.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
schema_version: "1.0.0"
ticket_id: "OMN-16824"
evidence_item_id: "occ-self-bind-pr-7393"
check_type: "command"
check_value: "gh pr view 7393 --repo OmniNode-ai/onex_change_control --json number,state,headRefName"
contract_sha256: "sha256:55a93b881574c32b4ebef17c45aa516779008293c805c628a6c249400f63fa39"
contract_entry_sha256: "sha256:5fc664865977b739518d05bac365d5d8aca8b0e12cde7d49ffbed2b244773e88"
status: PASS
run_timestamp: "2026-08-28T09:31:35Z"
commit_sha: "4426f1915f62b4e4f6e03a87a40ad1fd88a3c902"
runner: "testpasses-split-lane"
verifier: "live-command-readback"
probe_command: >-
gh api repos/OmniNode-ai/onex_change_control/pulls/7393/files --paginate --jq '.[].sha'
probe_stdout: |
b0b5effa3205d8937dcd73bfd9b6d43419731e3b 2f2b0b16a2c060309cc55031b83543641869974b
92449ecc514aaaead0534781e26f5cb2690692b3 4eb5c997e4bb7b9c09f855936c18b2f450945c11
9abf0e8ab796816f90728f37e86c95f5dfd65c94 244ed2407964709185a6b3710362432c2471aabf
b763de14d75843ea4976e771b9f4122f89ebd84d cbe7b8dc0a90ad1a9b15f0f4a04f7fa3f36c2705
948658ed19a6fc2e575ff58f03f0f5e1d935628b
actual_output: >-
PASS: OCC self-bind for OMN-16824 (OCC#7393), the in-repo PR carrying the runner fix, the shared cross-runner
case table, the replaced regression tests, and the authoring documentation. The probe read the PR's
nine changed-file blob SHAs back from the live GitHub API rather than restating them.
exit_code: 0
pr_number: 7393
branch: "jonah/omn-16824-testpasses-executes"
19 changes: 16 additions & 3 deletions src/onex_change_control/models/model_dod_check.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,12 @@ class ModelDodCheck(BaseModel):

Each check has a type that determines how check_value is interpreted:
- test_exists: check_value is a glob pattern for test files
- test_passes: check_value is a pytest marker or path to run
- test_passes: check_value is a shell command that runs tests -- an
EXECUTED alias of ``command`` (OMN-16824). Both the hosted Contract
Compliance Check and node_dod_verify run it under
``bash -o pipefail -c`` and read its exit status. It does NOT mean
"this PR's CI is green"; assert that explicitly with
``check_type: command`` + ``gh pr checks`` if you want it.
- file_exists: check_value is a glob pattern for expected files
- grep: check_value is a dict with 'pattern' and 'path' keys
- command: check_value is a shell command (exit 0 = pass)
Expand All @@ -49,8 +54,16 @@ class ModelDodCheck(BaseModel):
execute under. The runner expands ``${OMNI_HOME}``, ``${PR_NUMBER}``,
``${REPO}``, and ``${TICKET_ID}`` template tokens before invocation, and
is responsible for path-traversal containment checks. When ``cwd`` is
omitted the runner inherits its caller's working directory (legacy
behavior).
omitted the runner uses the product checkout under test (hosted) or
inherits its caller's working directory (local).

OMN-16824: a declared ``cwd`` a runner cannot resolve makes the check
NOT_EVALUATED -- it is never rerouted to the runner's own workspace, since
running the command in a different tree answers a different question under
this entry's name. The hosted gate checks out one repo, so a cross-repo
``cwd`` belongs to an item declaring
``execution_scope: local_done_gate`` (OMN-15392). Authoring reference:
``docs/CHECK_TYPES.md``.

OMN-10078: replaces the brittle ``cd ${OMNI_HOME}/<repo> && `` shell
prefix introduced as a temporary fix in OMN-10049 / PR #448.
Expand Down
Loading
Loading