ci: absent prerequisites SKIP with a named reason, and skipping is bounded

[stranske]

The first CI run of the public repo came back red: 21 pytest failures, 8 module selftests, 2 capability gates — against 330 green locally. Not one was a defect in this tree.

What was actually wrong

Nothing, in the code. Every failing check needed something that exists only on the machine the system runs on. I reproduced the runner exactly — committed files only (git archive), empty HOME, no agent CLIs, no sibling checkouts, unauthenticated gh, fresh state — and got 305 passed / 21 failed / 4 skipped / the same 8 selftests / the same 2 gates, byte-for-byte with CI. That made the diagnosis measurement rather than guesswork:

Prerequisite	Failures
the populated capability ledger	8 tests, 5 selftests, both gates
~/.codex/skills/.../audit_code_root.sh	8 tests
the version-capable Codex binary for exact profiles	2 tests, 2 selftests
an installed agent CLI / credential file for a seat	2 tests
(not a prerequisite — a stub leak)	1 test, 1 selftest

The ledger is the big one and worth stating plainly: it is machine-local state by design. A fresh bootstrap holds the 14 rows the code declares; this instance's holds 40. The other 26 are accumulated registration history and cannot be rebuilt from source.

The fix

Each genuinely-absent prerequisite now skips with a reason naming the missing thing, via detectors in the new env_prereq.py. Detection is of the prerequisite — never of $CI — so the same code is right on a runner, on the owner's box, and on a second instance with a different ORCH_STATE_DIR.

Assertions are untouched. A check that runs asserts exactly what it asserted before; only its applicability gate is new. Selftests gate the smallest section that needs the missing thing rather than the whole function, so CI keeps everything else — feedback.py's selftest still runs its several hundred other assertions.

One failure was not a prerequisite and is fixed by isolation, not a skip. test_experiment_arm_identity and exp_abcd's selftest replace subprocess.Popen wholesale, and model resolution spawns a catalog probe whenever the advertised-model cache is cold — which lands in the stub and dies on stdout.write with an int. Warm cache locally, cold on a fresh machine. Using adapters' own ORCH_MODEL_PROBE kill-switch makes both hermetic, so CI now runs more, not less.

The skip ceiling — the reason skipping was allowed at all

A suite that looks green while running less is the failure this repo exists to prevent, so verify.py now bounds skipping instead of permitting it:

• ceilings on skipped tests / selftests / gates in .verify-floor.json (24 / 7 / 2 — exactly what a machine with none of the local prerequisites skips). One more than agreed is a red that names the key to raise.
• the floor is now passed + skipped, so a check may move between passing and consciously-skipped but the two together may never shrink. --update-floor records passed + skipped and never re-measures the ceilings, so the floor cannot ratchet down behind a leak.
• a selftest or gate that exits 0 having skipped is reported as SKIPPED, not ok — the silent-zero-exit rule's twin, and the reason the 74 of 81 number can be trusted.
• every skip and its reason is printed, so green always states what did not run.
• every pytest failure is printed, not the last 12 lines. The first run reported 21 failures and named 7, which cost a whole round trip.

Deliberate break → revert (all three reverted byte-identical, cmp-confirmed)

1. One extra skipped file → 33 > 24 fails, naming skipped_max. With the ceiling removed, the identical break exits 0 and prints "VERIFIED". The ceiling is load-bearing, not decoration.
2. Blinding run_selftests to the skip mark → claims "81 of 81 modules ran" while 7 ran nothing; verify.py --selftest catches it.
3. Floor back to bare passed → CI goes red for being honest.

Two real bugs the reproduction surfaced

• capability_activation_audit._fleet_label_index took the whole audit down on a missing gh binary, while a failed gh call was already handled as "unknown for this repo". Same information, opposite outcome.
• ci.yml claimed the suite was state-independent, "verified by running it with ORCH_STATE_DIR pointed at an empty directory — 330 passed". That moved the wrong knob: the ledger and Brain live behind ORCH_LOCAL_RUNTIME. The experiment left the populated 40-row ledger in place and proved nothing about a fresh machine. Both are set now, and CLAUDE.md records the distinction.

Verification

	tests	selftests	gates
locally	330 passed, 0 failed, 0 of 24 skipped	81 of 81 ran	5 of 5 green
runner reproduction	306 passed, 0 failed, 24 of 24 skipped	74 of 81 ran	3 of 5 green, exit 0

Also re-run from the mirror layout via orch-sync-mirror.sh — cmp-clean is not agreement — same verdict. The local run is not weakened: it still executes all 330 and skips nothing.

Note for whoever merges. The live exec mirror was deliberately not synced: the canonical tree is on codex/partitioned-review-reconciliation with another session's uncommitted edits to two files this PR also touches (different regions of capability_activation_audit.py and test_model_tier_resolution.py — they should merge cleanly). Per CLAUDE.md §1 that mirror gap is the manual circuit breaker between an agent's change and the dispatcher. Sync it from the canonical tree after merge. That other branch also appends a test, so the collection floor will need to go 330 → 331 when both land.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your current included review allowance is based on your included PR review attempts over the past 7 days.

Next review available in: 22 minutes

Limit details: You’ve used the included review currently available. Your 64 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 7af3a077-e103-4145-baec-e07e8e2057b3

📥 Commits

Reviewing files that changed from the base of the PR and between 73eb96f and 285eb86.

📒 Files selected for processing (24)

• .github/workflows/ci.yml
• .verify-floor.json
• CLAUDE.md
• README.md
• adapters.py
• capability_activation_audit.py
• capability_admission.py
• capability_advisor.py
• capability_outcome_bridge.py
• capability_recurrence_check.py
• env_prereq.py
• exp_abcd.py
• feedback.py
• range_lane_rollout.py
• test_capabilities.py
• test_capability_admission.py
• test_capability_lifecycle_e2e.py
• test_capability_set_coverage.py
• test_capacity_profiles.py
• test_experiment_arm_identity.py
• test_feedback_model_provenance.py
• test_model_tier_resolution.py
• test_skill_compiler.py
• verify.py

---

_{Comment @coderabbitai help to get the list of available commands.}