Repository navigation
feat(OMN-16634): add the hcloud EC2 overflow row and make the host-identity refusal proofs host-independent - #1640
Merged
Conversation
jonahgabriel
pushed a commit
to OmniNode-ai/onex_change_control
that referenced
this pull request
Sep 1, 2026
…#8014) * evidence: OCC companion pass 1 for OmniNode-ai/omnibase_core#1640 * evidence: OCC companion self-bind for #8014 --------- Co-authored-by: node-occ-companion-effect <occ-companion-effect@omninode.ai>
added 3 commits
September 2, 2026 06:40
…or the governed pre-push picker The cloud pre-push host the operator ordered as a standing demand (OMN-16634 revival; cloud leg previously dropped twice). Row facts: c7a.4xlarge i-02a5f8bc0833bdacd, Elastic IP, 16 cores, lockdir, slots=1, mode=authorizing. Lab-first preserved by lifecycle: default state is STOPPED; a stopped host probes unreachable and is SKIPPED fail-closed. Provisioned only by omnibase_infra deploy/prepush-cloud/bootstrap.sh (no docker, no tailnet, no repo/cloud credentials on the host). Transport proof for THIS repo (bundle->scp->clone->uv sync->collect+slice on the branch tree) recorded on OMN-16634.
…ndependent Adding the hcloud row is what exposed this. A governed full-suite leg dispatched to onex-prepush-cloud1 against this branch's tree turned three tests in tests/scripts/test_prepush_hook_host_identity_guard.py red: test_guard_refuses_full_suite_escalation_on_non_200_host test_guard_refuses_selector_whole_tree_sentinel_when_flag_is_false test_guard_refuses_whole_suite_equivalent_selection_when_flag_is_false These are refusal proofs: they assert the hook refuses a heavy escalation when the local machine is not a designated host. They forced that condition with the two legacy aliases PREPUSH_200_HOSTNAME and PREPUSH_201_GATE_RUNNER_HOSTNAME, which displace only rows h200 and h201c. Every OTHER authorizing row -- h101, h105, h201, and now hcloud -- still carried its real hostname, so running the suite ON one of those hosts made the hook match a row, take the designated-host branch, and answer with a placement/slot refusal instead of the identity refusal the assertion read for. The reds say the proof was host-dependent, not that the hook is wrong. The same latent false-red applied to any full-suite dispatch of this repo to h101 or h105 from the moment OMN-17159 added those rows; hcloud is merely the row that made it observable. _nonmatching_host_overrides() reads the committed host table and points PREPUSH_HOST_OVERRIDE_<LABEL> at a guaranteed-non-matching hostname for EVERY authorizing row, so these proofs stay host-independent as rows are added rather than needing a new alias per row. Measured red -> green on the host that exposed it, plus h200: onex-prepush-cloud1 38 passed in 8.20s stickybeatz-studio 38 passed in 69.31s docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json records the dispatch, the three reds, both green runs, and two honest caveats: that leg never reached a terminal verdict (SIGTERM at 48%, exit=143) so no green full-suite claim is made from it, and its MARKER collected=0 is a wrapper sed bug, not a collection failure -- real collection was 44,695 items.
…ntropy false positives
docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json carries three
64-char sha256 digests -- argv_sha, head_sha and suite_log_sha256 -- which are
what make the receipt verifiable against the run it describes. detect-secrets
reads them as Hex High Entropy Strings and the Detect Secrets job failed:
Secret Type: Hex High Entropy String
Location: docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json:2
Location: docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json:30
Location: docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json:47
Same false-positive class the repo already documents for its other generated
baselines. Baselined via the repo's sanctioned mechanism rather than by
stripping the digests, which would leave the receipt unverifiable.
Regenerated with detect-secrets==1.5.0 under the exact exclude set the CI job
uses. What changed, checked field by field against the previous baseline:
* results: one file added (85 -> 86), carrying the three findings above.
No existing entry modified or removed.
* filters_used: gains detect_secrets.filters.common.is_baseline_file, which
the tool adds itself whenever it is run with --baseline. Nothing removed.
* plugins_used and the remaining top-level keys: identical.
* generated_at: refreshed, as the tool always does.
Verified locally with the job's own command --
`git ls-files -z | xargs -0 detect-secrets-hook --no-verify --baseline ...` --
which now exits 0.
jonahgabriel
disabled auto-merge
September 2, 2026 20:12
added 2 commits
September 3, 2026 05:01
OMN-17269 shipped per-host slot capacity; the mechanism has been INERT since, in this repo and in omnibase_infra. `LOCK.2` has never been created once anywhere in the fleet in three days, despite omnibase_infra carrying h105 slots=2 since 2026-08-30. Three defects in `_PREPUSH_SLOT_PROBE_SH` explain it, each measured on live hosts rather than inferred, and each fixed here. D1 -- the `held` glob dies under the remote login shell. The probe string is handed to ssh and runs under the TARGET's login shell, which on h101 and h105 is /bin/zsh 5.9 (measured 2026-09-02). zsh's default `nomatch` makes an unmatched glob a fatal error that aborts the command line BEFORE it runs, so `ls -d "$W"/LOCK "$W"/LOCK.* 2>/dev/null` printed nothing whenever no `LOCK.<k>` existed -- the redirection belonged to a command that never executed. `held` therefore read 0 on every remote Mac probe, in exactly the state slot 2 exists to serve. `find -name` does its own matching, so no glob reaches the shell. D2 -- one remote leg counted as two processes. The leg is launched as `zsh -c '...; ./prepush_smart_tests.sh ...'`, so the wrapper shell AND the script both carry the script name in argv. Measured with exactly one leg in flight on h105 (PID 86954 wrapper + 86956 script), and the same on h101 and h201: the old count returned 2 everywhere. D1 and D2 compound. The busy predicate is `p <= self + held`, and `self` is 0 for a remote probe, so a correctly-locked host read `2 <= 0 + 0` -> BUSY on every slot. Fixing either alone still refuses: `2 <= 1` and `1 <= 0`. D3 -- an existing-but-empty QUEUE file shifted every field. `grep -c .` on an empty file prints 0 AND exits 1, so the old `|| echo 0` fired as well and `q` became the two-line string "0\n0". `printf` then emitted FIVE words, `set -- $raw` split five positionals, and the parse read only the first four -- every field shifted left, and `l` was read out of `p`. Live on h201, whose ~/push-lanes/QUEUE exists and is 0 bytes: the 20:07Z refusal trail printed `h201=busy(queue=0 heavy_pids=0 lock=2 held=1)`, and `l` is assigned only 0 or 1, so `lock=2` is a value this code cannot otherwise produce. D3 is also FAIL-OPEN, not merely false-busy. With a real p=0 and l=1 -- the window between `_lock_acquire`'s mkdir and pytest spawning, which spans the whole `uv sync` -- the shifted read is q=0 p=0 l=0 held=1 and the probe returns FREE on a host whose LOCK is held. That is why the exact-four-field parse guard is load-bearing rather than cosmetic: a malformed probe is now UNKNOWN, skipped exactly like unreachable, instead of readable as a verdict. WHY IT SHIPPED: `_PREPUSH_SLOT_PROBE_SH` had never been EXECUTED by a test in any of the three repos. The only reference was a string grep, and every slot test went through PREPUSH_SLOT_OVERRIDE_MAP, which short-circuits the probe. Three defects shipped into a fail-closed merge gate behind assertions that only read the library's text. The seven tests added here extract the real probe body and run it -- under `bash -O failglob` and under real zsh, against a `ps` stub built from argv captured verbatim off h105, and against an existing-but-empty QUEUE file. All five defect tests were proven RED against the shipped library first; `test_an_unexplained_heavy_process_is_still_busy` passes BOTH before and after, which is what proves these changes restore the predicate's inputs rather than relax `p <= self + held`. The four library hunks are applied VERBATIM from .onex_state/omn17602-slot-probe-fix.patch, and the whole shared block they land in -- from `# multi-slot row it lets a legitimately-held` through the end of `prepush_slot_state` -- is now BYTE-IDENTICAL to the copy already merged on omnimarket dev (omnimarket#2276, omnimarket dev b79d8983), verified by `diff` at 2026-09-03T08:55Z. The one line that differed was the in-file citation: this commit originally wrote `(OMN-17602)` because that is the lane that measured the defects, while the merged omnimarket text names OMN-17606 as the landing lane and OMN-17602 as the finder. Aligning to the merged text is not cosmetic -- the comment is inside a file both repos vendor, so a divergent line means the next re-sync of lines 140-300 shows a spurious diff, and both repos ship a `test_the_picker_library_is_not_edited_into_a_repo_specific_fork` that rejects a repo name appearing in this file at all. The sha256 parity pin in tests/scripts/test_prepush_host_table.py is bumped in this same commit to 6813caf5...c67d, and its docstring now says what that digest actually tracks -- core's copy is 1114 lines to omnibase_infra's 1511 and has not been byte-identical since OMN-17159, so the pin means "not edited without review", not "equals upstream". Live proof of the fixed probe, read-only over the same ssh transport prepush_slot_state uses, 2026-09-02T21:33Z: h200 k=1 [0 11 0 0] busy: 11 heavy legs, genuinely saturated h201 k=1 [0 1 1 1] busy: lock held -- and queue now reads ONE field h101 k=1 [0 1 1 1] busy h101 k=2 [0 1 0 1] FREE h105 k=1 [0 1 1 1] busy h105 k=2 [0 1 0 1] FREE Both slot-2 rows are placeable for the first time. This repo's table still says slots=1 for h101/h105, so core sees that capacity only once OMN-17602's widening lands; omnibase_infra already carries slots=2 and gains it on merge. Ticket: OMN-17606
…e-push host table
The core host table pinned every capacity row at slots=1. OMN-17159 refused
to inherit omnibase_infra's slots=2 widening on an explicit premise: that
this repo's escalation is the whole tests/ tree, so two concurrent core
suites on h105's ten cores "would make both slower than one serialized pair
while degrading the load signal every other row is ranked on".
That premise was a projection about a real core leg, and the first real core
legs falsify it. Probed read-only 2026-09-02T19:22Z over ssh, on two hosts
each carrying exactly one live governed core full-suite leg:
h101 (12c/32GiB) run omnibase_core-4e116501fd0a-37091, 1h13m43s elapsed:
ONE pytest process at 100.0% of ONE core, RSS 670 MB; process-tree
totals cpu_sum=100.0%, rss_sum=671 MB, 2 procs; load1 2.96/12 = 0.25x.
h105 (10c/32GiB) run omnibase_core-e69568dd5e02-80404, 1h49m49s elapsed:
same shape, RSS 700 MB; load1 4.07/10 = 0.41x while that host was in
fact carrying TWO concurrent core pytest invocations.
The core remote leg is single-threaded, so a lane costs one core and ~0.7 GB
-- not the machine. The serialization was not protecting the hosts from
saturation; it was idling 8-11 cores per host while lanes queued for a
placement target. Each slot is still re-qualified on its own live load at
pick time, so a busy or loaded second slot still refuses.
Not widened past 2, and h200/h201/h201c are not widened at all: h200 is the
local/default identity host rather than a distribution target, h201 runs the
separate ~/push-lanes/QUEUE serializer (a different concurrency mechanism
this column does not govern), h201c never executes anything.
A new test asserts the behaviour against the REAL shipped table -- before
this change slots>1 was only ever exercised on a synthetic fixture, and a
read-only probe at 19:11Z confirmed LOCK.2 has never been created once
across h105 (121 run dirs) and h101 (73 run dirs), i.e. the path was
untaken fleet-wide.
LANDED AS PART OF omnibase_core#1640 (OMN-16634), not as its own PR. The
original commit deliberately excluded the hcloud row because #1640 already
carried it (c98bfc3) together with the host-identity refusal fix (f4b10ba)
that the row requires. Both halves are now in one PR instead, so the lab pays
ONE governed full-suite run for the three capacity changes rather than three.
The cherry-pick conflicted in `prepush_hosts.tsv` and in
`test_the_shipped_slots_column_is_pinned`; both were resolved as the UNION --
h101 and h105 go to slots=2, and #1640's hcloud row stays at slots=1.
hcloud stays at 1 on its own reasoning, not by inheriting the blanket pin this
commit removes: the h101/h105 widening is earned by a measurement of those two
hosts, no concurrent-pair measurement of the EC2 host exists, and the row is an
overflow target only reached once the lab is already saturated. The hcloud
note's old sentence ("slots=1 for the same reason as every row in THIS repo")
asserted a blanket rule this commit falsifies, so it is rewritten here rather
than left to contradict the table it sits in.
Ticket: OMN-17602
jonahgabriel
force-pushed
the
jonah/omn-16634-cloud-prepush-host
branch
from
September 3, 2026 13:32
f8b7919 to
c13ef5f
Compare
This was referenced Sep 4, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One PR, three tickets, one full-suite run. These three changes all touch the
governed pre-push picker and two of them touch the same test file
(
tests/scripts/test_prepush_host_table.py); landing them separately would havecost the lab three ~3.5h full-suite escalations instead of one.
c98bfc37+f4b10baa+69ea8bbbhcloudEC2 overflow row; make the host-identity refusal proofs host-independent5e135cf6c13ef5fch101/h105toslots=2in core's host tableThey are ordered so each is a strict precondition of the next: the identity fix
is what lets core's suite pass at all on a host other than h200, the probe fix
is what makes a second slot reachable, and the table widening is what declares
the second slot. Landing 17602 without 17606 would have declared slots that stay
permanently unpickable, which is exactly the inert state OMN-17606 documents.
dod_evidence — OMN-16634
Adds
hcloud(AWS EC2 c7a.4xlarge,i-02a5f8bc0833bdacd) as the off-LANoverflow row, and fixes the host-identity refusal proofs that adding it exposed
as host-dependent.
Three tests in
tests/scripts/test_prepush_hook_host_identity_guard.pyarerefusal proofs: they assert the hook refuses a heavy escalation when the local
machine is not a designated host. They forced that condition with the two legacy
aliases
PREPUSH_200_HOSTNAMEandPREPUSH_201_GATE_RUNNER_HOSTNAME, whichdisplace only rows
h200andh201c. Every other authorizing row —h101,h105,h201, and nowhcloud— still carried its real hostname, so runningthe suite on one of those hosts made the hook match a row, take the
designated-host branch, and answer with a placement/slot refusal instead of the
identity refusal the assertion read for.
_nonmatching_host_overrides()reads the committed host table and pointsPREPUSH_HOST_OVERRIDE_<LABEL>at a guaranteed-non-matching hostname forevery authorizing row, so these proofs stay host-independent as rows are
added rather than needing a new alias per row.
Proof, and it is the whole point of this ticket: the governed pre-push for
this very push placed the escalation on h105 (
omnibook), not h200, and thethree formerly host-dependent tests passed there:
Before this commit those three were guaranteed red on
h101/h105/h201/hcloud— they would have read a CAPACITY refusal where they assert an IDENTITY refusal.
An off-h200 green full suite is only reachable with this fix in the tree.
Governed full suite, green, on the exact pushed head:
/Users/Shared/onex-prepush/runs/omnibase_core-c13ef5fc43a4-93037on h105MARKER:head_sha=c13ef5fc43a40ae7f2430917d2a216b71ca0f57f,exit=0,host=omnibook,log_sha256=9506a2a8afb34bf7461dbf21821f469c708b3391d902c35496fc496c320e4893WRAPPER_EXIT=0;suite.logre-hashed on the host, digest matches the MARKERFAILEDstrings in the log are parametrised testnames such as
ONEX_CORE_005_VALIDATION_FAILED, each markedPASSED)collected=0in the MARKER is the known wrappersedbug, not a collectionfailure — the log shows the full 44,718-outcome run
Verbatim placement line from the hook:
That trail is also the first live evidence of a slot-2 placement anywhere in
the fleet (
h105.2), which is the OMN-17606 + OMN-17602 acceptance evidencebelow.
Instance disposition:
i-02a5f8bc0833bdacdis stopped, confirmed viaaws ec2 describe-instances. Disk and Elastic IP preserved, compute cost halted.A stopped host probes unreachable and is skipped fail-closed — the row's
documented default state, and why it reads
slot-unknownin the trail above.dod_evidence — OMN-17606 (core half only)
Ports the three measured slot-probe defects into core's vendored copy of the
shared picker library. The parent ticket also covers omnibase_infra and
omnimarket; only the core half lands here, so OMN-17606 stays open.
_PREPUSH_SLOT_PROBE_SHnomatchonLOCK.*aborted the command line, zeroingheldon every remote Mac probeheldviafind "$PREPUSH_WORKROOT" -maxdepth 1 -type d -name "LOCK*"—finddoes its own matching, no glob is handed to a shellzsh -cwrapper and the script both carry the script name in argv, counting one leg as two heavy processespviaps ax -o args= | grep prepush_smart_tests.sh | grep -v grep | grep -cv -e " -c "grep -c .on an existing-but-emptyQUEUEprints0and exits 1, so|| echo 0also fired and shifted every field leftqcomputed inside the existing[ -r ... ]guard without the|| echo 0, with a separate[ -n "$q" ] || q=0Plus a fail-closed parse guard:
prepush_slot_statenow requires exactly fourfields or returns 2 (unknown → skipped, same as unreachable) rather than
reading a malformed probe shifted.
These compound. The busy predicate is
p <= self + held; with D1 and D2 bothlive a host running exactly one correctly-locked leg evaluated
2 <= 0→ BUSY onevery slot including the free one, which is why
LOCK.2had never been createdonce anywhere in the fleet.
Vendored-library alignment (the thing that must not drift). The probe block
in this PR is byte-identical to the version already merged in omnimarket:
The comment wording is deliberately repo-neutral — no added line in the file
names any repo — so the next vendor re-sync does not fail omnimarket's
test_the_picker_library_is_not_edited_into_a_repo_specific_fork. Verified:git show 5e135cf6 -- scripts/hooks/prepush_dispatch.sh | grep '^+' | grep -E 'omnibase_core|omnimarket|omnibase_infra'returns nothing.The whole-file digests still differ between the two repos
(core
6813caf5…, omnimarketed993364…) because omnimarket carries theOMN-17392 memory-probe feature core has not ported. That is pre-existing
divergence, not introduced here; core's own pin test
test_the_picker_library_is_byte_identical_to_the_shipped_upstreamis updated tocore's new digest in the same commit, as the file header requires.
Acceptance evidence:
h105.2=fit(0.226,authorizing)in the placement trailabove, and the suite actually executing on slot 2. That is the first slot-2
placement ever recorded in this fleet — the proof gap OMN-17561 named.
dod_evidence — OMN-17602
scripts/hooks/prepush_hosts.tsv:h101andh105goslots=1 → 2, and thehcloudrow is added.The widening is on first-hand measurement of a real core leg, not by analogy
with omnibase_infra's table:
omnibase_core-e69568dd5e02-80404, 1h49m49s elapsed, ONE pytest process, RSS 700 MB; observed carrying two concurrent core invocations at load1 4.07/10 = 0.41xomnibase_core-4e116501fd0a-37091, 1h13m43s elapsed, ONE pytest process at 100.0% of one core, RSS 671 MB, load1 2.96/12 = 0.25xA core lane costs one core and ~0.7 GB, not the machine. Neither row is widened
past 2: under OMN-17603's restored
-n4two lanes would be 8 of 10 cores (0.8x)and a third would cross the 1.0x threshold.
Schema constraint honoured. The ticket is explicit that core's parser reads
slotsat field 10,repos_denied11,mode12,note13, and that addingheavy_local/placement_tierwould shift every field. Verified on the pushedtree — all six rows are exactly 13 fields, no new columns:
hcloudstaysslots=1: no concurrent-pair measurement of that host exists, anda widening is earned by measuring the row, which is how h101 and h105 earned
theirs.
ssh_targetkeeps core's currentjonah@convention rather thanhalf-migrating one row to OMN-17280's; that sweep lands all rows together.
Local gates
ruff format+ruff check --fixclean(3 identity-guard + 11 host-table)
test_prepush_host_table.pyoutcomes PASSED,0 failed
pre-commit run --all-files— every hook Passed or Skipped (no-files);prepush-smart-testsPassed in 13091.59smypy --strictreports oneimport-not-foundforgrimpinscripts/ci/test_selection_closure.py— verified pre-existing, that file isbyte-identical to
origin/devKnown residuals, not fixed here
slot-unknownrather thanunreachable, because the slot probe returns before the reachability proberuns. Visible as
hcloud=slot-unknownabove.prepush_dispatch.sh's refusalvocabulary is outside this lane's scope.
tightest in the fleet at 71 GiB free / 92% capacity.
tests/makespytest read
tests/pytest.ini, shadowing the pyproject-n4) is untouchedhere. That is why the h105 run took 3h37m for 44.7k tests.
Evidence-Ticket: OMN-16634
Evidence-Source: OCC#8014