Skip to content

feat(OMN-16634): add the hcloud EC2 overflow row and make the host-identity refusal proofs host-independent - #1640

Merged
github-actions[bot] merged 5 commits into
devfrom
jonah/omn-16634-cloud-prepush-host
Sep 3, 2026
Merged

github-actions[bot] merged 5 commits into
devfrom
jonah/omn-16634-cloud-prepush-host

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

What

One PR, three tickets, one full-suite run. These three changes all touch the
governed pre-push picker and two of them touch the same test file
(tests/scripts/test_prepush_host_table.py); landing them separately would have
cost the lab three ~3.5h full-suite escalations instead of one.

commit ticket change
c98bfc37 + f4b10baa + 69ea8bbb OMN-16634 add the hcloud EC2 overflow row; make the host-identity refusal proofs host-independent
5e135cf6 OMN-17606 (core half) port the three slot-probe defect fixes into core's vendored picker library
c13ef5fc OMN-17602 widen h101/h105 to slots=2 in core's host table

They are ordered so each is a strict precondition of the next: the identity fix
is what lets core's suite pass at all on a host other than h200, the probe fix
is what makes a second slot reachable, and the table widening is what declares
the second slot. Landing 17602 without 17606 would have declared slots that stay
permanently unpickable, which is exactly the inert state OMN-17606 documents.


dod_evidence — OMN-16634

Adds hcloud (AWS EC2 c7a.4xlarge, i-02a5f8bc0833bdacd) as the off-LAN
overflow row, and fixes the host-identity refusal proofs that adding it exposed
as host-dependent.

Three tests in tests/scripts/test_prepush_hook_host_identity_guard.py are
refusal proofs: they assert the hook refuses a heavy escalation when the local
machine is not a designated host. They forced that condition with the two legacy
aliases PREPUSH_200_HOSTNAME and PREPUSH_201_GATE_RUNNER_HOSTNAME, which
displace only rows h200 and h201c. Every other authorizing row — h101,
h105, h201, and now hcloud — still carried its real hostname, so running
the suite on one of those hosts made the hook match a row, take the
designated-host branch, and answer with a placement/slot refusal instead of the
identity refusal the assertion read for.

_nonmatching_host_overrides() reads the committed host table and points
PREPUSH_HOST_OVERRIDE_<LABEL> at a guaranteed-non-matching hostname for
every authorizing row, so these proofs stay host-independent as rows are
added rather than needing a new alias per row.

Proof, and it is the whole point of this ticket: the governed pre-push for
this very push placed the escalation on h105 (omnibook), not h200, and the
three formerly host-dependent tests passed there:

tests/scripts/test_prepush_hook_host_identity_guard.py::test_guard_refuses_full_suite_escalation_on_non_200_host PASSED
tests/scripts/test_prepush_hook_host_identity_guard.py::test_guard_refuses_selector_whole_tree_sentinel_when_flag_is_false PASSED
tests/scripts/test_prepush_hook_host_identity_guard.py::test_guard_refuses_whole_suite_equivalent_selection_when_flag_is_false PASSED

Before this commit those three were guaranteed red on h101/h105/h201/hcloud
— they would have read a CAPACITY refusal where they assert an IDENTITY refusal.
An off-h200 green full suite is only reachable with this fix in the tree.

Governed full suite, green, on the exact pushed head:

44656 passed, 59 skipped, 3 xpassed, 142 warnings in 13037.86s (3:37:17)
  • run dir /Users/Shared/onex-prepush/runs/omnibase_core-c13ef5fc43a4-93037 on h105
  • MARKER: head_sha=c13ef5fc43a40ae7f2430917d2a216b71ca0f57f, exit=0,
    host=omnibook, log_sha256=9506a2a8afb34bf7461dbf21821f469c708b3391d902c35496fc496c320e4893
  • WRAPPER_EXIT=0; suite.log re-hashed on the host, digest matches the MARKER
  • zero test failures (the four FAILED strings in the log are parametrised test
    names such as ONEX_CORE_005_VALIDATION_FAILED, each marked PASSED)
  • collected=0 in the MARKER is the known wrapper sed bug, not a collection
    failure — the log shows the full 44,718-outcome run

Verbatim placement line from the hook:

[prepush-smart-tests] remote leg: probed -> h200=busy(queue=0 heavy_pids=6 lock=0 held=0 slot=1) h201=busy(queue=0 heavy_pids=1 lock=1 held=1 slot=1) h101=fit(0.389,authorizing) h101.2=fit(0.389,authorizing) h105=busy(queue=0 heavy_pids=1 lock=1 held=1 slot=1) h105.2=fit(0.226,authorizing) hcloud=slot-unknown
[prepush-smart-tests] REMOTE LAB RUN PASS accepted in place of whole-suite-equivalent impacted selection: h105.2 ran green on c13ef5fc43a40ae7f2430917d2a216b71ca0f57f
[prepush-smart-tests] impacted tests passed; allowing push.

That trail is also the first live evidence of a slot-2 placement anywhere in
the fleet (h105.2), which is the OMN-17606 + OMN-17602 acceptance evidence
below.

Instance disposition: i-02a5f8bc0833bdacd is stopped, confirmed via
aws ec2 describe-instances. Disk and Elastic IP preserved, compute cost halted.
A stopped host probes unreachable and is skipped fail-closed — the row's
documented default state, and why it reads slot-unknown in the trail above.


dod_evidence — OMN-17606 (core half only)

Ports the three measured slot-probe defects into core's vendored copy of the
shared picker library. The parent ticket also covers omnibase_infra and
omnimarket; only the core half lands here, so OMN-17606 stays open.

defect fix in _PREPUSH_SLOT_PROBE_SH
D1 — zsh nomatch on LOCK.* aborted the command line, zeroing held on every remote Mac probe held via find "$PREPUSH_WORKROOT" -maxdepth 1 -type d -name "LOCK*" — find does its own matching, no glob is handed to a shell
D2 — the zsh -c wrapper and the script both carry the script name in argv, counting one leg as two heavy processes p via ps ax -o args= | grep prepush_smart_tests.sh | grep -v grep | grep -cv -e " -c "
D3 — grep -c . on an existing-but-empty QUEUE prints 0 and exits 1, so || echo 0 also fired and shifted every field left q computed inside the existing [ -r ... ] guard without the || echo 0, with a separate [ -n "$q" ] || q=0

Plus a fail-closed parse guard: prepush_slot_state now requires exactly four
fields
or returns 2 (unknown → skipped, same as unreachable) rather than
reading a malformed probe shifted.

These compound. The busy predicate is p <= self + held; with D1 and D2 both
live a host running exactly one correctly-locked leg evaluated 2 <= 0 → BUSY on
every slot including the free one, which is why LOCK.2 had never been created
once anywhere in the fleet.

Vendored-library alignment (the thing that must not drift). The probe block
in this PR is byte-identical to the version already merged in omnimarket:

sha256 99216313bd50d5a39c1f213cd22b7d8930fdab89fcb2f60c6e32c52f6d66e9ff
  == omnimarket origin/dev scripts/hooks/prepush_dispatch.sh (same region)
  == omnibase_core scripts/hooks/prepush_dispatch.sh @ c13ef5fc

The comment wording is deliberately repo-neutral — no added line in the file
names any repo — so the next vendor re-sync does not fail omnimarket's
test_the_picker_library_is_not_edited_into_a_repo_specific_fork. Verified: git show 5e135cf6 -- scripts/hooks/prepush_dispatch.sh | grep '^+' | grep -E 'omnibase_core|omnimarket|omnibase_infra' returns nothing.

The whole-file digests still differ between the two repos
(core 6813caf5…, omnimarket ed993364…) because omnimarket carries the
OMN-17392 memory-probe feature core has not ported. That is pre-existing
divergence, not introduced here; core's own pin test
test_the_picker_library_is_byte_identical_to_the_shipped_upstream is updated to
core's new digest in the same commit, as the file header requires.

Acceptance evidence: h105.2=fit(0.226,authorizing) in the placement trail
above, and the suite actually executing on slot 2. That is the first slot-2
placement ever recorded in this fleet — the proof gap OMN-17561 named.


dod_evidence — OMN-17602

scripts/hooks/prepush_hosts.tsv: h101 and h105 go slots=1 → 2, and the
hcloud row is added.

The widening is on first-hand measurement of a real core leg, not by analogy
with omnibase_infra's table:

host measurement (read-only, 2026-09-02T19:22Z)
h105 (10-core M4, 32 GB) run omnibase_core-e69568dd5e02-80404, 1h49m49s elapsed, ONE pytest process, RSS 700 MB; observed carrying two concurrent core invocations at load1 4.07/10 = 0.41x
h101 (12-core, 32 GB) run omnibase_core-4e116501fd0a-37091, 1h13m43s elapsed, ONE pytest process at 100.0% of one core, RSS 671 MB, load1 2.96/12 = 0.25x

A core lane costs one core and ~0.7 GB, not the machine. Neither row is widened
past 2: under OMN-17603's restored -n4 two lanes would be 8 of 10 cores (0.8x)
and a third would cross the 1.0x threshold.

Schema constraint honoured. The ticket is explicit that core's parser reads
slots at field 10, repos_denied 11, mode 12, note 13, and that adding
heavy_local / placement_tier would shift every field. Verified on the pushed
tree — all six rows are exactly 13 fields, no new columns:

13 fields: h200 slots=1     13 fields: h101   slots=2
13 fields: h201 slots=1     13 fields: h105   slots=2
13 fields: h201c slots=1    13 fields: hcloud slots=1

hcloud stays slots=1: no concurrent-pair measurement of that host exists, and
a widening is earned by measuring the row, which is how h101 and h105 earned
theirs. ssh_target keeps core's current jonah@ convention rather than
half-migrating one row to OMN-17280's; that sweep lands all rows together.


Local gates

  • ruff format + ruff check --fix clean
  • targeted re-run of both touched test files, on h200: 14 passed in 41.83s
    (3 identity-guard + 11 host-table)
  • on h105 in the full suite: 89 test_prepush_host_table.py outcomes PASSED,
    0 failed
  • pre-commit run --all-files — every hook Passed or Skipped (no-files);
    prepush-smart-tests Passed in 13091.59s
  • mypy --strict reports one import-not-found for grimp in
    scripts/ci/test_selection_closure.py — verified pre-existing, that file is
    byte-identical to origin/dev

Known residuals, not fixed here

  • A stopped or unreachable host reports slot-unknown rather than
    unreachable, because the slot probe returns before the reachability probe
    runs. Visible as hcloud=slot-unknown above. prepush_dispatch.sh's refusal
    vocabulary is outside this lane's scope.
  • The picker still has no free-disk precondition (OMN-17420). h101 is the
    tightest in the fleet at 71 GiB free / 92% capacity.
  • OMN-17603 (core's remote leg runs single-threaded because argv tests/ makes
    pytest read tests/pytest.ini, shadowing the pyproject -n4) is untouched
    here. That is why the h105 run took 3h37m for 44.7k tests.

Evidence-Ticket: OMN-16634
Evidence-Source: OCC#8014

@github-actions
github-actions Bot enabled auto-merge (squash) September 1, 2026 22:41
jonahgabriel pushed a commit to OmniNode-ai/onex_change_control that referenced this pull request Sep 1, 2026
…#8014)

* evidence: OCC companion pass 1 for OmniNode-ai/omnibase_core#1640

* evidence: OCC companion self-bind for #8014

---------

Co-authored-by: node-occ-companion-effect <occ-companion-effect@omninode.ai>
jonahgabriel added 3 commits September 2, 2026 06:40
…or the governed pre-push picker

The cloud pre-push host the operator ordered as a standing demand
(OMN-16634 revival; cloud leg previously dropped twice). Row facts:
c7a.4xlarge i-02a5f8bc0833bdacd, Elastic IP, 16 cores, lockdir, slots=1,
mode=authorizing. Lab-first preserved by lifecycle: default state is
STOPPED; a stopped host probes unreachable and is SKIPPED fail-closed.
Provisioned only by omnibase_infra deploy/prepush-cloud/bootstrap.sh
(no docker, no tailnet, no repo/cloud credentials on the host).
Transport proof for THIS repo (bundle->scp->clone->uv sync->collect+slice
on the branch tree) recorded on OMN-16634.
…ndependent

Adding the hcloud row is what exposed this. A governed full-suite leg
dispatched to onex-prepush-cloud1 against this branch's tree turned three
tests in tests/scripts/test_prepush_hook_host_identity_guard.py red:

  test_guard_refuses_full_suite_escalation_on_non_200_host
  test_guard_refuses_selector_whole_tree_sentinel_when_flag_is_false
  test_guard_refuses_whole_suite_equivalent_selection_when_flag_is_false

These are refusal proofs: they assert the hook refuses a heavy escalation
when the local machine is not a designated host. They forced that condition
with the two legacy aliases PREPUSH_200_HOSTNAME and
PREPUSH_201_GATE_RUNNER_HOSTNAME, which displace only rows h200 and h201c.
Every OTHER authorizing row -- h101, h105, h201, and now hcloud -- still
carried its real hostname, so running the suite ON one of those hosts made
the hook match a row, take the designated-host branch, and answer with a
placement/slot refusal instead of the identity refusal the assertion read
for.

The reds say the proof was host-dependent, not that the hook is wrong. The
same latent false-red applied to any full-suite dispatch of this repo to
h101 or h105 from the moment OMN-17159 added those rows; hcloud is merely
the row that made it observable.

_nonmatching_host_overrides() reads the committed host table and points
PREPUSH_HOST_OVERRIDE_<LABEL> at a guaranteed-non-matching hostname for
EVERY authorizing row, so these proofs stay host-independent as rows are
added rather than needing a new alias per row.

Measured red -> green on the host that exposed it, plus h200:

  onex-prepush-cloud1  38 passed in 8.20s
  stickybeatz-studio   38 passed in 69.31s

docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json records the
dispatch, the three reds, both green runs, and two honest caveats: that leg
never reached a terminal verdict (SIGTERM at 48%, exit=143) so no green
full-suite claim is made from it, and its MARKER collected=0 is a wrapper
sed bug, not a collection failure -- real collection was 44,695 items.
…ntropy false positives

docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json carries three
64-char sha256 digests -- argv_sha, head_sha and suite_log_sha256 -- which are
what make the receipt verifiable against the run it describes. detect-secrets
reads them as Hex High Entropy Strings and the Detect Secrets job failed:

  Secret Type: Hex High Entropy String
  Location:    docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json:2
  Location:    docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json:30
  Location:    docs/evidence/prepush-hcloud/hcloud-host-identity.receipt.json:47

Same false-positive class the repo already documents for its other generated
baselines. Baselined via the repo's sanctioned mechanism rather than by
stripping the digests, which would leave the receipt unverifiable.

Regenerated with detect-secrets==1.5.0 under the exact exclude set the CI job
uses. What changed, checked field by field against the previous baseline:

  * results: one file added (85 -> 86), carrying the three findings above.
    No existing entry modified or removed.
  * filters_used: gains detect_secrets.filters.common.is_baseline_file, which
    the tool adds itself whenever it is run with --baseline. Nothing removed.
  * plugins_used and the remaining top-level keys: identical.
  * generated_at: refreshed, as the tool always does.

Verified locally with the job's own command --
`git ls-files -z | xargs -0 detect-secrets-hook --no-verify --baseline ...` --
which now exits 0.
jonahgabriel added 2 commits September 3, 2026 05:01
OMN-17269 shipped per-host slot capacity; the mechanism has been INERT since,
in this repo and in omnibase_infra. `LOCK.2` has never been created once
anywhere in the fleet in three days, despite omnibase_infra carrying h105
slots=2 since 2026-08-30. Three defects in `_PREPUSH_SLOT_PROBE_SH` explain
it, each measured on live hosts rather than inferred, and each fixed here.

D1 -- the `held` glob dies under the remote login shell. The probe string is
handed to ssh and runs under the TARGET's login shell, which on h101 and h105
is /bin/zsh 5.9 (measured 2026-09-02). zsh's default `nomatch` makes an
unmatched glob a fatal error that aborts the command line BEFORE it runs, so
`ls -d "$W"/LOCK "$W"/LOCK.* 2>/dev/null` printed nothing whenever no
`LOCK.<k>` existed -- the redirection belonged to a command that never
executed. `held` therefore read 0 on every remote Mac probe, in exactly the
state slot 2 exists to serve. `find -name` does its own matching, so no glob
reaches the shell.

D2 -- one remote leg counted as two processes. The leg is launched as
`zsh -c '...; ./prepush_smart_tests.sh ...'`, so the wrapper shell AND the
script both carry the script name in argv. Measured with exactly one leg in
flight on h105 (PID 86954 wrapper + 86956 script), and the same on h101 and
h201: the old count returned 2 everywhere.

D1 and D2 compound. The busy predicate is `p <= self + held`, and `self` is 0
for a remote probe, so a correctly-locked host read `2 <= 0 + 0` -> BUSY on
every slot. Fixing either alone still refuses: `2 <= 1` and `1 <= 0`.

D3 -- an existing-but-empty QUEUE file shifted every field. `grep -c .` on an
empty file prints 0 AND exits 1, so the old `|| echo 0` fired as well and `q`
became the two-line string "0\n0". `printf` then emitted FIVE words, `set --
$raw` split five positionals, and the parse read only the first four -- every
field shifted left, and `l` was read out of `p`. Live on h201, whose
~/push-lanes/QUEUE exists and is 0 bytes: the 20:07Z refusal trail printed
`h201=busy(queue=0 heavy_pids=0 lock=2 held=1)`, and `l` is assigned only 0 or
1, so `lock=2` is a value this code cannot otherwise produce.

D3 is also FAIL-OPEN, not merely false-busy. With a real p=0 and l=1 -- the
window between `_lock_acquire`'s mkdir and pytest spawning, which spans the
whole `uv sync` -- the shifted read is q=0 p=0 l=0 held=1 and the probe
returns FREE on a host whose LOCK is held. That is why the exact-four-field
parse guard is load-bearing rather than cosmetic: a malformed probe is now
UNKNOWN, skipped exactly like unreachable, instead of readable as a verdict.

WHY IT SHIPPED: `_PREPUSH_SLOT_PROBE_SH` had never been EXECUTED by a test in
any of the three repos. The only reference was a string grep, and every slot
test went through PREPUSH_SLOT_OVERRIDE_MAP, which short-circuits the probe.
Three defects shipped into a fail-closed merge gate behind assertions that
only read the library's text. The seven tests added here extract the real
probe body and run it -- under `bash -O failglob` and under real zsh, against
a `ps` stub built from argv captured verbatim off h105, and against an
existing-but-empty QUEUE file. All five defect tests were proven RED against
the shipped library first; `test_an_unexplained_heavy_process_is_still_busy`
passes BOTH before and after, which is what proves these changes restore the
predicate's inputs rather than relax `p <= self + held`.

The four library hunks are applied VERBATIM from
.onex_state/omn17602-slot-probe-fix.patch, and the whole shared block they land
in -- from `# multi-slot row it lets a legitimately-held` through the end of
`prepush_slot_state` -- is now BYTE-IDENTICAL to the copy already merged on
omnimarket dev (omnimarket#2276, omnimarket dev b79d8983), verified by `diff`
at 2026-09-03T08:55Z. The one line that differed was the in-file citation: this
commit originally wrote `(OMN-17602)` because that is the lane that measured the
defects, while the merged omnimarket text names OMN-17606 as the landing lane
and OMN-17602 as the finder. Aligning to the merged text is not cosmetic -- the
comment is inside a file both repos vendor, so a divergent line means the next
re-sync of lines 140-300 shows a spurious diff, and both repos ship a
`test_the_picker_library_is_not_edited_into_a_repo_specific_fork` that rejects a
repo name appearing in this file at all.

The sha256 parity pin in tests/scripts/test_prepush_host_table.py is bumped in
this same commit to 6813caf5...c67d, and its docstring now says what that digest
actually tracks -- core's copy is 1114 lines to omnibase_infra's 1511 and has
not been byte-identical since OMN-17159, so the pin means "not edited without
review", not "equals upstream".

Live proof of the fixed probe, read-only over the same ssh transport
prepush_slot_state uses, 2026-09-02T21:33Z:
  h200 k=1 [0 11 0 0]   busy: 11 heavy legs, genuinely saturated
  h201 k=1 [0 1 1 1]    busy: lock held -- and queue now reads ONE field
  h101 k=1 [0 1 1 1]    busy   h101 k=2 [0 1 0 1]  FREE
  h105 k=1 [0 1 1 1]    busy   h105 k=2 [0 1 0 1]  FREE
Both slot-2 rows are placeable for the first time. This repo's table still
says slots=1 for h101/h105, so core sees that capacity only once OMN-17602's
widening lands; omnibase_infra already carries slots=2 and gains it on merge.

Ticket: OMN-17606
…e-push host table

The core host table pinned every capacity row at slots=1. OMN-17159 refused
to inherit omnibase_infra's slots=2 widening on an explicit premise: that
this repo's escalation is the whole tests/ tree, so two concurrent core
suites on h105's ten cores "would make both slower than one serialized pair
while degrading the load signal every other row is ranked on".

That premise was a projection about a real core leg, and the first real core
legs falsify it. Probed read-only 2026-09-02T19:22Z over ssh, on two hosts
each carrying exactly one live governed core full-suite leg:

  h101 (12c/32GiB) run omnibase_core-4e116501fd0a-37091, 1h13m43s elapsed:
    ONE pytest process at 100.0% of ONE core, RSS 670 MB; process-tree
    totals cpu_sum=100.0%, rss_sum=671 MB, 2 procs; load1 2.96/12 = 0.25x.
  h105 (10c/32GiB) run omnibase_core-e69568dd5e02-80404, 1h49m49s elapsed:
    same shape, RSS 700 MB; load1 4.07/10 = 0.41x while that host was in
    fact carrying TWO concurrent core pytest invocations.

The core remote leg is single-threaded, so a lane costs one core and ~0.7 GB
-- not the machine. The serialization was not protecting the hosts from
saturation; it was idling 8-11 cores per host while lanes queued for a
placement target. Each slot is still re-qualified on its own live load at
pick time, so a busy or loaded second slot still refuses.

Not widened past 2, and h200/h201/h201c are not widened at all: h200 is the
local/default identity host rather than a distribution target, h201 runs the
separate ~/push-lanes/QUEUE serializer (a different concurrency mechanism
this column does not govern), h201c never executes anything.

A new test asserts the behaviour against the REAL shipped table -- before
this change slots>1 was only ever exercised on a synthetic fixture, and a
read-only probe at 19:11Z confirmed LOCK.2 has never been created once
across h105 (121 run dirs) and h101 (73 run dirs), i.e. the path was
untaken fleet-wide.

LANDED AS PART OF omnibase_core#1640 (OMN-16634), not as its own PR. The
original commit deliberately excluded the hcloud row because #1640 already
carried it (c98bfc3) together with the host-identity refusal fix (f4b10ba)
that the row requires. Both halves are now in one PR instead, so the lab pays
ONE governed full-suite run for the three capacity changes rather than three.
The cherry-pick conflicted in `prepush_hosts.tsv` and in
`test_the_shipped_slots_column_is_pinned`; both were resolved as the UNION --
h101 and h105 go to slots=2, and #1640's hcloud row stays at slots=1.

hcloud stays at 1 on its own reasoning, not by inheriting the blanket pin this
commit removes: the h101/h105 widening is earned by a measurement of those two
hosts, no concurrent-pair measurement of the EC2 host exists, and the row is an
overflow target only reached once the lab is already saturated. The hcloud
note's old sentence ("slots=1 for the same reason as every row in THIS repo")
asserted a blanket rule this commit falsifies, so it is rewritten here rather
than left to contradict the table it sits in.

Ticket: OMN-17602
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant