fix(OMN-20798): lane census fails loud when it cannot publish drift; nightly sweep unit waits for its broker

[jonahgabriel]

lane-census-check.sh now exits 8 when a drift event is not published, while a published drift still exits 30; previously both cases exited 30. The drift event is produced through the broker container named by LANE_MEMORY_BROKER_CONTAINER, which the installer writes into the unit, using the same helper as the memory pass. The KAFKA_BOOTSTRAP_SERVERS plus host-side rpk branch is removed because rpk is not on a lab host PATH.

scripts/systemd/onex-nightly-sweep.service now carries the whole publish mechanism instead of running an untracked host script. It waits up to 120 s for the broker container to be healthy and fails before publishing if it is not, has TimeoutStartSec=300, names the SASL pair by variable inside the container, and sends a fresh correlation id per run. The previous nightly run had failed with "OCI runtime exec failed ... setns process: exit status 1"; the run before it had hung from 2026-10-07 03:00 to 2026-10-09 10:01. Not done in this PR: the second lab host (h202) could not be reached from this lane; h202 is out of lab placement per the operator's 2026-10-09 ruling under OMN-20769 and ssh to it from the lane host fails host key verification.

Acceptance criteria (OMN-20798)

• AC1: the census exits non-zero when its drift cannot be published. Falsifier uv run pytest tests -q -k lane_census_unpublishable_drift_fails on h201 at head 9c41170: the three *_fails_* tests were red before the change (exit 30 where 8 is expected) and are green after; the published-drift and dry-run companions in tests/unit/scripts/test_lane_census_unpublishable_drift_omn20798.py are green. test_lane_census_dry_run.py was updated to the new contract (no broker container is now exit 8, not 30).
• AC2: both units complete a run on their hosts.
  • h201 onex-nightly-sweep.service: MET. Unit installed from this branch's file, systemctl --user start onex-nightly-sweep.service returned 0, Result=success ExecMainStatus=0. journalctl --user -u onex-nightly-sweep -n 5:
    Starting onex-nightly-sweep.service - ONEX Nightly Sweep — publishes build-loop-start to the dev lane broker...
onex-nightly-sweep[2324809]: Produced to partition 0 at offset 41 with timestamp 1791568984154.
onex-nightly-sweep[2324775]: onex-nightly-sweep: build-loop-start bcaca6ae-c011-4167-bb22-f9257ebffa77 published via omnibase-infra-redpanda
Finished onex-nightly-sweep.service - ONEX Nightly Sweep — publishes build-loop-start to the dev lane broker.
  • h202 onex-lane-census.service: NOT PROVEN by this lane. h202 is out of lab placement (operator ruling 2026-10-09 under OMN-20769, so onex-lab-run --host h202 refuses) and ssh to it from the h201 lane host fails host key verification. The host steps are below.

h202 steps (for the operator or the launching host)

From the refreshed omnibase_infra clone on h202 at this PR's merge: bash deploy/lane-census/install-lane-census.sh --standalone --broker-container omnibase-infra-dev-202-redpanda --repo-root /data/omninode/omnibase_infra, then systemctl --user start onex-lane-census.service and journalctl --user -u onex-lane-census -n 5. Expected: with the broker container named, a drift is published (published lane-census-drift event ... via broker container) and the unit exits 30 only while a declared container is genuinely absent (the dev-202 Phoenix of the 2026-10-09 journal); with none named it exits 8 and says DRIFT event NOT published.

Notes

• The installed onex-disk-gc census drop-in on h201 is an older install (no --memory, no broker container) and its ExecStart carries the - prefix, so a census exit there, 8 included, does not fail that unit. Left unchanged here; reinstalling with install-lane-census.sh --broker-container omnibase-infra-redpanda and the fail-soft prefix are separate decisions.
• ~/.local/bin/onex-nightly-sweep-trigger.sh on h201 is no longer referenced by the unit; it is left in place and can be deleted.
• Plan section 1.2 / T6 (knowledge-base-internal PR feat: DelegationIntentBridge completes delegation chain end-to-end [OMN-7604] #1180) was not readable from the lane host; the brief's excerpt was used.

Lab: host=h201 (omninode-pc), lane=mon-reds-infra-r-5d21, command=cp scripts/systemd/onex-nightly-sweep.service ~/.config/systemd/user/ && systemctl --user daemon-reload && systemctl --user start onex-nightly-sweep.service then journalctl --user -u onex-nightly-sweep -n 5: unit finished Result=success and the broker answered Produced to partition 0 at offset 41; before the change the same unit failed OCI runtime exec failed ... setns process. Census exit codes exercised live on h201 with LANE_MANIFEST=<manifest plus one absent container> bash scripts/lane-census-check.sh --lane judge: no broker container named gave exit=8 and DRIFT event NOT published: LANE_MEMORY_BROKER_CONTAINER is unset; a nonexistent broker container gave exit=8 and produce ... FAILED; nothing reached the bus. h202 not reachable from this lane (see above). head=9c41170fac80ca266b497084da8df9d3d75efb2b

Delegation: commit messages and the summary paragraph were drafted through onex delegate via landing_text.py and checked against the facts.

Evidence-Ticket: OMN-20798
Evidence-Source: OCC#13455

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, local-studio-planner
Models failed: none
New finding threads: 4
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 8
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 8 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

• [CRITICAL] scripts/systemd/onex-nightly-sweep.service (local-studio-planner) — Command injection via broker container name in ExecStartPre | The ExecStartPre command in onex-nightly-sweep.service passes the broker container name as a positional argument to a bash script: `..
• [MAJOR] scripts/lane-census-check.sh (qwen3-review) — Shell injection via unquoted variable expansion in broker_produce | The broker_produce function constructs a shell command string for docker exec by interpolating $user_var, $pass_var, `$mecha
• [MAJOR] scripts/lane-census-check.sh (qwen3-review) — Incorrect exit code precedence in finish() function | The finish() function in lane-census-check.sh gives precedence to MEMORY_RC over the census exit code rc, except when rc is `EXIT_DRIFT_
• [MAJOR] scripts/systemd/onex-nightly-sweep.service (local-studio-planner) — Race condition in health check loop | The ExecStartPre loop in onex-nightly-sweep.service polls the container health status every 2 seconds. If the container becomes healthy between checks, it pro
• [MAJOR] tests/unit/scripts/test_onex_nightly_sweep_unit_omn20798.py (local-studio-planner) — Test shim does not validate SASL variable expansion safety | The test test_onex_nightly_sweep_unit_omn20798.py shims docker and checks the command line arguments. It verifies that `${DEV_KAFKA_SAS
• [MINOR] tests/unit/scripts/test_lane_census_unpublishable_drift_omn20798.py (qwen3-review) — Missing test for concurrent execution of lane-census-check.sh | The tests for lane-census-check.sh do not cover concurrent execution. If two instances of the script run simultaneously, they may inte
• [MINOR] scripts/lane-census-check.sh (qwen3-review) — Missing error handling for docker exec failure in broker_produce | The broker_produce function does not handle the case where docker exec fails due to a non-existent container or other Docker erro
• [MINOR] scripts/lane-census-check.sh (local-studio-planner) — Ambiguous exit code handling in finish function | In lane-census-check.sh, the finish function checks if MEMORY_RC is non-zero. If it is, it logs and exits with MEMORY_RC. However, if rc (th

Findings demoted from threads (anchor rejected)

• [MAJOR] hostile-reviewer (qwen3-review)

  Shell injection in onex-nightly-sweep.service ExecStart | The ExecStart directive in onex-nightly-sweep.service uses $$1 to pass the broker container name into a bash -c script. The variable $$1 is expanded by systemd, but the resulting value is then used in a docker exec command inside the bash script. If the ONEX_SWEEP_BROKER_CONTAINER environment variable is set to a value containing shell metacharacters, it could lead to command injection. The use of sh -c inside docker exec with unquo

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (local-studio-planner)

  Credential exposure in docker exec command | In onex-nightly-sweep.service, the SASL credentials are passed via environment variables to the docker container. While the comment claims they are not on argv, the docker exec command constructs a shell command string that includes variable expansion: sh -c 'RPK_USER="${DEV_KAFKA_SASL_USERNAME}" ...'. If ${DEV_KAFKA_SASL_USERNAME} or ${DEV_KAFKA_SASL_PASSWORD} contain single quotes or other shell metacharacters, they could break out of the string liter

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (qwen3-review)

  Inefficient health check loop in onex-nightly-sweep.service | The ExecStartPre directive in onex-nightly-sweep.service polls the broker container's health status every 2 seconds for up to 120 seconds. This is inefficient and could delay the start of the service unnecessarily. A more efficient approach would be to use a single docker wait command or a similar mechanism to wait for the container to become healthy. | Evidence: ExecStartPre=/bin/bash -c 'for i in $$(seq 1 60); do if [ "$$(docker inspect -

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MINOR] hostile-reviewer (local-studio-planner)

  Inefficient health check polling interval | The health check loop in ExecStartPre sleeps for 2 seconds between checks. With a maximum of 60 iterations, this allows up to 120 seconds (2 minutes) for the container to become healthy. This is consistent with the comment but may be too slow if the container typically starts faster. Conversely, if it fails, the unit waits the full duration. | Evidence: ExecStartPre=/bin/bash -c 'for i in $$(seq 1 60); do ... sleep 2; done' | Fix: Consider reducing the sleep int

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

[github-actions]

✅ Hostile Reviewer — REVIEWED

Critical findings: 3
Major findings: 5
Total findings: 13
Models succeeded: qwen3-review,local-studio-planner
Models unavailable: none

Action required: findings were posted as review threads. Address or reject each one, then resolve its thread — the Hostile Review Thread Gate fails while hostile-reviewer threads are unresolved (OMN-17492).

---

Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface	Meaning	Blocks merge?
Review threads	Per-finding, posted by the reviewer	No (informational)
Hostile Review Thread Gate	Deterministic: unresolved hostile-reviewer threads exist	Fails until resolved (not yet a required context)
degraded verdict	Fewer than 2 models succeeded (infra)	Fails this job with a named reason

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review (Qwen3.8-27B), local-studio-planner (Qwen3.6-35B-A3B) (OMN-8468/OMN-8524/OMN-17492)

[github-actions]

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: qwen3-review, local-studio-planner
Models failed: none
New finding threads: 2
Deduped (already posted on this PR): 0
Below quorum (one model only, reported not threaded): 10
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Below quorum: 10 finding(s) raised by one model only (OMN-18479)

These are reported and NOT dropped, but they get no thread and do not block: a single model's finding no other model reproduced is not evidence enough to stop a merge. Read them; act on them if they are right.

• [CRITICAL] scripts/systemd/onex-nightly-sweep.service (qwen3-review) — Shell injection via unquoted variable in docker exec command | The broker_produce function constructs a shell command string where the $topic variable is interpolated without proper quoting. The l
• [CRITICAL] scripts/systemd/onex-nightly-sweep.service (qwen3-review) — ExecStartPre uses wrong argument index for container name | The ExecStartPre directive invokes /bin/bash -c '...' sweep ${ONEX_SWEEP_BROKER_CONTAINER}. Inside the single-quoted script, $$1 expan
• [CRITICAL] scripts/lane-census-check.sh; scripts/systemd/onex-nightly-sweep.service (local-studio-planner) — Credential Injection via Broker Container Name | The broker_produce function in lane-census-check.sh and the ExecStart command in onex-nightly-sweep.service construct shell commands by interpo
• [MAJOR] tests/unit/scripts/test_onex_nightly_sweep_unit_omn20798.py (qwen3-review) — Test for nightly sweep unit will fail due to argument index bug | The test test_a_healthy_broker_gets_one_command_per_run_with_a_fresh_id runs the ExecStart command and expects it to succeed. Howe
• [MAJOR] scripts/systemd/onex-nightly-sweep.service (local-studio-planner) — Insecure Credential Expansion in Nightly Sweep | The nightly sweep unit expands credentials using ${DEV_KAFKA_SASL_USERNAME} and ${DEV_KAFKA_SASL_PASSWORD} inside the sh -c command. While these
• [MAJOR] tests/unit/scripts/test_lane_census_unpublishable_drift_omn20798.py (local-studio-planner) — Incomplete Test Coverage for Error Paths | The test test_lane_census_unpublishable_drift_fails_when_the_produce_fails mocks the docker exec command to return a non-zero exit code. However, it does
• [MINOR] scripts/lane-census-check.sh (qwen3-review) — SASL credentials exposed in process environment | The broker_produce function passes SASL credentials via environment variables to the docker exec command. While the credentials are expanded insid
• [MINOR] tests/unit/scripts/test_onex_nightly_sweep_unit_omn20798.py (qwen3-review) — Missing test for ExecStartPre argument index bug | The test suite does not include a test that would catch the argument index bug in the systemd unit file. The test `test_an_unhealthy_broker_fails_the
• [MINOR] scripts/systemd/onex-nightly-sweep.service (local-studio-planner) — Inefficient Broker Health Check Polling | The health check loop in onex-nightly-sweep.service uses a fixed sleep interval of 2 seconds. This may be too aggressive for some environments, causing unne
• [MINOR] scripts/lane-census-check.sh (local-studio-planner) — Tight Coupling Between Script and Broker Container | The lane-census-check.sh script is tightly coupled to the broker container's internal structure (e.g., expecting rpk to be available inside the

Findings demoted from threads (anchor rejected)

• [MAJOR] hostile-reviewer (qwen3-review)

  ExecStart uses wrong argument index for container name | The ExecStart directive has the same issue: it passes sweep as the first argument and ${ONEX_SWEEP_BROKER_CONTAINER} as the second. The script body uses $$1 to reference the container name in docker exec -i "$$1", which will resolve to sweep instead of the actual container name. This will cause the produce command to fail because it will try to exec into a non-existent container named sweep. | Evidence: ExecStart=/bin/bash -c 'set -euo p

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

• [MAJOR] hostile-reviewer (local-studio-planner)

  Race Condition in Broker Health Check | The ExecStartPre command in onex-nightly-sweep.service polls the broker container's health status every 2 seconds for up to 120 seconds. However, if the container becomes healthy between checks, the script will proceed. If the container becomes unhealthy immediately after the check passes but before the ExecStart command runs, the produce operation may fail. This is a standard TOCTOU race condition in health checking. While bounded, it does not guarantee that th

  Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).