Skip to content

fix(OMN-18073): deploy agent launches through a bash-sourced env and re-execs on self-update - #3364

Merged
jonahgabriel merged 1 commit into
devfrom
jonah/omn-18073-agent-env-transport
Sep 9, 2026
Merged

jonahgabriel merged 1 commit into
devfrom
jonah/omn-18073-agent-env-transport

Conversation

@jonahgabriel

@jonahgabriel jonahgabriel commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

OMN-18073 — deploy agent launches through a bash-sourced env and re-execs on self-update

The repo half of the guard landed in #3358. This lands the transport that guard
protects against, plus the reason the guard's own advice ("restart this unit")
would not by itself have been enough.

Cause

deploy-agent-dev.service declared EnvironmentFile= on the operator env store.
systemd's env-file parser does not implement bash ANSI-C $'...' quoting: it
keeps the literal wrapper and drops every backslash escape, so each escaped
newline collapses to a bare letter. The store carries exactly one line in that
form. Through bash source that value is 1679 bytes and parses as a PEM;
through EnvironmentFile= it is 1682 bytes, still wrapped, with zero
backslashes. scripts/deploy-runtime.sh has always bash-sourced the very same
file on the very same host and decoded it correctly, which is why only the agent
path produced broken containers.

Change

Half 1 — the transport. ExecStart now runs
scripts/deploy-agent/deploy/deploy-agent-launch.sh, which does what
deploy-runtime.sh does: set -a; source; set +a, then exec the interpreter.
There is no EnvironmentFile= on the unit at all.

DEPLOY_AGENT_ENV_PROTECTED is what makes that faithful rather than merely
different. systemd applies directives in file order, so the Environment= lines
that followed EnvironmentFile= used to win over the store; a plain
set -a; source inverts that. The store defines both KAFKA_BOOTSTRAP_SERVERS
and KAFKA_ENVIRONMENT, and its KAFKA_ENVIRONMENT disagrees with the dev
unit's — so an unprotected name is a silently relabelled lane, not a cosmetic
difference. The launcher snapshots the unit-owned names, sources, and restores
them, reproducing systemd's ordering exactly. A test derives the expected set
from the unit file itself, so adding an Environment= line without extending
the list is a red test.

Half 2 — the re-exec carries the environment, not just the code. The premise
this work was dispatched on, that the agent never re-execs on self-update, is
false and the journal says so: one pid re-execed four times in a day, advancing
its head normally. NRestarts=0 and an unchanged ExecMainStartTimestamp are
not evidence against that, because os.execv keeps the pid so systemd sees no
restart. What did not advance is the environment: os.execv inherits the
caller's environment verbatim, so re-execing sys.executable can only carry
forward what the process started with. self_update now re-execs
DEPLOY_AGENT_LAUNCHER when the launcher exported it, which re-sources the
store, so a repaired or rotated value is picked up at the next job boundary
instead of needing an operator restart. Off the unit (a developer running the
module directly, container mode's supervisor respawn) the pre-existing
interpreter re-exec is unchanged, with a negative control asserting it.

Why this is urgent

The #3358 fail-closed guard is currently firing on the dev lane: every deploy
since 13:05Z fails at compose_gen naming the affected key. That is the correct
behaviour on a still-mangled environment, and it also means a bare restart of
the OLD unit would refuse every job rather than fix anything. The unit change
plus a daemon-reload is what clears it.

Tests

scripts/deploy-agent/tests/unit/:

  • test_deploy_agent_launcher_omn18073.py — the unit declares no
    EnvironmentFile= and execs the launcher; the protected list covers every name
    the unit declares; a throwaway generated PEM proves the two parsers disagree
    (positive control) and that the launcher takes bash's answer; the store loses
    to the unit on a protected name and wins on an unprotected one (negative
    control); every missing/unreadable/malformed input refuses with exit 2 rather
    than starting on an un-sourced environment.
  • test_executor_self_update_launcher_omn18073.py — re-exec goes through the
    launcher when declared, through the interpreter when not, and not at all when
    the head is unchanged.
  • test_deploy_agent_service_source.py — asserts the unit file's own shape.

56 unit tests green.

Lab-first statement — stated honestly, not borrowed

This change was exercised by unit tests only. It carries no live lab proof.

The governed pre-push selector ran on the .201 lab host and passed
(scripts/tests/, tests/scripts/, tests/unit/scripts/ — 2861 passed, 5
skipped, 274s, host load 0.35x of 32 cores). That is a test-suite result on lab
hardware. It is not a runtime proof: nothing in this PR started the unit,
sourced the real store, or observed a decoded key inside a running container.

The live proof is an operator-run systemctl --user daemon-reload plus a restart
of the dev agent unit, followed by a readback of the effects container's key
digest (sha256-12 and byte length only, never the value). That restart is a
runtime mutation on the lab host and is pending operator approval; this PR
does not perform it and does not claim it.

Residual, stated rather than left implicit

deploy-agent.service (prod lane) still declares EnvironmentFile= on the same
store and would hit the same refusal on its next prod deploy. It is not changed
here because its tracked drop-in replaces ExecStart wholesale, so the launcher
cannot be adopted there without editing the prod drop-in and reinstalling the
prod agent — a prod-lane action outside this change's scope.

Gate notes

No bypass of any kind was used. This PR body contains no member of the
skip-token family and no merge-hold vocabulary; both are referred to here by
family name only, per the repo rule that those gates are substring-matched and a
literal mention fires them. The evidence-source stamp at the very bottom is the
one real stamp, on its own line, where it is meant to be parsed. The governed pre-push
hook chain ran in full on .201 with no PREPUSH_* override, no
ENABLE_SMART_TESTS override, no hooks-path override and no --no-verify.

OMN-18073

Evidence-Source: OCC#8810
Evidence-Ticket: OMN-18073

…re-execs on self-update

The repo half of #3358 landed the fail-closed guard. This lands the transport
it guards against, plus the reason the guard's own advice ("restart this unit")
would not by itself have been enough.

Cause, re-measured on the lab host 2026-09-09, read-only, values identified by
length and sha256-12 only.

Half 1 -- the transport. deploy-agent-dev.service declared
`EnvironmentFile=<the operator env store>`. systemd's env-file parser does not
implement bash ANSI-C $'...' quoting: it keeps the literal $' / ' wrapper and
drops every backslash escape, so each \n collapses to the bare letter n. The
store carries exactly one line in that form (ONEXBOT_OCC_PRIVATE_KEY, line
255). Through bash `source` it is 1679 bytes and parses as a PEM; through
EnvironmentFile= it is 1682 bytes, still wrapped, with zero backslashes.
scripts/deploy-runtime.sh (lines 89-93) has always bash-`source`d the very same
file on the very same host and decoded it correctly, which is why only the
agent path produced broken containers.

ExecStart now runs deploy/deploy-agent-launch.sh, which does what
deploy-runtime.sh does -- `set -a; source; set +a`, then exec the interpreter --
so the agent's process environment is the bash-decoded one and there is no
EnvironmentFile= on the unit at all.

DEPLOY_AGENT_ENV_PROTECTED is what makes that faithful rather than merely
different, and it is not decoration. systemd applies directives in file order,
so the Environment= lines that followed EnvironmentFile= used to WIN over the
store; a plain `set -a; source` inverts that. The store defines both
KAFKA_BOOTSTRAP_SERVERS and KAFKA_ENVIRONMENT, and its KAFKA_ENVIRONMENT is
`local` while the dev unit declares `dev` -- so an unprotected name is a
silently relabelled lane, not a cosmetic difference. The launcher snapshots the
unit-owned names, sources, and restores them, reproducing systemd's ordering
exactly. A test derives the expected set from the unit file itself, so adding
an Environment= line without extending the list is a red test.

Half 2 -- the re-exec carries the environment, not just the code. The premise
this lane was dispatched on -- that the agent never re-execs on self-update --
is false, and the journal says so: pid 2171795 re-execed at 01:55:03Z,
09:05:01Z, 12:41:12Z and 12:54:36Z, advancing b0b46c1 -> c65d8a8 normally.
NRestarts=0 and an unchanged ExecMainStartTimestamp are not evidence against
that; os.execv keeps the pid, so systemd sees no restart.

What did not advance is the environment. os.execv inherits the caller's
environment verbatim, so re-execing sys.executable can only ever carry forward
what the process started with at 01:38:54Z. self_update now re-execs
DEPLOY_AGENT_LAUNCHER when the launcher exported it, which re-`source`s the
store, so a repaired or rotated value is picked up at the next job boundary
instead of needing an operator restart. Off the unit (a developer running
`python -m deploy_agent`, container mode's supervisor respawn) the pre-existing
interpreter re-exec is unchanged, with a negative control asserting it.

#3358's UndecodedAnsiCQuotingError guard is untouched and stays as the
fail-closed backstop. It is currently firing in production: job
bd21f41c-d24a-45c4-b8a0-01c77a681a2c failed at 13:05:37Z at compose_gen naming
ONEXBOT_OCC_PRIVATE_KEY, which is the correct behaviour on a still-mangled
environment and also means a bare restart of the OLD unit would refuse every
job rather than fix anything. The unit change plus a daemon-reload is what
clears it.

Tests (scripts/deploy-agent/tests/unit/):
  test_deploy_agent_launcher_omn18073.py -- the unit declares no
    EnvironmentFile= and execs the launcher; the protected list covers every
    name the unit declares; a throwaway generated PEM proves the two parsers
    disagree (positive control) and that the launcher takes bash's answer; the
    store loses to the unit on a protected name and wins on an unprotected one
    (negative control); every missing/unreadable/malformed input refuses with
    exit 2 rather than starting on an un-sourced environment.
  test_executor_self_update_launcher_omn18073.py -- re-exec goes through the
    launcher when declared, through the interpreter when not, and not at all
    when the head is unchanged.

Residual, stated rather than left implicit: deploy-agent.service (prod lane)
still declares EnvironmentFile= on the same store and would hit the same
refusal on its next prod deploy. It is not changed here because its tracked
drop-in replaces ExecStart wholesale, so the launcher cannot be adopted there
without editing the prod drop-in and reinstalling the prod agent -- a prod-lane
action outside this lane's scope.

OMN-18073
@jonahgabriel
jonahgabriel enabled auto-merge (squash) September 9, 2026 16:39

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hostile Reviewer — adversarial findings (OMN-17492)

Models succeeded: glm-review
Models failed: codex
New finding threads: 2
Deduped (already posted on this PR): 0
Nit-level findings suppressed: 1

The model is the FINDER, never the gate: merge is gated only by the
deterministic Hostile Review Thread Gate, which blocks while
hostile-reviewer threads are unresolved. Resolve each thread after
addressing (or rejecting, with a reply) its finding.

Findings not anchored to a changed file

  • [CRITICAL] hostile-reviewer (glm-review)

    Root-context bash source of operator-owned file enables arbitrary code execution | The unit previously consumed the store as data via EnvironmentFile=, which cannot execute logic. The launcher now sources /home/jonah/.omnibase/.env from a script that runs under the unit's identity (no User= directive is shown, so root), and with set -a. Any account that can write that file, or any future line in it such as $(...) command substitution, backtick substitution, or an outright rm/curl | sh statement, n

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MAJOR] hostile-reviewer (glm-review)

    Protected-restore only covers names already set, diverging from systemd semantics | The snapshot loop records a name only when it is already set in the launcher's environment (__is_set). If the store defines a protected name that the unit did not set for some invocation (a developer running the launcher by hand, or a unit edited to remove a line), the store's value survives the restore and silently wins. systemd's ordering would never allow the store to override an explicit Environment= declaration; the r

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MAJOR] hostile-reviewer (glm-review)

    Positive control models systemd's parser instead of invoking it | test_systemd_environmentfile_mangles_what_bash_decodes does not run systemd; it hand-implements the claimed mangling (raw.replace("\\n", "n")) and then asserts the hand-implementation matches itself. If systemd's actual behavior on this input differs in any detail (it applies other transformations to unquoted values, including stripping certain characters), the test passes while production diverges. The entire incident rationale rests on th

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    Documented exit code 2 not produced on store parse failure | The contract comment states malformed input 'exits 2 and names the reason on stderr'. But errexit remains armed during the source; a store line whose command fails, or a store invoking exit 1, aborts the launcher with exit 1 (or the store's own code), not 2, and with no launcher-branded message. Monitoring keyed on exit code 2 will misclassify the failure. | Evidence: Errexit stays armed -- a store that fails to parse is a refusal, not something

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    os.execv of launcher raises unhandled OSError | The new self_update branch execs DEPLOY_AGENT_LAUNCHER without validating existence or executability. If the script was removed between the env snapshot and the re-exec (self-update path, git pull racing), os.execv raises FileNotFoundError from deep inside self_update, producing a traceback and an inconsistent agent state, whereas the bare-interpreter branch would have succeeded. DEPLOY_AGENT_LAUNCHER is also trusted from the environment without a sanity check

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    Env rotation claim overstates when the store is re-read | The executor comment says 'a repaired or rotated value is picked up at the next job boundary'. The launcher is only re-exec'd when self_update observes a code change; on an up-to-date tree no re-exec occurs (test_up_to_date_never_re_execs_the_launcher pins this). A credential rotation with no accompanying code change therefore still requires a systemd restart, the exact operational friction the comment implies is gone. Operators reading this comment

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

Findings demoted from threads (anchor rejected)

  • [MINOR] hostile-reviewer (glm-review)

    EnvironmentFile= detection misses the '-'-prefixed ignore-failure form | test_dev_unit_declares_no_environment_file_on_the_operator_store matches lines starting with 'EnvironmentFile=' only. systemd accepts 'EnvironmentFile=-path', which silently ignores a missing file; a future regression using that form reintroduces the mangled parser for an existing file while passing this test. | Evidence: if line.startswith("EnvironmentFile=") | Fix: Match the directive name with a regex tolerant of the leading '-' arg

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

  • [MINOR] hostile-reviewer (glm-review)

    Tests hardcode host-specific absolute paths | The launcher tests assert exact lab-host paths (/home/jonah/.omnibase/.env, /data/omninode/omnibase_infra/...) inside the repo's unit test suite. These tests fail on any developer checkout or CI runner lacking that layout, and they encode a personal home directory into the test suite. | Evidence: _OPERATOR_ENV_STORE = "/home/jonah/.omnibase/.env" | Fix: Derive expected paths from a single module-level constant or environment-driven configuration shared between t

    Resolve this thread when addressed — the Hostile Review Thread Gate blocks while hostile-reviewer threads are unresolved (OMN-17492).

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

⚠️ Hostile Reviewer — DEGRADED (informational)

Critical findings: 1
Major findings: 2
Total findings: 9
Models succeeded: glm-review

Note: Fewer than 2 reviewer models succeeded. Degraded results are informational (OMN-8468/OMN-8524) and do not block merge. Error: cli_review exit 2 (fewer than 2 models succeeded — partial/total outage)


Semantics (OMN-17492 — the model finds, thread resolution gates)

Surface Meaning Blocks merge?
Review threads Per-finding, posted by the reviewer No (informational)
Hostile Review Thread Gate Deterministic: unresolved hostile-reviewer threads exist Fails until resolved (not yet a required context)
degraded verdict Fewer than 2 models succeeded (infra) No

Powered by omniintelligence.review_pairing.cli_review — multi-model adversarial review: qwen3-review, qwen3-review-b, glm-review (OMN-8468/OMN-8524/OMN-17492)

@jonahgabriel
jonahgabriel merged commit f4714c2 into dev Sep 9, 2026
229 of 237 checks passed
@jonahgabriel
jonahgabriel deleted the jonah/omn-18073-agent-env-transport branch September 9, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant