A deployment converges on member identities: tree, executable and serve unit mutate only when they differ

[briansrls]

What

First slice of the convergence rework (operator direction 2026-09-06): a no-drift dashboard converge produces zero mutation effects for the release-consuming members.

deployment_apply_plan reconciled against observed: [], so every member was an add on every apply and an already-converged srv1 replayed its installation (measured at 19 min, four of five source-compiling evaluators downstream of unconditional restarts). The observed-side record refutes feeding the path-keyed roster an observed side, so the three release-consuming members get real identities and leave the apply-all roster:

member	identity	observed via
tree publication	revision + tracked projection clean (three-leg instrument from the deployed-tree record)	git.Inspect ops locally / same argv over ssh
executable	sha256 of the installed file	crypto.Sha256Sum.File / sha256sum argv
serve unit	content hash of the installed document + ActiveState	Filesystem + systemctl show

decide_member → AlreadyConverged | Create | Replace | MemberRefused (unobservable refuses; never skipped, never reinstalled). derive_effects orders install-executable before publish-tree (the publishing oneshot runs the admitted binary), emits daemon-reload once iff a unit document changed, restarts iff anything changed, and always carries the readiness readback. LiveDeployApply now carries the plan; the emitter renders the plan's effects from the same fragments and reconciles only the remaining owned artifacts apply-all (declared frontier).

Evidence

Fresh build, claim_batch --hermetic:

• member_identity_witness_test 7/7: converged → []; binary drift → [install-executable, restart]; unit drift → [write-unit, daemon-reload, restart]; failed process under current unit → restart; dirty projection → republish; absent → create, unobservable → refused; wrong-member observation → refused.
• live_deploy/emit_test 58/58 incl. new: zero-mutation plan renders no install / no repository-convergence oneshot / no serve restart while the full plan renders all three; binary-only plan does not publish the tree.
• deployed_tree_remote_witness_test, shell_exec_run_argv_embed_witness_test, ssh_transport_witness_test, apply_witness_test, intent_witness_test: green except witness_readback_requires_service_ready_gate, which fails identically on current main (pre-existing).
• Full converge CLI closure typechecks with the wired apply path.

Not in this PR

Thin service-user actuator for the repository transition (both publication oneshots still run inside PublishTree, only when the tree differs); identities for the timer members; folding mode=deploy into the planner. Live measurement of a no-drift converge follows on srv1 via the sum branch.

🤖 Generated with Claude Code

https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

[briansrls]

Live measurement on srv1 (sum branch 2081e8f, this PR included):

• Drift converge (new binary + tree): observed sha256 ×2, HEAD, ActiveState; published + installed + restarted; deploy-converged.
• No-drift converge immediately after: 6 min 38 s wall (was ~19 min), zero release mutations. Trace shows the three-leg tree observation (ReadTreeIntoIndex 225 ms, StatusAgainstIndex 535 ms, IgnoredAgainstIndex 16 ms), two Sha256Sum.File reads (~30 ms each), one ShowProperty; no tree-sync start, no serve restart (serve process start time unchanged), health polls 60 → 2.
• Remaining time: ~3 min converge-CLI closure compile, ~2 min apply-all remainder (sudoers, ~20 runner-slot memory-cap reverts, publication dirs, tailscale serve, 4 daemon-reloads), ~1 min readiness. Those are the next members to give identities, then the thin actuator and the closure compile.

[briansrls]

Repairs for the five P1 findings, plus the required-floor red

Every finding is accepted. Nothing here is a narrowing of a claim to accommodate the change.

P1-1 — Ordering: the plan no longer preempts the deployment

apply_intent_from_effects folded plan.mutations ahead of every membership effect, so on a fresh
or partially repaired host the new process started before the roots it writes existed.

The emitter now walks spec.steps — the deployment's own order — and projects each step to its
effects: an identity member's step renders that member's fragments (possibly none), every other step
renders the membership effects keyed to it. apply_step_effects adds no ordering; the authority is
deployment_steps_apply_order and only that.

The contradiction you named is repaired at the authority rather than overridden downstream:
deployment_owned_steps now lists ServeBinary before GunbcSourceTree, because the tree is
published by a oneshot that runs the admitted binary. The old comment's reason (the binary serves
the tree) is true of the running process, not of the installation.

New control the_full_plan_lands_its_members_in_the_deployments_order pins convergence → attempt-state
root → restart, so a regression to a prepended block reds.

P1-2 — The unit is three members, not one

ServiceUnit is split into UnitFile (document on disk), UnitManager (systemd's own
NeedDaemonReload, wire values measured on srv1), and ServiceProcess (the release the process
reports over /healthz, through gunbc.running_release_identity — the same reader readiness uses).

Both halves of your finding are now witnessed:

• a_stale_manager_reloads_and_restarts_without_rewriting_the_file — the interrupted deploy reloads and restarts, no rewrite.
• a_failed_process_restarts_and_nothing_else — no unnecessary write or reload.
• an_active_process_running_the_previous_release_restarts — the absorbing state you described: unit current, ActiveState Active, previous release running. Under the fused identity every member compared equal, the plan was empty, and readiness refused forever. It is now a Replace on the process.

ActiveState alone is not an identity; an Active process that cannot be probed is unobservable, which refuses rather than guessing.

P1-3 — Absent tree refuses by name

Correct on both counts. Create for the tree cannot work: the sync excludes .git, and the
convergence oneshot refuses a non-repository pre-state, so an attempted create converts a recoverable
absence into a permanently refusing one.

MemberRefusalCause gains RepositoryBootstrapUnavailable, and tree absence refuses with it.
Witness: an_absent_repository_refuses_with_the_bootstrap_cause.

The fresh-host half is separately fixed: entry_presence now walks up when a listing refuses. An
ancestor established absent (by listing its parent) means the child is absent; an ancestor present
but unlistable stays indeterminate. Before this, a never-deployed host refused the whole plan before
the effect that creates the directory could run.

P1-4 — Planned identity binds actuation

The operands were labels: the emitter discarded every one and copied the ambient path.

• ReleaseTarget { revision, executable, unit } is carried once on the plan. The duplicate revision argument to live_deploy_apply_script_for is gone.
• The install now brackets the copy with digest_shell_verify_line: source verified against the planned digest before the copy, destination after, each under set -e. Your constructible state — source changes between planning and copy — stops the line.
• The unit renders from plan.target.revision.
• Post-mutation re-observation: after readiness, the deploy re-derives its own plan and requires it empty (deployment_identity_readback). A remaining mutation names the member that did not land; a refusal names the member that could not be read back. Both are NotConverged.

P1-5 — The scratch index is an owned effect, not a readonly global

git.Inspect.ReadTreeIntoIndex is no longer declared readonly — it writes the file
GIT_INDEX_FILE names — with the reason stated above the service.

The consumer now mints the index with shell.Mktemp (unique per observation, so two observers of one
release under two principals cannot collide) and removes it on every arm, including refusing ones.
A scratch that cannot be removed refuses the observation: an observation that leaves state has not
finished.

Modeling correction — completeness is by construction

List<MemberDecision> made completeness and ordering a caller obligation, and an empty list derived
an accepted empty plan. Replaced by ReleaseMemberDecisions, one slot per member, and
ReleaseEffectPlan, one switch per member — so no plan can install twice or restart before it
writes. Slot/member agreement is still checked (a_decision_in_the_wrong_slot_refuses_the_plan).

EffectPlan.readback is deleted. It was decorative: readiness is the parent transaction's policy
and always was. Two apparent authorities for one fact, now one.

Required floor

Both interrupted claims are repaired by removing a second full render each, not by raising a budget:

• the_preamble_ensures_... now reads the preamble producer and step list — its actual subject is two commands.
• a_zero_mutation_plan_... keeps only the idle render; the positive control moved to the sibling claims that already hold the shared full render.

The publication helper (separate production matter, diagnosed not yet repaired)

You were right that it contaminates any measurement taken beside it. The cause is exact:

/opt/gunbc/publication-spool/requests/ holds two requests written under
roadmap-publication-request/v1; the helper reads /v2. It refuses both — correctly — writes
request-unreadable for each, exits 1, and re-reads the same two on the next tick. Both already have
answers written (17:36) and the requests were re-written after their answers (17:38), so nothing
is pending behind them.

The defect is not the refusal, it is that an undecodable request has no disposition. It is never
retired from the pending set, so one poison request stops every later publication forever, at
~2 CPU-minutes per tick, and the deficit's frequency is unbounded rather than counted. That is the
absorbing arm §5 names, arrived at from the refusing side: the line stops and can never restart.

The repair belongs in the helper's own module — an undecodable request gets a typed refusal answer
and leaves the pending set — not in this PR. Parking the two answered requests stops the burn today
but is the workaround, so it is named as one rather than presented as the fix.

[briansrls]

Required floor is green on the exact head, and both witnesses reach verdicts

08fbccff608 — heal-generated-artifacts, required-witnesses-build, required-witnesses-floor all pass.

planned=3541 executed=3541 terminal=3541 passed=3477 known_red_held=19
claims_failed=0 interrupted_before_verdict=0 interrupted_cpu_deadline=0

interrupted_before_verdict is 0 (was 2), and passed rose 3475 → 3477 — so the two claims reached verdicts rather than the deadline preempting them. That closes the last item in your recommended sequence.

How, and one correction worth recording

The first head still interrupted both, at 510 ms and 515 ms against the 500 ms ceiling, with claims_failed=0 — nothing wrong with the model, the deadline just preempted the verdict. Two things were wrong with my reasoning, and both were caught by measurement rather than by reading:

The module-scope render hoist made per-claim cost order-dependent. The first claim to touch the shared apply render pays for it. Locally an unchanged claim paid; in CI the_full_plan_lands_... did. So a local reading of a hoisted claim does not predict its CI cost — which is why my "331 ms locally, comfortably clear" was worthless as evidence.

The repair is that neither changed claim renders the apply script at all. Both now read emit_release_member_effects — the steps one member contributes. That is cheaper and a closer statement of the subject than grepping serialized bash for absent lines. a_zero_mutation_plan_... went 3606 ms → 2 ms locally.

The ordering half is no longer asserted over text either, and does not need to be: the emitter folds flat_map over spec.steps, so fragment order is step order by construction. What a regression would change is the authority's order, so that is what the claim pins now.

A falsified annotation, corrected in place rather than quietly dropped. I attributed the residual 469 ms to owned_step_index rebuilding the whole spec on each of four calls. Taking the step list once measured 496 ms — noise. The parameterization stays, because rebuilding a spec four times inside a fold is the cost-shape defect the standing rule fixes regardless of realized n, but the note now says explicitly that the measurement does not credit it and the residual is unattributed. A reader who took that as the repair would look for the next one in the wrong place.

Separately: the publication helper

Diagnosed and fixed as its own PR (#10707), not folded in here. Two requests written under a superseded schema stopped the helper every tick for a day. The refusal was correct; the defect was that it retained the request, while every other terminal outcome consumes its request — so one undecodable document re-stopped the line forever.

Production is repaired by hand ahead of that PR (8 ticks, 25 min, zero failures since). Worth noting for your measurement-contamination point: the helper's ~1 min 53 s CPU per tick is not caused by the failure — a successful run costs the same. That is the per-tick closure re-resolve, now measured at 178 s (belt) and 208 s (converge) of pure resolve with zero effects executed, i.e. roughly half of a no-drift converge spent before any observation happens.