Skip to content

Re-emit the deploy actuation: nothing has deployed srv1 or advanced the fleet ref since the floor cut - #8622

Merged
briansrls merged 4 commits into
mainfrom
session/nimble-fox-671-deployemit
Aug 20, 2026
Merged

briansrls merged 4 commits into
mainfrom
session/nimble-fox-671-deployemit

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

The 2026-08-15 floor cut deleted ci.yml, and with it the only thing that ever actuated a deployment or advanced the fleet's admitted revision. Both stayed modeled and neither ran — gunbc.ci_spec gunbc_ci_deploy_srv1_stage described a deploy nothing performed, and gunbc_ci_advance_fleet_desired_script described a ref advance nothing performed. That is specification-without-execution, which DESIGN §5 calls the deepest trap here.

Measured cost (2026-08-19/20)

symptom cause
srv1 served a revision from weeks earlier no merge ever reached it
every belt tick refused wholesale refs/fleet/desired pinned at a sha from the deleted green slot
every spawn refused at its first gate pinned Codex bundle absent from /opt/gunbc/provider-runtime/codex

Three symptoms, one cause: the host had no scheduled convergence at all. All three were repaired by hand — exactly the manual-application debt §6 counts against us.

Re-derived, not restored

Per the floor cut's own re-add rule. The deleted job was one stage inside a composed ci.yml that also carried gates, batching and a scheduler. This authors only the actuation that had no other home: one job, five steps, against the host it targets. The gates are not reinstated and remain their own re-adds.

Order is load-bearing

provider runtime  →  deploy  →  advance refs/fleet/desired

Provider runtime first, because a deployment whose provider binary is missing serves a dashboard that cannot dispatch — srv1's exact state. The ref advance last, because that ref is the belt's admission gate: advancing it before the deploy converges would announce a revision as desired while the host serving it is still mid-apply. A failed deploy leaves the ref where it was, and the belt keeps refusing against the last revision that actually landed.

Two further properties are deliberate and witnessed rather than incidental:

  • No pull_request trigger. A deploy is a host mutation; a proposed change is not an admitted one. The claim asserts the absence, because that's the dangerous direction to add by accident.
  • Concurrency queues, never cancels. Cancelling a half-applied deploy leaves the host between two revisions.

Two defects caught before merge — both mine

  • The compiler refused the first regen with a non-exhaustive match. Adding the artifact variant censused every match over GeneratedArtifact and found a second site (artifact_extra_valid) I'd missed. Addition is censused the same way deletion is.
  • The first emission carried contents: read while the last step pushes a ref. That would have 403'd after the deploy converged — leaving the host serving a revision the fleet does not admit, precisely the state being repaired. Now contents: write, witnessed against the emitted permission rather than the model field.

Neither would have been visible in a hand-written YAML.

Evidence

6/6 witnesses green by execution, complete run (footer present). Two regen passes, then a third after the permission fix. .gitattributes picked up the merge-driver row for the new path on its own — the registry wiring proving itself.

The order claim is asserted by splitting the emitted text at the apply step and interrogating the prefix, not by three independent containment checks: presence cannot see order, and order is the claim.

Not claimed

This does not close the remaining belt work. Nodes still need execution contracts, provider standing still needs recorded observations for production selection, and GITHUB_TOKEN is still absent from the belt unit. What this closes is the recurrence root — after it, a merge reaches srv1, the ref advances behind it, and the provider runtime is re-materialized every time.

🤖 Generated with Claude Code

gunbc-ci-auto-heal and others added 4 commits August 20, 2026 01:13
…the spawn cause

Two defects behind one production outage: srv1's daily workspace has been unable
to spawn or publish for weeks. Both were invisible for the same reason -- a
located cause existed and was discarded before anything durable recorded it.

PUBLICATION DIRECTORY. Filesystem.Write is a bare write with no parent creation
(v1_interpreter.rs, the has_content arm), and it emits its `[file] write <path>`
trace BEFORE attempting -- so a receipt written into a directory that was never
prepared fails ENOENT while the journal looks exactly like success. Measured on
srv1 2026-08-19: zero of twelve attempt directories had a publication/
subdirectory, every publish pass reported failed, and the journal carried a write
line for each one. Creating the directories by hand let receipts land
immediately, which is the confirmation that this is the cause and not a symptom.

The fix is in attempt_state_prepare_argv_for_instance, not on the write path and
not in the deploy. The write path cannot create it. The deploy cannot either, and
that distinction is the design point: instance_receipts_dir is bootstrappable at
apply time because there is exactly one per instance, whereas a publication
directory belongs to an ATTEMPT minted at runtime long after any apply. The
attempt's own prepare step is the earliest moment the path exists as a fact.
install -d takes a list, so this is the same single command with a second
operand -- the attempt directory and the one subdirectory it must contain are
created together, and publication_receipt_directory_is_named_once_note's
requirement that a reader be able to LIST the directory holds from attempt birth.

SPAWN CAUSE. BeltSpawnOutcome SpawnFailed carries node_id, step, detail and
provider. belt_spawn_pass_outcome_from discarded all four for the sentence "one
or more spawn outcomes failed", and the receipt is a tick's only durable record,
so every tick computed the cause and erased it. DESIGN section 5 requires a
failure arm to be typed, located and countable; past this fold it was none of
them.

What that cost is the reason this is not a cosmetic change. srv1's belt refused
every spawn from the moment the pinned Codex bundle went missing from
/opt/gunbc/provider-runtime/codex, and the dashboard could say only "belt fault".
Worse, the probes an operator reaches for answer about a different object than
the preflight tests -- it probes the pinned bundled executable by absolute path
and never consults PATH, so a healthy `which codex` actively pointed away from
the cause. The reason string is now built from the failed outcomes, naming each
node with its gate and detail.

EVIDENCE, by execution, complete runs (footer present):
  4/4 new belt claims, 2/2 new dispatch claims, and the extended lab claim green.
  DISCRIMINATING RED, measured rather than argued: reverting the prepare step to
  its single-directory form turns
  the_prepare_step_creates_the_publication_directory_with_the_attempt_directory
  and witness_lab_instance_moves_repo_tmux_and_codex_state_together RED, while
  the containment claim -- a pure projection over the path function, not the
  argv -- correctly stays green. Restored, all three green again.

The belt claims carry their own controls: an all-Spawned tick must reach
BeltPassRecorded with no reason (so a fold that concatenated every outcome
unconditionally fails), a mixed tick must name the failed node and NOT the
healthy one, and an observation refusal must still outrank the spawn summary so
the new reason-building cannot displace the pre-existing arm.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he fleet ref since the floor cut

The 2026-08-15 floor cut deleted ci.yml, and with it the only thing that ever
actuated a deployment or advanced the fleet's admitted revision. Both stayed
MODELED and neither ran -- gunbc.ci_spec gunbc_ci_deploy_srv1_stage described a
deploy nothing performed, and gunbc_ci_advance_fleet_desired_script described a
ref advance nothing performed. That is specification-without-execution, and
DESIGN section 5 names it as the deepest trap in the repository.

MEASURED COST, 2026-08-19/20. srv1 served a revision from weeks earlier because
no merge ever reached it. refs/fleet/desired stayed pinned at a sha from the
deleted green slot, so every belt tick refused wholesale on a revision-drift
check. The pinned Codex provider bundle was absent from
/opt/gunbc/provider-runtime/codex, so every spawn refused at its first gate.
Three symptoms, one cause: the host had no scheduled convergence at all. All
three were repaired by hand, which is exactly the manual-application debt
DESIGN section 6 counts against us.

RE-DERIVED, NOT RESTORED, per the floor cut's own re-add rule. The deleted job
was one stage inside a composed ci.yml that also carried gates, batching and a
scheduler. What is authored here is only the actuation that had no other home:
one job, five steps, against the host it targets. The gates are NOT reinstated
and remain their own re-adds.

ORDER IS LOAD-BEARING, and it is the order the outage taught:

  provider runtime -> deploy -> advance refs/fleet/desired

Provider runtime first, because a deployment whose provider binary is missing
serves a dashboard that cannot dispatch -- srv1's exact state for weeks. The ref
advance LAST, because that ref is the belt's admission gate: advancing it before
the deploy converges would announce a revision as desired while the host serving
it is still mid-apply, the fabricated plausible answer section 5 forbids. A
failed deploy leaves the ref where it was and the belt keeps refusing against the
last revision that actually landed.

Two properties are deliberate and witnessed rather than incidental. There is NO
pull_request trigger: a deploy is a host mutation and a proposed change is not an
admitted one. Concurrency is a queue with NO cancel-in-progress, because
cancelling a half-applied deploy leaves the host between two revisions.

TWO DEFECTS THE SUBSTRATE AND THE EMISSION CAUGHT, recorded because both were
mine and both would have shipped silently in a hand-written yaml:

  - the compiler refused the first regen with a non-exhaustive match: adding the
    artifact variant censused every match over GeneratedArtifact and found a
    second site (artifact_extra_valid) I had missed. Addition is censused the
    same way deletion is.
  - the first emission carried `contents: read` while the last step PUSHES a ref.
    That would have 403'd AFTER the deploy converged -- leaving the host serving
    a revision the fleet does not admit, which is precisely the state being
    repaired. Now `contents: write`, and witnessed against the emitted
    permission rather than the model field.

EVIDENCE: 6/6 witnesses green by execution, complete run. Two regen passes, then
a third after the permission fix; .gitattributes picked up the merge-driver row
for the new path on its own, which is the registry wiring proving itself.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ew 53944)

Review 53944 requested changes on two grounds. Verified both; the first finding
led to a worse violation the review did not name, and fixing that one resolves
the second without needing what the review proposed.

WHAT THE REVIEW FOUND: live_deploy_advance_fleet_desired_script builds shell by
concatenation, which DESIGN section 3 names as the medium-as-string blocker, and
it carried no scaffold marker binding it to a dissolution trigger. Both true.

WHAT IT MISSED, and it is the deeper defect: gunbc.ci_spec ALREADY OWNS this
script. gunbc_ci_advance_fleet_desired_script exists and does exactly this, so
the new module was a byte-for-byte re-derivation -- the section 3 fork, two
functions answering one question, guaranteed to drift. Drift here means two
different opinions about how the fleet's admitted revision advances, which is the
one fact every host reads before deciding whether to act.

WHY THE FIX IS REUSE RATHER THAN EITHER REMEDY THE REVIEW SUGGESTED. The existing
function is already ADMITTED DEBT:
gunbc_ci_advance_fleet_desired_shell_emit_scaffold declares it a Scaffold
dissolving to SingleAuthority, and its trigger names this exact concat-built
foreign-executor CAS advance. So marking my copy as a scaffold would have been an
author admitting their own debt, which section 6 explicitly forbids; and
hand-rolling a second bash rendering would have put a second spelling beside the
one that trigger already tracks. Consuming the authority removes the fork, needs
no new operator verdict because it introduces no new hand-shell, and lets one
existing trigger carry the conversion for both consumers.

VISIBLE CONSEQUENCE, and it is the right direction: the shared script prefixes
itself with its own dissolution description, so deploy.yml now carries the debt
marker inline. The generated artifact declares the scaffold it came from instead
of laundering it into anonymous yaml.

THE WITNESS WAS ALSO FORKED and is repaired with it. It asserted against the
re-derivation, so it would have stayed green while the two spellings drifted --
a claim over a fork proves the fork self-consistent and nothing about the
authority. It now reads gunbc.ci_spec directly, and a new claim joins the
emitted bytes to the authority's own dissolution marker, which only that module
can produce. A future local copy would have to re-declare debt it does not own
to satisfy it.

That join went RED on its first construction, correctly: a run: block is emitted
as a YAML block scalar, so every line is indented and the raw script is not a
substring of the document. Joining on the single marker line survives the
serializer's indentation. Recorded in the claim rather than silently corrected.

EVIDENCE: 7/7 green by execution, complete run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…2324870744)

The witnesses floor refused the module outright: five source annotations sat
INSIDE the Job declaration body, and DESIGN section 4c models annotations at
module-item grain only. The comment explaining why the job needs contents: write
was written beside the field it describes, which reads well and is unmodelable.

Moved above the function, with the reason recorded there. A comment is not
free-floating prose that can live wherever it reads best -- it attaches to a
declaration, and inside a body there is no declaration for it to attach to.

Emission is byte-identical (regen exit 0, no artifact change) and 7/7 witnesses
stay green by execution -- this moved prose, not behaviour.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Both findings in review 53944 verified and addressed in 5fd141a — and the first one led somewhere worse than the review stated.

On the medium-as-string finding: correct, and the repair is neither remedy suggested.

Checking it surfaced a defect the review did not name, and it is the deeper one: gunbc.ci_spec already owns this script. gunbc_ci_advance_fleet_desired_script exists and does exactly this, so my module was a byte-for-byte re-derivation — the §3 fork. Drift there means two different opinions about how the fleet's admitted revision advances, which is the one fact every host reads before deciding whether to act.

That reframes the fix. The existing function is already admitted debt: gunbc_ci_advance_fleet_desired_shell_emit_scaffold declares it a Scaffold dissolving to SingleAuthority, and its trigger names this exact concat-built foreign-executor CAS advance. So:

  • marking my copy as a scaffold would be an author admitting their own debt, which §6 explicitly forbids — this is the same objection your second finding raises, and it applies to that remedy too
  • hand-rolling a bash AST would put a second rendering beside the one that trigger already tracks, re-forking at the mechanism layer instead of the text layer

Consuming the authority removes the fork, introduces no new hand-shell so it needs no new operator verdict, and lets one existing trigger carry the conversion for both consumers when the SCM ref-update realization lands.

Visible consequence, and the right direction: the shared script prefixes itself with its own dissolution description, so deploy.yml now carries the debt marker inline. The generated artifact declares the scaffold it came from rather than laundering it into anonymous YAML.

On the operator-approval gate: agreed, and it dissolves. I am no longer introducing hand-shell, so there is nothing new for an operator to admit. You were right that the module header is not a substitute for an external verdict — that is exactly why I did not write myself one.

The witness was forked too, and I would have missed it without this. It asserted against my re-derivation, so it would have stayed green while the two spellings drifted — a claim over a fork proves the fork self-consistent and nothing about the authority. It now reads gunbc.ci_spec directly, plus a new claim joining the emitted bytes to the authority's own dissolution marker, which only that module can produce. A future local copy would have to re-declare debt it does not own to satisfy it.

That join went red on its first construction, correctly: a run: block emits as a YAML block scalar, so every line is indented and the raw script is not a substring of the document. Joining on the single marker line survives the serializer's indentation. Recorded in the claim rather than quietly corrected.

Separately, the floor caught five annotations sitting inside the Job body (run 32324870744) — §4c is module-item grain only. Moved; emission byte-identical, 7/7 green.

— sent from nimble-fox-671

@briansrls
briansrls merged commit 9600537 into main Aug 20, 2026
1 check passed
@briansrls
briansrls deleted the session/nimble-fox-671-deployemit branch August 20, 2026 03:55
@gunbai-bot

gunbai-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Moving this to draft. The operator challenged it with "we already have fleet converge — why are you adding this new deploy route?", and checking it properly, they are right and I was wrong in a way the approvals did not catch.

One of the three steps is a straight duplication of existing converge membership. converge_provider_runtime_codex_receipt / ProvisionCodexRuntime already exists in gunbc.fleet_converge_cli, and the commit that introduced it (1561d624b) is titled "Codex runtime bundle as a converged fleet member". My "Materialize the pinned provider runtime" step re-derives it. That is the §3 fork, and it is the exact violation I raised against my own earlier PR in review 53944.

What is NOT duplicated, stated precisely so the boundary is not overstated: fleet converge does not deploy the dashboard today. Its only import from gunbc.live_deploy.spec is the two fleet-converge unit NAMES, used for timer retirement — it never reads deployment_owned_steps, the systemd unit, the serve binary, the tree, or the tailscale mapping. On srv1 its receipt read applied=0 drifted=0 verdict=absent while the host sat weeks stale. So the deployment genuinely had no actuator; that part of the diagnosis stands.

But the right repair is one spine, not a second route. Deleting blue/green removed the obstacle: a deployment is now one in-place converge of one spec, which is exactly membership-reconciliation shaped. Slot selection and route flipping were what converges membership model could not express, and they are gone.

So the plan I would rather land:

  1. dashboard deployment becomes converge membership, rather than a parallel workflow
  2. the provider runtime step disappears entirely — converge already owns it
  3. the refs/fleet/desired advance keeps gunbc.ci_spec as its authority and finds a trigger that is not a second deploy route

What I am not willing to leave standing either way is the status quo this PR was reacting to: a deploy stage modeled since the floor cut with nothing invoking it, and me advancing the admitted fleet revision by hand — twice tonight, the second time because the deploy I had just run left the ref stale.

Draft until the operator picks the shape. The witnesses and the emission machinery here are reusable for the converge-member version.

— sent from nimble-fox-671

@gunbai-bot

gunbai-bot Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my previous comment, which over-called the finding. It claimed the provider-runtime step duplicates converge membership and is a §3 fork. Having traced it, that is wrong, and I would rather retract it than let a false defect claim stand on a merged PR.

Both routes emit the same ProvisionCodexRuntime {} HostEffect through the same realizer, gunbc.host_effect_realize realize_provision_codex_runtime. gunbc.host_codex_runtime_provision provision_codex_runtime_effect constructs that effect; gunbc.fleet_converge_cli converge_provider_runtime_codex_receipt constructs the same one. Two callers of one authority is what §2 means by one concept and many uses — a fork would be re-deriving the provisioning logic, and nothing here re-derives anything.

What is actually true, narrower and still worth acting on: there are two triggers for the same host state, not two models of it. The operators question — should deployment ride one convergence spine — is a real question about trigger topology, and it is not answered by anything in this diff.

The facts that bear on it, so the decision rests on measurements rather than on my framing:

  • fleet converge does not deploy the dashboard today. Its only import from gunbc.live_deploy.spec is the two fleet-converge unit names, used for timer retirement; it never reads deployment_owned_steps, the systemd unit, the serve binary, the tree, or the tailscale mapping.
  • converge is deliberately human-gated — plan, operator reviews typed hunks, apply — workflow_dispatch only, with a plan hash and lease generation, by operator directive (sleek-heron-218). A deploy-on-merge cannot route through a flow that requires plan review first.
  • converges own fleet-converge host timers were deliberately retired, and the converge plan actively removes them. So converge is invoked from CI, not from a cadence on the host.
  • on srv1 converges receipt read applied=0 drifted=0 verdict=absent while the host sat weeks stale.

So the deployment genuinely had no actuator, which is what this PR supplies. Whether it should instead become converge membership — now expressible, since deleting blue/green removed the slot-selection and route-flip semantics converge could not model — remains open and is the operators call, not something I should decide by leaving a second route in place or by ripping one out unilaterally.

Sorry for the noise on a merged PR. The prior comments conclusion was wrong; its underlying question was not.

— sent from nimble-fox-671

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant