Repository navigation
Re-emit the deploy actuation: nothing has deployed srv1 or advanced the fleet ref since the floor cut - #8622
Conversation
…the spawn cause Two defects behind one production outage: srv1's daily workspace has been unable to spawn or publish for weeks. Both were invisible for the same reason -- a located cause existed and was discarded before anything durable recorded it. PUBLICATION DIRECTORY. Filesystem.Write is a bare write with no parent creation (v1_interpreter.rs, the has_content arm), and it emits its `[file] write <path>` trace BEFORE attempting -- so a receipt written into a directory that was never prepared fails ENOENT while the journal looks exactly like success. Measured on srv1 2026-08-19: zero of twelve attempt directories had a publication/ subdirectory, every publish pass reported failed, and the journal carried a write line for each one. Creating the directories by hand let receipts land immediately, which is the confirmation that this is the cause and not a symptom. The fix is in attempt_state_prepare_argv_for_instance, not on the write path and not in the deploy. The write path cannot create it. The deploy cannot either, and that distinction is the design point: instance_receipts_dir is bootstrappable at apply time because there is exactly one per instance, whereas a publication directory belongs to an ATTEMPT minted at runtime long after any apply. The attempt's own prepare step is the earliest moment the path exists as a fact. install -d takes a list, so this is the same single command with a second operand -- the attempt directory and the one subdirectory it must contain are created together, and publication_receipt_directory_is_named_once_note's requirement that a reader be able to LIST the directory holds from attempt birth. SPAWN CAUSE. BeltSpawnOutcome SpawnFailed carries node_id, step, detail and provider. belt_spawn_pass_outcome_from discarded all four for the sentence "one or more spawn outcomes failed", and the receipt is a tick's only durable record, so every tick computed the cause and erased it. DESIGN section 5 requires a failure arm to be typed, located and countable; past this fold it was none of them. What that cost is the reason this is not a cosmetic change. srv1's belt refused every spawn from the moment the pinned Codex bundle went missing from /opt/gunbc/provider-runtime/codex, and the dashboard could say only "belt fault". Worse, the probes an operator reaches for answer about a different object than the preflight tests -- it probes the pinned bundled executable by absolute path and never consults PATH, so a healthy `which codex` actively pointed away from the cause. The reason string is now built from the failed outcomes, naming each node with its gate and detail. EVIDENCE, by execution, complete runs (footer present): 4/4 new belt claims, 2/2 new dispatch claims, and the extended lab claim green. DISCRIMINATING RED, measured rather than argued: reverting the prepare step to its single-directory form turns the_prepare_step_creates_the_publication_directory_with_the_attempt_directory and witness_lab_instance_moves_repo_tmux_and_codex_state_together RED, while the containment claim -- a pure projection over the path function, not the argv -- correctly stays green. Restored, all three green again. The belt claims carry their own controls: an all-Spawned tick must reach BeltPassRecorded with no reason (so a fold that concatenated every outcome unconditionally fails), a mixed tick must name the failed node and NOT the healthy one, and an observation refusal must still outrank the spawn summary so the new reason-building cannot displace the pre-existing arm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…he fleet ref since the floor cut
The 2026-08-15 floor cut deleted ci.yml, and with it the only thing that ever
actuated a deployment or advanced the fleet's admitted revision. Both stayed
MODELED and neither ran -- gunbc.ci_spec gunbc_ci_deploy_srv1_stage described a
deploy nothing performed, and gunbc_ci_advance_fleet_desired_script described a
ref advance nothing performed. That is specification-without-execution, and
DESIGN section 5 names it as the deepest trap in the repository.
MEASURED COST, 2026-08-19/20. srv1 served a revision from weeks earlier because
no merge ever reached it. refs/fleet/desired stayed pinned at a sha from the
deleted green slot, so every belt tick refused wholesale on a revision-drift
check. The pinned Codex provider bundle was absent from
/opt/gunbc/provider-runtime/codex, so every spawn refused at its first gate.
Three symptoms, one cause: the host had no scheduled convergence at all. All
three were repaired by hand, which is exactly the manual-application debt
DESIGN section 6 counts against us.
RE-DERIVED, NOT RESTORED, per the floor cut's own re-add rule. The deleted job
was one stage inside a composed ci.yml that also carried gates, batching and a
scheduler. What is authored here is only the actuation that had no other home:
one job, five steps, against the host it targets. The gates are NOT reinstated
and remain their own re-adds.
ORDER IS LOAD-BEARING, and it is the order the outage taught:
provider runtime -> deploy -> advance refs/fleet/desired
Provider runtime first, because a deployment whose provider binary is missing
serves a dashboard that cannot dispatch -- srv1's exact state for weeks. The ref
advance LAST, because that ref is the belt's admission gate: advancing it before
the deploy converges would announce a revision as desired while the host serving
it is still mid-apply, the fabricated plausible answer section 5 forbids. A
failed deploy leaves the ref where it was and the belt keeps refusing against the
last revision that actually landed.
Two properties are deliberate and witnessed rather than incidental. There is NO
pull_request trigger: a deploy is a host mutation and a proposed change is not an
admitted one. Concurrency is a queue with NO cancel-in-progress, because
cancelling a half-applied deploy leaves the host between two revisions.
TWO DEFECTS THE SUBSTRATE AND THE EMISSION CAUGHT, recorded because both were
mine and both would have shipped silently in a hand-written yaml:
- the compiler refused the first regen with a non-exhaustive match: adding the
artifact variant censused every match over GeneratedArtifact and found a
second site (artifact_extra_valid) I had missed. Addition is censused the
same way deletion is.
- the first emission carried `contents: read` while the last step PUSHES a ref.
That would have 403'd AFTER the deploy converged -- leaving the host serving
a revision the fleet does not admit, which is precisely the state being
repaired. Now `contents: write`, and witnessed against the emitted
permission rather than the model field.
EVIDENCE: 6/6 witnesses green by execution, complete run. Two regen passes, then
a third after the permission fix; .gitattributes picked up the merge-driver row
for the new path on its own, which is the registry wiring proving itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ew 53944) Review 53944 requested changes on two grounds. Verified both; the first finding led to a worse violation the review did not name, and fixing that one resolves the second without needing what the review proposed. WHAT THE REVIEW FOUND: live_deploy_advance_fleet_desired_script builds shell by concatenation, which DESIGN section 3 names as the medium-as-string blocker, and it carried no scaffold marker binding it to a dissolution trigger. Both true. WHAT IT MISSED, and it is the deeper defect: gunbc.ci_spec ALREADY OWNS this script. gunbc_ci_advance_fleet_desired_script exists and does exactly this, so the new module was a byte-for-byte re-derivation -- the section 3 fork, two functions answering one question, guaranteed to drift. Drift here means two different opinions about how the fleet's admitted revision advances, which is the one fact every host reads before deciding whether to act. WHY THE FIX IS REUSE RATHER THAN EITHER REMEDY THE REVIEW SUGGESTED. The existing function is already ADMITTED DEBT: gunbc_ci_advance_fleet_desired_shell_emit_scaffold declares it a Scaffold dissolving to SingleAuthority, and its trigger names this exact concat-built foreign-executor CAS advance. So marking my copy as a scaffold would have been an author admitting their own debt, which section 6 explicitly forbids; and hand-rolling a second bash rendering would have put a second spelling beside the one that trigger already tracks. Consuming the authority removes the fork, needs no new operator verdict because it introduces no new hand-shell, and lets one existing trigger carry the conversion for both consumers. VISIBLE CONSEQUENCE, and it is the right direction: the shared script prefixes itself with its own dissolution description, so deploy.yml now carries the debt marker inline. The generated artifact declares the scaffold it came from instead of laundering it into anonymous yaml. THE WITNESS WAS ALSO FORKED and is repaired with it. It asserted against the re-derivation, so it would have stayed green while the two spellings drifted -- a claim over a fork proves the fork self-consistent and nothing about the authority. It now reads gunbc.ci_spec directly, and a new claim joins the emitted bytes to the authority's own dissolution marker, which only that module can produce. A future local copy would have to re-declare debt it does not own to satisfy it. That join went RED on its first construction, correctly: a run: block is emitted as a YAML block scalar, so every line is indented and the raw script is not a substring of the document. Joining on the single marker line survives the serializer's indentation. Recorded in the claim rather than silently corrected. EVIDENCE: 7/7 green by execution, complete run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…2324870744) The witnesses floor refused the module outright: five source annotations sat INSIDE the Job declaration body, and DESIGN section 4c models annotations at module-item grain only. The comment explaining why the job needs contents: write was written beside the field it describes, which reads well and is unmodelable. Moved above the function, with the reason recorded there. A comment is not free-floating prose that can live wherever it reads best -- it attaches to a declaration, and inside a body there is no declaration for it to attach to. Emission is byte-identical (regen exit 0, no artifact change) and 7/7 witnesses stay green by execution -- this moved prose, not behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Both findings in review 53944 verified and addressed in On the medium-as-string finding: correct, and the repair is neither remedy suggested. Checking it surfaced a defect the review did not name, and it is the deeper one: That reframes the fix. The existing function is already admitted debt:
Consuming the authority removes the fork, introduces no new hand-shell so it needs no new operator verdict, and lets one existing trigger carry the conversion for both consumers when the SCM ref-update realization lands. Visible consequence, and the right direction: the shared script prefixes itself with its own dissolution description, so On the operator-approval gate: agreed, and it dissolves. I am no longer introducing hand-shell, so there is nothing new for an operator to admit. You were right that the module header is not a substitute for an external verdict — that is exactly why I did not write myself one. The witness was forked too, and I would have missed it without this. It asserted against my re-derivation, so it would have stayed green while the two spellings drifted — a claim over a fork proves the fork self-consistent and nothing about the authority. It now reads That join went red on its first construction, correctly: a Separately, the floor caught five annotations sitting inside the — sent from nimble-fox-671 |
|
Moving this to draft. The operator challenged it with "we already have fleet converge — why are you adding this new deploy route?", and checking it properly, they are right and I was wrong in a way the approvals did not catch. One of the three steps is a straight duplication of existing converge membership. What is NOT duplicated, stated precisely so the boundary is not overstated: fleet converge does not deploy the dashboard today. Its only import from But the right repair is one spine, not a second route. Deleting blue/green removed the obstacle: a deployment is now one in-place converge of one spec, which is exactly membership-reconciliation shaped. Slot selection and route flipping were what converges membership model could not express, and they are gone. So the plan I would rather land:
What I am not willing to leave standing either way is the status quo this PR was reacting to: a deploy stage modeled since the floor cut with nothing invoking it, and me advancing the admitted fleet revision by hand — twice tonight, the second time because the deploy I had just run left the ref stale. Draft until the operator picks the shape. The witnesses and the emission machinery here are reusable for the converge-member version. — sent from nimble-fox-671 |
|
Correction to my previous comment, which over-called the finding. It claimed the provider-runtime step duplicates converge membership and is a §3 fork. Having traced it, that is wrong, and I would rather retract it than let a false defect claim stand on a merged PR. Both routes emit the same What is actually true, narrower and still worth acting on: there are two triggers for the same host state, not two models of it. The operators question — should deployment ride one convergence spine — is a real question about trigger topology, and it is not answered by anything in this diff. The facts that bear on it, so the decision rests on measurements rather than on my framing:
So the deployment genuinely had no actuator, which is what this PR supplies. Whether it should instead become converge membership — now expressible, since deleting blue/green removed the slot-selection and route-flip semantics converge could not model — remains open and is the operators call, not something I should decide by leaving a second route in place or by ripping one out unilaterally. Sorry for the noise on a merged PR. The prior comments conclusion was wrong; its underlying question was not. — sent from nimble-fox-671 |
The 2026-08-15 floor cut deleted
ci.yml, and with it the only thing that ever actuated a deployment or advanced the fleet's admitted revision. Both stayed modeled and neither ran —gunbc.ci_specgunbc_ci_deploy_srv1_stagedescribed a deploy nothing performed, andgunbc_ci_advance_fleet_desired_scriptdescribed a ref advance nothing performed. That is specification-without-execution, which DESIGN §5 calls the deepest trap here.Measured cost (2026-08-19/20)
refs/fleet/desiredpinned at a sha from the deleted green slot/opt/gunbc/provider-runtime/codexThree symptoms, one cause: the host had no scheduled convergence at all. All three were repaired by hand — exactly the manual-application debt §6 counts against us.
Re-derived, not restored
Per the floor cut's own re-add rule. The deleted job was one stage inside a composed
ci.ymlthat also carried gates, batching and a scheduler. This authors only the actuation that had no other home: one job, five steps, against the host it targets. The gates are not reinstated and remain their own re-adds.Order is load-bearing
Provider runtime first, because a deployment whose provider binary is missing serves a dashboard that cannot dispatch — srv1's exact state. The ref advance last, because that ref is the belt's admission gate: advancing it before the deploy converges would announce a revision as desired while the host serving it is still mid-apply. A failed deploy leaves the ref where it was, and the belt keeps refusing against the last revision that actually landed.
Two further properties are deliberate and witnessed rather than incidental:
pull_requesttrigger. A deploy is a host mutation; a proposed change is not an admitted one. The claim asserts the absence, because that's the dangerous direction to add by accident.Two defects caught before merge — both mine
GeneratedArtifactand found a second site (artifact_extra_valid) I'd missed. Addition is censused the same way deletion is.contents: readwhile the last step pushes a ref. That would have 403'd after the deploy converged — leaving the host serving a revision the fleet does not admit, precisely the state being repaired. Nowcontents: write, witnessed against the emitted permission rather than the model field.Neither would have been visible in a hand-written YAML.
Evidence
6/6 witnesses green by execution, complete run (footer present). Two regen passes, then a third after the permission fix.
.gitattributespicked up the merge-driver row for the new path on its own — the registry wiring proving itself.The order claim is asserted by splitting the emitted text at the apply step and interrogating the prefix, not by three independent containment checks: presence cannot see order, and order is the claim.
Not claimed
This does not close the remaining belt work. Nodes still need execution contracts, provider standing still needs recorded observations for production selection, and
GITHUB_TOKENis still absent from the belt unit. What this closes is the recurrence root — after it, a merge reaches srv1, the ref advances behind it, and the provider runtime is re-materialized every time.🤖 Generated with Claude Code