Repository navigation
Conversation
…ns and attributable bindings for the five newer units, endpoints, procurement at seven The router table the operator pasted on 2026-09-05 holds seven Static, Active Spark reservations. Two were enrolled (srv5/srv6), two were named but not enrolled (srv7/srv8, SPARK-PAIR-1), and three were unknown to the model. This lands the network half of adoption for all five newer units: - gunbc.fleet_intent_network: operator_host_srv9/srv10/srv11 assigned in ascending address order (spark-0c75 .236, spark-3336 .237, spark-b66c .238), and host-lan endpoint rows for srv7..srv11 joined into the topology. - gunbc.spark.dgx_procurement: reservation rows (MAC, address, label) for the five, router bindings under an attributable OperatorAllocation dated 2026-09-05, the roster at seven, the intent at unit_count 7 with the batch history in its note. - gunbc.spark.cell_role: NO role for the five, recorded as a refusal rather than an omission: srv7/srv8 hold an undischarged llama.cpp RPC peer reservation the disjointness wall keeps out of the serving roster, and the serving desired state is still per-host Ollama, not the pair deployment the operator wants. Roles land with the pair-serving promotion and the fabric observation that decides pairing. Witnesses: dgx_spark 4/4 (also repairs the row still asserting the pre-August .222/.223 addresses against data that says .226/.225), cell-role retirement 4/4, fleet_intent_network typechecks with 0 blocking; the intent-network witness file resolves bare names only on the whole-population route and cannot execute by entry closure on main either, so its new srv7..srv11 row is evidence on CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…ter; the grant-install prelude selects the version for its target One SecretRef pinned to version 1 said nothing about which hosts it was for. Measured 2026-09-05: every one of the seven units authenticates and sudoes with the version-1 credential, so the roster maps all seven to version 1 today -- but the mapping is per host, its membership is an identity join with gunbc.spark.dgx_procurement's reserved identities (witnessed), and a host outside the roster refuses before any fetch (SparkAdministratorCredentialHostNotEnrolled). A future batch with its own password is one more version and one roster row. - gunbc.spark.credential_workflow: spark_administrator_credential_roster, spark_administrator_password_secret_ref_for(host), resolve_spark_administrator_credential(host, ..). - gunbc.spark.managed_grant_install: the administrator standing is per target host. - gunbc.spark.managed_access_apply: resolves for the host it applies to. - gunbc.ci_spec: the grant-install prelude emits one exact Secret Manager URL per rostered host, selected by SPARK_CONVERGE_TARGET, and refuses an unrostered target before any fetch; no shell variable is spliced into a modeled URL. fleet-converge.yml regenerated. Witnesses: credential workflow 4/4 (roster join, per-host version, pins), argv exposure 3/3, the managed_access_apply closure typechecks with 0 blocking. The grant-install witness file cannot produce a verdict by the entry-closure route on main either (AuthorizationRequest resolves differently there), so its per-host standing row is evidence on CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f70645242e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+1204
to
+1209
| join(map(spark_administrator_credential_roster, r => join([ | ||
| r.host as String, ") SM_URL=\"", | ||
| secret_ref_access_url(ref: spark_administrator_password_secret_ref_at_version(version: r.version)), | ||
| "\"; SM_RES=\"", | ||
| secret_ref_version_resource(ref: spark_administrator_password_secret_ref_at_version(version: r.version)), | ||
| "\" ;; ", |
There was a problem hiding this comment.
For spark_grants targeting srv7–srv11, these generated case arms can select and fetch the correct credential, but the operation can never reach the installer: .github/workflows/fleet-converge.yml:18-22 only offers srv5/srv6, and spark_converge_target_selection_of in dag/gunbc/fleet/fleet_multi_principal_probe.dag:328-337 rejects every value other than srv5/srv6. Thus five of the seven hosts added to this roster remain unable to run managed-grant reconciliation; extend both target-selection authorities to the same roster.
Useful? React with 👍 / 👎.
…he fleet's own identity, one target per run spark_managed_access_apply_wet is the operator-workstation shape: it mints a GCP token through gcloud and materializes the fleet key itself, neither of which a self-hosted runner has. The runner already holds both -- the fleet key in the in-run ssh-agent and the target host's administrator credential in the file the grant-install prelude writes -- so spark_managed_access_bootstrap_ci_wet is the same apply with its credentials taken from where the runner keeps them: fleet ssh context from prepare_fleet_ssh_agent_context, target from SPARK_CONVERGE_TARGET (single host, never fan-out), administrator credential through read_bootstrap_credential. It creates gunbc-automation, installs the fleet key, proves key-only authentication, and lays the narrow grants; the receipt is the same spark-apply receipt. Workflow: mode option spark_bootstrap, a mode-gated step sharing the grant-install prelude, CI target rostered (27 entries). Witness: the mode reuses the grant prelude, names its entry, and the emitted YAML gates on it. The apply closure typechecks with 0 blocking. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
… not spelled The literal pair refused the first bootstrap dispatch against srv7 with an HTTP 422 from GitHub: the workflow admitted two of seven enrolled units because the list was typed when there were two. The choice list is now the administrator credential roster, itself an identity join with procurement, so enrolling a Spark makes it dispatchable and nothing else can. fleet-converge.yml regenerated with srv5..srv11; the dispatch-input witness rows pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…ntial roster, not two spelled names The first spark_bootstrap run against srv7 (33948982086) materialized the right credential, prepared the fleet ssh context, and then refused at spark_converge_target_selection_of: the selector held its own srv5/srv6 literal, the same fact the workflow input had just stopped spelling. The target is now an identity match against spark_administrator_credential_roster_hosts(), the refusal names the roster, and the RED control names a host that is genuinely unrostered (srv1) instead of one that is now enrolled. Selector witness rows 4/4. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…one cloned key The spark_bootstrap run 33949801159 against srv7 refused at the transport: no ED25519 host key was enrolled for 192.168.1.232, so strict checking could not tell first contact from a changed key. The five new units (srv7..srv11) are scanned from the srv2 LAN position and enrolled as ScannedFirstContact rows, the same trust-on-first-use standing srv1..srv4 hold. The scan measured a fact the rows now carry rather than hide: srv7, srv8 and srv11 present one identical ED25519 host key, so they were imaged from a setup image that never regenerated its host keys. The key authenticates the image, not the unit, and the three are mutually indistinguishable by host key until bootstrap regenerates keys and the rows are re-scanned. srv9 and srv10 hold distinct keys; srv10 is enrolled for bootstrap only, being the RMA row in dgx_procurement. Anchor closure typechecks; enrolment witness 2/2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…es, not a second spelling spark_bootstrap run 33950942254 against srv7 converged every step (managed account, fleet key, nologin service identity, key-only authentication proven) and then failed at the step's final cat: ci_spec spelled the receipt as target/spark-apply-receipt.txt while managed_access_apply writes spark_apply_receipt_path. The CI invoke now carries that one authority, so the two cannot drift. fleet-converge.yml regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…ity instead of counting to two Three rows in v2.test.claim.spark_observation_scope asserted length(scope) == 2 against the production roster and went red the moment SPARK-BATCH-3 enrolled five more units, with the behaviour they describe unchanged. The scope is now checked as an identity join with dgx_spark_reserved_identities plus membership of srv5 and srv6; the serving-roster and training-cell rows assert coverage shape (no foreign, no duplicated) rather than a literal. 8/8. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…assword, not a key the unit has never seen srv9 (33951697853) and srv11 (33952611590) refused create-managed-account with "Permission denied (publickey,password)" while the same password opened a session from srv1 by hand. The legs offered the fleet key session and only piped the password to sudo -S; srv7 and srv8 converged because their earlier RPC-peer setup had already put the fleet key in the administrator's authorized_keys, so the password path had never executed. A first contact has nothing on the far side to offer a key to. The privileged legs now run over a password session: the same pinned known_hosts trust options, PreferredAuthentications=password,keyboard-interactive, BatchMode off so sshpass can answer the prompt, and the password in an owner-only file the existing materialized-secret bracket writes and shreds around the three legs. sshpass reads it with -f: measured on srv1 (sshpass 1.09), -d 0 over-reads the stdin pipe and the remote command receives nothing, -f passes every line through, so the sudo -S first line and the script still ride stdin. The key-only proof leg is unchanged. Witness 2/2 (password-only with the prompt enabled; no identity words, trust words kept); the apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
… and reads the lease's real fields CI's namespace-wave admission on #10517 refused one unadjudicated delta: managed_grant_install's requirement text still named the retired single secret ref after the roster made the reference per host, and the floor excluded the module. The same text also read bootstrap_principal, host and authorized_purpose straight off the AuthorizationRequest, which carries them under subject and purpose. The requirement now resolves the secret through the roster lookup, naming the host that has no enrolled secret when that is the case, and reads the lease's actual fields. Witness 25/25, closure typechecks, gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…ndering the whole workflow CI interrupted fleet_converge_spark_bootstrap_mode_reuses_the_grant_prelude_and_names_its_entry at 506ms against the 500ms per-claim CPU line, twice, with no verdict. The row's fourth conjunct rendered the entire fleet-converge workflow to search it for the mode condition; the condition is one data row, fleet_converge_spark_bootstrap_step_if, and the row now reads that. Same claim, bare-minimum cost. Passes locally. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
briansrls
pushed a commit
that referenced
this pull request
Sep 5, 2026
… and reads the lease's real fields CI's namespace-wave admission on #10517 refused one unadjudicated delta: managed_grant_install's requirement text still named the retired single secret ref after the roster made the reference per host, and the floor excluded the module. The same text also read bootstrap_principal, host and authorized_purpose straight off the AuthorizationRequest, which carries them under subject and purpose. The requirement now resolves the secret through the roster lookup, naming the host that has no enrolled secret when that is the case, and reads the lease's actual fields. Witness 25/25, closure typechecks, gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
briansrls
pushed a commit
that referenced
this pull request
Sep 5, 2026
… and reads the lease's real fields CI's namespace-wave admission on #10517 refused one unadjudicated delta: managed_grant_install's requirement text still named the retired single secret ref after the roster made the reference per host, and the floor excluded the module. The same text also read bootstrap_principal, host and authorized_purpose straight off the AuthorizationRequest, which carries them under subject and purpose. The requirement now resolves the secret through the roster lookup, naming the host that has no enrolled secret when that is the case, and reads the lease's actual fields. Witness 25/25, closure typechecks, gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
briansrls
pushed a commit
that referenced
this pull request
Sep 5, 2026
… and reads the lease's real fields CI's namespace-wave admission on #10517 refused one unadjudicated delta: managed_grant_install's requirement text still named the retired single secret ref after the roster made the reference per host, and the floor excluded the module. The same text also read bootstrap_principal, host and authorized_purpose straight off the AuthorizationRequest, which carries them under subject and purpose. The requirement now resolves the secret through the roster lookup, naming the host that has no enrolled secret when that is the case, and reads the lease's actual fields. Witness 25/25, closure typechecks, gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
briansrls
added a commit
that referenced
this pull request
Sep 6, 2026
…d the container bring-up rendered from it (slices 1-3 of the Ollama replacement) (#10540) * Enrol all seven DGX Sparks: identities srv9..srv11, router reservations and attributable bindings for the five newer units, endpoints, procurement at seven The router table the operator pasted on 2026-09-05 holds seven Static, Active Spark reservations. Two were enrolled (srv5/srv6), two were named but not enrolled (srv7/srv8, SPARK-PAIR-1), and three were unknown to the model. This lands the network half of adoption for all five newer units: - gunbc.fleet_intent_network: operator_host_srv9/srv10/srv11 assigned in ascending address order (spark-0c75 .236, spark-3336 .237, spark-b66c .238), and host-lan endpoint rows for srv7..srv11 joined into the topology. - gunbc.spark.dgx_procurement: reservation rows (MAC, address, label) for the five, router bindings under an attributable OperatorAllocation dated 2026-09-05, the roster at seven, the intent at unit_count 7 with the batch history in its note. - gunbc.spark.cell_role: NO role for the five, recorded as a refusal rather than an omission: srv7/srv8 hold an undischarged llama.cpp RPC peer reservation the disjointness wall keeps out of the serving roster, and the serving desired state is still per-host Ollama, not the pair deployment the operator wants. Roles land with the pair-serving promotion and the fabric observation that decides pairing. Witnesses: dgx_spark 4/4 (also repairs the row still asserting the pre-August .222/.223 addresses against data that says .226/.225), cell-role retirement 4/4, fleet_intent_network typechecks with 0 blocking; the intent-network witness file resolves bare names only on the whole-population route and cannot execute by entry closure on main either, so its new srv7..srv11 row is evidence on CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The Spark administrator credential is resolved per host through a roster; the grant-install prelude selects the version for its target One SecretRef pinned to version 1 said nothing about which hosts it was for. Measured 2026-09-05: every one of the seven units authenticates and sudoes with the version-1 credential, so the roster maps all seven to version 1 today -- but the mapping is per host, its membership is an identity join with gunbc.spark.dgx_procurement's reserved identities (witnessed), and a host outside the roster refuses before any fetch (SparkAdministratorCredentialHostNotEnrolled). A future batch with its own password is one more version and one roster row. - gunbc.spark.credential_workflow: spark_administrator_credential_roster, spark_administrator_password_secret_ref_for(host), resolve_spark_administrator_credential(host, ..). - gunbc.spark.managed_grant_install: the administrator standing is per target host. - gunbc.spark.managed_access_apply: resolves for the host it applies to. - gunbc.ci_spec: the grant-install prelude emits one exact Secret Manager URL per rostered host, selected by SPARK_CONVERGE_TARGET, and refuses an unrostered target before any fetch; no shell variable is spliced into a modeled URL. fleet-converge.yml regenerated. Witnesses: credential workflow 4/4 (roster join, per-host version, pins), argv exposure 3/3, the managed_access_apply closure typechecks with 0 blocking. The grant-install witness file cannot produce a verdict by the entry-closure route on main either (AuthorizationRequest resolves differently there), so its per-host standing row is evidence on CI. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * spark_bootstrap: the managed-access bootstrap runs from a runner as the fleet's own identity, one target per run spark_managed_access_apply_wet is the operator-workstation shape: it mints a GCP token through gcloud and materializes the fleet key itself, neither of which a self-hosted runner has. The runner already holds both -- the fleet key in the in-run ssh-agent and the target host's administrator credential in the file the grant-install prelude writes -- so spark_managed_access_bootstrap_ci_wet is the same apply with its credentials taken from where the runner keeps them: fleet ssh context from prepare_fleet_ssh_agent_context, target from SPARK_CONVERGE_TARGET (single host, never fan-out), administrator credential through read_bootstrap_credential. It creates gunbc-automation, installs the fleet key, proves key-only authentication, and lays the narrow grants; the receipt is the same spark-apply receipt. Workflow: mode option spark_bootstrap, a mode-gated step sharing the grant-install prelude, CI target rostered (27 entries). Witness: the mode reuses the grant prelude, names its entry, and the emitted YAML gates on it. The apply closure typechecks with 0 blocking. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The workflow's Spark target choices are the rostered Sparks, derived, not spelled The literal pair refused the first bootstrap dispatch against srv7 with an HTTP 422 from GitHub: the workflow admitted two of seven enrolled units because the list was typed when there were two. The choice list is now the administrator credential roster, itself an identity join with procurement, so enrolling a Spark makes it dispatchable and nothing else can. fleet-converge.yml regenerated with srv5..srv11; the dispatch-input witness rows pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The Spark converge target is resolved against the administrator credential roster, not two spelled names The first spark_bootstrap run against srv7 (33948982086) materialized the right credential, prepared the fleet ssh context, and then refused at spark_converge_target_selection_of: the selector held its own srv5/srv6 literal, the same fact the workflow input had just stopped spelling. The target is now an identity match against spark_administrator_credential_roster_hosts(), the refusal names the roster, and the RED control names a host that is genuinely unrostered (srv1) instead of one that is now enrolled. Selector witness rows 4/4. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Enrol the five new Spark host keys, recording that three units share one cloned key The spark_bootstrap run 33949801159 against srv7 refused at the transport: no ED25519 host key was enrolled for 192.168.1.232, so strict checking could not tell first contact from a changed key. The five new units (srv7..srv11) are scanned from the srv2 LAN position and enrolled as ScannedFirstContact rows, the same trust-on-first-use standing srv1..srv4 hold. The scan measured a fact the rows now carry rather than hide: srv7, srv8 and srv11 present one identical ED25519 host key, so they were imaged from a setup image that never regenerated its host keys. The key authenticates the image, not the unit, and the three are mutually indistinguishable by host key until bootstrap regenerates keys and the rows are re-scanned. srv9 and srv10 hold distinct keys; srv10 is enrolled for bootstrap only, being the RMA row in dgx_procurement. Anchor closure typechecks; enrolment witness 2/2. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The bootstrap step reads the receipt at the path the apply entry writes, not a second spelling spark_bootstrap run 33950942254 against srv7 converged every step (managed account, fleet key, nologin service identity, key-only authentication proven) and then failed at the step's final cat: ci_spec spelled the receipt as target/spark-apply-receipt.txt while managed_access_apply writes spark_apply_receipt_path. The CI invoke now carries that one authority, so the two cannot drift. fleet-converge.yml regenerated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The observation-scope witness joins the physical unit roster by identity instead of counting to two Three rows in v2.test.claim.spark_observation_scope asserted length(scope) == 2 against the production roster and went red the moment SPARK-BATCH-3 enrolled five more units, with the behaviour they describe unchanged. The scope is now checked as an identity join with dgx_spark_reserved_identities plus membership of srv5 and srv6; the serving-roster and training-cell rows assert coverage shape (no foreign, no duplicated) rather than a literal. 8/8. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The bootstrap's privileged legs authenticate with the administrator password, not a key the unit has never seen srv9 (33951697853) and srv11 (33952611590) refused create-managed-account with "Permission denied (publickey,password)" while the same password opened a session from srv1 by hand. The legs offered the fleet key session and only piped the password to sudo -S; srv7 and srv8 converged because their earlier RPC-peer setup had already put the fleet key in the administrator's authorized_keys, so the password path had never executed. A first contact has nothing on the far side to offer a key to. The privileged legs now run over a password session: the same pinned known_hosts trust options, PreferredAuthentications=password,keyboard-interactive, BatchMode off so sshpass can answer the prompt, and the password in an owner-only file the existing materialized-secret bracket writes and shreds around the three legs. sshpass reads it with -f: measured on srv1 (sshpass 1.09), -d 0 over-reads the stdin pipe and the remote command receives nothing, -f passes every line through, so the sudo -S first line and the script still ride stdin. The key-only proof leg is unchanged. Witness 2/2 (password-only with the prompt enabled; no identity words, trust words kept); the apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The grant-install credential requirement resolves its secret per host and reads the lease's real fields CI's namespace-wave admission on #10517 refused one unadjudicated delta: managed_grant_install's requirement text still named the retired single secret ref after the roster made the reference per host, and the floor excluded the module. The same text also read bootstrap_principal, host and authorized_purpose straight off the AuthorizationRequest, which carries them under subject and purpose. The requirement now resolves the secret through the roster lookup, naming the host that has no enrolled secret when that is the case, and reads the lease's actual fields. Witness 25/25, closure typechecks, gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Pair serving desired state: one vLLM TP=2 engine per fabric pair, resolved through the fabric and walled by the RPC reservation SPARK-BATCH-3: the serving unit is a pair, not a host. DeepSeek V4 Flash does not fit one Spark, so the desired state is one vLLM engine at tensor-parallel 2 across a fabric-paired couple. This is the root of the replacement for the per-host Ollama row in serving_desired, built in shadow: the Ollama row and the withholding rows keep standing until the one transition that flips the engine and discharges them, because a serving fleet is gap-intolerant. The engine tuple is the operator's observed-safe tuple (fp8_ds_mla KV, 82% utilization, 262144 context, Ray with its memory monitor disabled, admitted by the enforcer law with the kernel killer as the one enforcer). Each rank's address is resolved through fabric_link_desired, never spelled. max-num-seqs is sized by the measured KV pool, and until that measurement is a row the unit refuses rather than inventing a count. A pair holding an undischarged llama.cpp RPC peer reservation refuses. The promoted roster is srv5/srv6; srv7/srv8 wait on the release receipt, srv9/srv11 on their fabric pair, srv10 is the RMA unit. extdeps.vllm.engine_args gains the fp8_ds_mla KV arm read back on srv6. Witness 6/6; entry typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The enforcer kind's wire projection lives beside its type, and the pair module imports that CI's declarations phase refused #10540 with IMPORT-MEMBER-ABSENT on RayThresholdMonitor imported from gunbc.spark.serving_engine, although the coproduct declares it and the compiler resolved it; the index appears to drop the first variant of a single-line coproduct that sits under an annotation block, which is worth a failure-mode row once reproduced. Either way the pair module had no business enumerating another module's variants to spell them: memory_enforcer_kind_wire is now declared where the kind is, and the refusal text reads it. Witness 6/6; entry typechecks. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The bootstrap-mode witness reads the step condition row instead of rendering the whole workflow CI interrupted fleet_converge_spark_bootstrap_mode_reuses_the_grant_prelude_and_names_its_entry at 506ms against the 500ms per-claim CPU line, twice, with no verdict. The row's fourth conjunct rendered the entire fleet-converge workflow to search it for the mode condition; the condition is one data row, fleet_converge_spark_bootstrap_step_if, and the row now reads that. Same claim, bare-minimum cost. Passes locally. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The first pair as it runs: observed runtime, model, engine line, Ray head, pool and drift; the desired row reads the measurement gunbc.spark.pair_serving_observed records the first pair (head srv6 at 192.168.1.225, worker srv5) as read on 2026-09-05 from the hosts: vLLM 0.21.1rc1.dev339 inside docker container dsv4 from the local image dsv4-ray:local (a tag, not a reproducible identity, so no closure digest is claimed); the DeepSeek V4 Flash snapshot 60d8d707 on disk, 158G, mounted at /weights; the serve line field by field (deepseek-v4-flash:iq3s-split on :30000, TP=2 over Ray, 1048576 context, 32 sequences, fp8_ds_mla, 0.82, prefix caching, deepseek_v4 tool parser); the Ray head at the pair's rail-0 endpoint with a zero monitor refresh; the live collective environment; the units on both hosts including two leftovers of the abandoned earlier gunbc deployment; the stdout capture path; and the KV pool from the engine's own startup line, 4,632,182 tokens. Three drift rows name what the observation contradicted: TP_SOCKET_IFNAME differs between the launch script and the live engine, the head unit is inactive while the engine runs unsupervised from nohup, and the 2026-09-03 readback's 262144 context is a superseded configuration. pair_serving_desired now reads the measurement: max-num-seqs 32 with the pool as its basis, so the first pair's unit is Ready; the context is the launched 1048576; and the head is an observed row per pair rather than list order, which would have named srv5, the worker. Witness 8/8; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The first pair's container is a public base image plus a per-host commit, its env is baked at docker run, and the drift is fixed at its source Observed directly on both hosts on 2026-09-05. The engine's environment is the container's .Config.Env, baked at docker run on whatever machine first ran the image, and it carries three values foreign to these hosts: NODE_IP=192.168.50.58, MN_IF_NAME=enP7s7 and TP_SOCKET_IFNAME=enP7s7, where enP7s7 is the no-carrier RJ45 port on both units. The head script's exports reached only the Ray shell (and so the ranks, through the raylet), while the API server was launched by a second docker exec inheriting the container env. The head script on srv6 is now patched (backup kept) so the engine launch carries the same exports; the running engine keeps the old value until restart. dsv4-ray:local is a docker commit of the public base aidendle94/sparkrun-vllm-ds4-gb10:production-ready (b2eb4e6ee5cc), whose history is the whole build; the local layer is CMD sleep infinity plus 207M of JIT caches in the writable layer, and it differs per host (377dd4d0b0f9 on srv6, 0256d9df625d on srv5). So the runtime identity convergence carries is the base image plus a rendered container spec (host network and ipc, 10 GiB shm, unlimited memlock, all GPUs, /dev/infiniband, CAP_IPC_LOCK, unless-stopped, the hub bound read-only at /weights), never the local tag. Both recorded as drift. The two leftover gunbc units on srv5 are loaded with no unit file and sit in no wants directory. Witness 8/8; entry typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The pair's container bring-up is rendered from the desired row: pinned image, created env from the fabric, head and worker units Replaces the three hand-authored scripts observed on the first pair with a realization derived from gunbc.spark.pair_serving_desired. extdeps.docker.cli mints docker create/start/stop/exec/pull as typed argv (absolute program, so a unit's Exec line is the same words). The runtime is the public image aidendle94/sparkrun-vllm-ds4-gb10:production-ready pinned by registry digest (extdeps.docker.images.sparkrun_vllm_ds4_gb10), never the local dsv4-ray:local commits; the image row also carries the roster of interface-bearing environment names it bakes from its build host. The container is created with every one of those names overridden from the host's own rail endpoints (fabric_link_desired), plus VLLM_HOST_IP, a zero Ray monitor refresh and the long-context admission, so no docker exec can inherit the dead interface again: the 2026-09-05 drift is an identity join the witness holds, not a patch. The JIT cache is bound to a host directory. The head unit's ExecStartPre order releases the Ray session before the engine starts (the operator's launch precondition as an ordering fact), the engine line carries the whole tuple and never --max-num-batched-tokens, stdout is appended to a host log the pool line is read from, and Restart=on-failure is what waits for the worker. The worker holds its rank with ray start --block under Restart=always, which is the rejoin supervision the polling script did by hand. extdeps.systemd.unit_file gains ExecStartPre, ExecStop, StandardOutput and StandardError. Witness 6/6; realization, runner host deploy and Spark serving slice closures typecheck; gate clean. Install and apply of these units through the frozen-plan path come with the cut. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The first apply of the rendered pair: one privileged script per host, worker before head, over the password session gunbc.spark.pair_serving_apply projects the pair realization into one privileged script per host, made only of the realization's typed argv: log and cache directories, the pinned image pull, the JIT cache copied out of the retiring container, the container recreated from the rendered spec, the rendered unit written whole, the hand-authored unit and old container retired, the new unit enabled and read back. The worker is applied before the head so the head's first engine start finds its rank. The password session is shaped from the trust anchor alone (shape_fleet_ssh_password_exec_trusting), since an apply from an operator position holds no fleet key. Receipt under target/. This is the pair's first convergence, invoked by name; folding it into the frozen-plan path is the cut. Witness 2/2; entry typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The pair runtime is built on the pinned base plus ray[default]==2.58.0, because the base has no Ray The first apply of the rendered pair created the container from the public base image and the worker unit failed at its first ExecStartPre: /opt/env/bin/ray does not exist there. `import ray` fails in the base; the local dsv4-ray:local commits carry ray 2.58.0 pip-installed as a requested distribution, so the delta the pair depends on was a build, not caches. The image row now names the requirement and renders a two-line Dockerfile (FROM the digest, RUN python -m pip install), the apply writes it to /var/lib/gunbc/spark-pair/runtime and builds gunbc-spark-pair-runtime:ray-2.58.0 when absent, and the container spec names that tag. Second apply: srv5 and srv6 both active. Witness 6/6 and 2/2; entry typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The eighth Spark is enrolled and the switched fabric is observed from both ends: eight legs, two cages, two groups, a 50G ceiling with its cause spark-2196 (192.168.1.239, F8:3D:C6:6C:FE:98) appeared in the router table the day the fabric switch was cabled; it is srv12 by the same ascending-address rule, with its LAN endpoint, router binding, credential-roster row and scanned host key. Procurement now reads 8 units; the DGX witness counts arrivals against the bindings by identity rather than a literal, which also retires a stale "pending for both units" literal that could never have held over more than two rows. extdeps.mikrotik.crs812 cites the switch as RouterOS 7.20.8 presents it: two 400G QSFP56-DD cages exposed as eight 50G lanes each, a leg being a lane pair on the odd lane; the factory admin password location (printed on the underside label) recorded as a fact and never a value; the observed requirement that this passive breakout links only in forced mode; the REST paths convergence touches. gunbc.spark.fabric_switch_observed transcribes the two-sided LLDP join of 2026-09-05: each of the eight units on one odd lane, group A = cage 1 (srv5..srv8), group B = cage 2 (srv9..srv12), derived from the cage and never asserted. It records that the leg lands on the NIC's second-function ports and that both second-function netdevs come up from one leg, so only one may carry the rail address; and that the FS QDD-400G-4QPC015 legs are EEPROM-coded 100GBASE-CR4/25G (0x0B, 25.5 Gb/s), which is why the ConnectX-7 admits 50G_2X and refuses 100G_2X. srv10 links, so the RMA reading was wrong. Witnesses: dgx 35/35, credential 19/19, host key 2/2, pair desired 8/8, fabric 3/3; closures typecheck; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Rail addressing on the switched fabric is derived from the observed lane, rendered as netplan, and applied to all eight units gunbc.spark.fabric_rail_desired gives every fabric host one address in 192.168.110.0/24 as a function of the cage and lane it was observed on (.11-.14 on cage 1, .15-.18 on cage 2), jumbo MTU to match the switch bridge, and a NetworkManager-rendered netplan document that addresses the leg netdev and declares its twin unaddressed and optional, so exactly one of the two netdevs a leg raises carries the rail. gunbc.spark.fabric_rail_apply writes the document whole and applies it over the administrator password session, reusing the pair apply's privileged leg. Executed on 2026-09-05: all eight units read back their address on enp1s0f1np1. The fleet-converge workflow regenerates because the target roster now includes srv12. Witness 3/3; entry typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The pair resolves its ranks through the switched rail, and the units own the container's lifecycle With the direct cables out, the pair rows in fabric_link_desired describe no wire. Each rank now resolves through gunbc.spark.fabric_rail_desired over the host's observed lane: the rail address for Ray and VLLM_HOST_IP, the leg netdev for the socket-bound collectives, and the leg's RDMA device for NCCL and UCX so NCCL cannot pick the twin netdev's HCA on the same wire. The observed lane row gains that RDMA device. The unit is now the container's single authority: ExecStartPre pulls the pinned base, builds the runtime from the installed Dockerfile, removes whatever container stood before (tolerated when absent), creates one from the rendered spec and starts it, then the role's own Ray and engine steps follow. systemd runs these as root, so no executor grant covers docker. docker rm -f joins the argv mints. Witnesses: desired 8/8, realization 6/6, apply 2/2, fabric 3/3; apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * The serving unit is a fabric group at tensor-parallel N, and the llama.cpp reservation is discharged by a receipt The pair root widens to a group: membership is the four legs of one 400G cage (fabric_switch_observed), N is the member count, the head is an attributable row per group (srv6 for group A, srv9 for group B), and every rank resolves through the switched rail. The realization renders one head unit and N-1 worker units; the apply runs every worker before the head. Group A is the promoted roster; group B resolves and waits for A's launch line. The admission count inherits the first pair's measured 32 until a group's own pool is read, and the row says so. srv7 and srv8 held the undischarged llama.cpp RPC peer reservation, and the release the wall demanded is now a typed receipt: read on 2026-09-05 over the administrator session, neither host runs llama-server or rpc-server, only the hand-authored vLLM pair units supervise them (retrying a Ray session on the dead direct rail), nothing listens on 30000 or 50052, and each sits at 4 GB used. The wall witness flips to a regression control: every minted reservation carries a receipt, none stands undischarged, and no serving cell uses one. Witnesses: desired 5/5, realization 6/6, apply 2/2, RPC 21/21; apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Promote group B, plan every promoted group in one apply, and bounce a unit only when its text changed Group A's launch line was read back (11,767,856 KV tokens at tp=4), so the roster now names both groups. The wet apply took only the first realization and would have reported success for a desired state it never reached; it now plans every promoted group in roster order, refuses the whole apply on any refusal, and runs each group's workers before its head. The per-host script wrote the unit and restarted it unconditionally, so re-applying a converged roster paid a serving group's model load again. It now writes the rendered unit beside the live one, compares, and restarts only when the text changed or the unit is not active; the receipt line says which ("unchanged" / "restarted"). First run over both groups: srv5/srv8/srv7/srv6 unchanged and active, srv10/srv12/srv11/srv9 restarted and active. Witnesses: desired 6/6, apply 3/3, realization 6/6; apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Each group's capacity is grounded in its own launch line Both groups' KV pools are read back from their heads' captured stdout and carried as observed rows (group A 11,767,856 tokens, group B 11,952,124 tokens, each with the engine's own full-context concurrency figure and the read time). A promoted group's capacity now cites its own pool; only a group with no launch line falls back to the first pair's row, whose basis still says inherited. The admission count stays at the operator-observed 32 on both, and the row says why: no law from pool tokens to admitted sequences is modeled, and the pool's full-context concurrency bound is not that law. The rendered units are unchanged, so a re-apply reports every host unchanged and active. Witnesses: desired 7/7, apply 3/3, realization 6/6; apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Rebase onto main: the pair units log through the typed output stream, and the dispatch mode roster is an identity join Main typed StandardOutput/StandardError as a closed SystemdOutputStream while this branch had spelled them as open strings; the branch adopts the typed form and adds the one target it needs, `append:PATH` (systemd.exec(5)), so a restart continues the same log. ExecStop joins the directive set beside main's ExecStopPost. The converge mode options are the union of both sides' rows, and the workflow YAML is regenerated from the merged module. The dispatch-input witness pinned the mode option count at 10 while main already carried 16, a measurement copied from an older tree; it now joins the rendered options to fleet_converge_mode_options by identity and length, which is what the check was for. Witnesses: desired 7/7, realization 6/6, apply 3/3, systemd unit file 30/30, grain 3/3, overlap 7/7, guest image 6/6, dispatch input 8/8; apply closure typechecks; gate clean. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Rebase onto main: the converge mode roster is the union with the microvm modes, YAML regenerated Main added microvm_host_converge and microvm_boot_probe beside this branch's spark_bootstrap; the option list carries all of them and the workflow YAML is regenerated from the merged module. The dispatch-input witness joins by identity, so it holds without a count edit (8/8). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Contributor
Author
|
Superseded: every commit here landed via #10540 (spark-pair-serving), merged 2026-09-06. |
briansrls
added a commit
that referenced
this pull request
Sep 6, 2026
… the CI cost re-cut (#10631) * The harness roster names the pair heads through fleet endpoint rows: srv6 and srv9, not .232 The heads moved. .232 (srv7) was the first pair's front door and stayed in harness_engine_replica_addresses as a bare literal after Group B's head became srv9 (.236); nothing observed it until a dispatch on srv1 refused against it on 2026-09-05 while .225 and .236 both advertised the model. The roster now names both heads through gunbc.fleet_intent_network endpoint rows, and its annotation cites the authority it derives from once that module is on main: gunbc.spark.pair_serving_desired spark_pair_serving_head (spark-pair-serving, #10540). The srv9/srv10/srv11 identity slots and the srv7..srv11 LAN endpoint rows are ported from that branch verbatim so it merges onto them unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY * The harness roster is derived from the pair heads through the fleet topology; no address is authored #10540 put gunbc.spark.pair_serving_desired on main, so the roster stops naming endpoint rows and asks spark_pair_serving_head for each serving group, then gunbc.fleet_intent_network host_lan_address for that host's LAN endpoint. A head with no endpoint row makes the roster HarnessReplicasUnresolved, which the backend observer, dispatch selection, the belt preflight and every harness entry refuse on naming the host -- never a shorter list. The seed helpers over the authored list (replica_index / address / engine_backend_url) are deleted; the pick is harness_replica_pick over whatever roster is handed in. The LAN locality is still this module's choice and is annotated as the next move: the fabric owes a locus-to-locus resolution (dispatching host -> head front door), the relation fleet_locus_ssh_target already computes for ssh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY * Roadmap curation: thirty stale rows superseded with their successors named, five landed rows signed off, the CI cost re-cut The dispatch view showed 15 ready, 49 upcoming, 0 done: nothing had ever been signed off, and most of the ready set was either work merged this week or rows whose subject the 2026-08-15 floor cut deleted. Superseded (identity and id kept so edges still resolve; SupersededLine names what answers now): - ci-cost x15 (five-minute gate, warm admission, materialization, phased process, scoped worker, discovery snapshot, streaming, witness bundle, lens suite, arc ledger, ...) -> DESIGN Building & checks floor cut, re-cut as ci-required-run-cost against the live four-phase --required-ci run, measured on its own rows. - ci-placement x5 -> the 48-job ci.yml and two-host placement no longer exist. - ci-observation x2 (event grouping, collapse watchdog) -> the deleted job log; honest progress stays on observation-scoped-run-consumer. - roadmap-runtime x5 -> the harness replaced Codex (dispatch_harness_selection, operator ruling 2026-09-04); identity-join -> launch identity + roadmap event log; ready-override -> ManualReady + operator cause. - judgment x2 -> criterion reviewers + goal auditor on the fixed engine. - fleet-spark-host-enrollment -> #10512/#10517 rows and pair_serving_desired (#10540); serving stays open. Signed off with executed evidence (witness module/fn, run) and handback PRs: harness-thinking-and-write-briefs, roadmap-verify-to-publish-phase, roadmap-round-strip-pulse-follows-the-worker, roadmap-continuation-turn-label, roadmap-round-strip-percolator. The harness row's contract is re-pointed at the roster witness #10627 renamed. Not touched, deliberately: self-host (v1 delete half is parked by the v1 maintenance standing), namespace, compute, ci-control, toolchain, shell, source-intent. Those are the operator's call or are the work itself. The nine "closing-contract" ready rows are derived from nodes lacking a closing validation and follow. ROADMAP.md regenerated by generated_artifact_gate main_wet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #10512. See the commit message. All seven units measured to accept the version-1 credential on 2026-09-05, so the roster is uniform today; the mechanism is what changes: per-host resolution, roster membership as an identity join with procurement, unrostered targets refused before any fetch.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci