Model DSpark as its own authority, free srv6 from training, and make the fabric desired state

[gunbai-bot]

Three changes with one objective: make fleet convergence safe to run on the Sparks. The operator asked for this "ASAP, this or next PR", and separately for the correct DSpark to be modeled in extdeps and anchored in convergence.

1. DSpark is DeepSeek's, not NVIDIA's — a §3 meaning fork found by the operator

"dspark" was used across two sessions and an operator conversation to mean NVIDIA's dgx-spark-playbooks (the DGX Spark hardware recipes, connect-two-sparks and its vLLM multi-node section). The operator meant DSpark, DeepSeek's speculative-decoding algorithm. One spelling, two independently governed upstreams, nothing in common. A fabric, a Ray topology and a privileged-container decision were all pursued under the wrong referent before anyone asked.

extdeps.deepseek.deepspec is the DeepSeek authority. Every fact in it was verified against deepseek-ai/DeepSpec and the Hub rather than recalled: the three algorithms DeepSpec implements (DSpark, DFlash, Eagle3) with their papers as further citations, and the released (algorithm × target) checkpoint grid.

Its safety content is that a draft is bound to one target. Speculative decoding is output-equivalent by design — the draft proposes, the target verifies, rejected tokens are discarded. So a draft paired with the wrong target does not produce wrong text; it produces the same text, more slowly, while every acceptance-rate and throughput number measured from it describes a configuration nobody intended. A performance claim that silently detaches from its subject. The binding carries its target, and there is no constructor for an unbound draft.

The attachment shape is a second fact, and modelling it as one was a defect. A draft ships either as a separate artifact (the three *-DSpark-support.gguf files in the antirez distribution) or fused into the checkpoint: deepseek-ai/DeepSeek-V4-Flash-DSpark is 48 shards and 166.9 GB against the plain repository's 46 and 159.6 GB — same architecture, same fp8/e4m3/ue8m0 quantization, plus num_nextn_predict_layers: 1 and dspark_block_size: 5. DeepSeek's own model card: "not a new model. It is the same checkpoint with an additional speculative decoding module attached." In the fused shape there is no download that yields the base alone, so a consumer sizing from base_total plans for a residency no disk will ever hold, and 159.6 GB names a repository deliberately not in use.

Also grounded here, because it was being carried as an open project risk: vLLM does implement DSpark, via --speculative-config '{"method":"dspark",...}'. What remains unverified is whether a specific engine build carries it — an observation about that build, not a fact about this upstream.

2. srv6 is no longer reserved for training

Operator decision, recorded with its basis: "i wouldn't dedicate a whole node for any task - just have it converge and serve whatever task we converge it to." Role is converged desired state, not a dedication. Training remains a modeled capability in gunbc.spark.training_ready; it stops being a node reservation. Consequence in spark_serving_role_scoped_desired_members: srv6 receives full serving desired members rather than baseline-only.

Three witness claims transcribed the current roster (length(serving) == 1 && srv5 serves && srv6 trains) and one said in its own comment that it goes red the moment srv6 reverts. That turned an operator decision into a failing test. They are rewritten as the invariants they were reaching for — disjointness, and the desired-hosts/serving-cells bijection — which mention no host and survive any roster change. DESIGN §5: completeness is an identity join, not a count equality.

spark_cell_role_in and spark_serving_role_scoped_desired_members_in now take the assignment roster as a parameter, so the training arm stays exercisable regardless of which machines currently hold which role. The production-root retirement claims that needed a live training cell to have a subject are removed under a declared §4b(3) rung drop, with previous rung, temporary rung, reason, bounded population, and a restoration trigger that names the capability (roster threading through the plan artifact) rather than "put a training cell back" — which would re-create the very coupling the drop records.

3. The fabric is desired state, because runtime configuration was lost

The 200 Gb/s RoCE addressing on srv5/srv6 was applied at runtime by hand, verified working at MTU 9000 with a clean 8972-byte jumbo ping, reported by me as configured — and destroyed entirely by one power cycle. I reported a runtime observation as durable state.

The mechanism: these hosts run netplan with renderer: NetworkManager, and an nmcli write is intercepted by the netplan-NM integration into /run/NetworkManager/system-connections/, which is tmpfs. The durable artifact is the generated /etc/netplan/90-NM-<uuid>.yaml, and only that. So a convergence that actuates nmcli converges a host into a state that does not survive its next boot, while reporting success.

gunbc.spark.fabric_link_desired owns the netplan facts, read back off both hosts rather than authored from intent. A rail whose two ends name different interfaces has no constructor: the crossed configuration existed, was applied by hand, passed a jumbo ping, and was still wrong, because NCCL selects a rail by matching interface to subnet. That is a failure that tests green, which is why it is a constructor condition rather than a diagnostic.

extdeps.network.ipv4 gains PrefixLength, render_ipv4_cidr and ipv4_same_network — the CIDR spelling every router table and netplan file writes, and a same-network predicate that refuses for a prefix it cannot answer exactly rather than assuming a match.

srv7/srv8 are named, not enrolled. That module's own note makes the distinction structural: membership in endpoints is enrollment. Adoption is five separable acts and this is the second alone.

Evidence

All executed, all green:

• w_all_spark_fabric_claims_hold — new witness. Positive control plus five refusal arms (same host, crossed interfaces, unrelated subnets, identical addresses, MTU mismatch), the subnet-boundary discriminator, and the unanswerable-prefix refusal.
• w_the_planner_scopes_desired_state_by_role, w_the_production_roster_reaches_the_serving_arm, w_spark_roles_are_disjoint_over_the_assignment, w_serving_desired_hosts_are_the_serving_cells, w_serving_cell_is_not_offerable_to_the_training_fabric
• deepspec_released_dspark_bindings returns all four bindings with the correct attachment shapes and block sizes (7 for the Qwen grid, 5 for V4-Flash — block size is a property of the trained draft, not of the algorithm, which is why it is a field).

What this does not do

It does not yet repoint gunbc.spark.serving_desired off gpt-oss:120b, and it does not yet consume #9897's memory-fit function in serving_converge_plan. Those are the remaining two blockers before the wet slice can run, and they land next — the aggregate-memory fit across a two-host assembly is what the operator's "~160 GB across two Sparks, rest for slots" actually needs, and it depends on #9897 landing first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U397y4s3dSBof7vGPAX87G

[briansrls]

Operator-priority ruling: safe fleet-converge E2E is this PR or its immediate successor

The operator’s controlling objective is now explicit: stop hand-editing Sparks and prove a safe fleet-converge E2E as soon as possible — in this PR if the composition remains reviewable, otherwise in the immediately following PR with no intervening Spark feature work.

This PR correctly removes the premature permanent training dedication from srv6 and moves the fabric toward durable desired state. It does not yet satisfy the operational terminal: its own body says serving_desired is still on gpt-oss:120b and the #9897 fit/evidence join is not yet consumed by convergence. Therefore one of two paths is admissible:

1. Extend Model DSpark as its own authority, free srv6 from training, and make the fabric desired state #10009 only if the resulting exact head can still be reviewed as one coherent convergence transaction and can carry the wet receipt below; or
2. Land Model DSpark as its own authority, free srv6 from training, and make the fabric desired state #10009 on its accepted subject, then make SAFE-SPARK-CONVERGE-0: execute one exact-host plan → apply → full readback → terminal-health transaction #10001 SAFE-SPARK-CONVERGE-0 the immediate next new Spark-serving PR. No dspark expansion, P0a/P1/P2/P3 work, second-pair adoption, generic scheduler, or unrelated Spark modeling may interpose.

Required first terminal, on exactly one restored srv5 or srv6:

• exact modeled target; all-host execute remains impossible;
• exact desired artifact/runtime/configuration already present, with positive fit evidence; no hidden cold pull and no moving tag;
• observe → immutable typed plan/hash → prestate revalidation → one apply → whole-population readback → zero-write replan → terminal generation/identity/residency;
• second identical converge proves noop, zero effects, zero commands, and no restart;
• every refused/non-converged arm exits nonzero while preserving its receipt;
• stale/unstamped/unobservable active invocation refuses before effects until an exact quiescence/drain receipt exists.

Compose rather than rebuild: #9897 supplies exact evidence identity; #9984 supplies typed reactivation vocabulary; #9986 supplies running-definition/invocation observation; the existing spark_serving_converge_slice_wet already owns exact-host selection, frozen typed-row application, post-apply replan, and terminal health.

Operator decision carried: srv6 is not reserved for training. For this phase it may serve whatever desired task convergence assigns. Longer-term task movement belongs to a later allocation authority, not to a permanent per-node dedication.

Tracking/terminal: #10001.

[gunbai-bot]

CI investigated. Neither failing check is attributable to this PR, and I am not pushing a speculative fix over them. Evidence below so it can be checked rather than taken.

rust-unit-tests is red on main itself. Run 33596615712, head bb96afa61ba — test result: FAILED. 642 passed; 2 failed, the two being:

compiler_tests::compiler_tests::render_rust_applied_type_routes_qualified_base_through_leaf_name ... FAILED
compiler_tests::compiler_tests::shell_service_unmodeled_output_key_refuses ... FAILED

This branch changes no Rust whatsoever — git diff origin/main...HEAD -- src/ is empty. The diff is .dag modules plus the two generated-artifact projections, both written by the actuator (main_wet_one) rather than by hand. So the failure predates the branch and travels with the merge of main.

This branch's own rust-unit-tests job failed differently, and in a way that is not a test failure at all. Job 100152632335: started 06:47:07Z, completed 06:47:09Z — two seconds, with zero recorded steps. The job never ran; nothing was compiled and nothing was executed. That is a runner-level abort.

Same class on the earlier head. The build lane on 0fb42ed refused with:

spawn rustfmt: No such file or directory (os error 2) -- program …/cargo/bin/rustfmt was resolved at admission from PATH …, and ran there: --version was executed successfully before this phase began. So it has been removed, replaced or made unusable while this run was executing; this is not a missing or broken installation

The diagnostic distinguishes the two cases itself and says which one this is: the toolchain was disturbed mid-run. And on #9897 the floor refused with unexpected_failures=0 — the refusal was five INTERRUPTED-BEFORE-VERDICT rows that exceeded a 500ms CPU budget, all in modules that PR does not touch (self_host_compile_phase_frontier, v2.test.emit, v2.test.execution, lens_testgen), on a runner reporting pgmajfault=317120739 and pswpin=575989180.

So there are two independent things, and neither is this change: a genuine pre-existing red on main's rust-unit-tests, and self-hosted runners being disturbed mid-run (vanishing rustfmt, jobs aborting in two seconds, budget preemption under heavy swap).

What I verified locally instead, since the executing evidence is what matters:

• w_all_spark_fabric_claims_hold — green on the merged head 322affc6d50.
• w_the_planner_scopes_desired_state_by_role, w_the_production_roster_reaches_the_serving_arm, w_spark_roles_are_disjoint_over_the_assignment, w_serving_desired_hosts_are_the_serving_cells, w_serving_cell_is_not_offerable_to_the_training_fabric — all green.
• claim_executor --required-regen — std_measure.rs does not drift from this branch; the four files that do (compiler_tests.rs, gunbc_rust_source_type_bindings.rs, lib.rs, std_realization_schedule.rs) drift identically from main, and one of them adds pub mod std_interval; for a file this tree does not carry, so taking the actuator's output there would break the seed build. Left alone deliberately.
• docs/design-ledgers.md and DESIGN.md regenerated by main_wet_one after the merge, not hand-resolved. The gunbc.rung_drop conflict was resolved keeping both sides — that carrier has a documented merge hazard (merge_region_excludes_shared_tail in the ledger), and this PR's row is deliberately one line, which is the repair that ledger row prescribes.

Merge readiness is therefore blocked on infrastructure and on a main regression, not on this diff. I am not merging over that and not papering it with a re-push.

— sent from eager-pike-541

[briansrls]

Engine-pivot ruling — exact runtime scope, and no change to the first safe-converge terminal

1. Standalone llama.cpp + ggml-rpc-server is a new ServingRuntimeIdentity. Ollama’s embedded llama.cpp lineage may support an explicit lower-level InferenceKernelLineage relation, but it does not make the serving runtimes equal. The executable closure, scheduler, memory ownership, API process, residency behavior, RPC workers, transport and configuration all change. No Ollama fit/latency/prefill/residency evidence transfers implicitly to the RPC realization.

2. Do not anchor the global claim “vLLM cannot run DeepSeek-V4 on GB10.” Scope any refusal to the exact image/build/configuration whose bytes were inspected. The public vLLM source at the build revision advertised by the current recipe contains an mHC TileLang fallback selected when is_deep_gemm_supported() is false; its CUDA classifier nevertheless treats SM120-family devices as DeepGEMM-capable by default, while DeepGEMM’s hyperconnection dispatcher accepts only arch-major 9 or 10. That supports an exact default-path incompatibility and explains the failure, but it also means the public source contradicts “VLLM_USE_DEEP_GEMM=0 cannot affect mHC.” Before encoding a construction refusal, bind the exact image digest, in-container source digest, effective environment and selected backend. Operationally abandoning this exact image now is still sound; the model must not generalize the finding past its subject.

3. Compatibility is a constructor wall on the concrete deployment candidate, not on ExecutionAssemblyIdentity. The assembly identity should continue to identify the exact host/topology population; the same assembly may be usable by another runtime. A ServingDeploymentCandidate exists only after joining exact ArtifactRuntimeCompatibility × RuntimeHostKernelCompatibility × ParallelStrategyFeasibility × TransportFeasibility. A failure produces a typed construction refusal such as KernelDispatchUnsupported; it never reaches ranking as a losing candidate.

Head/expert divisibility is likewise strategy-specific. It may be a constructor condition for an exact tensor-parallel plan whose implementation requires even partitioning, but it is not a universal assembly law and must not be applied to llama.cpp RPC. RPC exposes remote devices and distributes model tensors/KV by its own rules; a three-host RPC candidate must be judged by that implementation, not by TP=3 arithmetic.

4. The immediate safe fleet-converge E2E is unchanged. It remains one exact srv5 or srv6 under Ollama, using the already-present exact no-h3-stop realization with positive same-host/configuration fit evidence. The llama.cpp RPC runtime, multi-host assembly, endpoint security and split configuration are P0a-and-later work and must not interpose before the SAFE-SPARK-CONVERGE-0: execute one exact-host plan → apply → full readback → terminal-health transaction #10001 wet receipt. This PR must not repoint convergence to the RPC candidate.

5. RPC later still has a network/security prerequisite. RoCE/RDMA is not required when plain TCP is selected, but exact private reachability is. The upstream RPC backend is explicitly proof-of-concept and insecure on open/sensitive networks. Pin bind addresses, endpoint population, firewall/isolation and GGML_RPC_NO_RDMA=1 if TCP-only is the intended configuration.

The 155 GiB safetensor copies are therefore ArtifactPresentButNoAdmittedRuntimeOnThisHost for the current exact vLLM materialization—not proof that the artifact family or vLLM project is universally unusable.

[briansrls]

Queue-stall boundary: off-tree RPC probing is allowed, but it does not interpose

The existing “no intervening Spark work” ruling constrains PR/authority order. During the shared-runner availability refusal, a quarantined llama.cpp RPC experiment may run without changing this PR or the immediate #10001 terminal.

It must remain off-tree and non-authoritative: no desired-state or role change, no PR, no evidence transport from Ollama, no claim of P0a/P0b completion, and no use of the result to delay the exact single-host Ollama convergence receipt. The probe must use exact runtime/artifact/host/configuration identities in its raw receipt, private-link RPC exposure, attempt-owned transient processes, and positive teardown/re-observation of the managed Ollama state. Any need to stop/rewrite/restart the managed unit or any host instability ends the probe rather than broadening #10009.

Publication order remains: #10009 on its accepted subject, then #10001 as the immediate Spark-serving successor unless the full wet terminal is absorbed here.

[briansrls]

New live divergence from the exploratory RPC run: adopt here or revert before #10001

The RPC probe created durable fabric state on srv5/srv6 after this PR's current rows were authored:

rail 102:
  enp1s0f0np0
  192.168.102.10/24 <-> 192.168.102.11/24
  MTU 9000

rail 103:
  enP2p1s0f0np0
  192.168.103.10/24 <-> 192.168.103.11/24
  MTU 9000

The files are /etc/netplan/90-NM-*.yaml and survive reboot. Current fabric_link_desired.dag instead models the separately observed srv7/srv8 rails on 192.168.100.0/24 and 192.168.101.0/24; it does not own this new srv5/srv6 state.

If the operator elects to keep the new links, this PR is the correct and only permitted absorption point before SAFE-SPARK-CONVERGE-0: add the exact srv5/srv6 links through the existing FabricLink constructor, preserve both pairs as distinct desired links, derive the host endpoint projections, and add discriminating roster/readback witnesses. Do not overwrite the srv7/srv8 rows or infer the new pair from the old one.

If the operator declines adoption, revert the persistent profiles and positively observe absence. In either arm, #10001 must begin from a fresh post-disposition observation of both hosts. Undeclared durable state may not be treated as harmless merely because the addresses improve an exploratory runtime.

This is not authorization to add llama.cpp RPC runtime/deployment authority here. The RPC measurements remain quarantined; #10009's subject is the durable fabric and role state, while the first production terminal remains the exact single-host Ollama converge.

[briansrls]

Sequencing correction: do not reopen this green head solely for rails created after it

The post-head RPC run created durable srv5/srv6 netplan state on rail 102 and rail 103. My earlier instruction named #10009 as the absorption point because this PR establishes the fabric-link authority. The load-bearing requirement is ownership before #10001, not that these later-created rows appear in this exact PR head.

Therefore, if the operator accepts the durable links, #10009 may land on its current accepted subject without being moved solely to add them. Put the exact srv5/srv6 links into the one immediate mandatory convergence-preparation successor alongside the already-required B1/B2 work. That cut must extend the same gunbc.spark.fabric_link_desired authority, wire the new endpoints into observation/plan/readback rather than merely declaring rows, and prove exact agreement with the durable /etc/netplan/90-NM-*.yaml files before #10001 runs.

If the operator declines the links, the alternative is explicit reversion plus observed absence. Until the operator selects one arm, make no further network mutation.

This amendment does not admit standalone llama.cpp RPC authority, transport its exploratory evidence, or move the first terminal. Publication remains:

#10009 current accepted subject
→ B1/B2 + exact live-rail disposition in one immediate prerequisite cut
→ fresh whole-host prestate
→ #10001 single-host Ollama safe-converge E2E

Separate correction: SHA-256 37bcd82c… named an Ollama blob that the exploratory RPC run did not load. The run used a distinct four-shard path, so its executed artifact identity remains unestablished and none of its performance numbers earns candidate standing.

[briansrls]

Landing caveat: rail sequencing does not waive current-main composition

The comment above removes the new rails as a reason to reopen this exact head. It is not merge authorization for a stale composition.

#10009@2e9a4bc56d4 records base 583ffd661c0, while current main has advanced to 2f59c2c6641. Main changed overlapping authority/generated surfaces after that base, including dag/gunbc/rung_drop.dag, DESIGN.md, and docs/design-ledgers.md—all files this PR changes. The earlier rule still applies: compose with then-current main, regenerate through the actuator rather than hand-resolving, and rerun the exact resulting head before landing. GitHub CLEAN is not a substitute for the repository merge driver or exact generated-surface convergence.

Consequently, if the operator selects Adopt before that mandatory recomposition, it is economical and admissible to add the exact srv5/srv6 rails in the same moved head rather than create another predecessor. If the operator has not adjudicated the rails when recomposition occurs, recompose #10009 without them and place the adjudicated disposition in the one immediate B1/B2 prerequisite cut. The semantic ruling remains: the rows must be authoritative before #10001, but no particular pre-existing commit must contain them.