Skip to content

Gap analysis: hand-edited Spark state with no modeled authority - #9991

Merged
briansrls merged 7 commits into
mainfrom
session/eager-pike-541-fleet-gap
Sep 3, 2026
Merged

briansrls merged 7 commits into
mainfrom
session/eager-pike-541-fleet-gap

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

The operator asked for an accumulation of what has been hand-edited on the live Sparks while the fleet is assembled by hand.

The reason it is worth a document rather than a plan line is one finding: a convergence run today would degrade the fleet without failing. The modeled desired unit differs from the live unit on BOTH serving hosts in three directions at once, all pointing the same way:

  • context 131072 modeled against 1048576 live -- below the 400k floor gunbc.model.choice exists to defend
  • served model gpt-oss modeled against hf.co/antirez/deepseek-v4-gguf live
  • OLLAMA_NUM_PARALLEL appears nowhere in the corpus, so convergence does not overwrite the slot count, it DELETES it and the runner falls back to a default

Nothing on that path refuses. The unit rewrites, the service restarts, it serves -- narrower and serialized.

Everything was read off the live hosts and compared against the module named as its authority. Desired side: gunbc.spark.serving_unit_render spark_serving_desired_user_unit_text. Live side: the unit file and /api/ps per host.

Also rostered: the second pair (spark-c2b1, spark-ac79) is serving while unknown to cell_role, the fleet slot roster, and network_identity_subsumption; residency is unmodeled on both sides; credentials were hand-carried past gunbc.spark.credential_workflow .

One row turns back on my own #9897: it keys realization identity on argv, and every runner here is bare ollama serve with its configuration in Environment= lines -- so on this fleet that identity discriminates almost nothing. Widening it needs a ruling on which variables are configuration and which are ambient.

Doc only -- no code, no substrate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01U397y4s3dSBof7vGPAX87G

gunbc-ci-auto-heal and others added 4 commits September 2, 2026 02:32
…ebuild

Every divergence read off the live hosts against the module named as its
authority, not recalled. The load-bearing finding is that convergence is armed
and wrong in three directions at once: modeled context is 131072 against a live
1048576, the served model is gpt-oss against a live DeepSeek-V4, and
OLLAMA_NUM_PARALLEL appears NOWHERE in the corpus -- so convergence does not
overwrite the slot count, it deletes it. None of those refuse. The unit rewrites,
the service restarts, and it serves, narrower and serialized, with every number
the serving decision is reasoned from silently invalidated.

Also rostered: the second Spark pair is provisioned and serving while unknown to
cell_role, the fleet slot roster and network identity subsumption; residency is
unmodeled on both sides so a 94 GB model evicts and reloads unmeasured; and
credentials were hand-carried past a modeled SecretRef path that already exists.

One finding turns back on my own PR: the realization carrier keys a runner on
argv, and every runner here is bare `ollama serve` with its whole configuration
in Environment= lines. On this fleet that identity discriminates almost nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U397y4s3dSBof7vGPAX87G
The context and slot-count divergences this document led with were false. Both
were derived by running the renderer from a session branch based on 373b8d1
while main had advanced: main desires 1048576 context, carries the slot count as
a typed PositiveSlotCount of 4 through extdeps.ollama.server_env, renders it, and
witnesses the rendering. The freeze recommendation that followed from them is
withdrawn.

The error is its own class and the document now says so: a real instrument was
run, a real number came back, and it was read as a fact about the fleet when it
was a fact about my branch. A divergence report is only as current as the tree it
was measured from.

What survives: the served model (desired gpt-oss, live DeepSeek-V4), residency
unmodeled on both sides, the second pair outside every authority, and
hand-carried credentials past a modeled SecretRef.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U397y4s3dSBof7vGPAX87G
@gunbai-bot

gunbai-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

CI on this branch is red for a reason that is not in this diff, and I want that on the record rather than silently retried.

This PR is one new markdown file, +145 lines, nothing else (git diff --stat origin/main...HEAD shows a single path under docs/plans/). It touches no .dag, no Rust, and no generated artifact.

The failing lane is required-witnesses-build, and its failure is:

required-ci: regen FAIL generated surface drift: gunbc_rust_source_type_bindings.rs

That same failure is present on main itself at d61f8c9bc, which is the commit this branch merged. Run 33584275111 on main fails with the identical line. The drifting artifact is untouched by this branch — git diff origin/main...HEAD -- gunbc_rust_source_type_bindings.rs is empty.

So this is inherited, not caused. I am deliberately not regenerating the artifact on a docs-only branch: that would put a generated-surface change in a PR whose entire subject is a planning document, and the regeneration belongs wherever the drift was introduced. Main has since advanced past d61f8c9bc (currently 3138c4617); once a green main is available I will merge it and let the lane re-run.

If the drift turns out to persist on current main, that is a main-side repair and I will raise it separately rather than smuggle it through here.

— sent from eager-pike-541

Copy link
Copy Markdown
Contributor

Operational supersession: #10001 is now the controlling immediate path to safe Spark fleet convergence.

Before this document can govern operations, its stale PR body and repeated paragraph must be corrected: context and OLLAMA_NUM_PARALLEL are not current divergences; the live blockers are the exact desired serving realization, safe treatment of an active stale process, and the current srv6 role statement. The operator has now withdrawn the dedicated-training assignment for srv6 for this phase.

Do not broaden #9991 into implementation. Link the corrected gap record to #10001, where completion requires one exact-host wet plan→apply→full readback→zero replan→terminal health transaction plus a second zero-effect noop.

gunbc-ci-auto-heal and others added 2 commits September 2, 2026 21:19
…rect the second pair

Two divergences this document reported have moved, and leaving either as written would
make the document assert a state the tree no longer has.

SERVED MODEL IS CLOSED. serving_desired names the DeepSeek manifest through
gunbc.ollama_model_resolution, admission requires an exact runtime version and a
manifest-identity match rather than a name match, and the stale `Description=` spelling is
now derived from the model ref instead of authored beside it. What did not close --
reproducing the local artifact from desired state -- is a declared rung drop rather than
silence, and the section says so rather than reporting the whole axis as done.

THE SECOND PAIR IS NOT TWO SERVING HOSTS. The section described two independent ollama
hosts; .232 is a llama-server front door whose weights sit on .233 behind ggml-rpc-server
over the RoCE fabric. That is one serving unit split across two machines, and the
correction matters because admitting them as two serving cells would plan an 86.7 GB load
onto a peer with ~16 GiB free and swap disabled -- an OOM kill that takes the front door
with it. The membership gap is unchanged; what membership would have to MEAN changed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01U397y4s3dSBof7vGPAX87G
@briansrls
briansrls merged commit 47ae825 into main Sep 3, 2026
10 of 12 checks passed
@briansrls
briansrls deleted the session/eager-pike-541-fleet-gap branch September 3, 2026 01:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant