Skip to content

Derive group B's serving route from the fabric assignment, not the retired GLM canary - #11617

Merged
briansrls merged 3 commits into
mainfrom
session/bright-eagle-728-fabric-serving-route
Sep 19, 2026
Merged

briansrls merged 3 commits into
mainfrom
session/bright-eagle-728-fabric-serving-route

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

What broke

A real dispatch on production (2026-09-18, shell-typed-invocation) spawned a worker that exited with no serving offer was admissible. For group B, the harness probed 192.168.1.236:30001, where nothing listens. That's the port of the retired exl3 GLM canary. The arm that actually serves group B answers on 192.168.1.236:30000 as glm-5.3-flash (checked live with GET /v1/models, snapshot eb9eb208).

The cause: group B's facts named an arm, not the fabric

Group A's subject is PairServingUnitOn { group }, which is anchored on the fabric. Group B's was PlacedServingArm { arm: glm_canary_arm }, anchored on one specific engine's declaration. Everything built on it inherited that snapshot, and none of it moved when the arm changed:

Consumer Stale fact
gunbc.serving.serving_enrollment route head, port 30001, ceiling (the canary's operator bound of 1)
gunbc.serving.admitted_model alias glm-5.3-flash-exl3-canary
gunbc.spark.serving_arm_placement claimed srv9 only. srv10–12, which were serving GLM, read as free
gunbc.spark.serving_subject reservation the operator's 2026-09-12 ruling …_group_b_is_glm reserved group B for the canary's subject
gunbc.spark.serving_deployment_selection the GLM fleet snapshot admitted no hosts to the running arm
gunbc.spark.host_commitment admit_unplaced_host the build/image seam admitted srv10–12 for builds while they served a TP4 GLM rank

The change

  • New gunbc.spark.fabric_serving_assignment: one row per group (glm_group_b_serving_assignment) holding the planner's inputs minus launch intent. fabric_serving_head takes the rank-zero node from the same arm_node_roster the launch is planned from. Ranks other than zero run --headless, so rank zero is the only front door. A roster refusal becomes a typed refusal, never a guessed head. The head is not the plan's master, which is the RoCE rail address.
  • Group B's holder is the assignment row. serving_arm_placement glm_group_b_serving_arm points at glm_group_b_serving_assignment, and the claims, subject, reservation, admissibility and selection snapshot all key on it. Changing which engine serves group B is now one edit inside the assignment row, and nothing else moves.
  • Enrollment: B's head is derived, its port is the arm profile's serve_port, and its ceiling is an attributed allocation of 4. It is deliberately not the engine's --max-num-seqs 16: the engine's scheduler limit and how many turns this repository offers are separate questions.
  • Admitted model: B's alias comes from the arm profile, glm-5.3-flash.
  • Relaunch CLI reads the same row, so the launch and the route can't drift apart.
  • Placement claims all four group-B hosts from group membership (fabric_group_hosts, which is total). They aren't taken from the launch roster, because a roster refusal would yield no claims and let occupied hosts read as free.
  • Build seam: admit_unplaced_host_among takes the unplaced set as a supplied value, and admit_unplaced_host is its unchanged production wrapper. Refusal arms run the real path. The admitting arm stays authorable as a positive control even though no production unit is unplaced any more.

Operational consequence to know about

With group B's arm claiming its hosts, every Spark unit is now placed: group A serves DeepSeek and group B serves GLM. So the v4.1 image-build lane has no admissible host until one is freed or a non-serving build host is provided. Before this change it would have silently built onto a host serving GLM.

Witnesses

Several witnesses were certifying the stale state: the :30001 endpoint, a ceiling of 1, the canary alias, "srv10–12 unplaced / reserved but idle", "srv10 admitted for a build", and srv10 as a ruling-only specimen. Each now asserts the true state and keeps its discriminating force. The annotations say what each used to assert and why that was wrong. The rung-drop population entry for harness_seat_ceiling_by_operator_policy names the new group-B row, and docs/design-rung-drops.md is regenerated.

Not in this PR

  • Group A's seat pool is still unreadable (cat-file 09ed6c96… refused). The fabric event log links events through a JSON parent field that git never fetches. That's P0-B and gets its own PR.
  • A declared arm has no stop receipt. Nothing notices when the process behind a claim goes away, which is how the canary lingered. serving_arm_placement's own annotation names that trigger. This PR removes the stale snapshot but doesn't build the liveness check.
  • The retired canary's rows (glm_canary_arm, glm_canary_host_claim) stay because its evidence rows still cite them. Nothing live reads them.

Verification

CI green at aa21ca2: witnesses, build, heal and floor all SUCCESS. Locally: the routing, admission, host-commitment, v4.1 build, selection and chat-shape witnesses returned true behind a control that must fail.

🤖 Generated with Claude Code

gunbc-ci-auto-heal and others added 2 commits September 18, 2026 16:04
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…sitive arm authorable

Group B's assigned arm now claims all four member hosts, so admit_unplaced_host -- the one
seam the source build, the image build and the published-image probe reach a host through --
refuses srv9-srv12. Two witnesses had certified the opposite: srv10 admitted for a build,
and srv10 admissible for the v4.1 image build. srv10 was running a rank of the native GLM
arm the whole time, with no placement claim, so a build would have competed with a serving
rank for the unit's unified memory.

admit_unplaced_host_among takes the unplaced set as a supplied value; admit_unplaced_host is
the unchanged production wrapper over it (spark_host_is_unplaced was already `any` over
spark_unplaced_hosts). Every refusal arm still runs the real path; the admitting arm is kept
as a positive control over a supplied set, because no production Spark unit is unplaced any
more and a seam that refused everything would otherwise read as tested.

authority_held_and_ruling_derived_causes_are_separable_by_a_fold: srv10 was the ruling-only
specimen only because the arm serving there had no claim. It now carries one cause of each
kind like srv9, and the fold still discriminates (a non-separating fold counts two of each).

Consequence to note: with every Spark unit placed, the v4.1 build lane has no admissible
host until one is freed or it is given a non-serving build host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot gunbai-bot Bot changed the title roadmap direction Derive group B's serving route from the fabric assignment, not the retired GLM canary Sep 18, 2026
@gunbai-bot
gunbai-bot Bot marked this pull request as ready for review September 18, 2026 19:06
… canary citations

Addresses review 67909 (REQUEST_CHANGES), both findings verified against the code.

1. spark_build_host_reachability_wet gated on a fold starting at true over
   map(spark_unplaced_hosts, ...). This PR makes that population empty in production
   (every Spark unit is placed), so the root would have written a receipt with no probe
   lines and exited zero -- the widening its own annotation records removing. The verdict
   is now spark_reachability_verdict, three arms: NoAdmissibleHost refuses by name,
   AllAnswered admits, SomeUnanswered refuses with the body. The earlier diff deleted the
   only assertion that would have caught this; the_reachability_verdict_refuses_an_empty_
   population_rather_than_passing pins all three arms against supplied readings.

2. gunbc.harness.harness_seat still said group B's tolerant ceiling derives from
   gunbc.spark.glm_canary_converge's arm, and cited serving_canary_route_ceiling, which
   this PR deletes. Both now name gunbc.serving.serving_enrollment's declared rows and
   ServingRouteTurnCeiling's CeilingFromRouteLaunch arm; a historical correction about the
   canary's bound is put in the past tense rather than left asserting current state.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor Author

Both findings in review 67909 were correct, and both are fixed in c5d7779.

1. The empty population exited zero. This was a real regression that this PR introduced. spark_build_host_reachability_wet gated on fold(readings, init: true, …), and this PR makes spark_unplaced_hosts empty in production, so the root would have written a receipt with no probe lines and returned ExitSuccess. The verdict is now spark_reachability_verdict, with three arms: SparkReachabilityNoAdmissibleHost refuses by name, …AllAnswered admits, and …SomeUnanswered refuses with the body. You were also right that I had deleted the only assertion that could catch it. the_reachability_verdict_refuses_an_empty_population_rather_than_passing now pins all three arms against supplied readings.

2. harness_seat still cited the canary. Both sentences now name gunbc.serving.serving_enrollment's declared rows and ServingRouteTurnCeiling's CeilingFromRouteLaunch arm. The nearby historical correction about the canary's bound is now in the past tense, so it no longer states something current. A repo-wide search for serving_canary_route_ceiling, glm_canary_serving_subject and glm_canary_fleet_snapshot now finds only this PR's own annotations recording the replacement.

CI's floor will verify the changed witnesses on this head. I couldn't finish the local run because the session container kept restarting partway through.

— sent from bright-eagle-728

@briansrls
briansrls added this pull request to the merge queue Sep 19, 2026
Merged via the queue into main with commit f370c25 Sep 19, 2026
4 checks passed
@briansrls
briansrls deleted the session/bright-eagle-728-fabric-serving-route branch September 19, 2026 01:05
gunbai-bot Bot pushed a commit that referenced this pull request Sep 19, 2026
…per host, a production backfill entry with receipts, and a nonempty retained population

Against the ninth side-chat hold and reviews 68027/68062:
- Liveness: a host-effect claim fences until released; the stated term marks it
  overdue, never free. A second claim on a held host is refused at the fold, at the
  claim and in the standings (OccupiedByHostEffect). Hosts of no claimed group claim
  on their own host-effect/<host> partition with the same events and fold.
- Backfill: fabric_event_log_event_refs_backfill_wet on the placed host over every
  refs/fabric/* partition through the envelope decoder, with a receipt ref per
  partition; the reader's refusal consults the receipt.
- Population: an empty retained host list refuses at decode, at the transition and in
  the fold.
- Merged origin/main (#11617's admit_unplaced_host_among: admit_unplaced_host_in is
  now that over a standings population); the stale spark_pair_apply_plan symbol
  reference is repointed (review 68062).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant