Repository navigation
M3: capacity is readiness, and restart is a separate decision behind its own preconditions - #10698
Merged
Merged
Conversation
…its own preconditions THE FLOOR IS ONE DECLARATION. "54 GiB of KV" and "three concurrent sessions" are the same requirement said twice, and once both are written down they drift — a change to the sequence budget moves one and leaves the other standing. So only the operational unit is declared, MinimumColdSessions, and the blocks are derived from it against the engine's own layout. The session count is not a second number either: it reads harness_seat_operator_policy_seats, because a fabric whose floor said four while the harness seated three would hold capacity nobody could use. COLD means no prefix-cache credit. A prospective hit belongs to a prefix another request may release at any moment; reserving against it spends a discount that was never granted. "BELOW FLOOR THEREFORE RESTART" IS ONE STEP WHERE THERE ARE TWO DECISIONS, and collapsing them is how a convergence run bounces a busy engine. Free memory on a unified-memory host is a moving target, so a restart taken because the pool came up small can come up smaller, and a loop that restarts on every unfavourable reading keeps restarting with requests in flight. The first move is therefore always withdrawal: stop sending new work. It costs nothing, is instantly reversible, and is correct under every failing standing including the ones where capacity is merely unknown — an unestablished capacity withdraws rather than assuming adequacy, which is today's real state for every engine in the fleet. Restart is then admitted, not triggered, behind five preconditions, and the fold names the FIRST unmet one. Four are decidable from what this repository models. The fifth refuses: no admitted unified-memory plan exists, and on a GB10 the pool is a residual of a supply shared with the OS page cache, the weights, the CUDA and NCCL contexts and this deployment's own Ray object store — restarting without a plan over it is what produced 23.6 GiB once and 54.01 GiB later from one unit text. Admitting a restart there would promise an outcome nothing modelled can predict. That is the M4/M6 slice; when it lands the arm is replaced by a real precondition rather than deleted. Re-admission is earned by a FRESH receipt. A completed restart proves a process started; on this hardware it proves nothing about the pool it came up with. Witnesses (7). The discriminating one: a busy engine below floor is withdrawn but not restarted, and with every decidable precondition met the restart still refuses naming the missing memory plan. Halving the declared budget halves the derived requirement, which a second literal would not do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
… was already taken gunbc.spark.vllm_rank_population has carried VllmEngineIncarnation since before this stack, with the SAME reasoning from the SAME incident -- three launches on one pair in one day, identical version and checkpoint, resolved capacity differing by 22%. M0 re-invented that concept under its own name, which is the section 3 nickname this repository forbids: one meaning, two names, and the derived work duplicated behind each. The merged type keeps the name. The two are different GRAINS of the same question and neither observer can produce the other's fields: VllmEngineIncarnation is what the ENGINE reports about itself (the engine-core identity, readable from its own output), and this type is what the HOST knows about the launch (systemd invocation, container, image and unit digests, start instant). So this one is renamed VllmServingLaunch, which is what it actually holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
… witness The group layout lost its detachable window and its unread page_size_bytes, and a startup reading now observes the revision it was captured from. This fixture follows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
Contributor
Author
|
Exact-head review — HOLD at
This head also inherits #10693's incorrect group-cost law and #10695's ungrounded printed-capacity pair. Finally, merging current |
Contributor
Author
|
…y is, and today gets "unestablished"
Every capacity claim in this stack has been green by typecheck. This one executes:
gunbc run --entry dag/gunbc/instruments/fabric_capacity_standing.dag \
--function fabric_capacity_standing
Run against the fleet just now, group A answered with its real launch — invocation
d140fa39c1174403bf2ce4d7ed8c2d74, inventory 54,475 blocks of 16, printed pool 11,767,856 tokens at
11.22x, 0 running, 0 waiting — and the receipt refused: the resolved cache groups are unobserved, so
the layout cannot reproduce the engine's own arithmetic. Floor unestablished, posture WITHDRAWN,
exit 1.
That refusal is the deliverable. Knowing mechanically that capacity cannot be established beats
reading a dashboard number that says 99% and means retained prefix cache. The instrument earns its
keep the day that answer changes without anyone intending it to.
ONE ROUND TRIP PER GROUP, and the far side does the text handling: the head curls its own metrics
endpoint, greps its own launch log, and normalises twelve fixed-position lines, so this module parses
integers at known offsets instead of carrying a Prometheus parser and a log parser it would have to
keep in step with two upstream formats. A field the far side cannot produce arrives as "-", fails to
parse, and refuses — it never arrives as a zero.
Two defects found by running it rather than by reading it:
- The first cut returned a String and decided admittance by searching it for "posture: admitting",
which re-ran group_report to ask the question — so every group was probed over ssh TWICE per
invocation. Visible in the run as two identical remote executions. The standing now carries the
verdict as a field beside the text, so the effect happens once and the decision is read rather
than recovered from prose.
- No brace appears in the remote script, which is a constraint of the reader rather than of shell:
the heads-only pass that indexes modules counts braces without knowing it is inside a string
literal, so a docker --format template ended the enclosing declaration early and the module
refused to parse.
Group B did not answer: the fleet-automation key is authorised on .225 and not on .236, so half the
serving fleet is unreachable to automation. Reported rather than patched by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
…tribution gap" that never existed The previous commit reported that the fleet key was authorised on .225 and not on .236, so half the serving fleet was unreachable to automation. That was wrong, and the error was in this file. The probe hardcoded "briansrls" — a human login that happens to exist on the group A head and not on the group B head. gunbc.spark.managed_access_apply already declares gunbc-automation as the convergence principal, managed_access_apply provisions it, and BOTH heads answer to it on the fleet key (verified: spark-3bd5 and spark-0c75 both return id -un = gunbc-automation). There is no key gap. There was a second spelling of "who automation is", and it read as evidence about the fleet. So the principal is no longer spelled here at all; it is read from its authority. A probe that names its own principal can disagree with the one convergence uses, and that disagreement is indistinguishable from a broken host. With the right principal, both groups now answer and the instrument refuses one step later, for a real reason: gunbc-automation is a member of exactly one group (its own), so docker inspect is refused and the container image digest comes back "-". The launch cannot be identified, and the instrument says so rather than assembling an incarnation around a placeholder that would join to anything. That is the honest state of "use the automation principal everywhere": it can read systemd, the unit text, the launch log and the metrics endpoint on every head, and it cannot read the container. The capability set convergence requires is undeclared, so a missing one surfaces as a "-" at the point of use instead of as an admission refusal at the boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci
File the heads-only brace-in-string parse deficit, stop spelling a home-path identity beside spark_managed_login, name the restart-admission frontier, and delete the harness_now_epoch alias. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
Addressed review 61596 (REQUEST_CHANGES):
|
Contributor
|
Verified review 61596 against current head
No further code change on this item. Merge tally is not met (0 approvals on this head; REQUEST_CHANGES still on the pre-fix SHA). — sent from gentle-hawk-44 |
…se missing running/waiting metrics. review 62609: a dash on the request gauges was becoming idle-zero, and the restart fold spoke Nat for a quantity std.measure already names. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
review 62609 (
|
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
The stacked M0 wall made a free Percent unwritable; the live probe and capacity-floor fixtures now admit through the same line-class reading. Co-authored-by: Cursor <cursoragent@cursor.com>
…ed batch cap. Token-aware indexing only lets braces inhabit a string; it does not emit this ssh script. A fabricated max_num_batched_tokens of 0 had no reader, so the field is gone rather than inhabited. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
review 62640 (
|
6 tasks
admit_kv_layout, seating matches, and ReAdmitted now bind launch, the same grain as VllmServingLaunch. Co-authored-by: Cursor <cursoragent@cursor.com>
… probe. Those call sites were still constructing reported concurrency as Percent after the M0 carrier split. Co-authored-by: Cursor <cursoragent@cursor.com>
…ved. Stamping vllm_source_revision_read made revision_is_modelled permanently green. The probe still prints inventory; it does not fill the modelled revision so a later group-metadata path cannot look matched by construction. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
review 62657 ( The live probe was stamping
— sent from gentle-hawk-44 |
…anding witness. The instrument still passes none rather than stamping the modelled revision; that gap now has a TRIGGER/SUFFICIENT-FOR row. The leftover max_num_batched_tokens argument on the rounding fixture is gone with the field. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
review 62676 (
— sent from gentle-hawk-44 |
Bring in the reported-capacity revision-agreement refusal from M0.
Contributor
|
review 62685 — already fixed on this branch before this head (
Head: — sent from gentle-hawk-44 |
Contributor
|
review 62685 () reviewed the older SHA
No further commit for this review. — sent from gentle-hawk-44 |
Contributor
|
Artifact for the review above: /api/reviews/62685/artifacts/stdout.log — sent from gentle-hawk-44 |
Map unavailable layout reproduction to KvLayoutUnreadable.
gunbai-bot Bot
pushed a commit
that referenced
this pull request
Sep 9, 2026
…its own preconditions (#10698) * M3: capacity is readiness, and restart is a separate decision behind its own preconditions THE FLOOR IS ONE DECLARATION. "54 GiB of KV" and "three concurrent sessions" are the same requirement said twice, and once both are written down they drift — a change to the sequence budget moves one and leaves the other standing. So only the operational unit is declared, MinimumColdSessions, and the blocks are derived from it against the engine's own layout. The session count is not a second number either: it reads harness_seat_operator_policy_seats, because a fabric whose floor said four while the harness seated three would hold capacity nobody could use. COLD means no prefix-cache credit. A prospective hit belongs to a prefix another request may release at any moment; reserving against it spends a discount that was never granted. "BELOW FLOOR THEREFORE RESTART" IS ONE STEP WHERE THERE ARE TWO DECISIONS, and collapsing them is how a convergence run bounces a busy engine. Free memory on a unified-memory host is a moving target, so a restart taken because the pool came up small can come up smaller, and a loop that restarts on every unfavourable reading keeps restarting with requests in flight. The first move is therefore always withdrawal: stop sending new work. It costs nothing, is instantly reversible, and is correct under every failing standing including the ones where capacity is merely unknown — an unestablished capacity withdraws rather than assuming adequacy, which is today's real state for every engine in the fleet. Restart is then admitted, not triggered, behind five preconditions, and the fold names the FIRST unmet one. Four are decidable from what this repository models. The fifth refuses: no admitted unified-memory plan exists, and on a GB10 the pool is a residual of a supply shared with the OS page cache, the weights, the CUDA and NCCL contexts and this deployment's own Ray object store — restarting without a plan over it is what produced 23.6 GiB once and 54.01 GiB later from one unit text. Admitting a restart there would promise an outcome nothing modelled can predict. That is the M4/M6 slice; when it lands the arm is replaced by a real precondition rather than deleted. Re-admission is earned by a FRESH receipt. A completed restart proves a process started; on this hardware it proves nothing about the pool it came up with. Witnesses (7). The discriminating one: a busy engine below floor is withdrawn but not restarted, and with every decidable precondition met the restart still refuses naming the missing memory plan. Halving the declared budget halves the derived requirement, which a second literal would not do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Rename the host-side launch identity off VllmEngineIncarnation, which was already taken gunbc.spark.vllm_rank_population has carried VllmEngineIncarnation since before this stack, with the SAME reasoning from the SAME incident -- three launches on one pair in one day, identical version and checkpoint, resolved capacity differing by 22%. M0 re-invented that concept under its own name, which is the section 3 nickname this repository forbids: one meaning, two names, and the derived work duplicated behind each. The merged type keeps the name. The two are different GRAINS of the same question and neither observer can produce the other's fields: VllmEngineIncarnation is what the ENGINE reports about itself (the engine-core identity, readable from its own output), and this type is what the HOST knows about the launch (systemd invocation, container, image and unit digests, start instant). So this one is renamed VllmServingLaunch, which is what it actually holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Complete the constructor rename in the import lists Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Carry the corrected layout and reading shapes into the capacity floor witness The group layout lost its detachable window and its unread page_size_bytes, and a startup reading now observes the revision it was captured from. This fixture follows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * A wet instrument that asks the live engines what our capacity actually is, and today gets "unestablished" Every capacity claim in this stack has been green by typecheck. This one executes: gunbc run --entry dag/gunbc/instruments/fabric_capacity_standing.dag \ --function fabric_capacity_standing Run against the fleet just now, group A answered with its real launch — invocation d140fa39c1174403bf2ce4d7ed8c2d74, inventory 54,475 blocks of 16, printed pool 11,767,856 tokens at 11.22x, 0 running, 0 waiting — and the receipt refused: the resolved cache groups are unobserved, so the layout cannot reproduce the engine's own arithmetic. Floor unestablished, posture WITHDRAWN, exit 1. That refusal is the deliverable. Knowing mechanically that capacity cannot be established beats reading a dashboard number that says 99% and means retained prefix cache. The instrument earns its keep the day that answer changes without anyone intending it to. ONE ROUND TRIP PER GROUP, and the far side does the text handling: the head curls its own metrics endpoint, greps its own launch log, and normalises twelve fixed-position lines, so this module parses integers at known offsets instead of carrying a Prometheus parser and a log parser it would have to keep in step with two upstream formats. A field the far side cannot produce arrives as "-", fails to parse, and refuses — it never arrives as a zero. Two defects found by running it rather than by reading it: - The first cut returned a String and decided admittance by searching it for "posture: admitting", which re-ran group_report to ask the question — so every group was probed over ssh TWICE per invocation. Visible in the run as two identical remote executions. The standing now carries the verdict as a field beside the text, so the effect happens once and the decision is read rather than recovered from prose. - No brace appears in the remote script, which is a constraint of the reader rather than of shell: the heads-only pass that indexes modules counts braces without knowing it is inside a string literal, so a docker --format template ended the enclosing declaration early and the module refused to parse. Group B did not answer: the fleet-automation key is authorised on .225 and not on .236, so half the serving fleet is unreachable to automation. Reported rather than patched by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Correct the capacity instrument's principal, and with it the "key distribution gap" that never existed The previous commit reported that the fleet key was authorised on .225 and not on .236, so half the serving fleet was unreachable to automation. That was wrong, and the error was in this file. The probe hardcoded "briansrls" — a human login that happens to exist on the group A head and not on the group B head. gunbc.spark.managed_access_apply already declares gunbc-automation as the convergence principal, managed_access_apply provisions it, and BOTH heads answer to it on the fleet key (verified: spark-3bd5 and spark-0c75 both return id -un = gunbc-automation). There is no key gap. There was a second spelling of "who automation is", and it read as evidence about the fleet. So the principal is no longer spelled here at all; it is read from its authority. A probe that names its own principal can disagree with the one convergence uses, and that disagreement is indistinguishable from a broken host. With the right principal, both groups now answer and the instrument refuses one step later, for a real reason: gunbc-automation is a member of exactly one group (its own), so docker inspect is refused and the container image digest comes back "-". The launch cannot be identified, and the instrument says so rather than assembling an incarnation around a placeholder that would join to anything. That is the honest state of "use the automation principal everywhere": it can read systemd, the unit text, the launch log and the metrics endpoint on every head, and it cannot read the container. The capability set convergence requires is undeclared, so a missing one surfaces as a "-" at the point of use instead of as an admission refusal at the boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Merge launch-attempt-receipt and clear the M3 review. File the heads-only brace-in-string parse deficit, stop spelling a home-path identity beside spark_managed_login, name the restart-admission frontier, and delete the harness_now_epoch alias. Co-authored-by: Cursor <cursoragent@cursor.com> * Carry floor and posture block counts as AllocatorBlockCount, and refuse missing running/waiting metrics. review 62609: a dash on the request gauges was becoming idle-zero, and the restart fold spoke Nat for a quantity std.measure already names. Co-authored-by: Cursor <cursoragent@cursor.com> * Build capacity-floor witness groups with TokenCount block size. Co-authored-by: Cursor <cursoragent@cursor.com> * Spell the capacity-floor callback as a fn type so the witness parses. Co-authored-by: Cursor <cursoragent@cursor.com> * Drop the capacity-floor callback helper so the witness parses. Co-authored-by: Cursor <cursoragent@cursor.com> * Bind M3 reported-capacity construction to the MaximumConcurrency line. The stacked M0 wall made a free Percent unwritable; the live probe and capacity-floor fixtures now admit through the same line-class reading. Co-authored-by: Cursor <cursoragent@cursor.com> * Split the probe's brace trigger from bash-emit, and drop the unobserved batch cap. Token-aware indexing only lets braces inhabit a string; it does not emit this ssh script. A fabricated max_num_batched_tokens of 0 had no reader, so the field is gone rather than inhabited. Co-authored-by: Cursor <cursoragent@cursor.com> * Follow the receipt field rename on the capacity-floor consumers. admit_kv_layout, seating matches, and ReAdmitted now bind launch, the same grain as VllmServingLaunch. Co-authored-by: Cursor <cursoragent@cursor.com> * Follow ConcurrentRequestHundredths on the capacity floor and standing probe. Those call sites were still constructing reported concurrency as Percent after the M0 carrier split. Co-authored-by: Cursor <cursoragent@cursor.com> * Refuse live probe readings when the engine source revision is unobserved. Stamping vllm_source_revision_read made revision_is_modelled permanently green. The probe still prints inventory; it does not fill the modelled revision so a later group-metadata path cannot look matched by construction. Co-authored-by: Cursor <cursoragent@cursor.com> * Name the live revision-observation frontier and drop the duplicate standing witness. The instrument still passes none rather than stamping the modelled revision; that gap now has a TRIGGER/SUFFICIENT-FOR row. The leftover max_num_batched_tokens argument on the rounding fixture is gone with the field. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
briansrls
added a commit
that referenced
this pull request
Sep 9, 2026
…that worked (#10697) * M0: one engine launch is one subject, and the unit is not it Every capacity, usage, restart and routing observation this repository takes about a serving engine has been joined on some mixture of host, group, model name and self-reported version. None of those identify a LAUNCH. On 2026-09-05 the group A head produced engine cores reporting 23.6 GiB, then 21.29 GiB, then 54.01 GiB of available KV — three incarnations from ONE rendered unit whose text never changed. Joined on host and unit those are three readings of one unstable object; joined on incarnation they are three objects with one reading each, and the live question becomes which one is serving now. It is a safety defect and not only a reporting one: a seat granted against the engine that answered a probe must not authorize a request against the engine that has since replaced it, and "FabricGroupA" cannot tell them apart. Restart is the event that invalidates every outstanding grant, so a grant that cannot name its incarnation cannot be fenced. VllmEngineIncarnation is sole_constructor over group, head, systemd invocation, container id, executable digest, unit digest and start instant. The engine's self-reported vLLM version is deliberately NOT a field: extdeps.vllm.server already carries VllmSourceRevision for what the laws were established against, and a version a process prints about itself is a claim, not provenance. same_incarnation discriminates on the systemd invocation — the one field guaranteed to differ across two starts of a unit — then folds a row list of agreement checks, so a same-invocation pair whose fields disagree returns IncarnationContradiction rather than a match. That arm is separate from DifferentIncarnation on purpose: ordinary turnover and "two observers disagree about one launch" call for different action. Cross-family digests return DigestsIncomparable and surface as a contradiction naming the incomparability. std.content_hash refuses to collapse a cross-family pair into a silent false precisely so callers cannot read "cannot compare" as "differ"; mapping it to false here would reintroduce that collapse one level up. Witness (4): two launches of one unit — identical group, head, container name, image digest and unit digest — are NOT one incarnation, which is exactly the case a group- or unit-keyed join answers wrong; a disagreeing field under one invocation is a contradiction; cross-family digests refuse as incomparable rather than reading as a rebuild. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * WIP M8 launch attempt receipt * Rename the host-side launch identity off VllmEngineIncarnation, which was already taken gunbc.spark.vllm_rank_population has carried VllmEngineIncarnation since before this stack, with the SAME reasoning from the SAME incident -- three launches on one pair in one day, identical version and checkpoint, resolved capacity differing by 22%. M0 re-invented that concept under its own name, which is the section 3 nickname this repository forbids: one meaning, two names, and the derived work duplicated behind each. The merged type keeps the name. The two are different GRAINS of the same question and neither observer can produce the other's fields: VllmEngineIncarnation is what the ENGINE reports about itself (the engine-core identity, readable from its own output), and this type is what the HOST knows about the launch (systemd invocation, container, image and unit digests, start instant). So this one is renamed VllmServingLaunch, which is what it actually holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Complete the constructor rename in the import lists Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Merge kv-layout-receipt and clear the M8 review. Exit 0 is LaunchExited, not serving. Apply reports serving_standing over the attempts it has, which is currently none, with a named observation trigger. Co-authored-by: Cursor <cursoragent@cursor.com> * Keep dissolution import at module scope. Co-authored-by: Cursor <cursoragent@cursor.com> * M3: capacity is readiness, and restart is a separate decision behind its own preconditions (#10698) * M3: capacity is readiness, and restart is a separate decision behind its own preconditions THE FLOOR IS ONE DECLARATION. "54 GiB of KV" and "three concurrent sessions" are the same requirement said twice, and once both are written down they drift — a change to the sequence budget moves one and leaves the other standing. So only the operational unit is declared, MinimumColdSessions, and the blocks are derived from it against the engine's own layout. The session count is not a second number either: it reads harness_seat_operator_policy_seats, because a fabric whose floor said four while the harness seated three would hold capacity nobody could use. COLD means no prefix-cache credit. A prospective hit belongs to a prefix another request may release at any moment; reserving against it spends a discount that was never granted. "BELOW FLOOR THEREFORE RESTART" IS ONE STEP WHERE THERE ARE TWO DECISIONS, and collapsing them is how a convergence run bounces a busy engine. Free memory on a unified-memory host is a moving target, so a restart taken because the pool came up small can come up smaller, and a loop that restarts on every unfavourable reading keeps restarting with requests in flight. The first move is therefore always withdrawal: stop sending new work. It costs nothing, is instantly reversible, and is correct under every failing standing including the ones where capacity is merely unknown — an unestablished capacity withdraws rather than assuming adequacy, which is today's real state for every engine in the fleet. Restart is then admitted, not triggered, behind five preconditions, and the fold names the FIRST unmet one. Four are decidable from what this repository models. The fifth refuses: no admitted unified-memory plan exists, and on a GB10 the pool is a residual of a supply shared with the OS page cache, the weights, the CUDA and NCCL contexts and this deployment's own Ray object store — restarting without a plan over it is what produced 23.6 GiB once and 54.01 GiB later from one unit text. Admitting a restart there would promise an outcome nothing modelled can predict. That is the M4/M6 slice; when it lands the arm is replaced by a real precondition rather than deleted. Re-admission is earned by a FRESH receipt. A completed restart proves a process started; on this hardware it proves nothing about the pool it came up with. Witnesses (7). The discriminating one: a busy engine below floor is withdrawn but not restarted, and with every decidable precondition met the restart still refuses naming the missing memory plan. Halving the declared budget halves the derived requirement, which a second literal would not do. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Rename the host-side launch identity off VllmEngineIncarnation, which was already taken gunbc.spark.vllm_rank_population has carried VllmEngineIncarnation since before this stack, with the SAME reasoning from the SAME incident -- three launches on one pair in one day, identical version and checkpoint, resolved capacity differing by 22%. M0 re-invented that concept under its own name, which is the section 3 nickname this repository forbids: one meaning, two names, and the derived work duplicated behind each. The merged type keeps the name. The two are different GRAINS of the same question and neither observer can produce the other's fields: VllmEngineIncarnation is what the ENGINE reports about itself (the engine-core identity, readable from its own output), and this type is what the HOST knows about the launch (systemd invocation, container, image and unit digests, start instant). So this one is renamed VllmServingLaunch, which is what it actually holds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Complete the constructor rename in the import lists Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Carry the corrected layout and reading shapes into the capacity floor witness The group layout lost its detachable window and its unread page_size_bytes, and a startup reading now observes the revision it was captured from. This fixture follows. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * A wet instrument that asks the live engines what our capacity actually is, and today gets "unestablished" Every capacity claim in this stack has been green by typecheck. This one executes: gunbc run --entry dag/gunbc/instruments/fabric_capacity_standing.dag \ --function fabric_capacity_standing Run against the fleet just now, group A answered with its real launch — invocation d140fa39c1174403bf2ce4d7ed8c2d74, inventory 54,475 blocks of 16, printed pool 11,767,856 tokens at 11.22x, 0 running, 0 waiting — and the receipt refused: the resolved cache groups are unobserved, so the layout cannot reproduce the engine's own arithmetic. Floor unestablished, posture WITHDRAWN, exit 1. That refusal is the deliverable. Knowing mechanically that capacity cannot be established beats reading a dashboard number that says 99% and means retained prefix cache. The instrument earns its keep the day that answer changes without anyone intending it to. ONE ROUND TRIP PER GROUP, and the far side does the text handling: the head curls its own metrics endpoint, greps its own launch log, and normalises twelve fixed-position lines, so this module parses integers at known offsets instead of carrying a Prometheus parser and a log parser it would have to keep in step with two upstream formats. A field the far side cannot produce arrives as "-", fails to parse, and refuses — it never arrives as a zero. Two defects found by running it rather than by reading it: - The first cut returned a String and decided admittance by searching it for "posture: admitting", which re-ran group_report to ask the question — so every group was probed over ssh TWICE per invocation. Visible in the run as two identical remote executions. The standing now carries the verdict as a field beside the text, so the effect happens once and the decision is read rather than recovered from prose. - No brace appears in the remote script, which is a constraint of the reader rather than of shell: the heads-only pass that indexes modules counts braces without knowing it is inside a string literal, so a docker --format template ended the enclosing declaration early and the module refused to parse. Group B did not answer: the fleet-automation key is authorised on .225 and not on .236, so half the serving fleet is unreachable to automation. Reported rather than patched by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Correct the capacity instrument's principal, and with it the "key distribution gap" that never existed The previous commit reported that the fleet key was authorised on .225 and not on .236, so half the serving fleet was unreachable to automation. That was wrong, and the error was in this file. The probe hardcoded "briansrls" — a human login that happens to exist on the group A head and not on the group B head. gunbc.spark.managed_access_apply already declares gunbc-automation as the convergence principal, managed_access_apply provisions it, and BOTH heads answer to it on the fleet key (verified: spark-3bd5 and spark-0c75 both return id -un = gunbc-automation). There is no key gap. There was a second spelling of "who automation is", and it read as evidence about the fleet. So the principal is no longer spelled here at all; it is read from its authority. A probe that names its own principal can disagree with the one convergence uses, and that disagreement is indistinguishable from a broken host. With the right principal, both groups now answer and the instrument refuses one step later, for a real reason: gunbc-automation is a member of exactly one group (its own), so docker inspect is refused and the container image digest comes back "-". The launch cannot be identified, and the instrument says so rather than assembling an incarnation around a placeholder that would join to anything. That is the honest state of "use the automation principal everywhere": it can read systemd, the unit text, the launch log and the metrics endpoint on every head, and it cannot read the container. The capability set convergence requires is undeclared, so a missing one surfaces as a "-" at the point of use instead of as an admission refusal at the boundary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci * Merge launch-attempt-receipt and clear the M3 review. File the heads-only brace-in-string parse deficit, stop spelling a home-path identity beside spark_managed_login, name the restart-admission frontier, and delete the harness_now_epoch alias. Co-authored-by: Cursor <cursoragent@cursor.com> * Carry floor and posture block counts as AllocatorBlockCount, and refuse missing running/waiting metrics. review 62609: a dash on the request gauges was becoming idle-zero, and the restart fold spoke Nat for a quantity std.measure already names. Co-authored-by: Cursor <cursoragent@cursor.com> * Build capacity-floor witness groups with TokenCount block size. Co-authored-by: Cursor <cursoragent@cursor.com> * Spell the capacity-floor callback as a fn type so the witness parses. Co-authored-by: Cursor <cursoragent@cursor.com> * Drop the capacity-floor callback helper so the witness parses. Co-authored-by: Cursor <cursoragent@cursor.com> * Bind M3 reported-capacity construction to the MaximumConcurrency line. The stacked M0 wall made a free Percent unwritable; the live probe and capacity-floor fixtures now admit through the same line-class reading. Co-authored-by: Cursor <cursoragent@cursor.com> * Split the probe's brace trigger from bash-emit, and drop the unobserved batch cap. Token-aware indexing only lets braces inhabit a string; it does not emit this ssh script. A fabricated max_num_batched_tokens of 0 had no reader, so the field is gone rather than inhabited. Co-authored-by: Cursor <cursoragent@cursor.com> * Follow the receipt field rename on the capacity-floor consumers. admit_kv_layout, seating matches, and ReAdmitted now bind launch, the same grain as VllmServingLaunch. Co-authored-by: Cursor <cursoragent@cursor.com> * Follow ConcurrentRequestHundredths on the capacity floor and standing probe. Those call sites were still constructing reported concurrency as Percent after the M0 carrier split. Co-authored-by: Cursor <cursoragent@cursor.com> * Refuse live probe readings when the engine source revision is unobserved. Stamping vllm_source_revision_read made revision_is_modelled permanently green. The probe still prints inventory; it does not fill the modelled revision so a later group-metadata path cannot look matched by construction. Co-authored-by: Cursor <cursoragent@cursor.com> * Name the live revision-observation frontier and drop the duplicate standing witness. The instrument still passes none rather than stamping the modelled revision; that gap now has a TRIGGER/SUFFICIENT-FOR row. The leftover max_num_batched_tokens argument on the rounding fixture is gone with the field. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com> * Transport the engine's startup lines instead of synthesizing them. The probe emits the last matching log line, grepping prefixes from vllm_startup_line_prefix, and the carrier is inhabited only from that text. Co-authored-by: Cursor <cursoragent@cursor.com> * Keep LaunchTerminal on serving standing instead of wired strings. A later consumer asking whether a failure was SIGKILL should match the typed arm, not re-parse launch_terminal_wire prose. Co-authored-by: Cursor <cursoragent@cursor.com> * Consume pair realization names and measure carriers on the capacity probe. Restore the retiring dsv4 container as the JIT-cache source, interpolate spark_pair_* authorities into the probe script, and type inventory fields as AllocatorBlockCount and TokenCount instead of bare Int. Co-authored-by: Cursor <cursoragent@cursor.com> * Derive probe emit and decode from one named field roster. Position is no longer restated as a command list, integer indices, and a fixture order; a reorder cannot silently swap equally-shaped slots. Co-authored-by: Cursor <cursoragent@cursor.com> * Cite the harness interactive-quality seat constant that actually exists. The capacity floor imported a nickname with no definition, so the session count could not resolve against the one harness authority. Co-authored-by: Cursor <cursoragent@cursor.com> * Refuse unobserved hash_block_size and extra probe lines. The probe was stamping scheduler block_size onto a separate upstream field, and decode accepted surplus lines so a newline in a slot shifted the rest. Co-authored-by: Cursor <cursoragent@cursor.com> * Withdraw the remote bash inventory probe rather than license it. A concat-built ssh script is out-of-band actuation, and ProbeReadingsReady was unreachable from the entry anyway. Decode and readings stay; the entry refuses without shipping that actuator. Co-authored-by: Cursor <cursoragent@cursor.com> * Refuse missing launch identity and type the start instant as EpochSecs. A literal dash on invocation, unit digest, container or image was decoding as a join key; those slots now refuse like the other captured fields. Co-authored-by: Cursor <cursoragent@cursor.com> * Declare the apply frontiers for OOM attribution and clean bring-up. Those folds are exercised only by witnesses today; the attempt-observation frontier already names serving_standing and does not cover this cluster. Co-authored-by: Cursor <cursoragent@cursor.com> * Refuse standing over mixed desired_digest or mixed-host attempt lists. A flap is many incarnations of one convergence decision on one head; folding another digest or host into recovered standing was fabricated. Co-authored-by: Cursor <cursoragent@cursor.com> * Drive serving standing from whether the run ends serving. A serve-then-kill list was folding as recovered from the first serving row. Only prior failures feed recovery, and the reverse order is a RED. Co-authored-by: Cursor <cursoragent@cursor.com> * Type the unobserved hash_block_size as Absent in the capacity witness. `none` inferred as Unit next to TokenCount?, so the floor refused the module; Absent is the optional constructor the fold already matches. Co-authored-by: Cursor <cursoragent@cursor.com> * Bind probe readings to a launch through admit_kv_layout. Identity slots were decoded and never read; Ready minted a free layout. launch_from_fields consumes the join keys, Ready carries the receipt, and running/waiting stay off the roster until a drain fold exists. Co-authored-by: Cursor <cursoragent@cursor.com> * Name VllmServingLaunch fields launch, not incarnation. The engine-core identity already owns incarnation; keeping that name on the host-side launch type reopened the grain collision the layout receipt already closed. Co-authored-by: Cursor <cursoragent@cursor.com> * Mint EpochSecs without a cast; size floor witnesses to one session. decode_probe cannot `as EpochSecs` after parse_int. The floor witnesses still assumed three seats, so 40000 blocks looked like a miss against a one-session requirement of 31250. Co-authored-by: Cursor <cursoragent@cursor.com> * Refuse off-scale concurrency prints and negative capacity slots. Deleting '.' and 'x' made 8.5x look like 85 hundredths. The two-digit Nx shape is now required, and allocator/block/context counts mint Nat only after a non-negative check. Co-authored-by: Cursor <cursoragent@cursor.com> * Restore main's layout and seat surfaces after the rebase. The replayed M0/M1 commits reintroduced a deleted host-side incarnation module and dropped max_num_batched_tokens. Carry that field from an observed TokenCount, the same way hash_block_size is carried. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Brian Searls <briansearls1@gmail.com> Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #10697 (M8) → #10695 (M2) → #10693 (M0+M1). Also merges current
main, so it carries the merged M5 constant.One declaration, derived blocks
MinimumColdSessionsis the only thing declared. "54 GiB" and "three sessions" are one requirement said twice, and written down twice they drift — change the sequence budget and one moves while the other stands.The session count isn't a second number either: it reads
harness_seat_operator_policy_seats. A fabric whose floor said four while the harness seated three would hold capacity nobody could use; one that said two would admit a seat the fabric never promised.Cold means no prefix-cache credit — a prospective hit belongs to a prefix another request may release at any moment.
Withdrawal is not restart
"Below floor therefore restart" is one step where there are two decisions, and collapsing them is how a run bounces a busy engine. Free memory here is a moving target: a restart taken because the pool came up small can come up smaller.
So the first move is always stop sending new work — free, instantly reversible, and correct even when capacity is merely unknown. An unestablished capacity withdraws rather than assuming adequacy, which is today's real state for every engine in the fleet.
Restart is admitted, not triggered
Five preconditions; the fold names the first unmet one. Four are decidable today. The fifth refuses:
On a GB10 the pool is a residual of a supply shared with the OS page cache, the weights, the CUDA/NCCL contexts and our own Ray object store. Restarting without a plan over it is what produced 23.6 GiB once and 54.01 GiB later from one unit text. That's the M4/M6 slice; the arm gets replaced by a real precondition, not deleted.
Re-admission is earned by a fresh receipt — a completed restart proves a process started, not what pool it came up with.
Evidence
7 witnesses. Discriminating: a busy engine below floor is withdrawn but not restarted; with every decidable precondition met the restart still refuses, naming the memory plan. Halving the declared budget halves the derived requirement — a second literal would not move.
🤖 Generated with Claude Code
https://claude.ai/code/session_01FbXKcgqW4Jx7e78BPWc8ci