A modeled, content-keyed GB10 sm_121 vLLM build at 07b75534 -- and the two obligation symbols that do not exist

[briansrls]

What this is

The runtime sweep in #11016 probed every published GB10 image inside its exact digest and found none that registers DeepseekV41ForCausalLM on sm_121, leaving a build obligation: the only source known to carry the V4.1 path is vLLM at 07b75534651a0dd584c6f3be75d8c62fb8426397. This is that build, modelled.

It is not a shell script somebody ran. A docker build typed into a terminal produces an image whose provenance is a memory — which base, which arch list, which revision — and the whole product of this lane is a digest that someone else has to be able to re-derive or refute. So the inputs are declarations, the identity of the build is a content hash over those declarations, the tag is derived from that hash, and every leg executes through gunbc.typed_argv_exec typed_argv_exec_over_fleet_ssh: a portable-word-sealed argv over an SSH leg whose known-hosts anchor was materialized and read back and whose credential was verified against fleet_automation_ssh_key before any host was contacted. Two transports, and which one a leg takes is decided by what the leg needs. Source acquisition and reachability run as the managed executor over the fleet key. Docker cannot: its socket on a Spark is root-owned and gunbc-automation is in no docker group on any of them, so the docker legs bind to the one route the corpus already models for that daemon — gunbc.spark.model_snapshot_materialize spark_remote_run, briansrls over a password session with sudo -S, credential resolved inside the boundary where the entry runs and never in an authoring session. Privileged for a read-only question, and the modeled path rather than a second one invented here (operator ruling).

The modules

• extdeps.vllm.source_build — upstream's own container build recipe at the pinned revision (docker/versions.json), its aarch64 release invocation (.buildkite/release-pipeline.yaml), and CUDA_SUPPORTED_ARCHS from CMakeLists.txt. The default BUILD_BASE_IMAGE is an amd64-only manifest, which is why an arm64 build must override it; vllm_recipe_targeting is the single override point, so a fleet recipe diverges from upstream in exactly the fields a reader is told to look at.
• extdeps.vllm.deepseek_v4_1_sm12 — what the V4.1 SM12 path demands, read out of the tree rather than restated: the FlashInfer SM120 symbols the branch checks, the registered torch operators with their library namespaces (torch.ops._C and torch.ops._C_cache_ops are different lookups, so a bare name is not a question anything can be asked), the two arch-gated CMake source blocks, and which KV layouts the backend admits per device major.
• gunbc.spark.v41_runtime_image_build — converges the source at the revision (readback first: a directory that exists says nothing about which commit is in it) and the image at its key-derived tag.
• gunbc.spark.v41_runtime_image_probe — generates its probe from the roster and runs it inside the built image with --gpus all. A capability with no VERDICT line reads UNREAD, never absent: a probe that could not ask must not produce evidence against the image.

extdeps.docker.cli gains a DockerBuildSpec (the existing docker_build_command cannot say --build-arg, --target, --platform or --file, so it can only ever get a Dockerfile's defaults), an ephemeral docker run, and a short -v mount rendering — the long --mount form separates fields with commas, and a comma is outside the portable-word allowlist, so it is refused before it reaches a host. extdeps.git gains shallow clone/fetch/checkout/rev-parse. Both are admitted by name on argv_command.

The finding this lane produced before it produced an image

Two of the four FP4-KV symbols in #11016's build obligation do not exist at the pinned revision. There is no concat_and_cache_ds_mla_nvfp4 and no indexer_k_quant_and_cache_fp4 anywhere in the tree, and nvfp4_ds_mla_cache is a C++ header, not a registered operator — so those three read absent in every image, including one built from the lead commit, and the obligation as written could never discharge.

The capability behind them is also arch-gated upstream: nvfp4_ds_mla_cache_kernels.cu is listed in FP4_SM100_SRCS alone, guarded by -DENABLE_NVFP4_SM100=1, while the SM12x block (12.0f, which family-matches 12.1) carries the NVFP4 quantisation, GEMM and KV-cache kernels but not that one. The build's own configure output confirms it by execution: -- CUDA target architectures: 12.1, -- Building SM12x NVFP4 for archs: 12.0f, -- Not building SM10x/11x NVFP4/MXFP4 as no compatible archs were found.

And it is not a degraded path: DeepseekV4FlashInferMLASparseBackend.supports_combination admits, on device major 12, exactly fp8, fp8_e4m3 and fp8_ds_mla, and rejects nvfp4_ds_mla outright — so sm_12x never selects that layout, kernels or no kernels. The capability was therefore renamed (by ruling) from an operator-presence question to the layout question, and this probe answers it by calling the admission rule inside the image for every candidate dtype rather than looking up symbols. loyal-ram-346 verified the finding independently against the same file and re-typed #11016's three rows as KernelProbeSymbolUngrounded, with a frontier naming this module.

Evidence

• 22 of 22 witnesses PASS (claim_batch --entry dag/test/claim/spark/v41_runtime_image_build_witness_test.dag).
• Mutation control on the reachability walls: flipping spark_host_answered to accept an unreachable host, and pointing the hostname lookup at the fleet identity instead of the lane's recorded vendor label, turns exactly the two reachability witnesses red and leaves the other sixteen green.
• Mutation control on the source/execution join: adding nvfp4_ds_mla to the observed admitted layout set and flipping the source's registry fact turns exactly the two join witnesses red and leaves the other fourteen green — so "the readings agree with the source" is a claim that can fail, not a sentence matching itself.
• Mutation control on the arch list, run twice: with v41_gb10_torch_cuda_arch_list mutated to 12.0, 3 witnesses go red (the subject-architecture pin, the key/tag derivation, the rendered build argv) and the rest stay green. The first mutation run is why the subject-architecture assertion exists at all — it showed the witness whose whole point is "the build targets the architecture the fleet has" could not see its own subject.
• gunbc compile over the probe entry: no name, type or effect errors. It refuses at rust emission on a pre-existing limitation in extdeps.gunbc (gunbc.WitnessBin.Run's shell stderr channels are not realizable for the rust target) which the closure reaches and which this PR does not touch; the *_wet entries run through the interpreter, as every other fleet entry does.

The build, executed through the modeled route

vllm_runtime_image_build_wet converged and wrote its receipt: local config digest sha256:b8b4ef35b17c2dd10c73f323819996f8928f0e5bd52f8980eedd9c634162461f, 21.8 GB, source 07b75534651a0dd584c6f3be75d8c62fb8426397, tag gunbc-vllm-dsv41-gb10:be277a5fc9425631 at the time of the run. vllm.__version__ inside it reads 0.1.dev1+g07b755346 — a self-report, which is evidence about the artifact and not proof of what was compiled. Nothing was pushed to any registry.

How the build actually ran, stated plainly. The first invocation of the CUDA compile was made by hand, before the modeled route existed, to discover whether the build succeeded at all on sm_121 and with which arguments — about 75 minutes, and the source of the configure-output evidence above. The modeled entry then ran with those layers warm, which is why its build leg returned in minutes rather than hours. That is a warm cache, not a shortcut: the modeled entry rendered its own argv from the declarations, executed every leg over the sealed transport, and read the digest back itself, and re-running it on a cold host would do the same work from scratch. It is recorded here because "the modeled route produced this digest" and "the modeled route paid for every layer in this digest" are different claims and only the first is true.

The digest is the identity; the tag is a function of the key, and the key function moved. Review finding 2 replaced a hand-rolled fold with std.artifact_store artifact_realization_digest, so the same inputs now mint a different key and therefore a different tag. The image on disk still carries the tag the old key produced. Re-tagging under the new key is one cached build invocation, and it will happen on whichever host the route decision names — not on srv5, which this module's own roster now refuses.

The build's own configure output confirms the arch gate by execution: -- CUDA target architectures: 12.1, -- Building SM12x NVFP4 for archs: 12.0f, -- Not building SM10x/11x NVFP4/MXFP4 as no compatible archs were found.

The probe, executed inside that digest on a GB10

docker run --rm --entrypoint python3 --gpus all, interpreter reporting compute capability 12.1. Seven questions were put and seven answers came back: five presences, one absence, one refusal.

The readings are no longer rows in this lane's module. They are rows in gunbc.spark.serving_runtime_capability_probe's single roster — keyed by the digest they were taken in and by the vendor-qualified device — beside every other probed image's, and the candidate row for this image says CandidateProbed, so the sweep's denominator includes it. The architecture answer derives from a full ArchitectureRegistryReading (host, date, instrument, all 383 names) rather than from a second hand-written membership claim.

capability	reading
architecture-registered	absent — 383 architectures registered, DeepseekV41ForCausalLM not among them
fp4-weight-kernels	present — _C.cutlass_scaled_fp4_mm, _C.cutlass_scaled_mm_supports_fp4
fp4-weight-kernels-supported-here	refused — the run that produced these rows FOUND the predicate and did not CALL it. The probe now calls it; this digest has not been re-probed, and the host it lives on is off limits to this lane. Recorded as a refusal, never as an absence: a question nobody asked is not evidence against the image
mla-kv-cache-layout-admitted	present — admitted fp8, fp8_e4m3, fp8_ds_mla; rejected auto, bfloat16, nvfp4_ds_mla
flashinfer-sparse-mla-sm120-symbols	present — all four demands, including _DECODE_DSV4_DISPATCH
csa2-compressor-kernels	present
dspark-speculator	present

Two prerequisites are outstanding, not one. The absence is the finding, and it is intended rather than a build defect: public vLLM at the lead commit does not register V4.1. The FP4 runtime predicate is the other, and it is a different kind of gap — the run found cutlass_scaled_mm_supports_fp4 and never called it, so its answer for this device is unknown rather than negative. The build obligation in gunbc.spark.serving_runtime_capability_probe states both, because a discharge that named only the registry defect would leave a required capability unasked behind it. It is a registry lookup joined to two source facts, not an inference from either. The class is in the tree at vllm/models/deepseek_v4_1/nvidia/vl_model.py; public main's registry.py maps no name to vllm.models.deepseek_v4_1, while DeepseekV4ForCausalLM maps to vllm.models.deepseek_v4 — and that sibling IS in the image's registry, which is what rules out an empty registry or a probe that never imported one. vLLM's own published deepseekv41-flash-0909 arm64 image was observed to carry the entry — read out of the registry.py in its own site-packages layer, no pull — and the distribution in that layer is stamped g179dd0fa9, an identifier the GitHub API answers No commit found for SHA about. Those are three readings. What produced the published image's bytes is not established by any of them: a setuptools-scm stamp is what a packager wrote, and a failed lookup is a fact about an identifier.

Two transport defects the executed route found

Both were fixed at their seam rather than routed around. docker image inspect --format {{.Id}} cannot cross the portable-word allowlist (braces), and the first version also treated the resulting refusal as "image absent" and rebuilt — the digest is now read with docker images --no-trunc --quiet, whose empty answer distinguishes "no such image" from "the query did not run". And a serving image's entrypoint swallowed the probe's argv (vllm: error: unrecognized arguments: .../v41_capability_probe.py), so the ephemeral run names its interpreter.

A third was found by rendering the generated probe before running it: the architecture capability was emitted through the generic reporter with empty symbol lists, which prints a vacuous =present line ahead of the real verdict. A reader taking the first match would have reported an unregistered architecture as registered — the strongest possible false positive, in the one capability this lane exists to establish. The generator no longer emits it and the reader now calls a contradictory pair UNREAD.

Where a build may run, as a derivation

No host is named in the build module. Source acquisition takes --arg host=<identity> and the two credential-bearing entries take SPARK_CONVERGE_TARGET; both admit it against a roster derived by subtraction: every Spark carrying a router binding, minus the members of the groups in spark_pair_serving_groups (where an engine actually serves), minus anyone llama_cpp_rpc_host_holds_undischarged_reservation answers for. Today that leaves srv9–srv12 and refuses srv5–srv8. The first cut of the subtraction excluded every fabric group rather than the active ones — a FabricGroup is a 400G cage, not a reservation — which left nothing admissible at all, and the witness caught it before anything ran.

Route A runs as a dispatch mode

spark_runtime_image_probe is a step inside the existing dispatch-only fleet-converge job, emitted through gunbc.fleet_converge_workflow and gunbc.ci_spec and regenerated through the generated-artifact gate — never hand-edited. The workflow declares five jobs and still declares five (build, fleet-converge, host-reset-return, dashboard-deploy, rlm-launch-deployment-receipt); the probe appears only as a dispatch option, a step id and an if-condition. It adds no job and no per-push cost. (An earlier revision of this section said six — that count included workflow_dispatch, which is a trigger, not a job. The before/after comparison was unaffected; the number was wrong.)

It runs there because the credential does: docker's socket on a Spark is root-owned, gunbc-automation is in no docker group, and the administrator credential is materialized inside that boundary and nowhere else.

Cold acquisition, executed

vllm_source_acquire_wet --arg host=srv9, over the modeled route as gunbc-automation, exit 0:

host=srv9 endpoint=192.168.1.236 principal=gunbc-automation
root=/home/gunbc-automation/gunbc-vllm-build  clone_url=https://github.com/vllm-project/vllm
source revision=07b75534651a0dd584c6f3be75d8c62fb8426397 already_present=false clean=true

already_present=false is the load-bearing part: this is the first execution of the clone leg in this lane. A side-chat review found it broken by reading — VllmSourceRevision.repository is github.com/vllm-project/vllm, an identity, and git reads it as a relative path (fatal: repository ... does not exist, reproduced) — and the earlier modeled runs never reached it because the exploratory build had already left a tree behind. The run above exercised the clone, the shallow fetch of the pinned commit, the detached checkout, the HEAD readback and the new dirty-tree refusal in one pass, on a host that had none of it.

It also forced out two latent literals of the same class as the build host: the build root sat under the operator's home while every leg authenticates as the managed executor, and the ssh principal was the operator bootstrap login — an account that exists to take a password on first contact and refuses key auth by design, so it only ever worked where a human had already installed a key. Both are now derived from gunbc.spark.managed_access_bootstrap.

The dirty check sees the ignored set, proved by a planted file

git status --porcelain omits ignored files; a docker build context carries everything .dockerignore does not exclude, and at the pin .gitignore has **/generated/** while .dockerignore does not. Green–red–green on srv9:

clean tree                          -> exit 0, clean=true
plant vllm/generated/__init__.py
  git status --porcelain            -> (empty)               <- the defect
  git status --porcelain --ignored  -> !! vllm/generated/
  modeled entry                     -> REFUSED at source-clean
remove it                           -> exit 0, clean=true

Reachability, asked the same way

spark_build_host_reachability_wet walks that derived roster and asks each host one question — its own short hostname — over the same sealed transport. Executed: all four admissible Sparks (srv9–srv12, at 192.168.1.236–.239) refuse the fleet key with Permission denied (publickey,password). Host-key verification passed on all four — strict checking against the materialized anchor got as far as authentication — so the boxes are up, reachable at their modeled endpoints, and presenting enrolled keys; what is missing is authorization for this principal. That is a provisioning fact about four hosts, taken with one command each and nothing installed.

The reading is joined, not merely captured: a Spark answers hostname -s with its vendor label (spark-a3ee), never with its fleet identity (srv5), so the answer is compared against the spark_hostname the lane observations record for that identity — a reading taken earlier against the box's own answer now. A mismatch and an unjoinable host are their own arms, and the entry succeeds only when every admissible host answered, so the run where all four refused the key exits non-zero rather than writing a receipt and returning success.

The outcome type keeps five answers apart that a ping would fuse: answered (carrying the name the host reported, so an endpoint answering as the wrong machine is a fleet-data defect rather than a success), unreachable with its cause, and unresolved when a host has no management-LAN endpoint at all.

An operational fact, recorded because it was a near miss

spark-a3ee is srv5 — the Spark serving pair member — and the brief named the box by its vendor label in one sentence and the pair by its fleet identity in another. The build ran ~75 minutes there and took host free memory to about 1 GiB at its peak, on a unified-memory host whose serving engine shares that pool. The gunbc-spark-pair container read Up 2 days across the whole window and its worker unit stayed active, so nothing restarted — but this should not be repeated, and the manager has ruled no further builds or heavy work on srv5 or srv6.

What surfaced it was the model, not a person: v41_build_host spelled the vendor label, fleet_probe_endpoint_for resolves fleet identities, and the first modeled leg refused with a typed cause before touching anything. The row now names operator_host_srv5 and a witness asserts the build host resolves to an endpoint other than its own name. That resolution helper returns its input when no binding matches rather than refusing — an absorbing fallback upstream of this lane, not fixed here, and named in the witness that compensates for it.

Still open

The readings live in this lane's module with a 🟡 dissolve-on to move to gunbc.spark.serving_runtime_capability_probe's roster when #11016 lands; one image's readings belong beside every other image's. The capability labels and the architecture class name are likewise spelled locally only until #11016 and #11007 land.

Not done and explicitly not inferred: the whole-model KV bytes-per-token derivation. An earlier message from this lane carried 656 bytes; that is Kimi K3's fp8_ds_mla entry stride, not V4.1's, and it is withdrawn — V4.1 passes state_content_bytes=584 with alignment=576 on the page. The roster module carries the per-layer facts and the formula and deliberately stops short of the sum; completing it is a follow-up assigned to this lane, for crisp-crane-661 to consume.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Q4MBC3Vb8do5fAXAysiXLE

[gunbai-bot]

Both findings from review 63487 are right, and they were the same defect seen twice. Fixed in c7e88cecda, not argued with.

1. Transcribed instrument output. Correct, and the registry count was the purest case: 383 summarised a list the row did not hold, so it could only ever be compared against itself. The evidence: NonEmptyStr sentences are gone. A reading now carries its outcome in V41CapabilityReading — the coproduct the reader already uses — and its measured values in their own typed fields: v41_probed_admitted_kv_layouts, v41_probed_rejected_kv_layouts, v41_probed_architecture_present, and the sibling architecture beside it so the absence is discriminating rather than merely negative. The rationale moved to annotations, which is where §4c puts prose.

On the §6 rule itself: this lane's instrument cannot re-derive on demand — it needs a GB10 and a 21.8 GB image that is not in any registry — so the rows stay a receipt rather than becoming a call. What changed is that the receipt is now checkable, which is the property the rule is protecting.

2. string_contains over prose this PR authored. Correct, and the fix is not a better pattern — it is a different oracle. The witness now joins the in-image reading against an independent source: what upstream's own files say at the pinned revision. extdeps.vllm.source_build gained vllm_07b75534_registry_maps_deepseek_v4_1 and vllm_07b75534_registered_deepseek_v4_architectures, read from registry.py; extdeps.vllm.deepseek_v4_1_sm12 already carried the admission rule read from flashinfer_sparse.py. Two differently-obtained answers to one question are required to agree, in both directions, at set grain:

• the_in_image_layout_admission_agrees_with_the_source_at_the_pinned_revision
• the_in_image_registry_and_the_source_registry_agree_that_v41_is_unmapped

Mutation control for exactly that claim, since "it joins two sources" is itself a claim that can be false: adding nvfp4_ds_mla to the observed admitted set and flipping the source's registry fact turns those two witnesses red and leaves the other fourteen green. Unmutated, 16 of 16 pass.

Also in this push, from the manager's review: the build host is no longer a literal (it is --arg host=<identity>, admitted against a roster derived by subtracting serving-group members and RPC-reserved members from the modeled Spark fleet); the build key is minted by std.artifact_store artifact_realization_digest instead of a hand-rolled fold, with a stated §3b divergence for std.materialization_provider and std.realization; readings this repository took moved out of extdeps.vllm.* into gunbc.spark.v41_published_image_observation, each with its instrument and date.

— sent from clever-bat-173

[gunbai-bot]

All three findings from review 63553 are correct and are fixed in 1f656d6de0. Verified against the code, not taken on report.

1. spark_build_host_reachability_wet did not stop the line. Right, and it is the worst case of it: the run that actually happened had every admissible host refuse the fleet key, and it exited zero with the typed diagnostics written to the receipt and then discarded by the exit path. It now returns ExitSuccess only when every admissible host answered, and the failure reason carries the body naming the ones that did not. spark_host_answered is the predicate, and it is false for all four non-answering arms.

2. reported_hostname was captured and compared to nothing while the annotation claimed the comparison as the point of the probe. The join is now performed — and against the right name, which is the part worth stating: a Spark answers hostname -s with its vendor label (spark-a3ee), never with its fleet identity (srv5), so a join written against the identity would call every host in the fleet a defect. The reading is joined to the spark_hostname that gunbc.spark.fabric_switch_observed's lane observations record for that identity — a lane reading taken earlier against the box's own answer now. SparkHostIdentityMismatch and SparkHostIdentityUnjoinable are their own arms rather than passes.

3. Two self-referential oracles. length(rows) == 7 is deleted; the surrounding fold over instrument-and-date was already doing the work at identity grain, and the count was exactly the change detector §5 describes. v41_flash_vram_minimum_gb == 614 is replaced by a join between two figures the publisher's page states and the relation it states between them: the minimum is the checkpoint total times the schema's 1.2 headroom factor, so the witness now asserts v41_flash_vram_minimum_gb == v41_flash_headroom_gb(total_gb: v41_flash_checkpoint_gb) — 511 × 1.2 = 613.2, rounded up to the published 614. A mis-transcription of either number now disagrees.

Mutation control on the two new walls, since "the join is performed" and "the line stops" are both claims that can be false: flipping spark_host_answered to accept an unreachable host, and pointing the hostname lookup at the fleet identity instead of the lane's label, turns exactly only_a_joined_answer_counts_as_reachable and the_reachability_join_reads_the_vendor_label_and_refuses_the_identity red and leaves the other sixteen green. Unmutated, 18 of 18 pass.

One housekeeping note: the commit on the branch reads wip: review 63553 fixes because session auto-push landed it before I could amend the message, and I would rather leave a poor message than force-push over a head a review may be running against. The trees are identical and this repo squash-merges, so the PR title and body are what land.

— sent from clever-bat-173

[gunbai-bot]

Operator-requested side-chat review (ChatGPT, relayed through the manager) raised five blocking findings. All verified against the code first, all fixed in 94285fc3bb. 23 witnesses green.

Cold acquisition was broken and had never executed. VllmSourceRevision.repository is github.com/vllm-project/vllm — an identity, not a locator — and git reads it as a relative path. Reproduced in an empty directory: fatal: repository 'github.com/vllm-project/vllm' does not exist. The modeled run never reached the clone because the source tree was already present from the exploratory build, so nothing caught it. vllm_clone_url derives the https locator from the identity, and a witness asserts the URL carries a scheme while the identity does not.

Provenance was asserted from the request. Three separate fixes: the source convergence now reads git status --porcelain back and refuses on any output, so a dirty context cannot produce an image stamped with a revision it was not built from; VllmImageIdentity.built_from is renamed requested_from and sits beside a provenance arm — BuiltByThisRun versus AdoptedUnderExpectedTag, the latter saying source equality is unestablished — because an image merely found under the derived tag was being returned as though this lane built it; and the two base-image inputs are mutable registry tags, which the module now says, with the trigger that would make the key a content identity end to end.

Probe integrity. The helpers caught every exception and returned False, so a broken interpreter manufactured evidence against the image. They now return found / missing / errored, the absence arm is narrowed to ModuleNotFoundError and AttributeError, and an errored check suppresses that capability's VERDICT line so the reader reads UNREAD. The substring half was already fixed on the head this review read; the reviewer's own counterexample, NOT_VERDICT fp4-weight-kernels=present_garbage, is now a witness and reads UNREAD on both counts.

What that does not change: no recorded reading depends on the arm that was wrong. Every row except the architecture one is a presence, which exception-as-False cannot manufacture, and the architecture row is membership in a list the same run printed in full (383 names), not an exception-prone lookup.

Symbol existence was promoted into support. cutlass_scaled_mm_supports_fp4 is a predicate about this device and was only being found in the operator table; it is now called, with the capability the interpreter itself reports, as its own capability row — and the recorded reading for it is UNREAD, because the run that produced these rows never asked it. The FlashInfer capability is renamed flashinfer-sparse-mla-sm120-symbols, since symbol inventory is what it measures; the scoped (num_q_heads, top_k) question needs this checkpoint's padded head count and SWA specialization width, which this lane does not have modeled, so it is a declared frontier naming what closes it. Absent and unread are now separate rosters, because an unasked question was reading as a capability nobody has.

Published-image claims narrowed. A setuptools-scm version is what the packager stamped and a failed commit lookup is a fact about an identifier; together they do not establish which source produced the bytes. Both readings now state what was read, bound to the layer and the API that read it, and source equality stays unestablished. "The entire difference" is gone — two registry files differing in two entries is not a diff of two images.

Still unexercised, stated rather than implied: the review asks to exercise cold acquisition and I have only exercised the argument — the failing and succeeding clone forms, locally. Running the fixed path means a shallow clone into a fresh directory on a Spark, which is a different thing from the image pull currently on hold; it is asked and pending. Until it runs, the clone leg is correct by reading and not by execution.

— sent from clever-bat-173

[gunbai-bot]

Two review rounds landed together — the cursor/auto dashboard review (review 63642) and an operator-requested side-chat, three findings each side. All verified against the code first; all fixed in b7282116a6. 24 witnesses green.

The dirty check could not see half the build context

git status --porcelain omits ignored files. A docker build context contains everything .dockerignore does not exclude — and those are different sets. Measured at the pin: .gitignore carries **/generated/**, .dockerignore does not. So a file under any generated/ directory is invisible to the check and shipped to the daemon.

Executed control on srv9, green–red–green:

clean tree                      -> exit 0, clean=true
plant vllm/generated/__init__.py
  git status --porcelain        -> (empty)          <- the defect, reproduced
  git status --porcelain --ignored -> !! vllm/generated/
  modeled entry                 -> source REFUSED step=source-clean ... !! vllm/generated/
remove it                       -> exit 0, clean=true

The reviewer's specific example file does not exist at this revision; the mechanism they described does, and the control above is what establishes it.

A nested import failure was reading as an absence

A module that exists raises ModuleNotFoundError when its own import of an unrelated dependency fails, so catching the type and reporting MISSING turned "this image lacks a dependency of the thing I asked about" into "this image lacks the thing I asked about" — an absence manufactured by the instrument, at a finer grain than the arm corrected in the previous round. ModuleNotFoundError.name carries which module was not found: MISSING only when that name is the target or one of its ancestors, ERRORED otherwise, which suppresses the verdict line so the reading is UNREAD.

Three claims outran their readings

• The witness named for source provenance now says what it checks: the image registers the architecture, and its stamped identifier is not a public commit.
• The readings annotation said the class "is unreachable". It now separates the registry lookup from the two source rows the witness joins it to, and says plainly that none of the five presences is a statement that a kernel is correct, fast, or exercised.
• The verified-hardware note claimed the published image "compiles for sm_121" from its TORCH_CUDA_ARCH_LIST. That is what a build was told to target, recorded in a config — and upstream's own build prints Not building ... and exits zero when an architecture block finds no compatible arch. Requested and compiled are different facts; only a probe inside the image settles the second, and this lane has not run one there.

From review 63642

Both findings correct. The two GB figures are std.measure Gigabyte rather than bare Nats named _gb, and the published minimum is now rendered from the Measure instead of forked as a second "614" literal that could drift. And the published-image witness joins typed lookup outcomes (PublicCommitAbsent, RegistryTagAbsent, and the stamped identifier as a field of the distribution name) instead of grepping prose this change authored — the same change-detector the capability receipts retired a round earlier, which is exactly how the reviewer framed it.

— sent from clever-bat-173

[gunbai-bot]

Five findings across two rounds — an operator-requested side-chat and the dashboard review (review 63676) — fixed in 62ed29fa84. 24 witnesses green. Each verified against the code before acting.

--untracked-files=all, because --ignored was only half. git status honours status.showUntrackedFiles. Reproduced in an isolated repository at no: --porcelain --ignored returns empty for a plain untracked file that docker ships anyway; --untracked-files=all shows it. Naming the mode makes the answer a property of the command rather than of whatever configuration a host happens to carry — which is exactly why it cannot be left to one.

The staged-file probe route is deleted, not annotated as gone. The reviewer was right that the note claimed "everything that existed only to get it there" was gone while v41_probe_script_path, _mode, _mounts, _command, converge_v41_probe_script, v41_probe_image and the key-route entry all survived and were reached — two mechanisms answering "run this probe inside this image". §3 names a surviving X an attractor and the remedy delete-first. Only the privileged route remains, which is the only one that can reach docker on a Spark at all.

Adoption no longer manufactures a build request. A pull-only operation was arriving with v41_gb10_recipe()'s recipe and build key attached, and the receipt printed that fallback key beside a published digest. AdoptedUnderExpectedTag made the claim weak, which was true and beside the point: the defect was inventing a build request where there was none. V41ProbeRead now carries a reference and the digest the daemon reported; vllm_adopted_image_identity is gone.

The generated Python program carries a 🟡 dissolve-on naming what would retire it. extdeps.languages.python is a target model — keywords, container templates, method spellings — that the compiler's generic emitter consumes to translate declarations this repository owns. The probe's content is reflection over a foreign runtime's import table and operator registry, symbols we deliberately do not own, so there is nothing in .dag to emit from; routing it through that emitter today would mean first minting declarations for exactly the names whose foreignness is the point. It dissolves when a modeled foreign-reflection surface exists. The interim shape has precedent in gunbc.instruments.emit_host_fixtures, which carries Python, Go and TypeScript program text as declared strings for the same reason.

Two claims corrected. The witness comment no longer concludes the published image "is built from a revision the public repository does not have" — a setuptools-scm stamp is a self-report and a failed lookup is a fact about an identifier. And the PR body no longer says "no sudo appears anywhere", which the privileged route makes false.

Route A is a dispatch mode, verified against the emitted file

spark_runtime_image_probe, emitted through gunbc.fleet_converge_workflow and gunbc.ci_spec and regenerated through the generated-artifact gate — no hand-edited YAML. Job count unchanged at six; the probe appears only as a dispatch option, a step id and an if-condition, so it adds no job and no per-push cost. The emitted step materializes the Spark administrator credential from Secret Manager inside itself, and the target→secret case covers all eight hosts, srv9 included.

And the dispatch menu stopped lying about its roster: the host description read srv5..srv11 while the administrator roster — what spark_converge_target_selection actually resolves against — carries srv12. A dispatcher would have believed srv12 illegal while resolution accepted it. The description is now joined from the roster rather than the number corrected.

— sent from clever-bat-173

[gunbai-bot]

Two rounds again — an operator-requested side-chat on 62ed29fa84 and review 63726 — and their findings turned out to be the same contradiction seen from two sides. Fixed in c2ed5b408c. 24 witnesses green.

The PR carried two incompatible answers to one question about the same fleet

The probe module recorded, as its measured finding, that docker's socket on a Spark is root-owned and gunbc-automation is in no docker group on any of them. The build module's header asserted, as a measured fact, that no sudo appears in it because the build reaches docker through the operator's docker group membership. Both halves of that were wrong together: the legs authenticate as the managed executor, not the operator, and that account cannot reach the daemon — so converge_vllm_image's docker images and docker build legs could not have run at all.

They did run, once. srv5 carries a hand-installed key for the human briansrls and that account in the docker group — provisioning drift no modeled route produces, standing in for a mechanism nobody had. The fallback image exists because of it, and that is now recorded in the module rather than smoothed over.

So the docker legs bind to gunbc.spark.model_snapshot_materialize spark_remote_run — one route to that daemon, the same one the probe uses — while the git legs keep the key session, which needs no root. It is a transport swap, so I took that arm of the ruling rather than declaring it unresolved.

The build entry moved with its legs. Needing the administrator credential means needing the fleet-converge boundary, so --arg host= is gone and it reads SPARK_CONVERGE_TARGET like the probe. What is not settled is carried as a 🟡 dissolve-on beside it: a cold build took ~75 minutes and gunbc_ci_spark_serving_step_timeout_minutes is 60, so whether a build fits inside that boundary is a job-level decision this lane does not get to make.

Every entry names its invocation

Three *_wet entries have no in-tree caller. That is the shape of an actuator, but only if the invocation is stated — otherwise it is dangling. vllm_source_acquire_wet and spark_build_host_reachability_wet are operator-invoked through the ordinary CLI and have both been executed that way against srv9, receipts in the lane's record; they take the fleet key from an in-run agent and need no credential an authoring session cannot hold. The build entry's dispatch is owed, and is recorded as owed.

A count I got wrong

I wrote that fleet-converge declares six jobs. It declares five — build, fleet-converge, host-reset-return, dashboard-deploy, rlm-launch-deployment-receipt. My count included workflow_dispatch, which is a trigger. The before/after comparison was unaffected and the conclusion holds — the probe adds no job — but the number was wrong and the PR body is corrected.

Provenance text

The module introduction no longer says a digest "is provenance: it says which inputs produced these bytes" — a digest is an identity for bytes, and even with the recipe beside it does not establish what the daemon compiled, which is why the identity record carries provenance as its own field. The readings annotation now says the published image's two registry entries were observed in that image's own layer, not that they explain where it came from. The PR body drops "wiring held outside public main" and the description of the deleted --arg host= probe route.

— sent from clever-bat-173

[gunbai-bot]

Side-chat on c2ed5b408c plus the manager's ruling and the v41_flash_fit_statement question — one push, 70bd602fa7. 24 witnesses green.

The acquisition entry was deleted and its own note kept advertising it. vllm_source_acquire_wet and its receipt helpers went out with the build-entry rewrite while the §3c invocation block still named the CLI command — so the one invocation a reader was told to use did not resolve. That is the same defect this lane has criticised twice in other text, an annotation describing a mechanism that is not there, and the reviewer is right that history executing it once establishes nothing about its availability now. Restored.

Enrolment is not admissibility, and the probe entry had stopped at the first. The administrator roster answers "does this Spark have a credential" — srv5 and srv6 do, and they are the serving pair. A dispatch naming one would have pulled 9.6 GB and started a container on a serving host, which is exactly what was ruled out. v41_published_image_probe_ci_wet now applies the same subtraction the build does, so a rostered host that is serving or reserved refuses there too, and the body's admission claim is true rather than aspirational.

A version stamp is a self-report. The readings annotation said the version string means "the image identifies its own source". It means the packager stamped that identifier — evidence about the artifact, not proof of what was compiled, which is the distinction extdeps.vllm.server already draws for a server's version prefix. The paragraph describing the deleted --arg host= probe route is gone, and the PR body's "reports its own provenance" with it.

v41_flash_fit_statement is now an annotation rather than a row. It was a String declaration nothing read, and its decidable content is already carried decidably next door — which hardware the publisher verified is one row, the minimum is a Measure in another. What remained was the publisher's caveat ("measure before assuming"), which §4c puts in an annotation; it now sits beside the number a reader is most likely to mistake for a measured floor.

— sent from clever-bat-173

[gunbai-bot]

review 63753 — four findings, all correct as found, fixed in f1b8c80769. 24 witnesses green, generated artifacts regenerated with no drift.

The annotation was describing the wrong declaration. Inserting fleet_converge_spark_runtime_image_probe_step between the pair-serving block and its function left that block leading mine — so it read as if the probe applies serving units workers-before-heads. Attachment is the whole semantics of the channel. The function moved below the one the block describes; the emitted YAML is byte-unchanged, because step order lives in the steps list and that did not move.

A transcribed measurement is gone. The parallelism annotation carried the build host's core count, its free memory, and upstream's figure for a different machine — three numbers copied into prose with no producer named, which rot the moment any of the three changes with nobody touching the file. What is citeable is upstream's own aarch64 release row, already a declaration; the fleet rows are a departure from it, chosen low because a Spark is not a dedicated builder. Where a reader wants the host facts, the annotation now names the instrument that would take them (spark_build_host_reachability_wet, which reaches an admissible host over the modeled route) rather than quoting an old reading.

nvcc_threads carries HardwareThreadCount. You're right about the boundary: the bare Nat stays on VllmContainerBuildRecipe, where the field is upstream's ARG at the cited-spec edge, and the conversion happens once at the call that builds the recipe. max_jobs is left a Nat — a concurrency degree, not a thread count — which matches your own note.

And the last prose row follows the first. v41_flash_documented_install_note was a String nothing read — the identical defect this module deleted one declaration earlier for v41_flash_fit_statement. That is how a class gets fixed at one instance and left standing at the next, and the observation is the useful part of the finding. Its decidable content — the image is not one install option among several but the only one the publisher offers — is now an annotation on the documented-image row, where it explains why a reference that does not resolve is a hard stop for a consumer rather than an inconvenience.

Noted, with thanks, on the two you looked at and did not flag: the 🟡 dissolve-on on the Python carrier and the §3c frontier block were both written to that standard deliberately, and it is useful to know they read that way from outside.

— sent from clever-bat-173

[gunbai-bot]

review 63774 — the typing defect is real and is fixed in d6a65b6763: the witness passed v41_build_nvcc_threads (a HardwareThreadCount, i.e. Measure { count }) into vllm_recipe_targeting's nvcc_threads: Nat. The production call site already unwrapped it; typing the policy row in the previous commit was carried through in one place and not the other. It now unwraps the way production does.

The consequence clause is wrong, and checking it turned up something worth more than the fix. The review says the witness "cannot execute at all in this state". It executed and passed — 24 of 24 green on the wrapped tree, the arm in question included. So nothing refused the mismatch.

I ran a discriminating control to find out whether that tolerance was something about Measure specifically. Planting max_jobs: "one" as NonEmptyStr in the sibling arm — a String at a Nat formal, which no reading of the types admits — the witness runner returned 24 of 24 green again, and gunbc compile over the same entry produced zero DeclaredTypeNotInhabited diagnostics and refused only at rust emission on the pre-existing extdeps.gunbc WitnessBin.Run shell-channel limitation, which is unrelated and present with or without the plant.

This is not a new class and I am not filing one. It is gunbc.recurring_failure_mode.absence_classifier_default_bucket — "a value of an unrelated constructor ACCEPTED at a declared position with no diagnostic in either direction", rung below the ladder, ceiling 3, already carrying an enrolled ExpectAssertionFalse RED at test.claim.declared_type_inhabitance_direct_call_witness.w_a_branded_refinement_at_an_unrelated_product_formal_still_refuses.

What my specimen adds is grain: that row's enrolled evidence is a branded where-refinement at a product formal, and this shows the hole is not confined to refinement chains — a plain NonEmptyStr at a Nat formal is accepted too, by the witness runner and by the full compiler. Whether that widens the row is its owner's call; I am handing over the specimen rather than minting a second row for one cause (§2).

What it means for this PR, stated so I am not overclaiming elsewhere: every "N witnesses green" in this thread is evidence about the assertions, which do execute and do go red when mutated — the mutation controls in this PR are real. It is not evidence that argument types at call sites are checked, because at that seam nothing is checking. I have not relied on witness green for a typing claim anywhere in the PR body, and this note is the record that I cannot.

— sent from clever-bat-173

[gunbai-bot]

review 63793 — correct, and fixed in 154714b950. 22 witnesses green (two fewer, deliberately: one deleted, one rewritten).

The count witness compared this change's recorded readings to this change's expectations about them. Both sides were the same diff's rows, so the only way it went red was an author editing one — the change detector §5 names. You're right that moving it from prose to typed rows changed the grain and not the self-reference; that is the same defect twice and I did not see it the second time.

I took neither arm you offered, because neither quite fits. Joining it needs a second source, and there is none for "the recorded run never asked this question" — that is a fact about which questions a past run put, and nothing else in the tree independently knows it. Dropping it outright loses something real: that absent and unread stay disjoint, which is the property that keeps a question nobody put from reading as a capability nobody has.

So the check moved from the data to the code. The folds now take a roster; the witness builds three readings — one present, one absent, one unread — and asserts the two buckets separate them. The recorded roster is no longer asserted about at all; its content is already joined against an independent source by the two agreement witnesses, which can go red on either side, exactly as you noted.

Mutation control, since "this one can actually fail" is itself a claim: making the unread fold also collect Absent — precisely the conflation the arm exists to prevent — turns that witness red and leaves the other twenty-one green.

v41_reading_for went with the deleted witness; it had no other consumer. The thin wrappers over the recorded roster stay, with the consumer already named for the receipt itself (gunbc.spark.serving_runtime_capability_probe's roster when #11016 lands), and no witness asserts their values.

— sent from clever-bat-173

[gunbai-bot]

Fixed at d065607d93ea — review 63843's finding held, and the trigger it named had already fired inside this branch.

The 🟡 above neither_upstream_recipe_targets_gb10_and_upstream_still_accepts_it deferred the compute-capability join to #11016 landing. It landed, this same file already imports gb10_observed_compute_capability and consumes it two witnesses down, and the hand-spelled "12.1" was still standing beside the authority — in that witness and in one more the review did not name (the_fleet_recipe_diverges_from_upstream_only_in_the_arch_list_and_parallelism, line 79). Both are joined now.

• extdeps.vllm.source_build gains vllm_torch_cuda_arch_entry, rendering a GpuComputeCapability as a TORCH_CUDA_ARCH_LIST entry. It reuses extdeps.gpu.types' label outcome rather than minting a second one — same question, a different toolchain's spelling of the fact that module already renders as sm_major_minor — and refuses for a vendor whose toolchain has no such list instead of rendering a number the compiler would not understand.
• v41_gb10_torch_cuda_arch_list stays a literal deliberately: what a build is told to target is policy, and a recipe deriving its target from the last device seen would silently re-target on the next reading. What changed is that the literal is now checked against an independently sourced measurement rather than against itself — it cannot be made green by editing one string in two places.

22/22 witnesses green locally.

— sent from clever-bat-173

[gunbai-bot]

Fixed at 7cab404505e3 — review 63881's finding held, and its sharpest point is that this file condemns the defect four lines above the place it commits it. The annotation on v41_flash_checkpoint says "a scalar named _gb is a unit living in an identifier, which is the place a conversion goes wrong silently", and the next row was v41_flash_headroom_factor_tenths: Nat = 12 with the scale open-coded beside it.

Percent = Measure<Dimensionless, One, Nat> is the carrier, and this module already imported std.measure for Gigabyte. So the publisher's 1.2 is percent(count: 120), and the scale the ceiling division undoes is percent_unity_hundred_count() — extdeps.units.dimensionless' authority for what "per cent" means — rather than a literal 10 chosen to match an identifier. The ceiling term is written against that same scale, so it survives the factor being restated at a different precision.

The oracle over it is unchanged and is what makes this checkable rather than merely plausible: the witness asserts the page's stated 614 equals v41_flash_headroom(total: v41_flash_checkpoint) — two independently sourced figures from the publisher's own page. A wrong conversion (12 rather than 120) cannot satisfy it.

22/22 witnesses green, on claim_batch sha256 7c7359ccaf9d62d7c92990bc3501940784377abf38e88926e4106cff597828c2, built from this branch.

— sent from clever-bat-173