diff --git a/ROADMAP.md b/ROADMAP.md index 170c0ff12da..26103478c72 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -71,6 +71,7 @@ The graph has sixteen lanes: SCM compatibility · namespace · P-derive · obser - [ ] **Bring the two new machines up as model-serving members, through the same path as everything else** — Both new machines join through the same standing-up sequence every other machine uses, reach a stated model and runtime, and serve requests on an endpoint. Which model is actually being served, and whether the endpoint is healthy, are read back independently, and each machine stays separately observable. They serve models — they are not job runners and not build caches. Why: Standing these up by hand would create two more machines nobody can answer questions about, which is the state this whole lane exists to end. Going through the convergence path also proves that path on a machine class it has never seen. [authority](dag/gunbc/spark/dgx_procurement.dag) — requires all: fleet-main-revision-authority, fleet-atomic-convergence-verdict, compute-exact-work-contract, fleet-spark-host-enrollment - [ ] **A serving route that has stopped serving stops being routable, and the seats held against it are freed** — Whether a route can take new work is one reading, bound to the endpoint process that answered it, and every consumer projects from that reading rather than deciding separately. The reading covers three ways a route stops serving: the front door does not answer inside a bounded deadline; the endpoint process was replaced between observations; and the engine core is no longer advancing while requests are admitted. Seats and grants held against a launch that is gone become unusable rather than waiting out their term. Why: Group A was dark for 43 hours in September 2026 and nothing in the repository could say so: the health probes had no deadline, so a wedged engine held the roster gate, dispatch selection and placement open with no refusal, and one harness POST was held open for over thirty minutes against a backend that would never answer. A stalled engine still answers its front door, so the route stayed offerable throughout. [authority](dag/gunbc/serving/serving_availability.dag) - [ ] **Serve DeepSeek V4.1 on four Sparks by building the runtime it needs, or say honestly that none exists** — One exact runtime candidate whose identity is fixed before anything is built: the pinned public vLLM source and build recipe, the kernel wheels re-admitted against their acquisition, the ordered engine patches by digest, the checkpoint and tokenizer manifests, which Engram tensors are read from storage and which stay resident, the materialization route, the topology and the device. Identity is not permission: a keyed candidate still owes an exact, time-bounded operator authorization before any host runs it. Why: The published V4.1 runtime cannot serve this checkpoint on four GB10s: gunbc.spark.serving_deployment_selection candidate_component_budget, reading gunbc.spark.pair_serving_observed group_rank_free_memory_observed, excludes resident Engram at TP4 by a measured shortfall on the Engram term after the artifact's other components — not a near miss against device total — and every Engram placement that runtime offers draws on that same pool. Without a candidate identity, a build receipt, a probe and a launch can each be attributed to bytes that produced none of them, which is how a deployment comes to be described by evidence it never generated. [authority](dag/gunbc/spark/v41_runtime_candidate.dag) — requires all: fleet-spark-inference-serving, serving-liveness-route-withdrawal + - [ ] **Cut D D0: suspend Group A under an exact operator consent, read the fleet inside it, settle to one typed terminal, and never free a host early** — gunbc.spark.pair_serving_d0 over gunbc.spark.pair_serving_authority_log on the fabric DB: one durable genesis per group partition, the host-placement partition as the linearization point, prepare -> append-claim -> authority -> finalize as one saga bound by a typed AuthorityWriteIntent, consumed/cancelled exclusivity, claim-bound quiescence evidence, total crash recovery by lifecycle identity, the operator consent slot as a fabric-DB head (gunbc.durable_cas_fabric_storage, generation-bearing objects), and a wet door that refuses while the store's write walls are missing. STATE (2026-09-20, wound down under operator direction to re-prioritize v1 performance and v2 migration): the whole stack is one branch, plan/dsv41-cut-d-2b, re-rooted onto the rewritten main and source-approved through twenty-one review rounds; the operator ruled it may land under the widened fabric-DB principal drop. Its stacked follow-up branch, plan/dsv41-cut-d-2b-admission-cleanup, deletes the twenty transition-admission rows the stack consumed and carries the detached-process wet-lane admission row. Why: Without D0 the V4.1 cut over Group A has no transaction: no consent that is spent exactly once across executors, no state in which the group is neither serving nor being mutated by two writers, and no receipt of the fleet as it was read before the authority moved -- the prior state, in which membership and placement were roster words and a crashed lane left no recoverable trace. [authority](docs/plans/dsv41-cut-d-redesign.md) - [ ] **One answer for the whole fleet, or exactly which machines are not there yet** — Each machine produces a receipt — what was wanted, what was there, what was planned, what was applied, and what was read back afterwards. Those fold into a single answer for the fleet, joined with the lifecycle cells enrollment constructs: every required host-and-phase cell is observed, refused, unreachable, drifted, or converged, and a required cell nobody has observed prevents convergence rather than defaulting green. Anything short of all of them is not converged, with the machines that refused, could not be reached, or have drifted each named separately. Why: One machine can hold the new value while the others hold the old one and nothing anywhere says so, which is exactly why the honest answer to whether something is applied is currently to go and check by hand. And the fleet phase matrix currently has exactly one real observation producer cell — one host, one phase, hard-coded — so every other lifecycle claim in the fleet is unobserved by construction; enrollment makes obligations countable, not satisfied. [authority](docs/plans/fleet-acceptance-criteria.md) - [ ] **Deploy every copy of the dashboard through the one deployment mechanism** — Deploying a dashboard uses the same mechanism as everything else we deploy: work out what should be there, compare it to what is, and change only the difference. Which machine it goes to becomes a parameter, not a second copy of the logic. Why: There are currently two deployment engines for the same three files. The second one does not know what it owns, and reaches the machine outside the checked path — so it can tear down something that was never ours. [authority](docs/plans/gunbc-served-dashboard-design.md) - [ ] **Serve the roadmap page from emitted native code, never the tree-walking interpreter** — The interpreted accept loop is a declared scaffold whose dissolution trigger has now fired in production: a two-hour page wedge, one core pinned in memcpy, thirteen connections queued behind a single thread. The fix is ordered. Wire roadmap_serve_handle through the existing emit-on-demand path FIRST — interpreted concat copies its accumulator on every call, quadratic; emitted concat moves, linear: a complexity-class change, not tuning. Then memoize page bodies by content hash of their inputs so a request is lookup-and-write. Concurrency only after bodies are immutable artifacts. A request deadline returns a typed refusal instead of eating the process. Why: The 2026-08-08 srv1 outage: gunbc serve wedged for two hours rendering the daily page, every route behind it dead, and it re-wedges on the next load of the same shape. The scaffold row already names emit-on-demand as its dissolution and the module carries zero references to it — declared and left standing. Separately, the belt reconcile has been dead as long as GcpProjectId has failed to resolve, burning about two minutes of CPU on every timer tick before exiting. [authority](docs/plans/gunbc-served-dashboard-design.md) diff --git a/dag/extdeps/docker/cli.dag b/dag/extdeps/docker/cli.dag index 33621d02b6a..173252607eb 100644 --- a/dag/extdeps/docker/cli.dag +++ b/dag/extdeps/docker/cli.dag @@ -375,6 +375,30 @@ fn docker_image_inspect_command(image: NonEmptyStr) -> ArgvCommand { // naming "No such container" when the reference names nothing. No --format, for the same // portable-word reason as the image form. The argv and the sudoers Cmnd_Spec that authorizes it are // the same words, so a grantee rendering both reads this one function. +// THE RUNNING CONTAINERS SELECTED BY ONE `docker ps` FILTER, one container id per line (`docker ps +// --filter = --quiet`): empty output is "no such running container". `docker ps` lists +// RUNNING containers only, which is the question a quiescence reading asks; a stopped container is +// not a residual realization of an effect. Two filters this repository asks by: `name=` +// (docker's name filter is an unanchored regular expression, so it matches any running container +// whose name contains the text -- wider than exact, and the wider reading is the safe one for a +// residue question) and `ancestor=` (containers created from that image reference, for an +// effect that starts an unnamed container). `--quiet` rather than a `--format` template because +// the argv crosses an ssh leg whose portable word alphabet has no braces. +type DockerPsFilter + = DockerPsByName { name: NonEmptyStr } + | DockerPsByAncestor { image: NonEmptyStr } + +fn docker_ps_filter_wire(f: DockerPsFilter) -> String { + match f { + DockerPsByName { name: n } => join(["name=", n as String], "") + DockerPsByAncestor { image: i } => join(["ancestor=", i as String], "") + } +} + +fn docker_ps_running_command(filter: DockerPsFilter) -> ArgvCommand { + argv_command(program: docker_binary_path, arguments: ["ps", "--filter", docker_ps_filter_wire(f: filter), "--quiet"]) +} + fn docker_container_inspect_command(name: NonEmptyStr) -> ArgvCommand { argv_command(program: docker_binary_path, arguments: ["container", "inspect", name as String]) } diff --git a/dag/extdeps/exec/command.dag b/dag/extdeps/exec/command.dag index dc230f8fb45..c298aef2d9c 100644 --- a/dag/extdeps/exec/command.dag +++ b/dag/extdeps/exec/command.dag @@ -79,6 +79,7 @@ fn argv_command(program: NonEmptyStr, arguments: List) -> ArgvCommand decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_exec_command"), decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_image_pull_command"), decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_container_inspect_command"), + decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_ps_running_command"), decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_build_command"), decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_build_spec_command"), decl_ref(module_path: "extdeps.docker.cli", decl_name: "docker_image_digest_command"), diff --git a/dag/extdeps/http/client.dag b/dag/extdeps/http/client.dag index c9d9b40f6de..43596819376 100644 --- a/dag/extdeps/http/client.dag +++ b/dag/extdeps/http/client.dag @@ -57,6 +57,16 @@ fn http_client_get_argv(url: NonEmptyStr, max_time: Second) -> List { // 5 rules that a bounded "forever" is not an "unknown" error. Measured on this deployment, a wedged // backend held one harness POST open for over thirty minutes while the router answered an unrelated // probe in 0.15ms -- so the turn was neither progressing nor failing, and no arm existed to say so. +// StatusWithin -- A STATUS IS AN ANSWER; ONLY NO STATUS IS SILENCE. Every GET carries -f, which +// turns a 4xx or 5xx into a failed operation -- right for a caller that wants a body, wrong for a +// caller asking whether anything is THERE: a front door returning 401 or 500 answered. That +// operation asks only for the status line, within the caller's deadline, and fails only when no +// HTTP response was received at all -- and WHY is carried as curl's exit code, because "no +// response" is not one fact: a failed connect (curl 7) and a deadline (curl 28) are different +// readings for a caller asking whether anything is THERE, and neither proves absence on its own. +// The consumer maps the code; the transport reports it. The codes are curl's own facts and live +// with the tool authority: extdeps.tools.curl curl_exit_outcome. +// // PostStdinWithin, A BOUNDED POST WHOSE TRANSPORT OUTCOME IS KEPT, NOT FOLDED INTO SUCCESS. The request body goes // on stdin (curl --data-binary @-), so no body byte passes through argv or a shared file. Without // -f, an HTTP error status is an ANSWER (exit 0), and the status is appended as the final line by @@ -142,6 +152,22 @@ service http.Client { } } + operation StatusWithin { + input { url: NonEmptyStr, max_seconds: NonEmptyStr } + output { status: String from "stdout", success: Bool from "exit_success", exit_code: Int from "exit_code" } + readonly + transport shell { + argv: ["curl", "-sS", "-o", "/dev/null", "-w", "%\{http_code}", "--max-time", "{max_seconds}", "{url}"] + } + exit { + 0 => Unit + nonzero => String "http client status GET within a caller deadline received no response" + } + mock_response { + 0 => { status: "", success: false, exit_code: 28 } "hermetic: no live HTTP endpoint; a bounded presence reading refuses rather than fabricate a status" + } + } + operation PostStdinWithin { input { url: NonEmptyStr, request_body: String, connect_seconds: NonEmptyStr, max_seconds: NonEmptyStr } output { diff --git a/dag/extdeps/linux/proc_net_tcp.dag b/dag/extdeps/linux/proc_net_tcp.dag new file mode 100644 index 00000000000..630206d1be2 --- /dev/null +++ b/dag/extdeps/linux/proc_net_tcp.dag @@ -0,0 +1,103 @@ +module extdeps.linux.proc_net_tcp + +import std.types { NonEmptyStr, List, Bool, Int, Port } +import std.string_type { String } +import v2.std.optional { Present, Absent } +import std.algebra { trim } +import extdeps.numeric.base16 { base16_word_value } +import extdeps.external_authority { ExternalAuthority } +import extdeps.uri { Uri, Https } + +// THE SHAPE OF /proc/net/tcp AND /proc/net/tcp6 (proc(5), "/proc/net/tcp"): one header line, then +// one row per socket, whitespace-separated. The fields this module reads are the second +// (local_address: hex IP, a colon, a 4-hex-digit port) and the fourth (st: the socket state as two +// hex digits, where 0A is TCP_LISTEN -- include/net/tcp_states.h). Everything else on the row is +// carried past. The port is the same 16-bit hex field in both files; only the address width +// differs (8 hex digits for v4, 32 for v6), and this parser reads the port from the right of the +// colon so the width is not its business. +// +// WHAT THIS ESTABLISHES: which local ports have a socket in LISTEN state on the host whose procfs +// was read, whatever bound them. It does not say which process owns the socket (that is the inode +// joined through /proc//fd, a separate read) and it says nothing about a listener in another +// network namespace -- a container with its own netns has its own /proc/net/tcp. A consumer that +// needs "nothing listens on this port on this host" reads both files and, when containers may +// hold the port, the host's namespace is the one the enrolled address reaches. +data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { + uri: Uri { + scheme: Https + locator: "man7.org/linux/man-pages/man5/proc.5.html#/proc/net/tcp" + } +} + +data tcp_listen_state_hex: String = "0A" + +// THE LOCAL PORT IS std.types Port WHERE ONE IS BOUND. The field is a 4-hex-digit 16-bit word; +// 0000 is a real value the kernel prints for a socket with no bound port, and Port (1..65535) has +// no constructor for it, so the row says which it read rather than carrying 0 as a port. +type TcpLocalPort + = TcpPortBound { port: Port } + | TcpPortUnbound + +type TcpSocketRow { + local_address: String + local_port: TcpLocalPort + state: String +} + +type ProcNetTcpTable + = ProcNetTcpParsed { rows: List } + | ProcNetTcpUnparseable { line: String } + +fn tcp_row_fields(line: String) -> List { + filter(split(s: trim(s: line), delimiter: " "), f => f != "") +} + +fn tcp_socket_row(line: String) -> TcpSocketRow? { + let fields = tcp_row_fields(line: line) + match get(xs: fields, index: 1) { + Absent => none + Present { value: local } => + match get(xs: fields, index: 3) { + Absent => none + Present { value: st } => { + let parts = split(s: local, delimiter: ":") + if length(parts) != 2 { none } else { + match get(xs: parts, index: 1) { + Absent => none + Present { value: port_hex } => + match base16_word_value(word: port_hex, max_digits: 4) { + Absent => none + Present { value: port } => Present { value: TcpSocketRow { local_address: local, local_port: if port == 0 { TcpPortUnbound } else { TcpPortBound { port: port } }, state: st } } + } + } + } + } + } + } +} + +// The table: the header line is skipped by its first field ("sl"), blank lines are skipped, and +// a row that does not carry the two fields this module reads refuses the whole table naming the +// line -- a socket table with an unreadable row is not a table with fewer sockets. +fn proc_net_tcp_table(text: String) -> ProcNetTcpTable { + let lines = filter(split(s: text, delimiter: "\n"), l => trim(s: l) != "") + fold(lines, init: ProcNetTcpParsed { rows: [] as List }, f: (acc, line) => + match acc { + ProcNetTcpUnparseable { line: _ } => acc + ProcNetTcpParsed { rows: rs } => + match get(xs: tcp_row_fields(line: line), index: 0) { + Absent => acc + Present { value: first } => + if first == "sl" { acc } else { + match tcp_socket_row(line: line) { + Absent => ProcNetTcpUnparseable { line: line } + Present { value: r } => ProcNetTcpParsed { rows: concat(rs, [r]) } + } + } + } + }) +} + +fn tcp_rows_listening_on(rows: List, port: Port) -> List { + filter(rows, r => r.state == tcp_listen_state_hex && (match r.local_port { TcpPortBound { port: p } => p == port TcpPortUnbound => false })) +} diff --git a/dag/extdeps/linux/procfs.dag b/dag/extdeps/linux/procfs.dag index 45864ec1d14..7e2721ffc88 100644 --- a/dag/extdeps/linux/procfs.dag +++ b/dag/extdeps/linux/procfs.dag @@ -25,6 +25,8 @@ type ProcfsPath | ProcSelfStatus | ProcSelfCgroup | ProcNetUnix + | ProcNetTcp + | ProcNetTcp6 | ProcSwaps | ProcSysKernelHostname | ProcNetPnp @@ -36,6 +38,8 @@ fn procfs_path_literal(p: ProcfsPath) -> NonEmptyStr { ProcSelfStatus => "/proc/self/status" ProcSelfCgroup => "/proc/self/cgroup" ProcNetUnix => "/proc/net/unix" + ProcNetTcp => "/proc/net/tcp" + ProcNetTcp6 => "/proc/net/tcp6" ProcSwaps => "/proc/swaps" ProcSysKernelHostname => "/proc/sys/kernel/hostname" ProcNetPnp => "/proc/net/pnp" @@ -121,6 +125,40 @@ service linux.Procfs { } } + operation ReadNetTcp { + input {} + output { + value: String from "stdout" + success: Bool from "exit_success" + } + readonly + transport shell { argv: ["cat", "/proc/net/tcp"] } + exit { + 0 => Unit + nonzero => String "/proc/net/tcp read failed" + } + mock_response { + 0 => { value: " sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n", success: true } "hermetic linux.Procfs.ReadNetTcp: a socket table with no rows" + } + } + + operation ReadNetTcp6 { + input {} + output { + value: String from "stdout" + success: Bool from "exit_success" + } + readonly + transport shell { argv: ["cat", "/proc/net/tcp6"] } + exit { + 0 => Unit + nonzero => String "/proc/net/tcp6 read failed" + } + mock_response { + 0 => { value: " sl local_address remote_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode\n", success: true } "hermetic linux.Procfs.ReadNetTcp6: a socket table with no rows" + } + } + operation ReadPidCgroup { input { pid: NonEmptyStr } output { @@ -185,6 +223,8 @@ fn procfs_path_provision(p: ProcfsPath) -> ProcfsPathProvision { ProcSwaps => ProvisionNotYetQualified ProcSysKernelHostname => ProvisionNotYetQualified ProcNetPnp => ProvidedWhenConfigured { option: "CONFIG_IP_PNP" as NonEmptyStr } + ProcNetTcp => ProvisionNotYetQualified + ProcNetTcp6 => ProvisionNotYetQualified } } diff --git a/dag/extdeps/numeric/base16.dag b/dag/extdeps/numeric/base16.dag index bbb29af6f74..a26434b9bd5 100644 --- a/dag/extdeps/numeric/base16.dag +++ b/dag/extdeps/numeric/base16.dag @@ -158,3 +158,31 @@ fn base16_decode_lower(hex: String) -> List? { base16_decode_lower_from(hex: hex, i: 0, len: len, acc: []) } } + +// A DIGIT'S VALUE IN EITHER CASE, over the same rows: the consumers below read fields the kernel +// prints in uppercase (/proc/net/tcp) or a wire that admits either spelling, and refuse a code +// point that is in neither column. +fn base16_digit_value_at(cp: Int) -> Int? { + fold(base16_digit_rows, init: none, f: (acc, row) => + if code_point(row.lower) == cp || code_point(row.upper) == cp { Present { value: row.value } } else { acc }) +} + +// A BOUNDED HEXADECIMAL WORD AS ONE INTEGER: one to `max_digits` digits in either case, most +// significant first, refused when empty, longer than the bound, or holding a non-digit. The bound +// is the consumer's field width -- four digits for a 16-bit port, six for a 24-bit run id -- and +// is capped at fifteen so the value stays inside Int; a consumer that needs a wider field is a +// different operation (base16_decode_lower yields octets). +fn base16_word_value(word: String, max_digits: Int) -> Int? { + let cps = word |> chars + if length(cps) == 0 || length(cps) > max_digits || max_digits > 15 { none } else { + fold(cps, init: Present { value: 0 }, f: (acc, cp) => + match acc { + Absent => none + Present { value: v } => + match base16_digit_value_at(cp: cp) { + Absent => none + Present { value: d } => Present { value: v * 16 + d } + } + }) + } +} diff --git a/dag/gunbc/boot_artifact_delivery.dag b/dag/gunbc/boot_artifact_delivery.dag index 40091f59040..0b0f1f14590 100644 --- a/dag/gunbc/boot_artifact_delivery.dag +++ b/dag/gunbc/boot_artifact_delivery.dag @@ -2,7 +2,7 @@ module gunbc.boot_artifact_delivery import std.types { Bool, EpochMs, Int, List, NonEmptyStr, String, Port, list_length } import std.decl_ref { DeclarationRef, WholeDeclaration } -import std.content_hash { ContentHash } +import std.content_hash { ContentHash, content_hash_equal } import extdeps.uri { Uri } import product.placement_supply { SiteIdentity } import extdeps.provisioning.dhcp_client_arch { DhcpProcessorArchitecture } @@ -91,7 +91,7 @@ import gunbc.boot_artifact { OperatingSystemInstaller, intake_artifact_for, } -import gunbc.machine_intake_subject { MachineIntakeSubject, content_hash_equal } +import gunbc.machine_intake_subject { MachineIntakeSubject } import gunbc.machine_intake_receipt { EvidenceRef, same_intake_subject } import gunbc.install_transport_qualification { ConfigfsTransportRequirement, diff --git a/dag/gunbc/ci/ci_layer_roots.dag b/dag/gunbc/ci/ci_layer_roots.dag index 82c0ade5378..5da6fa4460b 100644 --- a/dag/gunbc/ci/ci_layer_roots.dag +++ b/dag/gunbc/ci/ci_layer_roots.dag @@ -378,6 +378,10 @@ data excl_local_repo_wet_bash_materialized_reason: String = "real host-effect ex data excl_local_repo_wet_bash_materialized_dissolve: DissolutionCondition = unbound_dissolution(description: "mock_response coverage lands for shell.Exec.Run over a sealed transport script and the emitted-bytes assertions are re-checked under a fixed mock -- which the transport_script_stdin_byte_fidelity note already refuses as vacuous for a byte-fidelity claim -- OR the bin-witness wet class regains an executing per-PR consumer; then these fns re-enroll there, drop off local_repo_wet_schedule, and this row deletes") +data excl_local_repo_wet_executor_identity_reason: String = "real host-effect execution witness whose effects are the EXECUTOR'S OWN IDENTITY AND EVENT-LOG READ: the instrument entry reads the executing host's hostname (shell.Exec.RunArgv, no mock_response), resolves that host's event log layout, and reads each claimed group's serving authority from the log before judging any standing (gunbc.instruments.fabric_capacity_standing fabric_capacity_standing over gunbc.spark.pair_serving_authority_log). No temporary repository is built and nothing is written; on a runner that is no dashboard host the read refuses at the layout step and the entry reports every group unread, which is the route the claim asserts. It is the wet half of the pairing obligation whose hermetic half is test.claim.spark.fabric_capacity_standing_witness over a supplied roster." + +data excl_local_repo_wet_executor_identity_dissolve: DissolutionCondition = unbound_dissolution(description: "mock_response coverage lands for shell.Exec.RunArgv and the event-log head read, so the instrument entry can be driven hermetically with a fixed executor and a fixed partition head; then the claim re-enrolls as an ordinary hermetic discovery row, drops off local_repo_wet_schedule, and this row deletes") + data excl_local_repo_wet_host_probe_reason: String = "real host-effect execution witness whose ONLY effect is a READ-ONLY HOST PATH PROBE (shell.PosixCommandV.Check, i.e. command -v) -- refused by the hermetic envelope (no mock_response for operation Check), so excluded from the discovery corpus and executed by the required floor's local-repo wet lane. THIS IS A DISTINCT ADMITTED EFFECT, NOT THE THROWAWAY-REPOSITORY PROPERTY excl_local_repo_wet_reason NAMES: these fns build no temporary repository and write nothing. What they share with that class, and what admits them to the same lane, is the negative half -- no network, no cargo, no remote host, no install media. THE VERDICT IS HOST-INDEPENDENT ON THE TOOL-PRESENCE AXIS, MEASURED RATHER THAN ASSUMED (2026-09-02, PR #10055): every arm of both fns returns the literal true, and the npm fn was executed on one host in both conditions -- probe found, and probe exit=127 with sh still resolvable -- returning the same verdict in each. So admitting them imports no host-conditional red. The residual red these fns CAN produce is route-unreachability (no shell at all: TypeError failed to execute sh), which is the same reachability risk every member of this lane already carries and is not specific to the probed tool. PROVENANCE CARRIED FORWARD from the bin-wet row this replaces: falsifier Codex materialization reuses the same PosixCommandV.Check.exists path (migrated off shell.Which.Check, crisp-wren-896, PR #8590)." data excl_local_repo_wet_host_probe_dissolve: DissolutionCondition = unbound_dissolution(description: "mock_response coverage lands for shell.PosixCommandV.Check, so observe_host_cli_dependency can be asserted under a fixed mock; then these fns re-enroll as ordinary hermetic discovery rows, drop off local_repo_wet_schedule, and this row deletes") @@ -932,6 +936,36 @@ data witness_exclusion_frontier: List = [ classification: LocalRepoWetLane, reason: excl_local_repo_wet_tempdir_write_reason, dissolution: excl_local_repo_wet_dissolve}, + WitnessExclusionRow { + pattern: "pair_serving_authority_log_real_execution_witness_test.dag", + classification: LocalRepoWetLane, + reason: excl_local_repo_wet_reason, + dissolution: excl_local_repo_wet_dissolve}, + WitnessExclusionRow { + pattern: "pair_serving_d0_real_execution_witness_test.dag", + classification: LocalRepoWetLane, + reason: excl_local_repo_wet_reason, + dissolution: excl_local_repo_wet_dissolve}, + WitnessExclusionRow { + pattern: "durable_cas_fabric_storage_real_execution_witness_test.dag", + classification: LocalRepoWetLane, + reason: excl_local_repo_wet_reason, + dissolution: excl_local_repo_wet_dissolve}, + WitnessExclusionRow { + pattern: "pair_serving_d0_front_door_real_execution_witness_test.dag", + classification: LocalRepoWetLane, + reason: excl_local_repo_wet_reason, + dissolution: excl_local_repo_wet_dissolve}, + WitnessExclusionRow { + pattern: "fabric_capacity_standing_wet_witness_test.dag", + classification: LocalRepoWetLane, + reason: excl_local_repo_wet_executor_identity_reason, + dissolution: excl_local_repo_wet_executor_identity_dissolve}, + WitnessExclusionRow { + pattern: "spark_pair_serving_apply_wet_witness_test.dag", + classification: LocalRepoWetLane, + reason: excl_local_repo_wet_executor_identity_reason, + dissolution: excl_local_repo_wet_executor_identity_dissolve}, WitnessExclusionRow { pattern: "devboot_text_blob_real_execution_witness_test.dag", classification: LocalRepoWetLane, diff --git a/dag/gunbc/cloudflare/r2_permission_group_observe.dag b/dag/gunbc/cloudflare/r2_permission_group_observe.dag index f0573138535..47e52e184b7 100644 --- a/dag/gunbc/cloudflare/r2_permission_group_observe.dag +++ b/dag/gunbc/cloudflare/r2_permission_group_observe.dag @@ -221,7 +221,7 @@ fn r2_bucket_item_write_permission_group() -> PermissionGroupRef { // listing now has an executing route -- gunbc.fleet_converge_workflow's r2_mint_preflight mode // runs observe_account_permission_groups_wet and folds its standing into a process exit and an // uploaded receipt. Half of the original gap therefore closed. - // +// // THE HALF THAT REMAINS IS THE ONE THE ROW IS NOW ABOUT: the rows below are still TRANSCRIBED // from a run rather than derived by one. An executing caller that discards its own listing into // a text artifact, while the ids a consumer actually reads are hand-authored constants beside diff --git a/dag/gunbc/fabric/durable_cas_fabric_storage.dag b/dag/gunbc/fabric/durable_cas_fabric_storage.dag new file mode 100644 index 00000000000..d66425f817e --- /dev/null +++ b/dag/gunbc/fabric/durable_cas_fabric_storage.dag @@ -0,0 +1,192 @@ +module gunbc.durable_cas_fabric_storage + +import std.types { String, Bool, Int, NonEmptyStr, List } +import std.nat { Nat } +import v2.std.optional { Present, Absent } +import std.content_hash { ContentHash, serialize_content_hash, parse_content_hash } +import std.durable_compare_and_set { + CasGeneration, CasExpectation, ExpectSlotAbsent, ExpectSlotGeneration, + CasReadableSlot, CasReadableAbsent, CasReadablePresent, CasSlotVersion, + CasSlotObservation, CasObservedReadable, CasObservedUnreadable, + CasUnreadableSlot, CasUnreadableMalformed, CasUnreadableReadRefused, CasUnreadableObservationBoundExceeded, + CasOutcome, CasCommitted, CasPreconditionFailed, CasStoreRefused, + CasSlotObservationRefused, CasGenerationPublicationRefused, + cas_generation_count, cas_generation_next, cas_generation_first, +} +import gunbc.durable_cas_file_store { VerifiedCasAttempt } +import extdeps.languages.json.emit { JsonValue, json_kv, json_string, json_int, json_object, serialize_json } +import extdeps.languages.json.parse { JsonDocumentParsed, JsonDocumentUnreadable, parse_json_document } +import product.capacity.event_json { member_string, member_nonempty, member_nat } +import std.fabric_storage { + FabricObject, FabricObjectRef, FabricStoredObject, fabric_storage_fault_wire, + FabricHeadAbsent, FabricHeadAt, ExpectHeadAbsent, ExpectHeadAt, FabricHeadExpectation, + FabricObjectStored, FabricPutRefused, FabricHeadAdvanced, FabricHeadMoved, FabricHeadAdvanceRefused, + FabricClosureRead, FabricClosureBudgetExhausted, FabricClosureRefused, +} +import gunbc.fabric_storage_client { FabricStorageBinding, fabric_storage_put, fabric_storage_advance, fabric_storage_closure } + +// ── A SECOND REALIZATION OF THE DURABLE COMPARE-AND-SET: ONE SLOT IS ONE FABRIC-DB HEAD ──────── +// +// std.durable_compare_and_set is a shape, and gunbc.durable_cas_file_store is one handler bound to +// it: an exclusive create under a root on ONE filesystem, whose own authority says its guarantee +// holds only for writers addressing the same store instance. A claim that has to be single-use +// across the FLEET -- an operator's consent to a destructive transaction, consumed once whichever +// executor runs it -- cannot live there: two hosts have two roots, and each sees the other's slot +// absent. The fabric DB (std.fabric_storage, one placed store every host reaches through +// gunbc.fabric_storage_client) is exactly the fleet-wide linearization such a claim needs: a named head +// advanced by a compare-and-set on the expected object, whose generation the store itself keeps. +// So this module binds the same CAS shape to a head. DESIGN §3: transport is a realization handler +// bound to the shape, one of N, never a fact about the operation -- the attempt a caller builds is +// the same CasAttempt the file store takes, and the caller chooses the store. +// +// THE MAPPING. The slot key names the head `cas-slot-`; every committed generation is one +// link-free object whose body carries the value, its content digest AND THE GENERATION IT LANDS +// (the one the writer derived from its observation), so the object's identity is never the value +// alone: a value that cycles A -> B -> A lands three distinct objects, and a writer that observed +// A at generation 1 cannot advance against the A at generation 3 -- the head's expected-object +// compare-and-set would otherwise collapse to value identity. The slot's generation is the head's +// own (the store's compare-and-set generation), the object's carried generation must equal it on +// every read, and ExpectSlotAbsent / ExpectSlotGeneration are an advance against an absent head / +// against the object the observed generation holds -- so the store's head decides the race, and a +// writer that observed generation N while another landed N+1 is FabricHeadMoved, re-observed so +// the losing side reports what is now there. An unreadable head is CasStoreRefused, never an +// absent slot. +// +// THE DIGEST OBLIGATION IS DISCHARGED AT THE SAME MINT. The interface says the immutable object +// store that holds the bytes verifies bytes against digest; the file store's admit_cas_attempt is +// that mint and this store takes only its VerifiedCasAttempt, so one admission serves both +// handlers rather than each re-spelling the comparison. The head name is the key under one prefix +// and no separator, the one predicate the file store's slot addressing asks. + +data cas_slot_head_prefix: String = "cas-slot-" + +fn cas_slot_head(key: NonEmptyStr) -> NonEmptyStr { + join([cas_slot_head_prefix, key as String], "") as NonEmptyStr +} + +// The generation object: the value, the content digest the attempt was admitted with, and the +// generation it lands. +fn cas_slot_object(value: NonEmptyStr, content: ContentHash, generation: CasGeneration) -> FabricObject { + FabricObject { links: [] as List, body: serialize_json(v: json_object(members: [ + json_kv(key: "value", value: json_string(s: value as String)), + json_kv(key: "content", value: json_string(s: serialize_content_hash(hash: content) as String)), + json_kv(key: "generation", value: json_int(n: cas_generation_count(g: generation))), + ])) as NonEmptyStr } +} + +type CasSlotBody + = CasSlotBodyDecoded { value: NonEmptyStr, content: ContentHash, generation: Int } + | CasSlotBodyMalformed { detail: String } + +fn cas_slot_body_decode(text: String) -> CasSlotBody { + match parse_json_document(s: text) { + JsonDocumentUnreadable { gap: _ } => CasSlotBodyMalformed { detail: "the slot object is not a JSON document" } + JsonDocumentParsed { value: doc } => + match member_nonempty(doc: doc, key: "value") { + Absent => CasSlotBodyMalformed { detail: "the slot object carries no value" } + Present { value: v } => + match member_string(doc: doc, key: "content") { + Absent => CasSlotBodyMalformed { detail: "the slot object carries no content digest" } + Present { value: c } => + match parse_content_hash(wire: c) { + Absent => CasSlotBodyMalformed { detail: "the slot object's content digest is not a content-hash wire" } + Present { value: h } => + match member_nat(doc: doc, key: "generation") { + Absent => CasSlotBodyMalformed { detail: "the slot object carries no generation" } + Present { value: g } => CasSlotBodyDecoded { value: v, content: h, generation: g } + } + } + } + } + } +} + +// ── THE OBSERVATION ──────────────────────────────────────────────────────────────────────────── + +// What the head holds, with the object the next advance must expect. Kept beside the readable +// slot rather than folded into it because the interface's CasReadableSlot deliberately carries +// no store position -- the head's object is this realization's fact. +type CasSlotHead + = CasSlotHeadAbsent + | CasSlotHeadAt { object: FabricObjectRef, version: CasSlotVersion } + | CasSlotHeadUnreadable { cause: CasUnreadableSlot } + +fn fabric_storage_observe_cas_slot_head(store: FabricStorageBinding, key: NonEmptyStr) -> CasSlotHead { + match fabric_storage_closure(binding: store, name: cas_slot_head(key: key), bound: 1) { + FabricClosureRefused { fault: f } => CasSlotHeadUnreadable { cause: CasUnreadableReadRefused { detail: fabric_storage_fault_wire(fault: f) as NonEmptyStr } } + FabricClosureBudgetExhausted { at: _, bound: _ } => CasSlotHeadUnreadable { cause: CasUnreadableObservationBoundExceeded { bound: 1 } } + FabricClosureRead { reading: r, newest_first: xs } => + match r { + FabricHeadAbsent => CasSlotHeadAbsent + FabricHeadAt { object: o, generation: g } => + match first(xs) { + Absent => CasSlotHeadUnreadable { cause: CasUnreadableMalformed { detail: "the head names an object the closure did not return" as NonEmptyStr } } + Present { value: stored } => + match cas_slot_body_decode(text: stored.content.body as String) { + CasSlotBodyMalformed { detail: d } => CasSlotHeadUnreadable { cause: CasUnreadableMalformed { detail: d as NonEmptyStr } } + CasSlotBodyDecoded { value: v, content: c, generation: og } => + if og != cas_generation_count(g: g) { CasSlotHeadUnreadable { cause: CasUnreadableMalformed { detail: join(["the head is at generation ", to_string(cas_generation_count(g: g)), " but its object was written for generation ", to_string(og)], "") as NonEmptyStr } } } + else { CasSlotHeadAt { object: o, version: CasSlotVersion { generation: g, content: c, value: v } } } + } + } + } + } +} + +fn fabric_storage_observe_cas_slot(store: FabricStorageBinding, key: NonEmptyStr) -> CasSlotObservation { + match fabric_storage_observe_cas_slot_head(store: store, key: key) { + CasSlotHeadUnreadable { cause: c } => CasObservedUnreadable { cause: c } + CasSlotHeadAbsent => CasObservedReadable { readable: CasReadableAbsent } + CasSlotHeadAt { object: _, version: v } => CasObservedReadable { readable: CasReadablePresent { version: v } } + } +} + +// ── THE COMPARE-AND-SET ──────────────────────────────────────────────────────────────────────── +// +// THE EXPECTATION IS COMPARED TO WHAT THE STORE HOLDS, and the advance then names the observed +// object; the store's head decides the race. A moved head re-observes so the losing side reports +// what is now there; the committed generation is the one the store reports, never a prediction. +fn fabric_storage_compare_and_set(store: FabricStorageBinding, verified: VerifiedCasAttempt) -> CasOutcome { + let attempt = verified.attempt + match fabric_storage_observe_cas_slot_head(store: store, key: attempt.key) { + CasSlotHeadUnreadable { cause: c } => CasStoreRefused { cause: CasSlotObservationRefused { cause: c } } + CasSlotHeadAbsent => + match attempt.expected { + ExpectSlotGeneration { generation: _ } => CasPreconditionFailed { expected: attempt.expected, observed: CasReadableAbsent } + ExpectSlotAbsent => cas_slot_advance(store: store, verified: verified, expected: ExpectHeadAbsent, target: cas_generation_first()) + } + CasSlotHeadAt { object: o, version: v } => + match attempt.expected { + ExpectSlotAbsent => CasPreconditionFailed { expected: attempt.expected, observed: CasReadablePresent { version: v } } + ExpectSlotGeneration { generation: g } => + if cas_generation_count(g: g) == cas_generation_count(g: v.generation) { + cas_slot_advance(store: store, verified: verified, expected: ExpectHeadAt { object: o }, target: cas_generation_next(g: v.generation)) + } else { + CasPreconditionFailed { expected: attempt.expected, observed: CasReadablePresent { version: v } } + } + } + } +} + +fn cas_slot_advance(store: FabricStorageBinding, verified: VerifiedCasAttempt, expected: FabricHeadExpectation, target: CasGeneration) -> CasOutcome { + let attempt = verified.attempt + match fabric_storage_put(binding: store, object: cas_slot_object(value: attempt.proposed, content: attempt.proposed_content, generation: target)) { + FabricPutRefused { fault: f } => CasStoreRefused { cause: CasGenerationPublicationRefused { detail: fabric_storage_fault_wire(fault: f) as NonEmptyStr } } + FabricObjectStored { object: stored } => + match fabric_storage_advance(binding: store, name: cas_slot_head(key: attempt.key), expected: expected, target: stored) { + FabricHeadAdvanced { object: _, generation: g } => + if cas_generation_count(g: g) != cas_generation_count(g: target) { + CasStoreRefused { cause: CasGenerationPublicationRefused { detail: join(["the store advanced the head to generation ", to_string(cas_generation_count(g: g)), ", not the ", to_string(cas_generation_count(g: target)), " this writer derived from its observation"], "") as NonEmptyStr } } + } else { + CasCommitted { committed: CasSlotVersion { generation: g, content: attempt.proposed_content, value: attempt.proposed } } + } + FabricHeadAdvanceRefused { fault: f } => + CasStoreRefused { cause: CasGenerationPublicationRefused { detail: fabric_storage_fault_wire(fault: f) as NonEmptyStr } } + FabricHeadMoved { expected: _, observed: _ } => + match fabric_storage_observe_cas_slot(store: store, key: attempt.key) { + CasObservedUnreadable { cause: c } => CasStoreRefused { cause: CasSlotObservationRefused { cause: c } } + CasObservedReadable { readable: r } => CasPreconditionFailed { expected: attempt.expected, observed: r } + } + } + } +} diff --git a/dag/gunbc/fabric/fabric_event_log.dag b/dag/gunbc/fabric/fabric_event_log.dag index 5c3f55ff6bd..5225efab78a 100644 --- a/dag/gunbc/fabric/fabric_event_log.dag +++ b/dag/gunbc/fabric/fabric_event_log.dag @@ -20,9 +20,10 @@ import product.capacity.pool { Pool, PoolReading, PoolRead, PoolReadingRefused, import product.capacity.event_chain { PartitionId, EventId, ChainEvent, ChainEnvelope, HeadExpectation, HeadAbsent, HeadAt, head_expectation_eq, event_parent_expectation, AppendDecision, AppendAdmitted, AppendStale, ChainWalk, ChainWalked, ChainIncomplete, ChainBudgetExhausted, chain_from_head, + EventDecode, EventDecoded, EventUndecodable, } import product.capacity.pool_events { - PoolEvent, pool_event_wire_text, pool_event_decode, PoolEventDecode, PoolEventDecoded, PoolEventUndecodable, + PoolEvent, pool_event_wire_text, pool_event_decode, PoolFold, PoolFolded, PoolFoldRefused, pool_fold, SeatRequest, SeatProposal, SeatProposed, SeatRefused, propose_acquire, grant_from_admission, } import product.capacity.lease { LeasePolicy, LeaseGrant, release_law_eq, release_law_wire } @@ -78,8 +79,8 @@ type EventObject = EventObjectBuilt { object: FabricObject } | EventObjectRefused { cause: EventLogRefusal } -fn event_object(event: ChainEvent) -> EventObject { - let body = pool_event_wire_text(event: event) +fn event_object

(event: ChainEvent

, encode: fn(ChainEvent

) -> String) -> EventObject { + let body = encode(event) if body == "" { EventObjectRefused { cause: EventLogEventEmpty } } else { @@ -122,33 +123,44 @@ fn event_log_observe_head(store: FabricStorageBinding, partition: PartitionId) - // READING A PARTITION: one closure from the head, each held object decoded as an event, and the // collected envelopes handed to the pure walk so the chain's shape is decided by one authority. -type PartitionRead - = PartitionReadOk { head: HeadExpectation, walk: ChainWalk } +// THE ONE READ BUDGET FOR A PARTITION CLOSURE: how many events a single read of any partition may +// walk before it refuses as ChainBudgetExhausted. Owned here, beside the read it bounds, and +// consumed by every reader of the log (the seat and quota transactions through the pool-event +// read, the pair-serving authority and host-placement folds, the partition-read instrument), so a +// bound that moves moves once. +fn event_log_read_budget() -> Nat { 4096 } + +// THE LOG IS ONE CARRIER FOR EVERY PARTITION KIND. A partition's payload type and its codec are +// the partition owner's (pool events here; the pair-serving authority and host placement in +// gunbc.spark.pair_serving_authority_log), so the read and the append take the codec as a value: +// one store walk, one put-and-advance, N payload types -- never a second log per kind. +type PartitionRead

+ = PartitionReadOk { head: HeadExpectation, walk: ChainWalk

} | PartitionReadRefused { cause: EventLogRefusal } -type DecodeState { - envs: List> +type DecodeState

{ + envs: List> refused: EventLogRefusal? } -fn decode_envelopes(objects: List) -> DecodeState { - fold(objects, init: DecodeState { envs: [], refused: none }, f: (acc, s) => +fn decode_envelopes

(objects: List, decode: fn(String) -> EventDecode

) -> DecodeState

{ + fold(objects, init: DecodeState { envs: [] as List>, refused: none }, f: (acc, s) => match acc.refused { Present { value: _ } => acc Absent => - match pool_event_decode(text: s.content.body as String) { - PoolEventUndecodable { reason: why } => DecodeState { envs: acc.envs, refused: Present { value: EventLogEventUndecodable { id: event_id_of(object: s.object), reason: why } } } - PoolEventDecoded { event: e } => DecodeState { envs: concat(acc.envs, [ChainEnvelope { id: event_id_of(object: s.object), event: e }]), refused: none } + match decode(s.content.body as String) { + EventUndecodable { reason: why } => DecodeState { envs: acc.envs, refused: Present { value: EventLogEventUndecodable { id: event_id_of(object: s.object), reason: why } } } + EventDecoded { event: e } => DecodeState { envs: concat(acc.envs, [ChainEnvelope { id: event_id_of(object: s.object), event: e }]), refused: none } } }) } -fn event_log_read_partition(store: FabricStorageBinding, partition: PartitionId, budget: Nat) -> PartitionRead { +fn event_log_read_partition_with

(store: FabricStorageBinding, partition: PartitionId, budget: Nat, decode: fn(String) -> EventDecode

) -> PartitionRead

{ match fabric_storage_closure(binding: store, name: partition as String as NonEmptyStr, bound: budget) { FabricClosureRefused { fault: f } => PartitionReadRefused { cause: EventLogStoreFault { fault: f } } FabricClosureBudgetExhausted { at: a, bound: _ } => PartitionReadOk { head: HeadAt { id: event_id_of(object: a) }, walk: ChainBudgetExhausted { at: event_id_of(object: a) } } FabricClosureRead { reading: r, newest_first: xs } => { - let d = decode_envelopes(objects: xs) + let d = decode_envelopes(objects: xs, decode: decode) match d.refused { Present { value: c } => PartitionReadRefused { cause: c } Absent => PartitionReadOk { head: log_head(reading: r), walk: chain_from_head(envs: d.envs, head: log_head(reading: r), budget: budget) } @@ -157,6 +169,10 @@ fn event_log_read_partition(store: FabricStorageBinding, partition: PartitionId, } } +fn event_log_read_partition(store: FabricStorageBinding, partition: PartitionId, budget: Nat) -> PartitionRead { + event_log_read_partition_with(store: store, partition: partition, budget: budget, decode: fn(t) { pool_event_decode(text: t) }) +} + // APPEND: put the event's object, then advance the partition head from the expected one. The store // decides the race: a moved head is stale, carrying the head that won; every other failure is the // store's typed fault. The object is built from the event, so the bytes published are the bytes @@ -167,11 +183,11 @@ type EventAppend | EventAppendStale { expected: HeadExpectation, observed: HeadExpectation } | EventAppendRefused { cause: EventLogRefusal } -fn event_log_append(store: FabricStorageBinding, partition: PartitionId, event: ChainEvent, expected: HeadExpectation) -> EventAppend { +fn event_log_append_with

(store: FabricStorageBinding, partition: PartitionId, event: ChainEvent

, expected: HeadExpectation, encode: fn(ChainEvent

) -> String) -> EventAppend { if !head_expectation_eq(a: event_parent_expectation(e: event), b: expected) { EventAppendRefused { cause: EventLogParentMismatch } } else { - match event_object(event: event) { + match event_object(event: event, encode: encode) { EventObjectRefused { cause: c } => EventAppendRefused { cause: c } EventObjectBuilt { object: object } => match store_expectation(expected: expected) { @@ -191,6 +207,10 @@ fn event_log_append(store: FabricStorageBinding, partition: PartitionId, event: } } +fn event_log_append(store: FabricStorageBinding, partition: PartitionId, event: ChainEvent, expected: HeadExpectation) -> EventAppend { + event_log_append_with(store: store, partition: partition, event: event, expected: expected, encode: fn(e) { pool_event_wire_text(event: e) }) +} + // THE SEAT TRANSACTION, WET AND BOUNDED: read, fold, propose, append; a stale append re-reads and // decides again, at most `attempts` times, and exhaustion is its own typed outcome rather than a // seat. The grant is minted only from an admitted append. diff --git a/dag/gunbc/fleet/site_pxe_edge_converge.dag b/dag/gunbc/fleet/site_pxe_edge_converge.dag index fedd5948b66..13919ab7ea6 100644 --- a/dag/gunbc/fleet/site_pxe_edge_converge.dag +++ b/dag/gunbc/fleet/site_pxe_edge_converge.dag @@ -63,7 +63,7 @@ import gunbc.typed_remote_file_write { import gunbc.runner_host_grants { executor_bootstrap_principal } import gunbc.ci_spec { site_pxe_edge_converge_receipt_path } import gunbc.runner_host_file_converge { elevated_argv } -import gunbc.machine_intake_subject { content_hash_equal } +import std.content_hash { content_hash_equal } import extdeps.toolchain.types { Architecture, Aarch64 } import gunbc.network_boot_delivery { first_slice_admitted_architectures, diff --git a/dag/gunbc/floor_memory_demand.dag b/dag/gunbc/floor_memory_demand.dag index 264cbd2811f..00f5a30d88b 100644 --- a/dag/gunbc/floor_memory_demand.dag +++ b/dag/gunbc/floor_memory_demand.dag @@ -172,6 +172,12 @@ fn termination_withheld_completion(termination: SupervisedTermination) -> Bool { } } +// AN UNREADABLE LIMIT REFUSES BEFORE ANYTHING ELSE IS ASKED (the first arm of the qualification +// below). Every question after it -- was it pinned, did it fit -- is a comparison AGAINST the +// limits, so a limit that could not be read makes those questions unanswerable rather than +// negative. This is the arm whose absence let a run with an unknown throttle line publish as a +// demand. +// // THE JUDGMENT. Every input is a kernel counter or a wait status; nothing here is authored. // // The order of the tests is the order of severity, and each one independently establishes that the @@ -188,10 +194,6 @@ fn qualify_floor_memory_demand( events: CgroupMemoryEvents, termination: SupervisedTermination, ) -> FloorMemoryQualification { - // AN UNREADABLE LIMIT REFUSES BEFORE ANYTHING ELSE IS ASKED. Every question below -- was it - // pinned, did it fit -- is a comparison AGAINST the limits, so a limit that could not be read - // makes those questions unanswerable rather than negative. This is the arm whose absence let a - // run with an unknown throttle line publish as a demand. match limit_unparseable_body(limit: limit_high) { Present { value: hb } => DemandUnreadable { diff --git a/dag/gunbc/harness/harness_backend.dag b/dag/gunbc/harness/harness_backend.dag index 80bc4285fab..91bc8640326 100644 --- a/dag/gunbc/harness/harness_backend.dag +++ b/dag/gunbc/harness/harness_backend.dag @@ -40,6 +40,7 @@ import gunbc.spark.vllm_observed { } import gunbc.harness.harness_reasoning_wire { HarnessServingUnit, harness_serving_unit_wire } import extdeps.http.client +import extdeps.tools.curl { curl_exit_outcome, CurlCouldNotConnect, CurlOperationTimedOut } import extdeps.languages.json.parse { JsonDocumentParsed, JsonDocumentUnreadable, parse_json_document, json_document_gap_text, } @@ -73,6 +74,35 @@ fn harness_bounded_get(url: String) -> HarnessBoundedRead { HarnessBoundedRead { body: read.body, success: read.success } } +// PRESENCE, NOT CONTENT: whether ANY HTTP response came back within the front-door bound, and +// with what status. Consumed by the Cut D D0 incumbent question, for which a 401 or a 500 is an +// incumbent that answered and -f's failure would have read as absence. +// +// "NO RESPONSE" IS THREE READINGS, NOT ONE, AND NONE OF THEM IS ABSENCE. curl 7 ("failed to +// connect to host or proxy") covers a refused connection, an unreachable host and a proxy that +// would not connect -- it is not the operating system's ECONNREFUSED and proves no listener absent; +// curl 28 is the deadline, which says something may have accepted and did not answer; every other +// code is a transport that could not be read. The arms carry exactly what curl established and no +// more; whether an incumbent is absent is the consumer's to establish from a host-side reading. +type HarnessBoundedPresence + = HarnessResponded { status: String } + | HarnessConnectFailed { url: String, curl_exit: Int } + | HarnessDeadline { url: String } + | HarnessTransportUnread { url: String, curl_exit: Int } + +fn harness_bounded_presence(url: String) -> HarnessBoundedPresence { + let read = http.Client.StatusWithin(url: url as NonEmptyStr, max_seconds: to_string(second_count(s: harness_front_door_probe_bound)) as NonEmptyStr) + if read.success { + HarnessResponded { status: trim(s: read.status) } + } else { + match curl_exit_outcome(code: read.exit_code) { + CurlCouldNotConnect => HarnessConnectFailed { url: url, curl_exit: read.exit_code } + CurlOperationTimedOut => HarnessDeadline { url: url } + _ => HarnessTransportUnread { url: url, curl_exit: read.exit_code } + } + } +} + type HarnessBoundedRead { body: String success: Bool diff --git a/dag/gunbc/harness/harness_seat.dag b/dag/gunbc/harness/harness_seat.dag index bf00a79459f..4b6de567bae 100644 --- a/dag/gunbc/harness/harness_seat.dag +++ b/dag/gunbc/harness/harness_seat.dag @@ -62,6 +62,8 @@ import gunbc.spark.pair_serving_realization { } import gunbc.spark.serving_incarnation_observe { observe_endpoint_incarnation, endpoint_incarnation_standing, read_unit_property, UnitTextObserved, UnitTextRefused } import gunbc.spark.pair_serving_desired { spark_pair_group_head_host } +import gunbc.spark.pair_serving_authority { pair_serving_authority_for, pair_serving_apply_admits } +import gunbc.spark.pair_serving_authority_log { CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authority, authority_wire } import gunbc.spark.serving_group_launch { serving_group_launch, GroupRendezvous, serving_group_rendezvous } import gunbc.spark.rendezvous_participation { ServingGroupRendezvousReach, GroupRendezvousReachUnobserved, RendezvousEpoch, rendezvous_epoch, serving_group_rendezvous_reach, @@ -1108,7 +1110,39 @@ type HarnessAcquisition // which is the leak the ceiling of one exists to prevent, arriving through the recovery path // (review 62155). A refused release is therefore a refusal of the whole placement, and the seat it // could not give back is named in the reason. +// THE AUTHORITY IS READ AGAIN AFTER THE SEAT'S COMPARE-AND-SET, AND A MOVED AUTHORITY RELEASES THE +// SEAT. The pre-check in harness_acquire_on saves an attempt; it does not settle a race, because +// the authority and the seat live on different partitions and a suspension can land between the +// read and the seat's write. So the seat is provisional until the authority is read once more +// AFTER the grant: still admitting, the seat stands; moved, the seat is released on its own +// partition and no binding is returned -- and if that release cannot land, the refusal says so, +// because a seat that may still be held is not a turn to place elsewhere. This is the same shape +// harness_fence_grant already gives the engine's continuity, and it runs FIRST, so the seat never +// outlives a fence by the length of an HTTP read. +type SeatAfterGrant + = SeatStillAdmitted + | SeatReleasedAuthorityMoved { cause: String } + | SeatUnreleasedAuthorityMoved { cause: String, release_detail: String } + +fn harness_seat_after_grant_authority(store: FabricStorageBinding, group: FabricGroup, partition: PartitionId, ceiling: Nat, grant: LeaseGrant, now: EpochSecs) -> SeatAfterGrant { + match harness_seat_authority_gate(store: store, group: group, now: now) { + SeatAuthorityUngoverned => SeatStillAdmitted + SeatAuthorityAdmits => SeatStillAdmitted + SeatAuthorityRefuses { cause: why } => + match release_retry(store: store, partition: partition, ceiling: ceiling, reference: grant.reference, reason: "the pair-serving authority moved between admission and the seat's acquisition" as NonEmptyStr, now: now) { + HarnessSeatReleased { partition: _ } => SeatReleasedAuthorityMoved { cause: why } + HarnessSeatReleaseRefused { detail: d } => SeatUnreleasedAuthorityMoved { cause: why, release_detail: d } + } + } +} + fn harness_fence_grant(candidate: HarnessCandidate, class: ServingCoTenancyClass, store: FabricStorageBinding, partition: PartitionId, shape: HarnessWireShape, grant: LeaseGrant, now: EpochSecs) -> HarnessAcquisition { + match harness_seat_after_grant_authority(store: store, group: candidate.group, partition: partition, ceiling: candidate.ceiling, grant: grant, now: now) { + SeatReleasedAuthorityMoved { cause: why } => + AcquisitionRefused { detail: seat_bind_detail(group: candidate.group, text: join(["the pair-serving authority moved between admission and the seat's acquisition, and the seat was released: ", why], "")) } + SeatUnreleasedAuthorityMoved { cause: why, release_detail: d } => + AcquisitionRefused { detail: seat_bind_detail(group: candidate.group, text: join(["the pair-serving authority moved between admission and the seat's acquisition and the seat could NOT be released, so this turn is not placed elsewhere while that seat may still be held: ", why, " (release: ", d, ")"], "")) } + SeatStillAdmitted => match serving_availability_after_launch_read(endpoint: candidate.endpoint, expected: candidate.launch, observation: harness_observe_engine_at(endpoint: candidate.endpoint, at: now).launch, now: to_string(now)).reading { EndpointProcessContinuityUnread { endpoint: _, attempted_at: _, cause: o } => { let released = release_retry(store: store, partition: partition, ceiling: candidate.ceiling, reference: grant.reference, reason: "the engine could not be identified after the seat was granted" as NonEmptyStr, now: now) @@ -1140,10 +1174,46 @@ fn harness_fence_grant(candidate: HarnessCandidate, class: ServingCoTenancyClass }, } } + } } +// A SEAT IS MINTED ONLY UNDER AN ACTIVE PAIR-SERVING AUTHORITY, READ AT ACQUIRE TIME. The Cut D +// transaction's first write (gunbc.spark.pair_serving_d0) suspends a group's authority so that +// nothing starts on its hosts while it is read and settled; a seat is new work admitted to the +// answering engine, so it is one of the things that must not start. The offer was observed +// under whatever authority stood when the route was read; the authority is read AGAIN here, at +// the acquire, so a suspension that lands between observing the offer and taking the seat +// refuses the seat rather than using stale offer evidence. A group the pair-serving authority +// does not claim (the canary) is not governed by it and is not gated here; a claimed group whose +// authority cannot be read is refused -- unreadability is never permission. +type SeatAuthorityGate + = SeatAuthorityUngoverned + | SeatAuthorityAdmits + | SeatAuthorityRefuses { cause: String } + +fn harness_seat_authority_gate(store: FabricStorageBinding, group: FabricGroup, now: EpochSecs) -> SeatAuthorityGate { + match pair_serving_authority_for(group: group) { + Absent => SeatAuthorityUngoverned + Present { value: _ } => + match current_pair_serving_authority(store: store, group: group, at: now) { + CurrentAuthorityUnread { group: _, step: st, reason: why } => + SeatAuthorityRefuses { cause: join(["the pair-serving authority could not be read at ", st, ", so no seat is minted: ", why], "") } + CurrentAuthorityRead { authority: a, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + if pair_serving_apply_admits(a: a) { SeatAuthorityAdmits } + else { SeatAuthorityRefuses { cause: join(["the pair-serving authority does not admit new work: ", authority_wire(a: a)], "") } } + } + } +} fn harness_acquire_on(candidate: HarnessCandidate, class: ServingCoTenancyClass, store: FabricStorageBinding, shape: HarnessWireShape, actor: NonEmptyStr, attempt: NonEmptyStr, policy: LeasePolicy, now: EpochSecs, round: Nat) -> HarnessAcquisition { + match harness_seat_authority_gate(store: store, group: candidate.group, now: now) { + SeatAuthorityRefuses { cause: why } => AcquisitionRefused { detail: seat_bind_detail(group: candidate.group, text: why) } + SeatAuthorityUngoverned => harness_acquire_under_authority(candidate: candidate, class: class, store: store, shape: shape, actor: actor, attempt: attempt, policy: policy, now: now, round: round) + SeatAuthorityAdmits => harness_acquire_under_authority(candidate: candidate, class: class, store: store, shape: shape, actor: actor, attempt: attempt, policy: policy, now: now, round: round) + } +} + +fn harness_acquire_under_authority(candidate: HarnessCandidate, class: ServingCoTenancyClass, store: FabricStorageBinding, shape: HarnessWireShape, actor: NonEmptyStr, attempt: NonEmptyStr, policy: LeasePolicy, now: EpochSecs, round: Nat) -> HarnessAcquisition { match harness_seat_capacity(group: candidate.group, class: class) { AdmissionCeilingUnestablished { group: _, cause: why } => AcquisitionRefused { detail: seat_bind_detail(group: candidate.group, text: why as String) } diff --git a/dag/gunbc/host/host_operation_exec.dag b/dag/gunbc/host/host_operation_exec.dag index 62b4703cf04..ce61d524c7d 100644 --- a/dag/gunbc/host/host_operation_exec.dag +++ b/dag/gunbc/host/host_operation_exec.dag @@ -122,6 +122,8 @@ type HostOperation = | ProcfsReadStat | ProcfsReadPidStat { pid: NonEmptyStr } | ProcfsReadPidCgroup { pid: NonEmptyStr } + | ProcfsReadNetTcp + | ProcfsReadNetTcp6 | GetconfClockTicksPerSecond | Sha256SumFile { path: NonEmptyStr } @@ -237,6 +239,14 @@ fn host_operation_invocation(operation: HostOperation) -> BoundOperationInvocati at: procfs_operation_ref(operation: "ReadPidCgroup"), bindings: [operation_argv_bind_text(name: "pid", text: p as String)], ) + ProcfsReadNetTcp => bind_operation_invocation( + at: procfs_operation_ref(operation: "ReadNetTcp"), + bindings: [], + ) + ProcfsReadNetTcp6 => bind_operation_invocation( + at: procfs_operation_ref(operation: "ReadNetTcp6"), + bindings: [], + ) GetconfClockTicksPerSecond => bind_operation_invocation( at: OperationRef { path: getconf_operation_path, @@ -363,6 +373,14 @@ fn host_operation_exec_local(operation: HostOperation) -> HostOperationOutcome { let result = linux.Procfs.ReadPidCgroup(pid: p) HostOperationObserved { stdout: result.value, success: result.success, stderr: none } } + ProcfsReadNetTcp => { + let result = linux.Procfs.ReadNetTcp() + HostOperationObserved { stdout: result.value, success: result.success, stderr: none } + } + ProcfsReadNetTcp6 => { + let result = linux.Procfs.ReadNetTcp6() + HostOperationObserved { stdout: result.value, success: result.success, stderr: none } + } GetconfClockTicksPerSecond => { let result = posix.Getconf.ClockTicksPerSecond() HostOperationObserved { stdout: result.value, success: result.success, stderr: none } diff --git a/dag/gunbc/instruments/fabric_capacity_standing.dag b/dag/gunbc/instruments/fabric_capacity_standing.dag index ac037201b4e..a92a817089b 100644 --- a/dag/gunbc/instruments/fabric_capacity_standing.dag +++ b/dag/gunbc/instruments/fabric_capacity_standing.dag @@ -11,10 +11,14 @@ import gunbc.spark.fabric_switch_observed { FabricGroup, fabric_group_wire } import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest } import product.placement_supply { HostIdentity } import gunbc.spark.pair_serving_authority { - PairServingGroupAuthority, spark_pair_serving_authorities, pair_serving_capacity_subject, + PairServingGroupAuthority, pair_serving_capacity_subject, CapacityOrdinarySubject, CapacitySuspendedSubject, CapacityReleasedSubject, } import gunbc.spark.vllm_serving_launch { VllmServingLaunch, vllm_serving_launch } +import gunbc.spark.pair_serving_authority_log { CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authorities } +import gunbc.spark.fabric_reach { ExecutorReachKnown, ExecutorReachUnknown, observe_executor_reach } +import std.resources { Network } +import gunbc.fabric_event_log_host { now_epoch_seconds } import gunbc.spark.vllm_kv_layout_observe { VllmKvLayoutReceipt, admit_kv_layout, } @@ -552,6 +556,35 @@ fn capacity_standing_verdict(standings: List) -> ProcessExit { } } -fn fabric_capacity_standing() -> ProcessExit { - capacity_standing_verdict(standings: map(spark_pair_serving_authorities, a => standing_for_authority(a: a))) +// THE AUTHORITY IS READ FROM THE LOG, NOT THE SOURCE ROW. gunbc.spark.pair_serving_authority_log +// folds the resting row through every recorded transition, so a group a transaction has suspended +// is reported suspended here without a commit. An authority that could not be read is a refused +// standing naming the read's step -- the instrument does not stand for a group whose state it +// could not establish, and it does not fall back to the row. +fn standing_for_current(c: CurrentAuthority) -> GroupStanding { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => standing_for_authority(a: a) + CurrentAuthorityUnread { group: g, step: s, reason: why } => + refused_standing(text: join([fabric_group_wire(g: g) as String, ": serving authority could not be read at ", s, ": ", why], "")) + } +} + +// The verdict over a SUPPLIED current roster: the real group_report route for every read +// authority, so a witness can hand in the resting row at generation 0 and still execute the +// producer and the binding, while the live entry below hands in what the executor's log says. +fn fabric_capacity_standing_current(current: List) -> ProcessExit { + capacity_standing_verdict(standings: map(current, c => standing_for_current(c: c))) +} + +fn fabric_capacity_standing() -> ProcessExit + uses net: Network +{ + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["fabric capacity standing: ", c], "")) + ExecutorReachKnown { short_hostname: executor, path: _ } => + match now_epoch_seconds() { + Absent => exit_failure(reason: "fabric capacity standing: the clock could not be read as epoch seconds, so the lease standing of the authority cannot be derived") + Present { value: at } => fabric_capacity_standing_current(current: current_pair_serving_authorities(short_hostname: executor, at: at)) + } + } } diff --git a/dag/gunbc/instruments/fabric_ci_evidence.dag b/dag/gunbc/instruments/fabric_ci_evidence.dag index 8e124c336cf..7b070a5a33d 100644 --- a/dag/gunbc/instruments/fabric_ci_evidence.dag +++ b/dag/gunbc/instruments/fabric_ci_evidence.dag @@ -3,6 +3,7 @@ module tools.fabric_ci_evidence import std.types { Bool, Int, List, NonEmptyStr, String } import std.process { ProcessExit, ExitSuccess, exit_failure } import v2.std.optional { Present, Absent } +import extdeps.numeric.base16 { base16_word_value } import std.content_hash { ContentHash, content_hash_of_value } import extdeps.filesystem.filesystem_io { Filesystem } import tools.fabric_ci_evidence_projection_fixture { @@ -76,33 +77,8 @@ fn fabric_ci_shell_crossing_coordinates_hash(values: List) -> Conte content_hash_of_value(value: fabric_ci_framed_coordinates(values: values)) } -fn fabric_ci_hex_value(c: String) -> Int? { - let found = fold([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], init: -1, f: fn(acc, i) { - if substring(s: "0123456789ABCDEF", start: i, end: i + 1) == c { i } else { acc } - }) - if found < 0 { none } else { Present { value: found } } -} - fn fabric_ci_decode_scalar(wire: String, at: Int) -> Int? { - match fabric_ci_hex_value(c: char_at(s: wire, pos: at)) { - Absent => Absent - Present { value: a } => match fabric_ci_hex_value(c: char_at(s: wire, pos: at + 1)) { - Absent => Absent - Present { value: b } => match fabric_ci_hex_value(c: char_at(s: wire, pos: at + 2)) { - Absent => Absent - Present { value: c } => match fabric_ci_hex_value(c: char_at(s: wire, pos: at + 3)) { - Absent => Absent - Present { value: d } => match fabric_ci_hex_value(c: char_at(s: wire, pos: at + 4)) { - Absent => Absent - Present { value: e } => match fabric_ci_hex_value(c: char_at(s: wire, pos: at + 5)) { - Absent => Absent - Present { value: f } => Present { value: a * 1048576 + b * 65536 + c * 4096 + d * 256 + e * 16 + f } - } - } - } - } - } - } + base16_word_value(word: substring(s: wire, start: at, end: at + 6), max_digits: 6) } type FabricCiDecodeState diff --git a/dag/gunbc/instruments/fabric_partition_read.dag b/dag/gunbc/instruments/fabric_partition_read.dag index e147f266d71..dc04dd49ac2 100644 --- a/dag/gunbc/instruments/fabric_partition_read.dag +++ b/dag/gunbc/instruments/fabric_partition_read.dag @@ -12,7 +12,7 @@ import product.capacity.pool_events { PoolEvent, pool_event_wire_text } import gunbc.fabric_storage_client { FabricStorageBinding, FabricStorageServed, FabricStorageLocalFiles } import gunbc.fabric_storage_file_store { fabric_storage_file_root } import gunbc.fabric_event_log { - PartitionRead, PartitionReadOk, PartitionReadRefused, event_log_read_partition, event_log_refusal_wire, + PartitionRead, PartitionReadOk, PartitionReadRefused, event_log_read_partition, event_log_refusal_wire, event_log_read_budget, } // READING A PARTITION WITHOUT WRITING TO IT, and that is the whole point of a second entry beside @@ -47,7 +47,7 @@ fn head_word(head: HeadExpectation) -> String { // One line per event, oldest first: its id and its wire text. The text is the event's own // serialization, so a reader sees exactly the bytes the store holds rather than a rendering minted // here. Rendered by map-then-join rather than a fold over a copied accumulator: a readout is bounded -// at partition_read_budget() events, and an accumulator re-copied per event is quadratic in bytes at +// at event_log_read_budget() events, and an accumulator re-copied per event is quadratic in bytes at // that bound (DESIGN section 6 -- a copied accumulator is fixed regardless of the realized n). fn readout_lines(readout: PartitionReadout) -> String { join(map(readout.events, env => join([env.id as String, " ", pool_event_wire_text(event: env.event), "\n"], "")), "") @@ -94,7 +94,6 @@ fn partition_read_exit(report: PartitionReadReport, receipt: NonEmptyStr) -> Pro } } -fn partition_read_budget() -> Nat { 4096 } // THE SERVED READ: any host on the tailnet, against the placed store's endpoint. This is the entry // the live group A receipt runs, e.g. @@ -111,7 +110,7 @@ fn fabric_partition_read_served(endpoint: NonEmptyStr, partition: NonEmptyStr, r report: partition_read_report( store: FabricStorageServed { endpoint: endpoint }, partition: (partition as String) as PartitionId, - budget: partition_read_budget(), + budget: event_log_read_budget(), ), receipt: receipt, ) @@ -124,7 +123,7 @@ fn fabric_partition_read_local(root: NonEmptyStr, partition: NonEmptyStr, receip report: partition_read_report( store: FabricStorageLocalFiles { root: fabric_storage_file_root(root: root) }, partition: (partition as String) as PartitionId, - budget: partition_read_budget(), + budget: event_log_read_budget(), ), receipt: receipt, ) diff --git a/dag/gunbc/machine_intake/access.dag b/dag/gunbc/machine_intake/access.dag index c984e3b96b0..fec49c0c310 100644 --- a/dag/gunbc/machine_intake/access.dag +++ b/dag/gunbc/machine_intake/access.dag @@ -1,7 +1,7 @@ module gunbc.machine_intake_access import std.types { Bool, Int, List, NonEmptyStr, String } -import std.content_hash { ContentHash } +import std.content_hash { ContentHash, content_hash_equal } import extdeps.bmc.endpoint { BmcControllerEndpoint } import extdeps.bmc.capability { BmcFirmwareReleaseCapabilityRow } import extdeps.bmc.access_profile { @@ -22,7 +22,7 @@ import extdeps.bmc.access_profile { ProfilePromotedFromObservation, bmc_access_profile_for, } -import gunbc.machine_intake_subject { MachineIntakeSubject, content_hash_equal } +import gunbc.machine_intake_subject { MachineIntakeSubject } import gunbc.machine_intake_receipt { EvidenceRef, same_intake_subject } // THE ACCESS CONTEXT IS BOUND ONCE, BEFORE THE DELIVERY SOLVER RUNS (machine-intake ruling 3 diff --git a/dag/gunbc/machine_intake/disposition.dag b/dag/gunbc/machine_intake/disposition.dag index d4c62db9de2..2673bfb063b 100644 --- a/dag/gunbc/machine_intake/disposition.dag +++ b/dag/gunbc/machine_intake/disposition.dag @@ -1,7 +1,7 @@ module gunbc.machine_intake_disposition import std.types { Bool, EpochMs, Int, List, NonEmptyStr, String, list_length } -import std.content_hash { ContentHash } +import std.content_hash { ContentHash, content_hash_equal } import gunbc.machine_intake_subject { QualificationSubject, SubjectCurrency, @@ -9,7 +9,6 @@ import gunbc.machine_intake_subject { SubjectStale, merge_subject_currency, compare_qualification_subject, - content_hash_equal, } import gunbc.machine_intake_phase { ArrivalHardwareQualification, diff --git a/dag/gunbc/machine_intake/host_capture_historical_binding.dag b/dag/gunbc/machine_intake/host_capture_historical_binding.dag index 6aa09585c68..44214de42ac 100644 --- a/dag/gunbc/machine_intake/host_capture_historical_binding.dag +++ b/dag/gunbc/machine_intake/host_capture_historical_binding.dag @@ -235,6 +235,16 @@ fn census_roster_is_fully_bound() -> Bool { ) } +// (Hoisted to module grain from the body of the item below: a source annotation inside a +// declaration body is not modeled and refuses to parse.) +// NEWLY INTRODUCED IS THE ONE DISPOSITION THAT REQUIRES THE SECTION TO BE ABSENT, so it is +// decided before the historical section is demanded. Folded in with the others it was a trap: +// the capture has no such section, so the Absent arm answered false before this arm was +// reached, while roster_matches_bindings counted the stage as carried and passed. The first +// stage anyone added would have gone green on the dry claim and permanently red on the wet one. +// The row still has to EARN the disposition: if the capture does carry the section, the row is +// wrong about history and this fails. +// // WHAT THE BOUND ENVELOPE MUST SAY FOR ONE BINDING TO HOLD. Exact means the file's ARGV line IS // the roster's command; Normalized and AuthoredFromDescription mean the file's ARGV line is the row's // `original` (so an edit to that quote is caught against the bytes); DeliberatelyOmitted means the @@ -244,13 +254,6 @@ fn binding_holds_against_capture(envelope: BoundHostCaptureEnvelope, stages: Lis let historical = host_capture_bound_section_argv(envelope: envelope, name: b.name as String) let carried = stage_named(stages: stages, name: b.name as String) match b.disposition { - // NEWLY INTRODUCED IS THE ONE DISPOSITION THAT REQUIRES THE SECTION TO BE ABSENT, so it is - // decided before the historical section is demanded. Folded in with the others it was a trap: - // the capture has no such section, so the Absent arm answered false before this arm was - // reached, while roster_matches_bindings counted the stage as carried and passed. The first - // stage anyone added would have gone green on the dry claim and permanently red on the wet one. - // The row still has to EARN the disposition: if the capture does carry the section, the row is - // wrong about history and this fails. HistoricalStageNewlyIntroduced { reason: _ } => match historical { Present { value: _ } => false diff --git a/dag/gunbc/machine_intake/receipt.dag b/dag/gunbc/machine_intake/receipt.dag index a8b9e4a5263..512b7ea647b 100644 --- a/dag/gunbc/machine_intake/receipt.dag +++ b/dag/gunbc/machine_intake/receipt.dag @@ -2,6 +2,7 @@ module gunbc.machine_intake_receipt import std.types { Bool, EpochMs, Int, List, NonEmptyStr, String, list_length } import std.content_hash { + content_hash_equal, ContentHash, ContentHashEqual, ContentHashDifferent, @@ -15,7 +16,7 @@ import std.content_hash { } import extdeps.toolchain.types { Architecture } import gunbc.boot_artifact { BootArtifact } -import gunbc.machine_intake_subject { IntakeAttemptId, MachineIntakeSubject, QualificationSubject, UnitKey, content_hash_equal } +import gunbc.machine_intake_subject { IntakeAttemptId, MachineIntakeSubject, QualificationSubject, UnitKey } import gunbc.machine_intake_phase { IntakePhase, QualificationRefusalOwner, intake_phase_label, intake_phase_rank, refusal_owner_label } // THE RECEIPT SPINE (machine-intake ruling 2026-08-29 §12). Every phase emits one normalized diff --git a/dag/gunbc/machine_intake/subject.dag b/dag/gunbc/machine_intake/subject.dag index 5bce7af4335..242c7934d35 100644 --- a/dag/gunbc/machine_intake/subject.dag +++ b/dag/gunbc/machine_intake/subject.dag @@ -2,6 +2,7 @@ module gunbc.machine_intake_subject import std.types { Bool, EpochMs, Int, List, NonEmptyStr, String, list_length } import std.content_hash { + content_hash_equal, ContentHash, ContentHashComparison, ContentHashEqual, @@ -216,22 +217,10 @@ fn qualification_subject_of( } } -// THE ONE DIGEST-EQUALITY DECISION FOR THIS LAYER (review 57612). It was -// authored twice -- here and again in gunbc.machine_intake_receipt -- which is -// two names for one predicate and the hollow_alias class DESIGN section 3 -// names. It is homed here because the qualification subject is what the -// digests identify and every other machine-intake module already imports this -// one. std.content_hash deliberately offers no union-level Bool: the -// cross-family arm is a DECISION, and this layer makes it once, in the -// refusing direction -- two digests from different families are never the same -// subject, so they cannot be treated as unchanged. -fn content_hash_equal(left: ContentHash, right: ContentHash) -> Bool { - match compare_content_hash(left: left, right: right) { - ContentHashEqual => true - ContentHashDifferent => false - ContentHashCrossFamilyIncomparable => false - } -} +// content_hash_equal lived here (the refusing-direction Bool over the union, made once); it is now +// std.content_hash content_hash_equal, beside the comparison it folds, since two more consumers in +// gunbc.spark had re-minted it (review 67905). + // Which axes moved between an admitted subject and the one observed now. All // three are reported, not the first: a repair that changed a DIMM and flashed diff --git a/dag/gunbc/namespace/namespace_reference_derived_residency_qualification.dag b/dag/gunbc/namespace/namespace_reference_derived_residency_qualification.dag index cf7c3cb0a9f..50b80cf86c6 100644 --- a/dag/gunbc/namespace/namespace_reference_derived_residency_qualification.dag +++ b/dag/gunbc/namespace/namespace_reference_derived_residency_qualification.dag @@ -148,6 +148,10 @@ fn first_whole_corpus_syntax_carrier( whole_corpus_syntax_carriers(readings: readings) |> first } +// NOTHING DEFINITE FOUND IS NOT THE SAME AS A BOUND (the last arm of the fold below). A carrier +// the derivation could not read is checked LAST, after the definite failures, so a known-bad +// producer still reports its specific carrier -- but it refuses, because qualifying there would +// let a producer buy the admission by renaming a type. // Construction peak is read BEFORE retention, so the whole-pool-build-then-release specimen refuses on // the strength of `constructed` with `retained_at_return` empty. An empty `constructed` is refused // rather than qualified: a producer that reports no carriers has supplied no evidence of a bound. @@ -175,10 +179,6 @@ fn qualify_bounded_realization( failure: WholeCorpusSyntaxRetainedAtReturn { carrier: held.carrier }, } Absent => - // NOTHING DEFINITE WAS FOUND, WHICH IS NOT THE SAME AS A BOUND. A carrier the - // derivation could not read is checked LAST, after the definite failures, so a - // known-bad producer still reports its specific carrier -- but it refuses, because - // qualifying here would let a producer buy the admission by renaming a type. match first_undecidable_failure( readings: concat(reading.constructed, reading.retained_at_return), ) { diff --git a/dag/gunbc/network_boot_delivery.dag b/dag/gunbc/network_boot_delivery.dag index 24d497e8faf..631d4078a6f 100644 --- a/dag/gunbc/network_boot_delivery.dag +++ b/dag/gunbc/network_boot_delivery.dag @@ -4,7 +4,7 @@ import std.types { Bool, EpochMs, List, NonEmptyStr, String } import std.decl_ref { DeclarationRef, decl_ref } import std.roster_frontier { FrontierRow, frontier_row_decl } import std.dissolution { unbound_dissolution } -import std.content_hash { ContentHash } +import std.content_hash { ContentHash, content_hash_equal } import extdeps.uri { Uri } import extdeps.crypto.mac { MacKeyId } import extdeps.toolchain.types { Architecture, Aarch64 } @@ -20,7 +20,7 @@ import extdeps.provisioning.dhcp_client_arch { import extdeps.provisioning.network_boot { ClientNetworkBootMode, DhcpBootDirection, UefiHttpBoot, UefiPxe, LegacyBiosPxe } import product.placement_supply { SiteIdentity, site_identity_eq } import gunbc.machine_intake_access { BootDeliveryTarget, same_boot_delivery_target } -import gunbc.machine_intake_subject { UnitKey, IntakeAttemptId, content_hash_equal } +import gunbc.machine_intake_subject { UnitKey, IntakeAttemptId } // THE NETWORK-BOOT ESTABLISHMENT VOCABULARY, in its own authority (machine-intake ruling 3, // namespace ruling). Its four evidence axes have independent producers — client firmware mode, diff --git a/dag/gunbc/product/capacity/event_chain.dag b/dag/gunbc/product/capacity/event_chain.dag index 8f22bae7878..7afb6fb52a2 100644 --- a/dag/gunbc/product/capacity/event_chain.dag +++ b/dag/gunbc/product/capacity/event_chain.dag @@ -34,6 +34,15 @@ type ChainEnvelope

{ event: ChainEvent

} +// WHAT A PARTITION'S BYTES DECODE TO, for any payload. The log stores one object per event and +// reads it back through the payload's own codec; this is the codec's answer, and it is generic +// because the log is: a second decode result per payload type would re-spell this arm pair once +// per partition kind. Undecodable is a typed refusal carried up to the reader, never a skipped +// event. +type EventDecode

+ = EventDecoded { event: ChainEvent

} + | EventUndecodable { reason: String } + // The head a writer believes the partition is at. Absent is a real value: the first event of a // partition is admitted against an absent head, and a writer that observed absence while another // wrote first is stale exactly like any other. diff --git a/dag/gunbc/product/capacity/event_json.dag b/dag/gunbc/product/capacity/event_json.dag new file mode 100644 index 00000000000..690dcd4a8e3 --- /dev/null +++ b/dag/gunbc/product/capacity/event_json.dag @@ -0,0 +1,140 @@ +module product.capacity.event_json + +import std.types { String, NonEmptyStr, List } +import std.nat { Nat } +import v2.std.optional { Present, Absent } +import extdeps.languages.json.emit { JsonValue, JsonString, JsonNumber, JsonKeyValue, json_object, json_kv, json_string, json_int, serialize_json } +import extdeps.languages.json.parse { + parse_json_document, JsonDocumentParsed, JsonDocumentUnreadable, json_document_gap_text, + json_object_unique_member, JsonMemberFound, +} +import product.capacity.event_chain { PartitionId, EventId, ChainEvent, EventDecode, EventDecoded, EventUndecodable } + +// ── THE ENVELOPE ON THE WIRE, ONCE, FOR EVERY PARTITION KIND ──────────────────────────────────── +// +// A chain event is an envelope -- schema, partition, parent, recorded_at, actor -- around a +// payload the partition kind owns. The pool events were the first kind and their codec spelled +// the envelope inline; the pair-serving authority partition (gunbc.spark.pair_serving_authority_log) +// is the second, and a second inline spelling would be the envelope written twice with nothing +// keeping the two agreeing (DESIGN §2). So the envelope is encoded and decoded HERE for any +// payload, and a kind supplies only its schema word, its kind word and its members -- the three +// things that are actually about it. +// +// THE WIRE IS THE EVENT'S CANONICAL TEXT AND NOTHING ELSE. The id is not inside it: the store's +// content address of these bytes IS the id, so carrying it inside would make the bytes depend on +// their own hash. + +fn chain_event_json

(schema: String, e: ChainEvent

, kind: String, members: List) -> JsonValue { + json_object(members: concat([ + json_kv(key: "schema", value: json_string(s: schema)), + json_kv(key: "partition", value: json_string(s: e.partition as String)), + json_kv(key: "parent", value: json_string(s: match e.parent { Present { value: p } => p as String Absent => "" })), + json_kv(key: "recorded_at", value: json_int(n: e.recorded_at)), + json_kv(key: "actor", value: json_string(s: e.actor as String)), + json_kv(key: "kind", value: json_string(s: kind)), + ], members)) +} + +fn chain_event_wire_text

(schema: String, e: ChainEvent

, kind: String, members: List) -> String { + serialize_json(v: chain_event_json(schema: schema, e: e, kind: kind, members: members)) +} + +fn member_string(doc: JsonValue, key: String) -> String? { + match json_object_unique_member(v: doc, key: key) { + JsonMemberFound { value: JsonString { value: s } } => Present { value: s } + _ => none + } +} + +fn member_nonempty(doc: JsonValue, key: String) -> NonEmptyStr? { + match member_string(doc: doc, key: key) { + Present { value: s } => if s == "" { none } else { Present { value: s as NonEmptyStr } } + Absent => none + } +} + +fn member_nat(doc: JsonValue, key: String) -> Nat? { + match json_object_unique_member(v: doc, key: key) { + JsonMemberFound { value: JsonNumber { lexeme: lex } } => + match parse_int(s: lex as String) { + Present { value: n } => if n < 0 { none } else { Present { value: n } } + Absent => none + } + _ => none + } +} + +// THE ENVELOPE IS READ ONCE. One walk over the members every event carries -- schema, partition, +// parent (the empty string is "no parent"), recorded_at, actor, kind -- answers the envelope with +// the kind word as its payload, the schema word beside it and the document for a payload decoder. +// Both decoders below consume this and nothing else spells the envelope: a member added here is +// added for every reader (DESIGN §2). +type EnvelopeRead + = EnvelopeReadAt { schema: String, doc: JsonValue, event: ChainEvent } + | EnvelopeUnreadable { reason: String } + +fn chain_envelope_read(text: String) -> EnvelopeRead { + match parse_json_document(s: text) { + JsonDocumentUnreadable { gap } => EnvelopeUnreadable { reason: join(["not JSON: ", json_document_gap_text(gap: gap)], "") } + JsonDocumentParsed { value: doc } => + match member_string(doc: doc, key: "schema") { + Absent => EnvelopeUnreadable { reason: "schema missing" } + Present { value: s } => + match member_nonempty(doc: doc, key: "partition") { + Absent => EnvelopeUnreadable { reason: "partition missing" } + Present { value: part } => + match member_nat(doc: doc, key: "recorded_at") { + Absent => EnvelopeUnreadable { reason: "recorded_at missing or not a non-negative integer" } + Present { value: at } => + match member_nonempty(doc: doc, key: "actor") { + Absent => EnvelopeUnreadable { reason: "actor missing" } + Present { value: actor } => + match member_string(doc: doc, key: "kind") { + Absent => EnvelopeUnreadable { reason: "kind missing" } + Present { value: kind } => + EnvelopeReadAt { schema: s, doc: doc, event: ChainEvent { + partition: (part as String) as PartitionId, + parent: match member_string(doc: doc, key: "parent") { Present { value: p } => if p == "" { none } else { Present { value: p as EventId } } Absent => none }, + recorded_at: at, + actor: actor, + payload: kind, + } } + } + } + } + } + } + } +} + +// THE DECODE, generic in the payload: the envelope is read once and the kind word and document are +// handed to the payload's own decoder, which answers a payload or nothing. A schema that is not +// this kind's, a missing envelope member, or a payload that did not decode are each a typed +// refusal naming what was missing -- never a skipped event. +fn chain_event_decode

(text: String, schema: String, kind_decode: fn(JsonValue, String) -> P?) -> EventDecode

{ + match chain_envelope_read(text: text) { + EnvelopeUnreadable { reason: why } => EventUndecodable { reason: why } + EnvelopeReadAt { schema: s, doc: doc, event: env } => + if s != schema { + EventUndecodable { reason: join(["schema ", s, " is not ", schema], "") } + } else { + match kind_decode(doc, env.payload) { + Absent => EventUndecodable { reason: join(["kind ", env.payload, " did not decode"], "") } + Present { value: payload } => + EventDecoded { event: ChainEvent { partition: env.partition, parent: env.parent, recorded_at: env.recorded_at, actor: env.actor, payload: payload } } + } + } + } +} + +// THE ENVELOPE ALONE, FOR ANY SCHEMA: the same read without the schema check and with the kind word +// as the payload. A reader that only needs the chain's shape -- the parent links of a partition +// whose payload family it does not select, as the event-ref backfill does over every live +// partition -- reads envelopes and nothing else; it decides nothing from a payload it did not +// decode. +fn chain_envelope_decode(text: String) -> EventDecode { + match chain_envelope_read(text: text) { + EnvelopeUnreadable { reason: why } => EventUndecodable { reason: why } + EnvelopeReadAt { schema: _, doc: _, event: env } => EventDecoded { event: env } + } +} diff --git a/dag/gunbc/product/capacity/pool_events.dag b/dag/gunbc/product/capacity/pool_events.dag index 265c2bcae5e..a7c765f7c2c 100644 --- a/dag/gunbc/product/capacity/pool_events.dag +++ b/dag/gunbc/product/capacity/pool_events.dag @@ -3,11 +3,8 @@ module product.capacity.pool_events import std.types { String, Bool, Int, NonEmptyStr, List, EpochSecs } import std.nat { Nat } import std.measure { Measure } -import extdeps.languages.json.emit { JsonValue, JsonString, JsonNumber, JsonKeyValue, json_object, json_kv, json_string, json_int, serialize_json } -import extdeps.languages.json.parse { - parse_json_document, JsonDocumentParsed, JsonDocumentUnreadable, json_document_gap_text, - json_object_unique_member, JsonMemberFound, -} +import extdeps.languages.json.emit { JsonValue, JsonKeyValue, json_kv, json_string, json_int } +import product.capacity.event_json { chain_event_wire_text, chain_event_decode, member_nonempty, member_nat } import product.capacity.pool { Pool, PoolOutcome, PoolAdvanced, PoolRefused, PoolRefusal, LedgerRefusal, PoolReading, PoolRead, PoolReadingRefused, @@ -16,7 +13,7 @@ import product.capacity.pool { import extdeps.accounting.encumbrance { EncumbranceRefusal, ExceedsAppropriation } import product.capacity.event_chain { PartitionId, EventId, ChainEvent, ChainEnvelope, HeadExpectation, HeadAbsent, HeadAt, - AppendDecision, AppendAdmitted, AppendStale, + AppendDecision, AppendAdmitted, AppendStale, EventDecode, EventDecoded, EventUndecodable, } import product.capacity.lease { LeasePolicy, LeaseGrant, lease_grant, lease_fence_of } @@ -75,51 +72,8 @@ fn pool_event_kind_members(e: PoolEvent) -> List { } } -// THE WIRE IS THE EVENT'S CANONICAL TEXT AND NOTHING ELSE. The id is not inside it: the store's -// content address of these bytes IS the id, so carrying it inside would make the bytes depend on -// their own hash. -fn pool_event_canonical_json(e: ChainEvent) -> JsonValue { - json_object(members: concat([ - json_kv(key: "schema", value: json_string(s: pool_event_schema)), - json_kv(key: "partition", value: json_string(s: e.partition as String)), - json_kv(key: "parent", value: json_string(s: match e.parent { Present { value: p } => p as String Absent => "" })), - json_kv(key: "recorded_at", value: json_int(n: e.recorded_at)), - json_kv(key: "actor", value: json_string(s: e.actor as String)), - json_kv(key: "kind", value: json_string(s: pool_event_kind_word(e: e.payload))), - ], pool_event_kind_members(e: e.payload))) -} - fn pool_event_wire_text(event: ChainEvent) -> String { - serialize_json(v: pool_event_canonical_json(e: event)) -} - -type PoolEventDecode - = PoolEventDecoded { event: ChainEvent } - | PoolEventUndecodable { reason: String } - -fn member_string(doc: JsonValue, key: String) -> String? { - match json_object_unique_member(v: doc, key: key) { - JsonMemberFound { value: JsonString { value: s } } => Present { value: s } - _ => none - } -} - -fn member_nonempty(doc: JsonValue, key: String) -> NonEmptyStr? { - match member_string(doc: doc, key: key) { - Present { value: s } => if s == "" { none } else { Present { value: s as NonEmptyStr } } - Absent => none - } -} - -fn member_nat(doc: JsonValue, key: String) -> Nat? { - match json_object_unique_member(v: doc, key: key) { - JsonMemberFound { value: JsonNumber { lexeme: lex } } => - match parse_int(s: lex as String) { - Present { value: n } => if n < 0 { none } else { Present { value: n } } - Absent => none - } - _ => none - } + chain_event_wire_text(schema: pool_event_schema, e: event, kind: pool_event_kind_word(e: event.payload), members: pool_event_kind_members(e: event.payload)) } fn pool_event_kind_decode(doc: JsonValue, word: String) -> PoolEvent? { @@ -175,49 +129,10 @@ fn pool_event_kind_decode(doc: JsonValue, word: String) -> PoolEvent? { } } -fn pool_event_decode(text: String) -> PoolEventDecode { - match parse_json_document(s: text) { - JsonDocumentUnreadable { gap } => PoolEventUndecodable { reason: join(["not JSON: ", json_document_gap_text(gap: gap)], "") } - JsonDocumentParsed { value: doc } => - match member_string(doc: doc, key: "schema") { - Present { value: s } => - if s != pool_event_schema { - PoolEventUndecodable { reason: join(["schema ", s, " is not ", pool_event_schema], "") } - } else { - match member_nonempty(doc: doc, key: "partition") { - Absent => PoolEventUndecodable { reason: "partition missing" } - Present { value: part } => - match member_nat(doc: doc, key: "recorded_at") { - Absent => PoolEventUndecodable { reason: "recorded_at missing or not a non-negative integer" } - Present { value: at } => - match member_nonempty(doc: doc, key: "actor") { - Absent => PoolEventUndecodable { reason: "actor missing" } - Present { value: actor } => - match member_string(doc: doc, key: "kind") { - Absent => PoolEventUndecodable { reason: "kind missing" } - Present { value: kind } => - match pool_event_kind_decode(doc: doc, word: kind) { - Absent => PoolEventUndecodable { reason: join(["kind ", kind, " did not decode"], "") } - Present { value: payload } => - PoolEventDecoded { event: ChainEvent { - partition: (part as String) as PartitionId, - parent: match member_string(doc: doc, key: "parent") { Present { value: p } => if p == "" { none } else { Present { value: p as EventId } } Absent => none }, - recorded_at: at, - actor: actor, - payload: payload, - } } - } - } - } - } - } - } - Absent => PoolEventUndecodable { reason: "schema missing" } - } - } +fn pool_event_decode(text: String) -> EventDecode { + chain_event_decode(text: text, schema: pool_event_schema, kind_decode: fn(doc, word) { pool_event_kind_decode(doc: doc, word: word) }) } -// APPLYING ONE EVENT IS THE POOL TRANSITION IT NAMES, at the instant the chain recorded it. fn pool_apply_event(pool: Pool, env: ChainEnvelope) -> PoolOutcome { let at = env.event.recorded_at match env.event.payload { diff --git a/dag/gunbc/roadmap/roadmap_authority.dag b/dag/gunbc/roadmap/roadmap_authority.dag index 5fbd426326f..c63e5bc9136 100644 --- a/dag/gunbc/roadmap/roadmap_authority.dag +++ b/dag/gunbc/roadmap/roadmap_authority.dag @@ -2722,6 +2722,22 @@ fn declared_roadmap_nodes() -> List { ) ), + active( + identity: "rn_CUTD0TRANSACTION2026091DS41", + id: "serving-v41-cut-d-d0-transaction", + owner: "fleet", + path: "docs/plans/dsv41-cut-d-redesign.md", + t: fields( + headline: "Cut D D0: suspend Group A under an exact operator consent, read the fleet inside it, settle to one typed terminal, and never free a host early", + boundary: "gunbc.spark.pair_serving_d0 over gunbc.spark.pair_serving_authority_log on the fabric DB: one durable genesis per group partition, the host-placement partition as the linearization point, prepare -> append-claim -> authority -> finalize as one saga bound by a typed AuthorityWriteIntent, consumed/cancelled exclusivity, claim-bound quiescence evidence, total crash recovery by lifecycle identity, the operator consent slot as a fabric-DB head (gunbc.durable_cas_fabric_storage, generation-bearing objects), and a wet door that refuses while the store's write walls are missing. STATE (2026-09-20, wound down under operator direction to re-prioritize v1 performance and v2 migration): the whole stack is one branch, plan/dsv41-cut-d-2b, re-rooted onto the rewritten main and source-approved through twenty-one review rounds; the operator ruled it may land under the widened fabric-DB principal drop. Its stacked follow-up branch, plan/dsv41-cut-d-2b-admission-cleanup, deletes the twenty transition-admission rows the stack consumed and carries the detached-process wet-lane admission row.", + displaced_cost: "Without D0 the V4.1 cut over Group A has no transaction: no consent that is spent exactly once across executors, no state in which the group is neither serving nor being mutated by two writers, and no receipt of the fleet as it was read before the authority moved -- the prior state, in which membership and placement were roster words and a crashed lane left no recoverable trace.", + first_slice: "LANDED ON THE BRANCH: the authority log and its folds, the saga and both joined reads, D0 recovery, the fabric-DB CAS, the quiescence observer, the front-door and head-host absence readings, 32 hermetic + 9 wet authority-log claims, 21 hermetic + 13 wet + 6 front-door D0 claims, 2 CAS wet claims, host-commitment and seat gates over the current authority. REMAINING, IN ORDER: (1) land plan/dsv41-cut-d-2b, then its admission-cleanup follow-up (operator merge). (2) The fabric-DB write walls: retire gunbc.rung_drop fabric_storage_append_principal_unrefused by its trigger (observed writer-principal roster in fabric_storage_serve) AND restore gunbc.spark.pair_serving_d0 d0_store_operation_wall (per-operation D0 authorization verified at the store) -- the wet door consumes both and refuses until both stand. (3) Cut 0 keys the V4.1 candidate and the escalation -> ScopedAuthorization producer lands, so resolve_d0_authorization can admit; only then is a fleet run of D0 possible. (4) D1 converger: freeze the host population on PairServingActive (retires the one lane-roster dependence stated on authority_fold) and consume released_baseline_retains_reservation. (5) Retire the detached-process wet-lane row (gunbc.ci_layer_roots excl_local_repo_wet_detached_process_reason): a modeled process/listener fixture and mock_response seams for python.Interpreter.RunFile, the pgrep leg, http.Client.StatusWithin and /proc/net/tcp. (6) A v1 resolver defect found and reproduced on the way (a braced import of extdeps.http.client, or v2.std.algebra { filter }, makes the builtins split/last unresolvable inside a match-arm block; minimal fixture recorded on the branch's pull request, round 18) -- owned by the seed lane, avoided here by homing the curl exit codes with extdeps.tools.curl.", + red_control: "A stranger's abort of a prepared or claimed preparation refuses with nothing written; a cancelled preparation cannot be consumed and a consumed one cannot be cancelled; an authority event that is not the exact write its append claim named is unread on every join; a cycled CAS value does not let a stale writer advance at fabric_storage_advance; the wet door refuses while either store wall is missing; a 401/500 front door is an answer and a failed connect is absence only when the head host is quiet.", + out_of_scope: "Cut 0 candidate keying and the authorization producer; the fabric-DB principal and per-operation walls (fabric-DB lane); D1 convergence; P1 Cut 3 held-seat invalidation; any fleet mutation before the walls stand.", + handback: "Group A suspended under an exact consent with the fleet reading recorded on the log, settled to Suspended / restored Active / FencedRefusal, the consent spent once, every host fence released only by observed quiescence, and the placement finalized -- by execution against the fleet, which no branch has yet done.", + ) + ), + ticket_row( identity: "rn_RPKBFCD41FZGDV48TJKRJYQ8J2", id: "confidence-semantic-impact-query", @@ -3176,6 +3192,7 @@ fn declared_roadmap_edges() -> List { edge(child: "serving-liveness-route-withdrawal", parent: "fleet-spark-inference-serving"), edge(child: "serving-v41-first-party-runtime", parent: "fleet-spark-inference-serving"), edge(child: "serving-v41-first-party-runtime", parent: "serving-liveness-route-withdrawal"), + edge(child: "serving-v41-cut-d-d0-transaction", parent: "serving-v41-first-party-runtime"), edge(child: "toolchain-single-resolver", parent: "toolchain-pin-model"), edge(child: "toolchain-rust-hermetic", parent: "toolchain-single-resolver"), diff --git a/dag/gunbc/rung_drop/fabric_storage_append_principal_unrefused.dag b/dag/gunbc/rung_drop/fabric_storage_append_principal_unrefused.dag index f16ba3fd203..4c5227048ec 100644 --- a/dag/gunbc/rung_drop/fabric_storage_append_principal_unrefused.dag +++ b/dag/gunbc/rung_drop/fabric_storage_append_principal_unrefused.dag @@ -16,7 +16,22 @@ import gunbc.guarantee_rung { Mitigatable, MechanicallyPreventable } // and absent for tagged ones, and which the fleet hosts are has not been read. A roster written // before that reading would either refuse every real writer or admit every one -- a guess standing // where an observation belongs. - +// +// WIDENED BY THE CUT D D0 TRANSACTION (gunbc#11555): the pair-serving authority partitions, the +// host-placement partition and the D0 operator-consent claim slot are written through the same +// unrefused door, so a tailnet member could move a group's authority or take the consent slot +// without passing gunbc.spark.pair_serving_d0's authorization. STATED PLAINLY, THE LOSS IS WIDER +// THAN THIS ROW'S SUBJECT: the trigger below restores OUTSIDER exclusion only -- the served door +// exports generic put/advance and carries no partition, action, D0 subject, grant or attempt to +// the store, so once the roster lands every rostered fleet writer is trusted to mutate every head, +// including the D0 facts, and per-operation D0 authorization at the store is a wall this row does +// not declare restored. Until the operator rules on that trust boundary (gunbc#11555 escalation) +// the wet D0 door consumes THIS ROW'S STANDING and refuses while it stands +// (gunbc.spark.pair_serving_d0 d0_store_write_wall_standing), and refuses again while the +// per-operation wall this row does not cover is missing (gunbc.spark.pair_serving_d0 +// d0_store_operation_wall, a missing construction rather than a drop: it never existed under git +// either) -- so retiring this row alone does not open D0, and Cut 0 with the escalation -> +// authorization producer cannot make D0 executable before both walls stand, by construction. data fabric_storage_append_principal_unrefused: RungDrop = RungDrop { identity: "fabric_storage_append_principal_unrefused" as NonEmptyStr, @@ -33,7 +48,10 @@ data fabric_storage_append_principal_unrefused: RungDrop = RungDrop { population: [ "gunbc.fabric_storage_serve fabric_storage_serve_handle", "gunbc.fabric_event_log event_log_append", + "gunbc.fabric_event_log event_log_append_with", "gunbc.fabric_event_log fabric_seat_acquire", + "gunbc.spark.pair_serving_authority_log (every writer to a pair-serving-authority partition and to host-placement: establishment, transition, entry state, cancellation, host-effect claim and release, placement prepare / claim / finalize / abort)", + "gunbc.durable_cas_fabric_storage fabric_storage_compare_and_set (the D0 operator-consent claim slot)", ], restoration_trigger: "a modeled roster of fabric writer principals, grounded in an observed reading of the identity each fleet writer presents through tailscale serve on srv1 -- SUFFICIENT FOR fabric_storage_serve_handle to refuse a put or advance from any identity outside the roster, including an absent identity, with a discriminating RED over the real handler", } diff --git a/dag/gunbc/runner/runner_microvm_boot_probe.dag b/dag/gunbc/runner/runner_microvm_boot_probe.dag index 87d125acf61..cbf2652d333 100644 --- a/dag/gunbc/runner/runner_microvm_boot_probe.dag +++ b/dag/gunbc/runner/runner_microvm_boot_probe.dag @@ -251,6 +251,12 @@ fn executor_home_wet() -> String? { } } +// (Hoisted to module grain from the body of the item below: a source annotation inside a +// declaration body is not modeled and refuses to parse.) +// THE LOGIN COMES FROM THE HOST'S ROSTER ROW, because the standing now carries the executor's +// group enrolment and that is a fact about a named account. A probe on a host with no roster +// row has no executor to ask about, which is a BootNotAttempted cause like any other. +// // THE HOST STANDING IS CHECKED FIRST AND ITS REFUSAL IS CARRIED FORWARD WHOLE. A host with no // writable /dev/kvm or no Firecracker cannot boot anything, and a probe that discovered that by // failing to produce a console would report an IMAGE verdict for a HOST fact -- sending a reader to @@ -262,9 +268,6 @@ fn runner_microvm_boot_probe_wet() -> ProcessExit Absent => exit_failure(reason: "runner_microvm_boot_probe: HOME is unset, so neither the image root nor the Firecracker install root can be named") Present { value: home } => { let fc_root = firecracker_install_root(executor_home: home) - // THE LOGIN COMES FROM THE HOST'S ROSTER ROW, because the standing now carries the executor's - // group enrolment and that is a fact about a named account. A probe on a host with no roster - // row has no executor to ask about, which is a BootNotAttempted cause like any other. match microvm_host_binding_wet() { MicrovmHostBindingRefused { cause: c } => runner_microvm_boot_probe_receipt_wet( diff --git a/dag/gunbc/spark/fabric_switch_observed.dag b/dag/gunbc/spark/fabric_switch_observed.dag index 0ed971d1a17..82f2a403ffe 100644 --- a/dag/gunbc/spark/fabric_switch_observed.dag +++ b/dag/gunbc/spark/fabric_switch_observed.dag @@ -288,6 +288,13 @@ fn fabric_group_wire(g: FabricGroup) -> NonEmptyStr { } } +// The exact inverse of fabric_group_wire; a word naming no group is no group. +fn parse_fabric_group(wire: String) -> FabricGroup? { + if wire == "group-a" { Present { value: FabricGroupA } } + else if wire == "group-b" { Present { value: FabricGroupB } } + else { none } +} + fn fabric_group_hosts(g: FabricGroup) -> List { map(filter(fabric_lane_observations, o => o.cage == fabric_group_cage(g: g)), o => o.host) } diff --git a/dag/gunbc/spark/host_commitment.dag b/dag/gunbc/spark/host_commitment.dag index 992c3a681e6..d0635d3558d 100644 --- a/dag/gunbc/spark/host_commitment.dag +++ b/dag/gunbc/spark/host_commitment.dag @@ -1,6 +1,6 @@ module gunbc.spark.host_commitment -import std.types { Bool, List, String, NonEmptyStr } +import std.types { Bool, List, String, NonEmptyStr, EpochSecs } import std.algebra { FreeMonoid, Empty, Cons, FreeSemigroup } import v2.std.optional { Present, Absent } import std.decl_ref { declaration_ref_display_key } @@ -16,8 +16,31 @@ import gunbc.spark.cell_role { spark_cell_role_in, spark_cell_role_assignments, } -import gunbc.spark.fabric_switch_observed { FabricGroup, fabric_group_hosts, fabric_group_wire } -import gunbc.spark.pair_serving_authority { PairServingGroupAuthority, spark_pair_serving_authorities, pair_serving_commits_hosts, pair_serving_authority_group } +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, FabricGroupB, fabric_group_hosts, fabric_group_wire, fabric_group_of_host } +import gunbc.spark.pair_serving_authority { PairServingGroupAuthority, spark_pair_serving_authorities, pair_serving_commits_hosts, pair_serving_authority_group, pair_serving_committed_hosts, pair_serving_authority_for } +import gunbc.spark.pair_serving_authority_log { + CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authorities, current_pair_serving_authority, + HostEffectClaim, HostEffectClaimed, HostEffectClaimRefused, HostEffectRelease, HostEffectReleasedAt, HostEffectReleaseRefused, + HostEffectRecord, host_effect_admit, host_effect_release, + PlacementRead, PlacementReadAt, PlacementUnread, placement_read, + LiveClaimLookup, LiveClaimFound, LiveClaimAbsent, LiveClaimUnread, live_claim_on, + GroupPartitionRead, GroupPartitionAt, GroupPartitionUnread, read_group_partition, AuthorityFold, AuthorityFolded, AuthorityUnestablished, AuthorityFoldRefused, fold_refusal_text, current_authority_over, GroupSnapshot, placement_read_over, placement_fold_read, +} +import gunbc.spark.host_effect_quiescence { + HostEffectResidueSpec, HostQuiescenceEvidence, HostQuiescence, HostQuiet, HostResidue, HostQuiescenceUnread, ArgvRun, ArgvRan, ArgvLegDidNotRun, observe_host_effect_quiescence, + ReleaseProvenance, ClaimantTerminal, RecoveryAuthorized, parse_claimant_standing, +} +import gunbc.fleet_ssh_locus { prepare_fleet_ssh_agent_context, FleetSshContextReady, FleetSshContextRefused, fleet_locus_ssh_target } +import gunbc.fleet_known_hosts_anchor { FleetSshExecutionContext } +import gunbc.typed_argv_exec { typed_argv_exec_over_fleet_ssh, TypedArgvExecConverged, TypedArgvExecRefused } + +import gunbc.fabric_storage_client { FabricStorageBinding } +import gunbc.fabric_event_log_host { HostStoreResolved, HostStoreRefused, event_log_store_for_host } +import product.capacity.event_chain { EventId, PartitionId } +import std.measure { Second, second_count } +import std.process { ProcessExit, ExitSuccess, ExitFailure, exit_failure } +import gunbc.spark.fabric_reach { ExecutorReachKnown, ExecutorReachUnknown, observe_executor_reach } +import gunbc.fabric_event_log_host { now_epoch_seconds } import gunbc.spark.llama_cpp_rpc_observed { llama_cpp_rpc_undischarged_peer_reservation_hosts } import gunbc.spark.serving_arm_placement { ServingArmHostClaim, spark_serving_arm_host_claims } import gunbc.spark.dgx_procurement { dgx_spark_reserved_identities } @@ -65,7 +88,7 @@ import gunbc.spark.serving_subject { // The container build and the published-image probe ask it, and the answer reads only OBSERVED // causes -- a role, a claimed group, a placed arm, a held reservation -- because those are the facts // that something is already running or reserved on the machine. Its projection is -// `spark_unplaced_hosts` and its admission seam is `admit_unplaced_host`. +// `spark_unplaced_hosts` and its admission seam is `admit_unplaced_host_live` (an effect reads the current authority first; `admit_unplaced_host` is the pure projection over the source rows). // // The second is ADMISSIBILITY TO A SERVING SUBJECT: which units a particular deployment may use. A // host is admissible to subject S iff EVERY commitment cause on it is held by S. That reading is @@ -173,6 +196,7 @@ type SparkHostAuthorityCommitment | MemberOfClaimedServingGroup { group: FabricGroup } | OccupiedByServingArm { claim: ServingArmHostClaim } | HoldsUndischargedRpcReservation + | OccupiedByHostEffect { purpose: NonEmptyStr, executor: NonEmptyStr } | CellRoleUndecidable { assignments: List } // THE ONE RULING-DERIVED CAUSE, AND ITS BASIS IS A CITATION RATHER THAN A SENTENCE. gunbc.spark.serving_subject @@ -216,6 +240,7 @@ fn spark_host_commitments_in( arm_claims: List, held_reservation_hosts: List, serving_reservations: List, + host_effects: List, host: HostIdentity, ) -> FreeMonoid { let ruled = @@ -226,11 +251,17 @@ fn spark_host_commitments_in( Cons { head: RulingDerivedCommitment { cause: ReservedForServingSubject { subject: r.subject, basis: r.basis } }, tail: acc } }, ) + let effected = + fold( + filter(host_effects, r => !r.released && host_identity_eq(a: r.host, b: host)), + init: ruled, + f: fn(acc, r) { Cons { head: AuthorityHeldCommitment { cause: OccupiedByHostEffect { purpose: r.purpose, executor: r.executor } }, tail: acc } }, + ) let reserved = if any(held_reservation_hosts, h => host_identity_eq(a: h, b: host)) { - Cons { head: AuthorityHeldCommitment { cause: HoldsUndischargedRpcReservation }, tail: ruled } + Cons { head: AuthorityHeldCommitment { cause: HoldsUndischargedRpcReservation }, tail: effected } } else { - ruled + effected } let armed = fold( @@ -295,6 +326,7 @@ fn spark_host_standings_in( arm_claims: List, held_reservation_hosts: List, serving_reservations: List, + host_effects: List, ) -> List { map(units, h => match spark_host_commitments_in( assignments: assignments, @@ -302,6 +334,7 @@ fn spark_host_standings_in( arm_claims: arm_claims, held_reservation_hosts: held_reservation_hosts, serving_reservations: serving_reservations, + host_effects: host_effects, host: h, ) { Empty => SparkHostUncommitted { host: h } @@ -320,6 +353,7 @@ fn spark_host_standing_in( arm_claims: List, held_reservation_hosts: List, serving_reservations: List, + host_effects: List, host: HostIdentity, ) -> SparkHostStanding? { spark_standing_of( @@ -330,6 +364,7 @@ fn spark_host_standing_in( arm_claims: arm_claims, held_reservation_hosts: held_reservation_hosts, serving_reservations: serving_reservations, + host_effects: host_effects, ), host: host, ) @@ -415,6 +450,7 @@ fn serving_subject_claims_host(subject: ServingSubject, siblings: FreeSemigroup< OccupiedByServingArm { claim: cl } => serving_subject_eq(a: subject, b: PlacedServingArm { arm: cl.arm }) CommittedToCellRole { role: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false } RulingDerivedCommitment { cause: d } => @@ -436,6 +472,7 @@ fn spark_commitment_held_by(c: SparkHostCommitment, siblings: FreeSemigroup serving_subject_eq(a: subject, b: PairServingUnitOn { group: g }) OccupiedByServingArm { claim: cl } => serving_subject_eq(a: subject, b: PlacedServingArm { arm: cl.arm }) HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false } RulingDerivedCommitment { cause: d } => @@ -500,12 +537,15 @@ fn spark_admissible_hosts_to(standings: List, subject: Servin // still commits its four hosts, so nothing reads the pause as free capacity and lands a build or a // second lane on a host we are holding. gunbc.spark.pair_serving_authority pair_serving_commits_hosts // is where that answer lives, and ReleasedToFleet is the only arm that drops the commitment. +// WHICH hosts is the authority's own answer (pair_serving_committed_hosts): a transaction's states +// carry the exact population they were authorized over, so a roster that moves mid-transaction +// changes nothing here until the transaction ends. fn spark_claimed_members_under(authorities: List) -> List { flat_map( filter(authorities, a => pair_serving_commits_hosts(a: a)), a => { let g = pair_serving_authority_group(a: a) - map(fabric_group_hosts(g: g), h => ClaimedServingGroupMember { group: g, host: h }) + map(pair_serving_committed_hosts(a: a), h => ClaimedServingGroupMember { group: g, host: h }) }) } @@ -517,17 +557,114 @@ fn spark_claimed_serving_group_hosts() -> List { map(spark_claimed_serving_group_members(), m => m.host) } -data spark_host_standings: List = spark_host_standings_in( - units: dgx_spark_reserved_identities(), - assignments: spark_cell_role_assignments, - claimed_group_members: spark_claimed_serving_group_members(), - arm_claims: spark_serving_arm_host_claims, - held_reservation_hosts: llama_cpp_rpc_undischarged_peer_reservation_hosts(), - serving_reservations: spark_serving_reservations, -) +// THE STANDINGS OVER SUPPLIED AUTHORITIES. The resting projection below hands in the source rows +// (pure, the population every pure consumer reads); the effectful admission seams hand in the +// CURRENT authorities read from the event log (spark_host_standings_current), so an effect that +// runs while a transaction is open commits the hosts the transaction froze, not the roster. +fn spark_host_standings_under(authorities: List, host_effects: List) -> List { + spark_host_standings_in( + units: dgx_spark_reserved_identities(), + assignments: spark_cell_role_assignments, + claimed_group_members: spark_claimed_members_under(authorities: authorities), + arm_claims: spark_serving_arm_host_claims, + held_reservation_hosts: llama_cpp_rpc_undischarged_peer_reservation_hosts(), + serving_reservations: spark_serving_reservations, + host_effects: host_effects, + ) +} + +data spark_host_standings: List = spark_host_standings_under(authorities: spark_pair_serving_authorities, host_effects: [] as List) data spark_unplaced_hosts: List = spark_unplaced_hosts_of(standings: spark_host_standings) +// ── THE CURRENT STANDINGS, FOR AN EFFECT ─────────────────────────────────────────────────────── +// +// An effect that places work on a Spark host (a build, a probe, a source acquisition) may run +// while a pair-serving transaction is open, and then the resting row is not the authority: the +// event log is (gunbc.spark.pair_serving_authority_log). So the admission an effect asks is over +// the CURRENT authorities of every claimed group, read on the executor's store, and refuses when +// any of them could not be read -- an unread authority is not an empty commitment. +type CurrentStandings + = CurrentStandingsRead { standings: List } + | CurrentStandingsUnread { cause: NonEmptyStr } + +// THE LIVE HOST EFFECTS ARE COMMITMENTS TOO: they are supplied by the caller, who read the one +// placement partition (spark_host_standings_over). An effect that is live -- unreleased, however +// old -- commits its host until its release lands with quiescence evidence. +fn spark_host_standings_current(current: List, host_claims: List) -> CurrentStandings { + let unread = flat_map(current, c => match c { + CurrentAuthorityUnread { group: g, step: st, reason: why } => [join([fabric_group_wire(g: g) as String, " at ", st, ": ", why], "")] + CurrentAuthorityRead { authority: _ } => [] as List + }) + if length(unread) != 0 { + CurrentStandingsUnread { cause: join(["the current pair-serving authority could not be read for every claimed group, so host commitment cannot be derived: ", join(unread, "; ")], "") as NonEmptyStr } + } else { + CurrentStandingsRead { standings: spark_host_standings_under(authorities: flat_map(current, c => match c { CurrentAuthorityRead { authority: a } => [a] CurrentAuthorityUnread { group: _, step: _, reason: _ } => [] as List }), host_effects: host_claims) } + } +} + +// THE CURRENT STANDINGS OVER THE LOG. Each group partition is read by the one authority fold from +// its durable genesis: an established group contributes its current authority; a group that is +// unestablished AND undeclared in the source roster contributes nothing; a group the source +// roster declares but the log has not established is UNREAD -- the desired row has not been +// actuated (pair_serving_authority_establish_wet), and until it is, nothing may treat its hosts +// as free. The live host effects come from the one placement partition. Any unread partition is +// the whole reading unread. +type GroupStanding + = GroupStandingEstablished { current: CurrentAuthority } + | GroupStandingUnclaimed + | GroupStandingUnread { cause: String } + +fn group_standing_over(group: FabricGroup, partition: GroupPartitionRead, placement: PlacementRead, at: EpochSecs) -> GroupStanding { + match partition { + GroupPartitionUnread { step: st, reason: why } => GroupStandingUnread { cause: join([fabric_group_wire(g: group) as String, " at ", st, ": ", why], "") } + GroupPartitionAt { head: _, fold: f } => + match f { + AuthorityFoldRefused { at_event: e, reason: why } => GroupStandingUnread { cause: join([fabric_group_wire(g: group) as String, ": ", fold_refusal_text(at_event: e, reason: why)], "") } + AuthorityUnestablished { cancelled: _ } => + match pair_serving_authority_for(group: group) { + Absent => GroupStandingUnclaimed + Present { value: _ } => GroupStandingUnread { cause: join([fabric_group_wire(g: group) as String, ": the source roster declares an authority that has not been established on the log (pair_serving_authority_establish_wet)"], "") } + } + AuthorityFolded { current: _ } => GroupStandingEstablished { current: current_authority_over(group: group, partition: partition, placement: placement, at: at) } + } + } +} + +// ONE READ PER PARTITION: the placement fold and each group's partition are read once, and every +// standing and the joined placement are decided over that one snapshot -- never over independent +// re-reads of a live store that could disagree with each other. +fn spark_host_standings_over(store: FabricStorageBinding, at: EpochSecs) -> CurrentStandings { + let placement = placement_fold_read(store: store) + let groups = map([FabricGroupA, FabricGroupB], g => GroupSnapshot { group: g, partition: read_group_partition(store: store, group: g) }) + let standings = map(groups, gs => group_standing_over(group: gs.group, partition: gs.partition, placement: placement, at: at)) + let unread = flat_map(standings, st => match st { GroupStandingUnread { cause: c } => [c] _ => [] as List }) + if length(unread) != 0 { + CurrentStandingsUnread { cause: join(["a group partition could not be read, so host commitment cannot be derived: ", join(unread, "; ")], "") as NonEmptyStr } + } else { + let current = flat_map(standings, st => match st { GroupStandingEstablished { current: c } => [c] _ => [] as List }) + match placement_read_over(placement: placement, groups: groups) { + PlacementUnread { step: st, reason: why } => CurrentStandingsUnread { cause: join(["the host-placement partition could not be read at ", st, ", so host commitment cannot be derived: ", why], "") as NonEmptyStr } + PlacementReadAt { head: _, effects: effs, preparations: _ } => spark_host_standings_current(current: current, host_claims: effs) + } + } +} + +fn spark_host_standings_live() -> CurrentStandings { + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => CurrentStandingsUnread { cause: join(["the executor could not be identified, so the current authority cannot be read: ", c], "") as NonEmptyStr } + ExecutorReachKnown { short_hostname: executor, path: _ } => + match now_epoch_seconds() { + Absent => CurrentStandingsUnread { cause: "the clock could not be read as epoch seconds, so the lease standing of the current authority cannot be derived" as NonEmptyStr } + Present { value: at } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => CurrentStandingsUnread { cause: d as NonEmptyStr } + HostStoreResolved { store: store, executor: _ } => spark_host_standings_over(store: store, at: at) + } + } + } +} + fn spark_host_is_unplaced(host: HostIdentity) -> Bool { any(spark_unplaced_hosts, h => host_identity_eq(a: h, b: host)) } @@ -617,6 +754,7 @@ fn spark_authority_commitment_wire(c: SparkHostAuthorityCommitment) -> NonEmptyS MemberOfClaimedServingGroup { group: g } => join(["a member of a serving group this repository claims (", fabric_group_wire(g: g) as String, ")"], "") as NonEmptyStr OccupiedByServingArm { claim: c2 } => join(["a serving arm is placed on it (", declaration_ref_display_key(ref: c2.arm), ")"], "") as NonEmptyStr HoldsUndischargedRpcReservation => "holding an undischarged llama.cpp RPC peer reservation" as NonEmptyStr + OccupiedByHostEffect { purpose: p, executor: x } => join(["held by a live host effect (", p as String, " by ", x as String, ") until it is released"], "") as NonEmptyStr CellRoleUndecidable { assignments: _ } => "carrying more than one cell role row, so its commitment cannot be decided" as NonEmptyStr } } @@ -667,19 +805,35 @@ fn spark_host_commitment_wire(c: SparkHostCommitment) -> NonEmptyStr { // unplaced-set observation -> candidate ELIGIBILITY only // target-host claim acquired -> PERMISSION to mutate // terminal release receipt -> host becomes claimable again -// This module answers the first and nothing else. TRIGGER for the second: the fabric/capacity lease -// authority (product.capacity.lease LeaseGrant, gunbc.fabric_event_log fabric_seat_acquire) gaining a -// subject shaped like a whole Spark host rather than a serving seat, at which point admission becomes -// acquire-and-hold and this function's refusal arm becomes the lease's. SUFFICIENT FOR: two -// concurrent runs against one target, dispatched through different executors, cannot both be -// admitted. Until then an admitted answer means "nothing has placed work on it as of this read", and -// the name says so rather than implying a hold. +// This function answers the first and nothing else. THE SECOND AND THIRD NOW EXIST FOR AN EFFECT: +// admit_unplaced_host_over lands a host-effect claim on the fabric event log's host-placement +// partition against the head it read, so two concurrent runs against one +// target, dispatched through different executors, cannot both be admitted -- the second's claim +// is stale or held; and release_host_effect_live is the terminal that observes the host quiet and +// releases, which makes the host claimable again. This pure projection over the resting rows +// still holds no claim, and its name says so. +// AN ADMITTED HOST CARRIES THE CLAIM THAT FENCES IT. The admission is a claim event on the one +// host-placement partition (gunbc.spark.pair_serving_authority_log), appended against the head the +// admission read -- the same head every authority commitment is appended against, which is what +// makes it a fence and not a check. The handle carries what the release must observe: the host +// and the residue the effect declared. The resting projection admits with no claim (it places +// nothing). +type HostEffectHandle { + host: HostIdentity + residue: HostEffectResidueSpec + claim: EventId +} + type SparkHostAdmission - = SparkHostAdmitted { host: HostIdentity } + = SparkHostAdmitted { host: HostIdentity, claim: HostEffectHandle? } | SparkHostRefused { cause: NonEmptyStr } fn spark_authority_held_causes_wire(host: HostIdentity) -> NonEmptyStr { - match spark_host_standing(host: host) { + spark_authority_held_causes_in(standings: spark_host_standings, host: host) +} + +fn spark_authority_held_causes_in(standings: List, host: HostIdentity) -> NonEmptyStr { + match spark_standing_of(standings: standings, host: host) { Absent => "that host is not a DGX Spark unit in gunbc.spark.dgx_procurement, so this fleet has no standing for it" as NonEmptyStr Present { value: standing } => match standing { @@ -712,10 +866,195 @@ fn admit_unplaced_host_among(raw: String, purpose: NonEmptyStr, unplaced: List) -> SparkHostAdmission { + admit_unplaced_host_among(raw: raw, purpose: purpose, unplaced: spark_unplaced_hosts_of(standings: standings), held_causes: fn(h) { spark_authority_held_causes_in(standings: standings, host: h) as String }) +} + + + +// THE EFFECT'S ADMISSION IS A CLAIM, NOT A READ. The current standings admit, then the claim lands +// on the host-placement partition against the head that was read: stale when the partition moved +// in between, committed when a live authority commitment holds the host, held when another live +// effect has it. Every admitted host carries its claim; the effect releases it at its terminal +// with the observation that the host is quiet. `term` is the effect's own stated bound: past it +// the claim reads OVERDUE, and it still holds until released. `residue` is what the release must +// find absent -- the effect declares it here because only the effect knows what it leaves. +fn admit_unplaced_host_over(raw: String, purpose: NonEmptyStr, term: Second, residue: HostEffectResidueSpec, store: FabricStorageBinding, executor: NonEmptyStr, at: EpochSecs) -> SparkHostAdmission { + match spark_host_standings_over(store: store, at: at) { + CurrentStandingsUnread { cause: c } => SparkHostRefused { cause: join([raw, " is not admissible for ", purpose as String, ": ", c as String], "") as NonEmptyStr } + CurrentStandingsRead { standings: st } => + match admit_unplaced_host_in(raw: raw, purpose: purpose, standings: st) { + SparkHostRefused { cause: c } => SparkHostRefused { cause: c } + SparkHostAdmitted { host: host, claim: _ } => { + match host_effect_admit(store: store, host: host, purpose: purpose, executor: executor, term: term, residue: residue, at: at) { + HostEffectClaimed { id: id } => + SparkHostAdmitted { host: host, claim: Present { value: HostEffectHandle { host: host, residue: residue, claim: id } } } + HostEffectClaimRefused { partition: pt, step: st2, reason: why } => SparkHostRefused { cause: join([raw, " is not admissible for ", purpose as String, ": the host-effect claim did not land on ", pt as String, " at ", st2, ": ", why], "") as NonEmptyStr } + } + } + } + } +} + +fn admit_unplaced_host_live(raw: String, purpose: NonEmptyStr, term: Second, residue: HostEffectResidueSpec) -> SparkHostAdmission { + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => SparkHostRefused { cause: join([raw, " is not admissible for ", purpose as String, ": the executor could not be identified: ", c], "") as NonEmptyStr } + ExecutorReachKnown { short_hostname: executor, path: _ } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => SparkHostRefused { cause: join([raw, " is not admissible for ", purpose as String, ": ", d], "") as NonEmptyStr } + HostStoreResolved { store: store, executor: _ } => + match now_epoch_seconds() { + Absent => SparkHostRefused { cause: join([raw, " is not admissible for ", purpose as String, ": the clock could not be read as epoch seconds"], "") as NonEmptyStr } + Present { value: at } => admit_unplaced_host_over(raw: raw, purpose: purpose, term: term, residue: residue, store: store, executor: executor as NonEmptyStr, at: at) + } + } + } +} + +// THE RELEASE IS THE EVIDENCE LANDING: given a quiescence reading of the claim's host, append the +// release. A release that did not land is returned as its cause, never hidden -- the host stays +// held until one lands. +fn release_host_effect_over(handle: HostEffectHandle, evidence: HostQuiescenceEvidence, provenance: ReleaseProvenance, store: FabricStorageBinding, executor: NonEmptyStr, at: EpochSecs) -> String? { + match host_effect_release(store: store, claim: handle.claim, evidence: evidence, provenance: provenance, executor: executor, at: at) { + HostEffectReleasedAt { id: _ } => none + HostEffectReleaseRefused { partition: pt, step: st, reason: why } => Present { value: join(["the host-effect claim ", handle.claim as String, " was not released on ", pt as String, " at ", st, ": ", why, "; the host stays held until a release with quiescence evidence lands"], "") } + } +} + +// THE OBSERVATION LEG TO A SPARK: the fleet automation principal over the runner's agent-held key, +// the same route the build seams take. A leg that could not be prepared, or that was refused +// before it ran, is a leg that did not run -- the observer reads that as UNREAD, never as quiet. +data host_effect_release_attempt_raw: String = "host-effect-release" + +fn fleet_argv_run(context: FleetSshExecutionContext, host: HostIdentity, argv: List) -> ArgvRun { + match typed_argv_exec_over_fleet_ssh(target: fleet_locus_ssh_target(host: host), context: context, argv: argv) { + TypedArgvExecRefused { reason: why } => ArgvLegDidNotRun { cause: why } + TypedArgvExecConverged { result: r } => ArgvRan { exit_code: r.exit_code, stdout: r.stdout, stderr: r.stderr } + } +} + +// OBSERVE, THEN RELEASE. The observation is taken on the host after the effect's body returned, +// whatever it returned: a quiet host releases the claim; residue or an unread scan leaves the +// claim live and says so on the exit. Nothing here reads the exit as evidence. +type HostEffectSettlement + = HostEffectSettled + | HostEffectStillHeld { cause: String } + +fn settle_host_effect_over(handle: HostEffectHandle, provenance: ReleaseProvenance, run: fn(List) -> ArgvRun, store: FabricStorageBinding, executor: NonEmptyStr, at: EpochSecs) -> HostEffectSettlement { + match observe_host_effect_quiescence(run: run, claim: handle.claim, host: handle.host, spec: handle.residue, observer: executor, at: at) { + HostQuiescenceUnread { host: h, cause: c } => HostEffectStillHeld { cause: join([h as String, " could not be read for quiescence, so the host-effect claim ", handle.claim as String, " stays live: ", c], "") } + HostResidue { host: h, detail: d } => HostEffectStillHeld { cause: join([h as String, " still carries the effect, so the host-effect claim ", handle.claim as String, " stays live: ", d], "") } + HostQuiet { evidence: ev } => + match release_host_effect_over(handle: handle, evidence: ev, provenance: provenance, store: store, executor: executor, at: at) { + Absent => HostEffectSettled + Present { value: why } => HostEffectStillHeld { cause: why } + } + } +} + +fn settle_host_effect_live(handle: HostEffectHandle) -> HostEffectSettlement { + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => HostEffectStillHeld { cause: join(["the executor could not be identified for the release: ", c], "") } + ExecutorReachKnown { short_hostname: executor, path: _ } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => HostEffectStillHeld { cause: d } + HostStoreResolved { store: store, executor: _ } => + match now_epoch_seconds() { + Absent => HostEffectStillHeld { cause: "the clock could not be read as epoch seconds for the release" } + Present { value: at } => + match prepare_fleet_ssh_agent_context(attempt_raw: host_effect_release_attempt_raw) { + FleetSshContextRefused { cause: c } => HostEffectStillHeld { cause: join(["the host could not be reached to observe quiescence, so the host-effect claim ", handle.claim as String, " stays live: ", c], "") } + FleetSshContextReady { context: context, receipt: _ } => + settle_host_effect_over(handle: handle, provenance: ClaimantTerminal { claim: handle.claim, claimant_executor: executor as NonEmptyStr }, run: fn(argv) { fleet_argv_run(context: context, host: handle.host, argv: argv) }, store: store, executor: executor as NonEmptyStr, at: at) + } + } + } + } +} + +fn release_host_effect_live(handle: HostEffectHandle?, exit: ProcessExit) -> ProcessExit { + match handle { + Absent => exit + Present { value: h } => + match settle_host_effect_live(handle: h) { + HostEffectSettled => exit + HostEffectStillHeld { cause: why } => + match exit { + ExitSuccess => exit_failure(reason: why) + ExitFailure { code: _, reason: r } => exit_failure(reason: join([r, "\n", why], "")) + } + } + } +} + +// ── RECOVERING A CRASHED LANE'S CLAIM ────────────────────────────────────────────────────────── +// +// A lane that died between its claim and its release left the claim live, and nothing releases +// it on a clock. This entry is the authorized route, and it consumes TWO facts the normal +// terminal has by construction: the exact claim (never "whatever is live on the host"), and the +// claimant's standing -- terminated or abandoned -- as a typed operator disposition with the +// receipt that established it (a cancelled run's id, a dead process's last observation). A quiet +// host alone is not enough: a live lane between two legs is quiet too, and releasing it would +// admit another placement under it. The observation is then taken against THAT claim's residue, +// and the release lands only when the host is quiet. +// gunbc run --source-root dag --source-root src/v2 --entry dag/gunbc/spark/host_commitment.dag +// --function host_effect_recover_wet --arg host= --arg claim= +// --arg claimant=terminated|abandoned --arg receipt= +fn host_effect_recover_wet(host: String, claim: String, claimant: String, receipt: String) -> ProcessExit + uses net: Network +{ + if host == "" || claim == "" { + exit_failure(reason: "a host and the exact claim (event id) are required; this entry does not pick a claim by host") + } else if receipt == "" { + exit_failure(reason: "a receipt establishing the claimant's standing is required (what showed the lane terminated or abandoned)") + } else { + match parse_claimant_standing(wire: claimant) { + Absent => exit_failure(reason: join(["`", claimant, "` is not a claimant standing; say terminated or abandoned"], "")) + Present { value: standing } => { + let h = host as HostIdentity + let c = (claim as NonEmptyStr) as EventId + match observe_executor_reach() { + ExecutorReachUnknown { cause: cause } => exit_failure(reason: join(["the executor could not be identified: ", cause], "")) + ExecutorReachKnown { short_hostname: executor, path: _ } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => exit_failure(reason: d) + HostStoreResolved { store: store, executor: _ } => + match now_epoch_seconds() { + Absent => exit_failure(reason: "the clock could not be read as epoch seconds") + Present { value: at } => { + let provenance = RecoveryAuthorized { claim: c, claimant: standing, authorized_by: executor as NonEmptyStr, receipt: receipt as NonEmptyStr } + match live_claim_on(store: store, host: h, claim: c) { + LiveClaimUnread { step: st, reason: why } => exit_failure(reason: join(["the host-placement partition could not be read at ", st, ": ", why], "")) + LiveClaimAbsent => exit_failure(reason: join(["no live host-effect claim ", claim, " on ", host, "; nothing to recover"], "")) + LiveClaimFound { record: r } => + match prepare_fleet_ssh_agent_context(attempt_raw: host_effect_release_attempt_raw) { + FleetSshContextRefused { cause: cause } => exit_failure(reason: cause) + FleetSshContextReady { context: context, receipt: _ } => { + let handle = HostEffectHandle { host: h, residue: r.residue, claim: r.id } + match settle_host_effect_over(handle: handle, provenance: provenance, run: fn(argv) { fleet_argv_run(context: context, host: h, argv: argv) }, store: store, executor: executor as NonEmptyStr, at: at) { + HostEffectSettled => ExitSuccess + HostEffectStillHeld { cause: why } => exit_failure(reason: why) + } + } + } + } + } + } + } + } + } + } } } +// The admission over the RESTING rows -- the pure projection, for consumers with no effect to +// admit. An effect asks admit_unplaced_host_live, which reads the current authority and claims. fn admit_unplaced_host(raw: String, purpose: NonEmptyStr) -> SparkHostAdmission { admit_unplaced_host_among( raw: raw, diff --git a/dag/gunbc/spark/host_effect_quiescence.dag b/dag/gunbc/spark/host_effect_quiescence.dag new file mode 100644 index 00000000000..4294d33adb0 --- /dev/null +++ b/dag/gunbc/spark/host_effect_quiescence.dag @@ -0,0 +1,275 @@ +module gunbc.spark.host_effect_quiescence + +import std.types { String, Bool, Int, NonEmptyStr, List, EpochSecs } +import v2.std.optional { Present, Absent } +import std.algebra { trim } +import product.placement_supply { HostIdentity, host_identity_eq } +import product.capacity.event_chain { EventId } +import extdeps.exec.command { argv_words } +import extdeps.procps.pgrep { PgrepMatched, PgrepNoMatch, PgrepScanFailed, pgrep_match_full_command_line_argv, pgrep_scan_outcome } +import extdeps.docker.cli { DockerPsFilter, DockerPsByName, DockerPsByAncestor, docker_ps_filter_wire, docker_ps_running_command } +import extdeps.languages.json.emit { JsonValue, JsonKeyValue, json_kv, json_string, json_int } +import product.capacity.event_json { member_nonempty, member_nat } + +// ── A HOST EFFECT IS RELEASED BY EVIDENCE THAT NOTHING OF IT REMAINS, NEVER BY ITS RETURN ────── +// +// A build, a probe or a source acquisition returns; whether the host is QUIET with respect to it +// is a different fact. A leg can exit non-zero while a build client or a started container +// remains, and a crashed lane never returns at all. So the fact that clears a host-effect fence is +// an OBSERVATION taken on the host after the effect: the effect declares, when it is admitted, +// what its residue would look like (a process command-line pattern, and the container it starts, +// if any), and the release carries the reading that found neither. That reading is +// sole-constructed here -- only the observer mints it, from an executed scan -- so a release +// cannot be written from a return code, an elapsed term or prose. THE EVIDENCE NAMES THE CLAIM IT +// WAS TAKEN FOR and the instant it was taken: a quiet reading of the same host and residue made +// for an earlier claim, or before this claim was admitted, is not evidence that THIS effect is +// gone, and the fold refuses it (release_admission in the authority log). +// +// WHAT THE READING IS: pgrep over the full command line for the pattern (the same leg the serving +// incarnation observer uses; exit 1 is "no match", any other non-zero is a scan that did not +// reach a verdict) and, when a container is declared, `docker ps` under that filter (empty output +// is "not running"). A scan that could not run, or docker that could not be asked, is UNREAD -- +// the claim stays live -- never quiet. The pattern is pgrep's extended regular expression, so a +// path or an image reference in it matches slightly more than itself (`.` is any character); that +// direction is the safe one, since a wider match reports residue and refuses the release. +type HostEffectResidueSpec { + process_pattern: NonEmptyStr + container: DockerPsFilter? +} + +type HostQuiescenceEvidence sole_constructor { + claim: EventId + host: HostIdentity + process_pattern: NonEmptyStr + container: DockerPsFilter? + observer: NonEmptyStr + observed_at: EpochSecs +} + +// ── WHY A RELEASE IS ADMITTED: THE CLAIMANT'S STANDING ───────────────────────────────────────── +// +// Quiet is one fact; that the claimant will do nothing further on the host is another. The normal +// terminal has it by construction: the claimant itself observes after its body returned and +// releases its own claim. A crashed or cancelled lane cannot say so, and a quiet reading taken +// while a live lane is between two legs is not that fact either. So a recovery release carries a +// typed operator disposition BOUND TO THE EXACT CLAIM -- the claimant is terminated (its process +// or run is known ended) or abandoned (its holder has surrendered it) -- with the receipt that +// established it. That fact lies outside what the log can observe (DESIGN §4b: outside the modeled +// guarantee), so it enters at this declared boundary as a typed value, never as prose on a String +// receipt, and the fold refuses a recovery provenance naming another claim. +type ClaimantStanding + = ClaimantTerminated + | ClaimantAbandoned + +type ReleaseProvenance + = ClaimantTerminal { claim: EventId, claimant_executor: NonEmptyStr } + | RecoveryAuthorized { claim: EventId, claimant: ClaimantStanding, authorized_by: NonEmptyStr, receipt: NonEmptyStr } + +fn parse_claimant_standing(wire: String) -> ClaimantStanding? { + if wire == "terminated" { Present { value: ClaimantTerminated } } + else if wire == "abandoned" { Present { value: ClaimantAbandoned } } + else { none } +} + +fn claimant_standing_wire(c: ClaimantStanding) -> String { + match c { ClaimantTerminated => "terminated" ClaimantAbandoned => "abandoned" } +} + +fn release_provenance_members(p: ReleaseProvenance) -> List { + match p { + ClaimantTerminal { claim: c, claimant_executor: x } => [json_kv(key: "provenance", value: json_string(s: "claimant-terminal")), json_kv(key: "terminal_claim", value: json_string(s: c as String)), json_kv(key: "terminal_claimant", value: json_string(s: x as String))] + RecoveryAuthorized { claim: c, claimant: st, authorized_by: by, receipt: r } => [ + json_kv(key: "provenance", value: json_string(s: "recovery")), + json_kv(key: "recovery_claim", value: json_string(s: c as String)), + json_kv(key: "recovery_claimant", value: json_string(s: claimant_standing_wire(c: st))), + json_kv(key: "recovery_authorized_by", value: json_string(s: by as String)), + json_kv(key: "recovery_receipt", value: json_string(s: r as String)), + ] + } +} + +fn release_provenance_decode(doc: JsonValue) -> ReleaseProvenance? { + match member_nonempty(doc: doc, key: "provenance") { + Absent => none + Present { value: w } => + if (w as String) == "claimant-terminal" { + match member_nonempty(doc: doc, key: "terminal_claim") { + Absent => none + Present { value: c } => + match member_nonempty(doc: doc, key: "terminal_claimant") { + Absent => none + Present { value: x } => Present { value: ClaimantTerminal { claim: (c as String) as EventId, claimant_executor: x } } + } + } + } + else if (w as String) != "recovery" { none } + else { + match member_nonempty(doc: doc, key: "recovery_claim") { + Absent => none + Present { value: c } => + match member_nonempty(doc: doc, key: "recovery_claimant") { + Absent => none + Present { value: st } => + match parse_claimant_standing(wire: st as String) { + Absent => none + Present { value: standing } => + match member_nonempty(doc: doc, key: "recovery_authorized_by") { + Absent => none + Present { value: by } => + match member_nonempty(doc: doc, key: "recovery_receipt") { + Absent => none + Present { value: r } => Present { value: RecoveryAuthorized { claim: (c as String) as EventId, claimant: standing, authorized_by: by, receipt: r } } + } + } + } + } + } + } + } +} + +// A provenance is for a claim when it is the claimant's own terminal -- naming this claim and its +// claimant, and WRITTEN BY that claimant (the event's actor) -- or a recovery naming this claim. +fn release_provenance_covers(p: ReleaseProvenance, claim: EventId, claimant: NonEmptyStr, actor: NonEmptyStr) -> Bool { + match p { + ClaimantTerminal { claim: c, claimant_executor: x } => (c as String) == (claim as String) && (x as String) == (claimant as String) && (actor as String) == (claimant as String) + RecoveryAuthorized { claim: c, claimant: _, authorized_by: _, receipt: _ } => (c as String) == (claim as String) + } +} + +type HostQuiescence + = HostQuiet { evidence: HostQuiescenceEvidence } + | HostResidue { host: HostIdentity, detail: String } + | HostQuiescenceUnread { host: HostIdentity, cause: String } + +// THE LEG IS SUPPLIED: production hands the fleet-ssh typed-argv leg to the host, a witness hands a +// local one. The observer decides from exit codes and stdout only. +type ArgvRun + = ArgvRan { exit_code: Int, stdout: String, stderr: String } + | ArgvLegDidNotRun { cause: String } + +fn observe_host_effect_quiescence(run: fn(List) -> ArgvRun, claim: EventId, host: HostIdentity, spec: HostEffectResidueSpec, observer: NonEmptyStr, at: EpochSecs) -> HostQuiescence { + match run(pgrep_match_full_command_line_argv(pattern: spec.process_pattern)) { + ArgvLegDidNotRun { cause: c } => HostQuiescenceUnread { host: host, cause: join(["the process scan did not run: ", c], "") } + ArgvRan { exit_code: code, stdout: out, stderr: err } => + match pgrep_scan_outcome(exit_code: code, stdout: out, stderr: err) { + PgrepScanFailed { exit_code: c2, stderr: e2 } => HostQuiescenceUnread { host: host, cause: join(["the process scan failed (exit ", to_string(c2), "): ", e2], "") } + PgrepMatched { listing: l } => HostResidue { host: host, detail: join(["a process matching `", spec.process_pattern as String, "` is running: ", trim(s: l)], "") } + PgrepNoMatch => + match spec.container { + Absent => HostQuiet { evidence: HostQuiescenceEvidence { claim: claim, host: host, process_pattern: spec.process_pattern, container: none, observer: observer, observed_at: at } } + Present { value: f } => + match run(argv_words(command: docker_ps_running_command(filter: f))) { + ArgvLegDidNotRun { cause: c } => HostQuiescenceUnread { host: host, cause: join(["docker ps did not run: ", c], "") } + ArgvRan { exit_code: code2, stdout: out2, stderr: err2 } => + if code2 != 0 { + HostQuiescenceUnread { host: host, cause: join(["docker ps exited ", to_string(code2), ": ", trim(s: err2)], "") } + } else { + if trim(s: out2) != "" { + HostResidue { host: host, detail: join(["a container under `", docker_ps_filter_wire(f: f), "` is running: ", trim(s: out2)], "") } + } else { + HostQuiet { evidence: HostQuiescenceEvidence { claim: claim, host: host, process_pattern: spec.process_pattern, container: Present { value: f }, observer: observer, observed_at: at } } + } + } + } + } + } + } +} + +fn container_filter_eq(a: DockerPsFilter?, b: DockerPsFilter?) -> Bool { + match a { + Absent => match b { Absent => true Present { value: _ } => false } + Present { value: x } => match b { Absent => false Present { value: y } => docker_ps_filter_wire(f: x) == docker_ps_filter_wire(f: y) } + } +} + +// Evidence is FOR a claim only when it names that claim, its host and its own residue spec, and +// was taken no earlier than the claim was admitted: a quiet reading for another claim, of another +// host or pattern, or from before the admission, releases nothing. +fn quiescence_evidence_covers(e: HostQuiescenceEvidence, claim: EventId, admitted_at: EpochSecs, host: HostIdentity, spec: HostEffectResidueSpec) -> Bool { + (e.claim as String) == (claim as String) + && e.observed_at >= admitted_at + && host_identity_eq(a: e.host, b: host) + && (e.process_pattern as String) == (spec.process_pattern as String) + && container_filter_eq(a: e.container, b: spec.container) +} + +// ── THE WIRE ─────────────────────────────────────────────────────────────────────────────────── +// +// Emitted and decoded here because the evidence is sole-constructed: the event log's codec +// consumes these fns rather than constructing the evidence itself, so a decoded event's evidence +// is exactly what an observer once wrote (the codec is total over the wire and refuses a document +// that does not carry every member). +fn container_filter_members(prefix: String, f: DockerPsFilter?) -> List { + match f { + Absent => [] as List + Present { value: DockerPsByName { name: n } } => [json_kv(key: join([prefix, "container_name"], ""), value: json_string(s: n as String))] + Present { value: DockerPsByAncestor { image: i } } => [json_kv(key: join([prefix, "container_image"], ""), value: json_string(s: i as String))] + } +} + +type ContainerFilterMember + = ContainerFilterAbsent + | ContainerFilterPresent { filter: DockerPsFilter } + | ContainerFilterIncoherent + +fn container_filter_decode(doc: JsonValue, prefix: String) -> ContainerFilterMember { + let by_name = member_nonempty(doc: doc, key: join([prefix, "container_name"], "")) + let by_image = member_nonempty(doc: doc, key: join([prefix, "container_image"], "")) + match by_name { + Present { value: n } => match by_image { Absent => ContainerFilterPresent { filter: DockerPsByName { name: n } } Present { value: _ } => ContainerFilterIncoherent } + Absent => match by_image { Absent => ContainerFilterAbsent Present { value: i } => ContainerFilterPresent { filter: DockerPsByAncestor { image: i } } } + } +} + +fn residue_spec_members(prefix: String, s: HostEffectResidueSpec) -> List { + concat([json_kv(key: join([prefix, "process_pattern"], ""), value: json_string(s: s.process_pattern as String))], container_filter_members(prefix: prefix, f: s.container)) +} + +fn residue_spec_decode(doc: JsonValue, prefix: String) -> HostEffectResidueSpec? { + match member_nonempty(doc: doc, key: join([prefix, "process_pattern"], "")) { + Absent => none + Present { value: p } => + match container_filter_decode(doc: doc, prefix: prefix) { + ContainerFilterIncoherent => none + ContainerFilterAbsent => Present { value: HostEffectResidueSpec { process_pattern: p, container: none } } + ContainerFilterPresent { filter: f } => Present { value: HostEffectResidueSpec { process_pattern: p, container: Present { value: f } } } + } + } +} + +fn quiescence_evidence_members(e: HostQuiescenceEvidence) -> List { + concat( + [ + json_kv(key: "evidence_claim", value: json_string(s: e.claim as String)), + json_kv(key: "evidence_host", value: json_string(s: e.host as String)), + json_kv(key: "evidence_observer", value: json_string(s: e.observer as String)), + json_kv(key: "evidence_observed_at", value: json_int(n: e.observed_at)), + ], + residue_spec_members(prefix: "evidence_", s: HostEffectResidueSpec { process_pattern: e.process_pattern, container: e.container }), + ) +} + +fn quiescence_evidence_decode(doc: JsonValue) -> HostQuiescenceEvidence? { + match member_nonempty(doc: doc, key: "evidence_claim") { + Absent => none + Present { value: c } => + match member_nonempty(doc: doc, key: "evidence_host") { + Absent => none + Present { value: h } => + match member_nonempty(doc: doc, key: "evidence_observer") { + Absent => none + Present { value: o } => + match member_nat(doc: doc, key: "evidence_observed_at") { + Absent => none + Present { value: at } => + match residue_spec_decode(doc: doc, prefix: "evidence_") { + Absent => none + Present { value: spec } => Present { value: HostQuiescenceEvidence { claim: (c as String) as EventId, host: (h as String) as HostIdentity, process_pattern: spec.process_pattern, container: spec.container, observer: o, observed_at: at } } + } + } + } + } + } +} diff --git a/dag/gunbc/spark/pair_serving_apply.dag b/dag/gunbc/spark/pair_serving_apply.dag index 6176f0c06d7..3ab4443bc47 100644 --- a/dag/gunbc/spark/pair_serving_apply.dag +++ b/dag/gunbc/spark/pair_serving_apply.dag @@ -1,6 +1,7 @@ module gunbc.spark.pair_serving_apply -import std.types { String, Bool, List, NonEmptyStr, Secret, FilePath } +import std.types { String, Bool, List, NonEmptyStr, Secret, FilePath, EpochSecs } +import gunbc.fabric_event_log_host { now_epoch_seconds } import std.algebra { trim } import std.process { ProcessExit, ExitSuccess, exit_failure } import std.resources { Network } @@ -15,7 +16,10 @@ import gunbc.fleet_bootstrap_principal_session { } import extdeps.docker.images.sparkrun_vllm_ds4_gb10 { gunbc_spark_pair_runtime_dockerfile } import gunbc.spark.fabric_switch_observed { FabricGroup } -import gunbc.spark.pair_serving_authority { PairServingGroupAuthority, pair_serving_apply_admits, pair_serving_authority_for } +import gunbc.spark.pair_serving_authority { PairServingGroupAuthority, pair_serving_apply_admits } +import gunbc.spark.pair_serving_authority_log { + CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authorities, current_authority_for, current_authority_wire, +} import gunbc.auth.materialized_secret { with_materialized_ssh_key_file, MaterializedSshKeyFileBracketCompleted, MaterializedSshKeyFileBracketRefused } import gunbc.clock_read { clock_now_probed_at, probed_at_word } import gunbc.spark.fabric_reach { SparkReachPath, ReachManagementLan, ReachFabricRail, spark_reach_path_wire, spark_reach_endpoint, ExecutorReach, ExecutorReachKnown, ExecutorReachUnknown, observe_executor_reach } @@ -166,7 +170,7 @@ fn pair_serving_authority_admits_apply(a: PairServingGroupAuthority?) -> Bool { // decorative. An earlier shape looked the authority up inside this fold, so the only way to reach the // refusing arm was to edit the live row -- no control could SUPPLY a suspended authority, and deleting // the gate entirely would have left every claim green. A check whose red is unauthorable is worse than -// absent because it gets cited as coverage (DESIGN section 4b). spark_pair_apply_plan below is the thin +// absent because it gets cited as coverage (DESIGN section 4b). spark_pair_apply_plans_current below is the thin // binding that supplies the live authority, exactly as fabric_capacity_standing binds its verdict. fn spark_pair_apply_plan_under(r: SparkPairRealization, authority: PairServingGroupAuthority?) -> SparkPairApplyPlan { match r { @@ -194,8 +198,22 @@ fn spark_pair_apply_plan_under(r: SparkPairRealization, authority: PairServingGr } } -fn spark_pair_apply_plan(r: SparkPairRealization) -> SparkPairApplyPlan { - spark_pair_apply_plan_under(r: r, authority: pair_serving_authority_for(group: spark_pair_realization_group(r: r))) +// THE LIVE AUTHORITY IS THE LOG'S, NOT THE SOURCE ROW'S. gunbc.spark.pair_serving_authority_log +// folds the row through every recorded transition, so a group a transaction has suspended is +// suspended HERE, from the transaction's own tree. A group whose authority could not be read is +// refused with the read's cause -- never planned on the row it rests on -- and a group the roster +// does not name at all is refused as having no authority (absence is not permission). +fn spark_pair_apply_plan_current(r: SparkPairRealization, current: List) -> SparkPairApplyPlan { + match current_authority_for(current: current, group: spark_pair_realization_group(r: r)) { + Absent => spark_pair_apply_plan_under(r: r, authority: none) + Present { value: c } => + match c { + CurrentAuthorityUnread { group: _, step: _, reason: _ } => + SparkPairApplyRefused { cause: join(["the serving authority could not be read, so nothing is converged: ", current_authority_wire(c: c)], "") } + CurrentAuthorityRead { authority: a, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + spark_pair_apply_plan_under(r: r, authority: Present { value: a }) + } + } } fn spark_pair_realization_group(r: SparkPairRealization) -> FabricGroup { @@ -205,11 +223,15 @@ fn spark_pair_realization_group(r: SparkPairRealization) -> FabricGroup { } } -// EVERY PROMOTED GROUP IS PLANNED, IN ROSTER ORDER. One refusal anywhere refuses the whole apply: -// a converge that brought up group A while silently skipping group B would report success for a -// desired state it did not reach. -fn spark_pair_apply_plans() -> List { - map(spark_pair_realizations(), r => spark_pair_apply_plan(r: r)) +// EVERY PROMOTED GROUP IS PLANNED, IN ROSTER ORDER, UNDER THE AUTHORITY THE EXECUTOR READS. One +// refusal anywhere refuses the whole apply: a converge that brought up group A while silently +// skipping group B would report success for a desired state it did not reach. +fn spark_pair_apply_plans_current(current: List) -> List { + map(spark_pair_realizations(), r => spark_pair_apply_plan_current(r: r, current: current)) +} + +fn spark_pair_apply_plans(executor: String, at: EpochSecs) -> List { + spark_pair_apply_plans_current(current: current_pair_serving_authorities(short_hostname: executor, at: at)) } fn spark_pair_apply_plan_refusals(plans: List) -> List { @@ -293,16 +315,19 @@ fn spark_pair_serving_apply_wet() -> ProcessExit uses net: Network { let now = clock_now_probed_at() - let plans = spark_pair_apply_plans() - let refusals = spark_pair_apply_plan_refusals(plans: plans) - if length(plans) == 0 { - exit_failure(reason: "spark_pair_serving_apply: no group is promoted to pair serving") - } else if length(refusals) != 0 { - exit_failure(reason: join(["spark_pair_serving_apply: a group realization refused: ", join(refusals, "; ")], "")) - } else { - match observe_executor_reach() { - ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["spark_pair_serving_apply: ", c], "")) - ExecutorReachKnown { short_hostname: executor, path: path } => + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["spark_pair_serving_apply: ", c], "")) + ExecutorReachKnown { short_hostname: executor, path: path } => { + match now_epoch_seconds() { + Absent => exit_failure(reason: "spark_pair_serving_apply: the clock could not be read as epoch seconds, so the lease standing of the authority cannot be derived") + Present { value: at } => { + let plans = spark_pair_apply_plans(executor: executor, at: at) + let refusals = spark_pair_apply_plan_refusals(plans: plans) + if length(plans) == 0 { + exit_failure(reason: "spark_pair_serving_apply: no group is promoted to pair serving") + } else if length(refusals) != 0 { + exit_failure(reason: join(["spark_pair_serving_apply: a group realization refused: ", join(refusals, "; ")], "")) + } else { match read_bootstrap_credential() { BootstrapCredentialNotMaterialized { cause: c } => exit_failure(reason: c as String) BootstrapCredentialReady { secret: administrator } => @@ -351,3 +376,6 @@ fn spark_pair_serving_apply_wet() -> ProcessExit } } } +} +} +} diff --git a/dag/gunbc/spark/pair_serving_authority.dag b/dag/gunbc/spark/pair_serving_authority.dag index 622c5b11137..03c89f12c1c 100644 --- a/dag/gunbc/spark/pair_serving_authority.dag +++ b/dag/gunbc/spark/pair_serving_authority.dag @@ -4,7 +4,9 @@ import std.types { Bool, NonEmptyStr, List } import std.content_hash { ContentHash } import std.temporal_effect { HeldLease, HeldLeaseObservedState, held_lease_observed_verdict } import std.upsert_decision { ObservationVerdict, Converged, Absent, Drifted, Conflict, Inaccessible, UnknownRefused } -import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, fabric_group_wire } +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, fabric_group_wire, fabric_group_hosts } +import product.placement_supply { HostIdentity } +import gunbc.spark.released_baseline { ReleasedBaselineSpec } // WHOSE HOSTS WE CLAIM AND WHAT WE MAY DO WITH THEM ARE TWO QUESTIONS WITH ONE ANSWER, AND IT IS HERE. // A bare roster of claimed groups could say which groups are ours and nothing about whether we are free @@ -46,18 +48,22 @@ type PairServingGroupAuthority } | SuspensionPendingReconciliation { group: FabricGroup + hosts: List transaction: NonEmptyStr + authorization_binding: ContentHash lease: HeldLease } | SuspendedForAuthorizedSuccessor { group: FabricGroup + hosts: List authorization: NonEmptyStr exact_candidate_realization: PairRealizationIdentity lease: HeldLease - cleanup: NonEmptyStr + cleanup: ReleasedBaselineSpec } | FencedRefusal { group: FabricGroup + hosts: List cause: NonEmptyStr disposition_authority: NonEmptyStr } @@ -74,7 +80,7 @@ type PairServingGroupAuthority fn pair_serving_authority_group(a: PairServingGroupAuthority) -> FabricGroup { match a { PairServingActive { group: g, exact_realization: _ } => g - SuspensionPendingReconciliation { group: g, transaction: _, lease: _ } => g + SuspensionPendingReconciliation { group: g, transaction: _, authorization_binding: _, lease: _ } => g SuspendedForAuthorizedSuccessor { group: g, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => g FencedRefusal { group: g, cause: _, disposition_authority: _ } => g ReleasedToFleet { group: g, release_receipt: _ } => g @@ -89,7 +95,7 @@ fn pair_serving_authority_group(a: PairServingGroupAuthority) -> FabricGroup { fn pair_serving_apply_admits(a: PairServingGroupAuthority) -> Bool { match a { PairServingActive { group: _, exact_realization: _ } => true - SuspensionPendingReconciliation { group: _, transaction: _, lease: _ } => false + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: _ } => false SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => false FencedRefusal { group: _, cause: _, disposition_authority: _ } => false ReleasedToFleet { group: _, release_receipt: _ } => false @@ -102,10 +108,28 @@ fn pair_serving_apply_admits(a: PairServingGroupAuthority) -> Bool { // because fencing is precisely the state in which we refuse to act while refusing to let go. Only an // explicit release to the fleet drops the commitment. Answering `false` under suspension would let // another lane take a host mid-transaction with no cleanup and no readback. +// WHICH HOSTS ARE COMMITTED IS FROZEN WITH THE TRANSACTION. A group's membership is derived from +// the current lane observations, and a source edit can move a host in or out while the wire still +// says `group-a`. Active outside a transaction commits the resting population, because that is +// what it serves; every other state carries the EXACT hosts the transaction was authorized over +// (a fence keeps the population it fenced), so the commitment a suspended or fenced group holds is +// the one the operator saw -- not one re-derived from a roster that moved under it, which would +// release an authorized host and claim one nobody authorized while the fence says otherwise. +// gunbc.spark.host_commitment consumes this; it is the reservation, not the group word. +fn pair_serving_committed_hosts(a: PairServingGroupAuthority) -> List { + match a { + PairServingActive { group: g, exact_realization: _ } => fabric_group_hosts(g: g) + SuspensionPendingReconciliation { group: _, hosts: hs } => hs + SuspendedForAuthorizedSuccessor { group: _, hosts: hs } => hs + FencedRefusal { group: _, hosts: hs } => hs + ReleasedToFleet { group: _, release_receipt: _ } => [] as List + } +} + fn pair_serving_commits_hosts(a: PairServingGroupAuthority) -> Bool { match a { PairServingActive { group: _, exact_realization: _ } => true - SuspensionPendingReconciliation { group: _, transaction: _, lease: _ } => true + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: _ } => true SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => true FencedRefusal { group: _, cause: _, disposition_authority: _ } => true ReleasedToFleet { group: _, release_receipt: _ } => false @@ -137,7 +161,7 @@ type PairServingCapacitySubject fn pair_serving_capacity_subject(a: PairServingGroupAuthority) -> PairServingCapacitySubject { match a { PairServingActive { group: g, exact_realization: _ } => CapacityOrdinarySubject { group: g } - SuspensionPendingReconciliation { group: g, transaction: t, lease: _ } => + SuspensionPendingReconciliation { group: g, transaction: t, authorization_binding: _, lease: _ } => CapacitySuspendedSubject { group: g, reason: t } SuspendedForAuthorizedSuccessor { group: g, authorization: au, exact_candidate_realization: _, lease: _, cleanup: _ } => CapacitySuspendedSubject { group: g, reason: au } @@ -191,7 +215,7 @@ fn pair_serving_lease_is_live(o: HeldLeaseObservedState) -> Bool { fn pair_serving_successor_may_launch(a: PairServingGroupAuthority) -> Bool { match a { PairServingActive { group: _, exact_realization: _ } => false - SuspensionPendingReconciliation { group: _, transaction: _, lease: _ } => false + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: _ } => false SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: l, cleanup: _ } => pair_serving_lease_is_live(o: l.observed) FencedRefusal { group: _, cause: _, disposition_authority: _ } => false @@ -208,7 +232,7 @@ fn pair_serving_successor_may_launch(a: PairServingGroupAuthority) -> Bool { fn pair_serving_may_finish(a: PairServingGroupAuthority) -> Bool { match a { PairServingActive { group: _, exact_realization: _ } => false - SuspensionPendingReconciliation { group: _, transaction: _, lease: _ } => true + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: _ } => true SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => true FencedRefusal { group: _, cause: _, disposition_authority: _ } => true ReleasedToFleet { group: _, release_receipt: _ } => false @@ -252,17 +276,20 @@ fn pair_serving_authority_for(group: FabricGroup) -> PairServingGroupAuthority? (fabric_group_wire(g: pair_serving_authority_group(a: a)) as String) == (fabric_group_wire(g: group) as String))) } -// THE OBLIGATION THIS ROW CARRIES IS NOW A D0 OBLIGATION, NOT A MISSING PRODUCER. The producer that -// was missing has landed: gunbc.spark.pair_incumbent_identity keys the DECLARED pair realization -// from every rank's rendered unit bytes, observes the LIVE four-rank incarnation, and folds the two -// into a PairRealizationAgreement. What this row may not do is copy that key in here as if the -// declaration alone established what is live: the key is EXPECTED identity, the agreement is the -// reading, and a reading taken before the authority transfer can go stale before suspension. So -// the Keyed arm is minted by D0 -- inside its fence, from an Established agreement -- and by -// nothing else. TRIGGER: the Cut D D0 transfer fold lands, reads pair_realization_agreement under -// its held claim, and writes PairRealizationKeyed { key } on the suspended authority's -// exact_realization from the Established arm (a Drifted reading records the drift and does not -// become a restoration target; Unread and Conflict fence). SUFFICIENT FOR: PairServingActive can -// name the incumbent exactly, so a successor transition is discriminated by realization rather +// THE OBLIGATION THIS ROW CARRIES IS THE IMAGE AXIS, AND D0 DOES NOT DISCHARGE IT. The producers +// that were missing have landed: gunbc.spark.pair_incumbent_identity keys the DECLARED pair +// realization from every rank's rendered unit bytes, observes the LIVE four-rank incarnation, and +// folds the two into a PairRealizationAgreement; gunbc.spark.pair_serving_d0 takes that reading +// inside its held claim. What no producer has is a declared IMAGE identity to compare the running +// container's image id against -- the pair runtime image is built on the host and the corpus +// declares its reference, not its id -- so PairRealizationEstablished is rank-and-unit agreement +// with the image carried observed-and-uncompared, and it may NOT mint an exact incumbent key: a +// rebuilt image under the same reference with unchanged unit bytes passes it. D0 therefore restores +// a live declared incumbent to the Active state it held, unchanged. The only Keyed realization D0 +// writes is the authorized SUCCESSOR's, on the suspended authority. TRIGGER: a produced-image +// receipt for the pair runtime image (the build materialization gunbc.spark.vllm_runtime_image_build +// is the shape) is carried on the declared realization and compared in the agreement, so +// Established means the image too. SUFFICIENT FOR: the incumbent's exact realization can be minted +// from an Established agreement, and a successor transition is discriminated by realization rather // than by a serving subject V4 and V4.1 deliberately share. -data pair_realization_key_obligation: NonEmptyStr = "the live V4 pair realization is keyed by gunbc.spark.pair_incumbent_identity declared_pair_realization and read by pair_realization_agreement, but no transaction has yet taken that reading inside a held claim; the Keyed arm is minted by Cut D D0 from an Established agreement and by nothing else. SUFFICIENT FOR: PairServingActive can name the incumbent exactly, so a successor transition is discriminated by realization rather than by a serving subject V4 and V4.1 deliberately share" as NonEmptyStr +data pair_realization_key_obligation: NonEmptyStr = "the live V4 pair realization is keyed by gunbc.spark.pair_incumbent_identity declared_pair_realization and read by pair_realization_agreement inside D0's held claim, but that agreement carries the running image observed and uncompared (no produced-image digest exists to compare it against), so it may not mint an exact incumbent key; the Keyed arm for the incumbent waits on a produced-image receipt compared in the agreement, and D0 restores a live declared incumbent to its Active state unchanged. SUFFICIENT FOR: PairServingActive can name the incumbent exactly, so a successor transition is discriminated by realization rather than by a serving subject V4 and V4.1 deliberately share" as NonEmptyStr diff --git a/dag/gunbc/spark/pair_serving_authority_log.dag b/dag/gunbc/spark/pair_serving_authority_log.dag new file mode 100644 index 00000000000..5fb2e76c33c --- /dev/null +++ b/dag/gunbc/spark/pair_serving_authority_log.dag @@ -0,0 +1,2343 @@ +module gunbc.spark.pair_serving_authority_log + +import std.types { String, Bool, Int, NonEmptyStr, List, EpochSecs } +import std.nat { Nat } +import std.measure { Second, second, second_count } +import gunbc.spark.host_effect_quiescence { HostEffectResidueSpec, HostQuiescenceEvidence, residue_spec_members, residue_spec_decode, quiescence_evidence_members, quiescence_evidence_decode, quiescence_evidence_covers, ReleaseProvenance, ClaimantTerminal, RecoveryAuthorized, release_provenance_members, release_provenance_decode, release_provenance_covers, parse_claimant_standing } +import v2.std.optional { Present, Absent } +import std.content_hash { ContentHash, serialize_content_hash, parse_content_hash, content_hash_equal } +import std.temporal_effect { + HeldLease, HeldLeaseObservedState, LeaseEpoch, held_lease, held_lease_with_observed, + LeaseRunningExpected, LeaseRunningStale, LeaseInaccessible, +} +import product.capacity.lease { LeaseGrant, LeaseFence, GrantIdentity, lease_grant_expired_at } +import extdeps.languages.json.emit { JsonValue, JsonKeyValue, JsonString, JsonArray, json_kv, json_string, json_int, json_array, json_object, serialize_json } +import extdeps.languages.json.parse { json_object_unique_member, JsonMemberFound } +import product.capacity.event_chain { + PartitionId, EventId, ChainEvent, ChainEnvelope, HeadExpectation, HeadAbsent, HeadAt, head_expectation_eq, + EventDecode, ChainWalk, ChainWalked, ChainIncomplete, ChainBudgetExhausted, +} +import product.capacity.event_json { chain_event_wire_text, chain_event_decode, member_string, member_nonempty, member_nat } +import gunbc.fabric_event_log { + PartitionRead, PartitionReadOk, PartitionReadRefused, event_log_read_partition_with, + EventAppend, EventAppended, EventAppendStale, EventAppendRefused, event_log_append_with, event_log_refusal_wire, event_log_read_budget, +} +import gunbc.fabric_storage_client { FabricStorageBinding } +import gunbc.fabric_event_log_host { HostStoreResolved, HostStoreRefused, event_log_store_for_host, now_epoch_seconds } +import std.process { ProcessExit, ExitSuccess, exit_failure } +import gunbc.spark.fabric_reach { ExecutorReachKnown, ExecutorReachUnknown, observe_executor_reach } +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, FabricGroupB, fabric_group_wire, parse_fabric_group, fabric_group_of_host } +import product.placement_supply { HostIdentity, host_identity_eq } +import gunbc.spark.released_baseline { ReleasedBaselineSpec, released_baseline_wire, parse_released_baseline } +import gunbc.spark.pair_serving_authority { + PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, SuspendedForAuthorizedSuccessor, FencedRefusal, ReleasedToFleet, + PairRealizationIdentity, PairRealizationKeyed, PairRealizationUnestablished, + pair_serving_authority_group, pair_serving_authority_for, spark_pair_serving_authorities, pair_serving_committed_hosts, pair_serving_commits_hosts, +} + +// ── THE AUTHORITY THAT CAN CHANGE WITHOUT A COMMIT ───────────────────────────────────────────── +// +// gunbc.spark.pair_serving_authority spark_pair_serving_authorities is source data: it says which +// groups we CLAIM and what each group's RESTING authority is, and it is edited by a pull request. +// A replacement transaction (docs/plans/dsv41-cut-d-redesign.md, D0) has to move a group from +// Active to SuspensionPendingReconciliation to SuspendedForAuthorizedSuccessor and back, under a +// lease, between two readings of the same fleet -- and a pure function returning the next state +// changes nothing any consumer reads. The state a transaction is in has to be WRITTEN somewhere +// every effectful consumer reads, and written by a compare-and-set, so two transactions cannot +// both believe they hold the group. +// +// THE STORE IS THE FABRIC EVENT LOG, ON ITS OWN PARTITION PER GROUP. DESIGN §3b puts serving-side +// occupancy on the event chain (a termed LeaseGrant settled by fabric_seat_acquire over a +// partition head), and this is the same linearization fact about a different subject: one +// partition `pair-serving-authority/`, one event per transition, the head advanced by a +// leased push. gunbc.fabric_event_log was generic in nothing but its seats until this module; it +// now reads and appends for any payload, and this module supplies the authority's codec. +// +// DESIRED AND ESTABLISHED ARE TWO FACTS, AND THIS IS WHERE THE SPLIT IS DECIDED. The source row is +// the DESIRED authority: the pure membership projection (the desired roster, the resting host +// commitment) reads it, and the design says suspension KEEPS the claim, so membership does not +// move during a transaction. The log is the ESTABLISHED authority, folded from one durable genesis +// per partition: an empty partition is UNESTABLISHED (no authority, nothing committed), the source +// row becomes the group's authority only by an AuthorityEstablished event on that partition, and +// every effectful consumer reads the log and never the row -- a row without its establishment is +// Unread at "claim", and an UNREADABLE partition refuses, because a consumer that fell back to +// the row when the log could not be read would act on a state a transaction may have left behind. +// Absence is a read fact; unreadability is never permission. +// +// A TRANSITION NAMES THE STATE IT LEAVES, AND THE FOLD CHECKS IT. Every event carries the previous +// authority beside the next one, and the fold refuses a chain whose event claims to leave a state +// that was not current -- so a transaction that decided on a stale reading and still won the +// push (impossible under the head lease, but the fold does not assume the push) is a typed refusal +// at the event that lied, never a silently rewritten history. + +data pair_authority_event_schema: String = "pair-serving-authority-event/v1" +data pair_authority_partition_prefix: String = "pair-serving-authority-" + +fn pair_authority_partition(group: FabricGroup) -> PartitionId { + join([pair_authority_partition_prefix, fabric_group_wire(g: group) as String], "") as PartitionId +} + +// THE GRANT RIDES WITH THE TRANSITION THAT HOLDS IT. product.capacity.lease LeaseGrant owns the +// term -- reference, fence, granted_at, expiry -- and std.temporal_effect HeldLease owns what an +// observer currently sees for that lease's epoch. They are two facts (DESIGN §3b leasing row), so +// the authority carries the HeldLease and the EVENT carries the grant: a transition into a leased +// state names the grant it holds, a transition out carries none, and the fold keeps the latest. +// The observed state on a read is then DERIVED from the grant and the reading instant, never +// trusted from the bytes -- see current_pair_serving_authority. +// THE ENTRY-STATE RECEIPT IS JOINED TO THE TRANSITION BY ITS DIGEST. A transaction that settles +// the authority from a reading of the fleet names that reading here, so a later consumer (D1, D2) +// can prove which four-rank reading a suspension was settled from rather than trusting a local +// file that may have been replaced. Absent on transitions that were not settled from a reading. +// THE ENTRY-STATE RECEIPT IS AN EVENT OF ITS OWN ON THE SAME PARTITION. A transaction that settles +// the authority from a reading of the fleet first APPENDS that reading's bytes as an +// EntryStateRecorded event -- content-addressed by the store (its id is the object's ref), durable +// wherever the log is, fetchable by any executor -- and then names that event's id on the +// settling transition. Recording moves the head but not the authority: the fold passes it +// through with the current state, generation and lease unchanged. +// `lifecycle` IS THE AUTHORIZATION LIFECYCLE THAT WROTE THE TRANSITION -- the transaction's +// authorization binding (a SHA-256 over escalation, attempt, intent, subject, expiry and claim +// generation; the executor is deliberately not in it, so every executor computes the same +// identity), when a D0 transaction wrote it; Absent for an operator's or a cleanup's +// write. The transaction word is a branded string and nothing allocates it uniquely, so a later +// escalation may reuse it: recovery that selected history by the word alone would complete a new +// consent from an old transaction's terminal. The identity is on the event so the fold can keep it. +type PairAuthorityEvent + = AuthorityEstablished { + authority: PairServingGroupAuthority + placement: PlacementPreparationRef + } + | AuthorityTransitioned { + previous: PairServingGroupAuthority + next: PairServingGroupAuthority + transaction: NonEmptyStr + lifecycle: ContentHash? + grant: LeaseGrant? + entry_state: ContentHash? + placement: PlacementPreparationRef + } + | EntryStateRecorded { + transaction: NonEmptyStr + body: String + } + | AuthorityAppendCancelled { + preparation: EventId + provenance: ReleaseProvenance + } + | HostEffectAdmitted { + host: HostIdentity + purpose: NonEmptyStr + executor: NonEmptyStr + term: Second + residue: HostEffectResidueSpec + } + | HostEffectReleased { + admitted_by: EventId + evidence: HostQuiescenceEvidence + provenance: ReleaseProvenance + } + | PlacementPrepared { + operation: NonEmptyStr + group: FabricGroup + previous_hosts: List + next_hosts: List + } + | PlacementAppendClaimed { + preparation: EventId + intent: AuthorityWriteIntent + } + | PlacementFinalized { + preparation: PlacementPreparationRef + authority_event: EventId + } + | PlacementAborted { + preparation: EventId + reason: String + provenance: ReleaseProvenance + } + +// ── HOST PLACEMENT IS ONE PARTITION, AND IT IS THE LINEARIZATION POINT ──────────────────────── +// +// A build, a probe or a source acquisition PLACES WORK on a host; an authority state COMMITS +// hosts to a serving group. The one invariant both must respect is that no host is committed by +// an authority while a host effect is live on it, and a check made by reading one partition and +// writing another cannot hold it: the two heads linearize independently, so an effect that read +// "not committed" and a transition that read "no live effect" can both land. So every fact about +// host placement -- every effect claim, every release, and every authority write's PREPARATION, +// FINALIZATION or ABORT -- is an event on ONE partition, `host-placement`, and every writer of one +// is a compare-and-set against its head. +// +// AN AUTHORITY WRITE IS A SAGA ACROSS TWO LOGS, AND THE SAGA NEVER FREES A HOST EARLY. A +// transition or an establishment that changes what a group commits runs prepare → authority → +// finalize: the PREPARATION lands here first and FENCES the union of the previous and the next +// population (refused while any of those hosts has a live effect or is fenced by another group; +// stale when the partition moved); the authority event on the group partition CARRIES THE +// PREPARATION'S ID, so the two histories name each other exactly; the FINALIZATION lands here +// last and retains exactly the next population. A crash before the authority event over-fences; +// a crash after it over-fences until the finalization; neither frees the previous population +// before the authority moved. A preparation whose authority event did not land is ABORTED, and +// the abort is admitted only when no authority event on the group's chain names it -- so a +// preparation a successful write consumed cannot be voided from under it (the check and the +// operator's entries are in this module; the fold also refuses an abort of a finalized +// preparation). An effect claim is refused while any unfinalized preparation or live finalized +// commitment fences its host, or another live claim holds it (one effect per host). A source edit +// that moves a host between groups moves nothing here. +// +// LIVENESS IS QUIESCENCE-REQUIRED, NOT TIMED: the term the effect states is its own expected +// bound and marks the claim OVERDUE past it, but an overdue claim still fences -- expiry proves +// the claim is old, not that the build or the container is gone (product.capacity.lease +// QuiescenceRequired, the same law the serving lease follows). WHAT RELEASES IT IS EVIDENCE, NOT A +// RETURN: the claim declares the residue its effect would leave (gunbc.spark.host_effect_quiescence +// HostEffectResidueSpec), and a release carries a HostQuiescenceEvidence -- sole-constructed by +// the observer that scanned the host -- naming this claim, its host and its residue, taken no +// earlier than the admission, with a provenance that is the claimant's own terminal or an +// operator's recovery authorization naming the claim; the fold refuses anything else. A crashed +// lane's claim is released by whoever runs that same observation later +// (gunbc.spark.host_commitment host_effect_recover_wet); nothing releases it on a clock. +data host_placement_partition: PartitionId = "host-placement" as PartitionId + +// THE PREPARATION, NAMED EXACTLY, ON BOTH CHAINS. An authority event and a finalization carry a +// typed copy of the preparation they consume -- its id, group, both populations and operation -- +// so each fold verifies its own side against the copy (the group fold: the group is the +// partition's and the populations are the states' the event leaves and enters; the placement +// fold: the copy is the record) and the live join (preparation_consumption) verifies the copies +// are the record. Co-location of two ids is not provenance; the copy is what makes a lying pair +// of documents refuse on either chain. +type PlacementPreparationRef { + id: EventId + group: FabricGroup + previous_hosts: List + next_hosts: List + operation: NonEmptyStr +} + +type HostEffectRecord { + id: EventId + host: HostIdentity + purpose: NonEmptyStr + executor: NonEmptyStr + recorded_at: EpochSecs + term: Second + residue: HostEffectResidueSpec + released: Bool +} + +// One authority write's preparation, as landed on the placement partition. While Prepared it +// fences the union of both populations; Finalized it is the group's live commitment (its next +// population) until a later finalization supersedes it; Aborted it fences nothing. +// Prepared: the fence is up, no authority event has been attempted. AppendClaimed: the writer +// named itself on the placement chain before its group append -- from here an abort is the +// claimant's own or a recovery naming the claimant's standing, exactly as a host effect's release +// is. Finalized: the authority event landed and the retained population is the next one. +// Aborted: the fence is down. +type PreparationState + = Prepared + | AppendClaimed { claimant: NonEmptyStr, expected_head: HeadExpectation, intent: AuthorityWriteIntent } + | Finalized { authority_event: EventId, finalized_by: EventId, intent: AuthorityWriteIntent } + | Aborted + +// The record carries WHO PREPARED (the preparing event's actor): while the preparation is +// Prepared its fence is that writer's, and an abort of it is the preparer's own (ClaimantTerminal +// naming it, written by the preparer) or a recovery naming it -- never a stranger's. +type PlacementPreparationRecord { + id: EventId + operation: NonEmptyStr + group: FabricGroup + previous_hosts: List + next_hosts: List + recorded_at: EpochSecs + prepared_by: NonEmptyStr + state: PreparationState +} + +fn host_effect_released(r: HostEffectRecord) -> HostEffectRecord { + HostEffectRecord { id: r.id, host: r.host, purpose: r.purpose, executor: r.executor, recorded_at: r.recorded_at, term: r.term, residue: r.residue, released: true } +} + +// A release lands only for a live claim whose residue spec and host the evidence names. +type ReleaseAdmission + = ReleaseAdmitted { record: HostEffectRecord } + | ReleaseNotLive + | ReleaseEvidenceForAnother { record: HostEffectRecord } + | ReleaseProvenanceForAnother { record: HostEffectRecord } + +// `actor` is the event's actor: a claimant-terminal release must be written by the executor that +// acquired the claim (and name it), so another executor that observes a momentarily quiet host +// cannot label itself the claimant; it has the recovery route, which names the claimant's standing. +fn release_admission(records: List, claim: EventId, evidence: HostQuiescenceEvidence, provenance: ReleaseProvenance, actor: NonEmptyStr) -> ReleaseAdmission { + match first(filter(records, r => (r.id as String) == (claim as String) && !r.released)) { + Absent => ReleaseNotLive + Present { value: r } => + if !quiescence_evidence_covers(e: evidence, claim: claim, admitted_at: r.recorded_at, host: r.host, spec: r.residue) { ReleaseEvidenceForAnother { record: r } } + else if !release_provenance_covers(p: provenance, claim: claim, claimant: r.executor, actor: actor) { ReleaseProvenanceForAnother { record: r } } + else { ReleaseAdmitted { record: r } } + } +} + +fn release_refusal_reason(a: ReleaseAdmission, claim: EventId) -> String { + match a { + ReleaseAdmitted { record: _ } => "" + ReleaseNotLive => join(["no live host effect claim ", claim as String, " on the placement partition"], "") + ReleaseProvenanceForAnother { record: r } => join(["the release provenance is not for claim ", claim as String, ": a claimant-terminal release must name this claim and be written by its claimant ", r.executor as String, "; any other writer needs a recovery authorization naming this claim"], "") + ReleaseEvidenceForAnother { record: r } => join(["the quiescence evidence is not for claim ", claim as String, ": it must name that claim, host ", r.host as String, " and the claim's own residue (process pattern `", r.residue.process_pattern as String, "`), and be observed no earlier than ", to_string(r.recorded_at)], "") + } +} + +fn host_effect_is_live(r: HostEffectRecord, at: EpochSecs) -> Bool { + !r.released +} + +fn host_effect_is_overdue(r: HostEffectRecord, at: EpochSecs) -> Bool { + !r.released && at >= r.recorded_at + second_count(s: r.term) +} + +fn live_host_effects_on(records: List, hosts: List, at: EpochSecs) -> List { + filter(records, r => host_effect_is_live(r: r, at: at) && any(hosts, h => host_identity_eq(a: h, b: r.host))) +} + +fn host_effect_line(r: HostEffectRecord) -> String { + join([r.purpose as String, " on ", r.host as String, " by ", r.executor as String, " (claim ", r.id as String, ", term ", to_string(second_count(s: r.term)), "s from ", to_string(r.recorded_at), ")"], "") +} + +fn host_effect_line_at(r: HostEffectRecord, at: EpochSecs) -> String { + join([host_effect_line(r: r), if host_effect_is_overdue(r: r, at: at) { " -- OVERDUE: past its stated term and unreleased; it fences until released with quiescence evidence" } else { "" }], "") +} + +fn same_group(a: FabricGroup, b: FabricGroup) -> Bool { + (fabric_group_wire(g: a) as String) == (fabric_group_wire(g: b) as String) +} + +fn hosts_union(a: List, b: List) -> List { + concat(a, filter(b, h => !any(a, x => host_identity_eq(a: x, b: h)))) +} + +fn same_host_set(a: List, b: List) -> Bool { + length(a) == length(b) && all(a, x => any(b, y => host_identity_eq(a: x, b: y))) && all(b, y => any(a, x => host_identity_eq(a: x, b: y))) +} + +fn preparation_is_pending(r: PlacementPreparationRecord) -> Bool { + match r.state { Prepared => true AppendClaimed { claimant: _, expected_head: _, intent: _ } => true _ => false } +} + +fn preparation_ref_of(r: PlacementPreparationRecord) -> PlacementPreparationRef { + PlacementPreparationRef { id: r.id, group: r.group, previous_hosts: r.previous_hosts, next_hosts: r.next_hosts, operation: r.operation } +} + +fn preparation_ref_eq(a: PlacementPreparationRef, b: PlacementPreparationRef) -> Bool { + (a.id as String) == (b.id as String) && same_group(a: a.group, b: b.group) && same_host_set(a: a.previous_hosts, b: b.previous_hosts) && same_host_set(a: a.next_hosts, b: b.next_hosts) && (a.operation as String) == (b.operation as String) +} + +// THE AUTHORITY-WRITE INTENT: everything one authority write is, as a typed value -- the +// partition's group, the head it is appended at, its actor, the state it leaves (none before the +// establishment), the state it enters, its operation, its lifecycle, grant and entry-state +// identities. The append claim carries it; the authority event carries nothing of it, because an +// authority event's intent IS the event (`event_write_intent` derives it from the envelope's +// parent and actor and the payload's fields), so the joined reads compare the claim's carried +// intent to the consuming event's derived one field for field. No digest stands in for the +// comparison: a structural fingerprint has collisions and this is the one cross-log proof that +// the event which consumed a claim is the exact write the claim was for. +type AuthorityWriteIntent { + group: FabricGroup + expected_head: HeadExpectation + actor: NonEmptyStr + previous: PairServingGroupAuthority? + next: PairServingGroupAuthority + operation: NonEmptyStr + lifecycle: ContentHash? + grant: LeaseGrant? + entry_state: ContentHash? +} + +fn authority_write_intent(group: FabricGroup, head: HeadExpectation, actor: NonEmptyStr, previous: PairServingGroupAuthority?, next: PairServingGroupAuthority, operation: NonEmptyStr, lifecycle: ContentHash?, grant: LeaseGrant?, entry_state: ContentHash?) -> AuthorityWriteIntent { + AuthorityWriteIntent { group: group, expected_head: head, actor: actor, previous: previous, next: next, operation: operation, lifecycle: lifecycle, grant: grant, entry_state: entry_state } +} + +// The intent an authority event proves, derived from the event: its parent is the head, its +// actor the actor, and the payload's fields the rest. +fn event_write_intent(env: ChainEnvelope, group: FabricGroup, previous: PairServingGroupAuthority?, next: PairServingGroupAuthority, operation: NonEmptyStr, lifecycle: ContentHash?, grant: LeaseGrant?, entry_state: ContentHash?) -> AuthorityWriteIntent { + let head = match env.event.parent { Absent => HeadAbsent Present { value: p } => HeadAt { id: p } } + authority_write_intent(group: group, head: head, actor: env.event.actor, previous: previous, next: next, operation: operation, lifecycle: lifecycle, grant: grant, entry_state: entry_state) +} + +fn optional_hash_eq(a: ContentHash?, b: ContentHash?) -> Bool { + match a { + Absent => (match b { Absent => true Present { value: _ } => false }) + Present { value: x } => (match b { Absent => false Present { value: y } => content_hash_equal(left: x, right: y) }) + } +} + +fn optional_authority_eq(a: PairServingGroupAuthority?, b: PairServingGroupAuthority?) -> Bool { + match a { + Absent => (match b { Absent => true Present { value: _ } => false }) + Present { value: x } => (match b { Absent => false Present { value: y } => authority_eq(a: x, b: y) }) + } +} + +fn grant_wire(g: LeaseGrant?) -> String { + serialize_json(v: json_object(members: grant_members(g: g))) +} + +fn write_intent_eq(a: AuthorityWriteIntent, b: AuthorityWriteIntent) -> Bool { + same_group(a: a.group, b: b.group) + && head_expectation_eq(a: a.expected_head, b: b.expected_head) + && (a.actor as String) == (b.actor as String) + && optional_authority_eq(a: a.previous, b: b.previous) + && authority_eq(a: a.next, b: b.next) + && (a.operation as String) == (b.operation as String) + && optional_hash_eq(a: a.lifecycle, b: b.lifecycle) + && grant_wire(g: a.grant) == grant_wire(g: b.grant) + && optional_hash_eq(a: a.entry_state, b: b.entry_state) +} + +// The intent on the wire: its own members under `intent_`, the previous state under +// `intent_prev_` when there is one, the next under `intent_next_`, and the lifecycle, grant and +// entry-state members as the authority event spells them (an append claim carries no other). +fn write_intent_members(i: AuthorityWriteIntent) -> List { + concat( + [ + kv(prefix: "intent_", key: "group", value: fabric_group_wire(g: i.group) as String), + kv(prefix: "intent_", key: "head", value: match i.expected_head { HeadAbsent => "" HeadAt { id: h } => h as String }), + kv(prefix: "intent_", key: "actor", value: i.actor as String), + kv(prefix: "intent_", key: "operation", value: i.operation as String), + ], + concat( + match i.previous { Absent => [] as List Present { value: p } => authority_members(prefix: "intent_prev_", a: p) }, + concat(authority_members(prefix: "intent_next_", a: i.next), concat(lifecycle_members(l: i.lifecycle), concat(grant_members(g: i.grant), entry_state_members(d: i.entry_state)))), + ), + ) +} + +// The preparation an authority write over `previous` → `next` under `operation` must carry. +fn preparation_ref_expected(id: EventId, group: FabricGroup, previous: List, next: List, operation: NonEmptyStr) -> PlacementPreparationRef { + PlacementPreparationRef { id: id, group: group, previous_hosts: previous, next_hosts: next, operation: operation } +} + +// THE GROUP'S LIVE COMMITMENT: the next population of its newest finalized preparation (none +// before the first finalization). +fn group_live_commitment(preps: List, group: FabricGroup) -> List { + match last(filter(preps, r => same_group(a: r.group, b: group) && (match r.state { Finalized { authority_event: _, finalized_by: _, intent: _ } => true _ => false }))) { + Absent => [] as List + Present { value: r } => r.next_hosts + } +} + +// EVERYTHING A GROUP FENCES: its live commitment plus the union of both populations of every +// pending preparation of its own. +fn group_fenced_hosts(preps: List, group: FabricGroup) -> List { + fold(filter(preps, r => same_group(a: r.group, b: group) && preparation_is_pending(r: r)), init: group_live_commitment(preps: preps, group: group), f: (acc, r) => hosts_union(a: acc, b: hosts_union(a: r.previous_hosts, b: r.next_hosts))) +} + +// The groups that fence a host. +fn groups_fencing(preps: List, host: HostIdentity) -> List { + filter([FabricGroupA, FabricGroupB], g => any(group_fenced_hosts(preps: preps, group: g), h => host_identity_eq(a: h, b: host))) +} + +fn preparation_line(r: PlacementPreparationRecord) -> String { + join([fabric_group_wire(g: r.group) as String, " ", r.operation as String, " (preparation ", r.id as String, ": ", join(map(r.previous_hosts, h => h as String), ","), " -> ", join(map(r.next_hosts, h => h as String), ","), ")"], "") +} + +fn preparation_with_state(r: PlacementPreparationRecord, state: PreparationState) -> PlacementPreparationRecord { + PlacementPreparationRecord { id: r.id, operation: r.operation, group: r.group, prepared_by: r.prepared_by, previous_hosts: r.previous_hosts, next_hosts: r.next_hosts, recorded_at: r.recorded_at, state: state } +} + +// ── THE CODEC ────────────────────────────────────────────────────────────────────────────────── +// +// Flat members with a prefix per side (prev_ / next_), one state word, and only the fields the +// arm carries. The decode is the exact inverse and refuses a document whose fields do not inhabit +// the arm its state word names. + +fn kv(prefix: String, key: String, value: String) -> JsonKeyValue { + json_kv(key: join([prefix, key], ""), value: json_string(s: value)) +} + +fn realization_members(prefix: String, r: PairRealizationIdentity) -> List { + match r { + PairRealizationKeyed { key: k } => [kv(prefix: prefix, key: "key", value: serialize_content_hash(hash: k) as String)] + PairRealizationUnestablished { obligation: o } => [kv(prefix: prefix, key: "obligation", value: o as String)] + } +} + +// THE LEASE'S OBSERVED STATE IS NOT PART OF THE AUTHORITY'S IDENTITY. The durable authority +// carries the lease's EPOCH -- key, resource, owner, generation -- which is stable; what an +// observer sees for that epoch is a fact about the reading instant, derived from the grant at +// every read (authority_with_lease_observed). An earlier shape serialized the observation too, +// which made two readings of one state unequal on the wire the moment a term expired, so a +// transition whose `previous` was read after expiry could never fold against the event that +// wrote it before -- and a pending state appended before its grant existed read back unequal to +// itself. The wire is the epoch; the observation is the read's. +fn lease_members(prefix: String, l: HeldLease) -> List { + [ + kv(prefix: prefix, key: "lease_key", value: l.epoch.lease_key as String), + kv(prefix: prefix, key: "lease_resource", value: serialize_content_hash(hash: l.epoch.resource_fingerprint) as String), + kv(prefix: prefix, key: "lease_owner", value: serialize_content_hash(hash: l.epoch.owner_fingerprint) as String), + json_kv(key: join([prefix, "lease_generation"], ""), value: json_int(n: l.epoch.generation)), + ] +} + +fn authority_state_word(a: PairServingGroupAuthority) -> String { + match a { + PairServingActive { group: _, exact_realization: _ } => "active" + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: _ } => "suspension-pending-reconciliation" + SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => "suspended-for-authorized-successor" + FencedRefusal { group: _, cause: _, disposition_authority: _ } => "fenced-refusal" + ReleasedToFleet { group: _, release_receipt: _ } => "released-to-fleet" + } +} + +fn authority_members(prefix: String, a: PairServingGroupAuthority) -> List { + let head = [ + kv(prefix: prefix, key: "state", value: authority_state_word(a: a)), + kv(prefix: prefix, key: "group", value: fabric_group_wire(g: pair_serving_authority_group(a: a)) as String), + ] + match a { + PairServingActive { group: _, exact_realization: r } => + concat(head, realization_members(prefix: join([prefix, "realization_"], ""), r: r)) + SuspensionPendingReconciliation { group: _, hosts: hs, transaction: t, authorization_binding: ab, lease: l } => + concat(head, concat([hosts_member(prefix: prefix, hs: hs), kv(prefix: prefix, key: "transaction", value: t as String), kv(prefix: prefix, key: "authorization_binding", value: serialize_content_hash(hash: ab) as String)], lease_members(prefix: prefix, l: l))) + SuspendedForAuthorizedSuccessor { group: _, hosts: hs, authorization: au, exact_candidate_realization: r, lease: l, cleanup: c } => + concat(head, concat( + [hosts_member(prefix: prefix, hs: hs), kv(prefix: prefix, key: "authorization", value: au as String), kv(prefix: prefix, key: "cleanup", value: released_baseline_wire(s: c) as String)], + concat(realization_members(prefix: join([prefix, "candidate_"], ""), r: r), lease_members(prefix: prefix, l: l)), + )) + FencedRefusal { group: _, hosts: hs, cause: c, disposition_authority: d } => + concat(head, [hosts_member(prefix: prefix, hs: hs), kv(prefix: prefix, key: "cause", value: c as String), kv(prefix: prefix, key: "disposition_authority", value: d as String)]) + ReleasedToFleet { group: _, release_receipt: r } => + concat(head, [kv(prefix: prefix, key: "release_receipt", value: serialize_content_hash(hash: r) as String)]) + } +} + +// THE ONE EQUALITY. Two authorities are the same state exactly when their canonical members +// serialize identically; there is no second, hand-written field-by-field comparison to drift. +// THE FROZEN HOSTS ON THE WIRE: A JSON ARRAY OF STRINGS, one element per host. HostIdentity is a +// branded NonEmptyStr with no grammar excluding any character, so a joined-and-split spelling +// would give two populations one wire (["a,b","c"] and ["a","b","c"]) and would normalize a +// malformed wire ("srv5,,srv6") instead of refusing it -- and the reason the hosts are on the +// authority at all is to carry the EXACT population the operator saw (review, round 7). The array +// is injective over every HostIdentity; a non-string or empty element refuses the document. +fn hosts_member(prefix: String, hs: List) -> JsonKeyValue { + json_kv(key: join([prefix, "hosts"], ""), value: json_array(elements: map(hs, h => json_string(s: h as String)))) +} + +// An EMPTY array refuses too: every state that carries hosts commits them, and a committing state +// that retains no population is the absorbing failure the field exists to prevent. The same wall +// stands at the transition and in the fold, so no accepted event on the log holds it. +fn host_elements_decode(elements: List) -> List? { + if length(elements) == 0 { none } else { host_elements_decode_allowing_empty(elements: elements) } +} + +// A commitment record may name no host (the group commits nothing); every element still has to be +// a nonempty string. +fn host_elements_decode_allowing_empty(elements: List) -> List? { + fold(elements, init: Present { value: [] as List }, f: (acc, e) => + match acc { + Absent => none + Present { value: hs } => + match e { + JsonString { value: h } => if h == "" { none } else { Present { value: concat(hs, [h as HostIdentity]) } } + _ => none + } + }) +} + +fn hosts_decode_allowing_empty(doc: JsonValue, prefix: String) -> List? { + match json_object_unique_member(v: doc, key: join([prefix, "hosts"], "")) { + JsonMemberFound { value: JsonArray { elements: es } } => host_elements_decode_allowing_empty(elements: es) + _ => none + } +} + +fn preparation_ref_members(r: PlacementPreparationRef) -> List { + [kv(prefix: "", key: "prep_id", value: r.id as String), kv(prefix: "", key: "prep_group", value: fabric_group_wire(g: r.group) as String), hosts_member(prefix: "prep_previous_", hs: r.previous_hosts), hosts_member(prefix: "prep_next_", hs: r.next_hosts), kv(prefix: "", key: "prep_operation", value: r.operation as String)] +} + +fn preparation_ref_decode(doc: JsonValue) -> PlacementPreparationRef? { + match member_nonempty(doc: doc, key: "prep_id") { + Absent => none + Present { value: id } => + match member_string(doc: doc, key: "prep_group") { + Absent => none + Present { value: gw } => + match parse_fabric_group(wire: gw) { + Absent => none + Present { value: g } => + match hosts_decode_allowing_empty(doc: doc, prefix: "prep_previous_") { + Absent => none + Present { value: ph } => + match hosts_decode_allowing_empty(doc: doc, prefix: "prep_next_") { + Absent => none + Present { value: nh } => + match member_nonempty(doc: doc, key: "prep_operation") { + Absent => none + Present { value: o } => Present { value: PlacementPreparationRef { id: (id as String) as EventId, group: g, previous_hosts: ph, next_hosts: nh, operation: o } } + } + } + } + } + } + } +} + +fn hosts_decode(doc: JsonValue, prefix: String) -> List? { + match json_object_unique_member(v: doc, key: join([prefix, "hosts"], "")) { + JsonMemberFound { value: JsonArray { elements: es } } => host_elements_decode(elements: es) + _ => none + } +} + +fn authority_wire(a: PairServingGroupAuthority) -> String { + serialize_json(v: json_object(members: authority_members(prefix: "", a: a))) +} + +fn authority_eq(a: PairServingGroupAuthority, b: PairServingGroupAuthority) -> Bool { + authority_wire(a: a) == authority_wire(a: b) +} + +fn grant_members(g: LeaseGrant?) -> List { + match g { + Absent => [] as List + Present { value: gr } => [ + kv(prefix: "grant_", key: "reference", value: gr.reference as String), + kv(prefix: "grant_", key: "fence_grant", value: gr.fence.grant as String), + json_kv(key: "grant_fence_generation", value: json_int(n: gr.fence.generation)), + json_kv(key: "grant_granted_at", value: json_int(n: gr.granted_at)), + json_kv(key: "grant_maximum_duration_seconds", value: json_int(n: gr.maximum_duration_seconds)), + json_kv(key: "grant_expires_at", value: json_int(n: gr.expires_at)), + ] + } +} + +fn lifecycle_members(l: ContentHash?) -> List { + match l { + Absent => [] as List + Present { value: h } => [kv(prefix: "", key: "lifecycle", value: serialize_content_hash(hash: h) as String)] + } +} + +fn entry_state_members(d: ContentHash?) -> List { + match d { + Absent => [] as List + Present { value: h } => [kv(prefix: "", key: "entry_state_digest", value: serialize_content_hash(hash: h) as String)] + } +} + +fn pair_authority_event_members(e: PairAuthorityEvent) -> List { + match e { + AuthorityEstablished { authority: a, placement: pp } => concat(authority_members(prefix: "est_", a: a), preparation_ref_members(r: pp)) + AuthorityTransitioned { previous: p, next: n, transaction: t, lifecycle: lc, grant: g, entry_state: d, placement: pp } => + concat([kv(prefix: "", key: "transaction", value: t as String)], concat(preparation_ref_members(r: pp), concat(lifecycle_members(l: lc), concat(authority_members(prefix: "prev_", a: p), concat(authority_members(prefix: "next_", a: n), concat(grant_members(g: g), entry_state_members(d: d))))))) + EntryStateRecorded { transaction: t, body: b } => + [kv(prefix: "", key: "transaction", value: t as String), kv(prefix: "", key: "body", value: b)] + AuthorityAppendCancelled { preparation: pr, provenance: pv } => + concat([kv(prefix: "", key: "preparation", value: pr as String)], release_provenance_members(p: pv)) + HostEffectAdmitted { host: h, purpose: p, executor: x, term: t, residue: rs } => + concat([kv(prefix: "", key: "host", value: h as String), kv(prefix: "", key: "purpose", value: p as String), kv(prefix: "", key: "executor", value: x as String), json_kv(key: "term_seconds", value: json_int(n: second_count(s: t)))], residue_spec_members(prefix: "residue_", s: rs)) + HostEffectReleased { admitted_by: a, evidence: ev, provenance: pv } => + concat([kv(prefix: "", key: "admitted_by", value: a as String)], concat(quiescence_evidence_members(e: ev), release_provenance_members(p: pv))) + PlacementPrepared { operation: o, group: g, previous_hosts: ph, next_hosts: nh } => + [kv(prefix: "", key: "operation", value: o as String), kv(prefix: "", key: "group", value: fabric_group_wire(g: g) as String), hosts_member(prefix: "previous_", hs: ph), hosts_member(prefix: "next_", hs: nh)] + PlacementAppendClaimed { preparation: pr, intent: i } => + concat([kv(prefix: "", key: "preparation", value: pr as String)], write_intent_members(i: i)) + PlacementFinalized { preparation: pr, authority_event: ae } => + concat(preparation_ref_members(r: pr), [kv(prefix: "", key: "authority_event", value: ae as String)]) + PlacementAborted { preparation: pr, reason: r, provenance: pv } => + concat([kv(prefix: "", key: "preparation", value: pr as String), kv(prefix: "", key: "reason", value: r)], release_provenance_members(p: pv)) + } +} + +fn pair_authority_event_kind(e: PairAuthorityEvent) -> String { + match e { + AuthorityEstablished { authority: _, placement: _ } => "established" + AuthorityTransitioned { previous: _, next: _, transaction: _, lifecycle: _, grant: _, entry_state: _, placement: _ } => "transitioned" + EntryStateRecorded { transaction: _, body: _ } => "entry-state-recorded" + HostEffectAdmitted { host: _, purpose: _, executor: _, term: _, residue: _ } => "host-effect-admitted" + HostEffectReleased { admitted_by: _, evidence: _, provenance: _ } => "host-effect-released" + PlacementPrepared { operation: _, group: _, previous_hosts: _, next_hosts: _ } => "placement-prepared" + PlacementAppendClaimed { preparation: _, intent: _ } => "placement-append-claimed" + AuthorityAppendCancelled { preparation: _, provenance: _ } => "append-cancelled" + PlacementFinalized { preparation: _, authority_event: _ } => "placement-finalized" + PlacementAborted { preparation: _, reason: _, provenance: _ } => "placement-aborted" + } +} + +fn pair_authority_event_wire_text(event: ChainEvent) -> String { + chain_event_wire_text(schema: pair_authority_event_schema, e: event, kind: pair_authority_event_kind(e: event.payload), members: pair_authority_event_members(e: event.payload)) +} + +fn member_hash(doc: JsonValue, key: String) -> ContentHash? { + match member_string(doc: doc, key: key) { + Absent => none + Present { value: s } => parse_content_hash(wire: s) + } +} + +fn realization_decode(doc: JsonValue, prefix: String) -> PairRealizationIdentity? { + match member_hash(doc: doc, key: join([prefix, "key"], "")) { + Present { value: k } => Present { value: PairRealizationKeyed { key: k } } + Absent => + match member_nonempty(doc: doc, key: join([prefix, "obligation"], "")) { + Present { value: o } => Present { value: PairRealizationUnestablished { obligation: o } } + Absent => none + } + } +} + +fn lease_decode(doc: JsonValue, prefix: String) -> HeldLease? { + match member_nonempty(doc: doc, key: join([prefix, "lease_key"], "")) { + Absent => none + Present { value: k } => + match member_hash(doc: doc, key: join([prefix, "lease_resource"], "")) { + Absent => none + Present { value: res } => + match member_hash(doc: doc, key: join([prefix, "lease_owner"], "")) { + Absent => none + Present { value: own } => + match member_nat(doc: doc, key: join([prefix, "lease_generation"], "")) { + Absent => none + Present { value: g } => + Present { value: held_lease(epoch: LeaseEpoch { lease_key: k, resource_fingerprint: res, owner_fingerprint: own, generation: g }, observed: LeaseInaccessible) } + } + } + } + } +} + +fn member_cleanup(doc: JsonValue, prefix: String) -> ReleasedBaselineSpec? { + match member_string(doc: doc, key: join([prefix, "cleanup"], "")) { + Absent => none + Present { value: w } => parse_released_baseline(wire: w) + } +} + +fn authority_decode(doc: JsonValue, prefix: String) -> PairServingGroupAuthority? { + match member_string(doc: doc, key: join([prefix, "group"], "")) { + Absent => none + Present { value: gw } => + match parse_fabric_group(wire: gw) { + Absent => none + Present { value: g } => + match member_string(doc: doc, key: join([prefix, "state"], "")) { + Absent => none + Present { value: state } => + if state == "active" { + match realization_decode(doc: doc, prefix: join([prefix, "realization_"], "")) { + Absent => none + Present { value: r } => Present { value: PairServingActive { group: g, exact_realization: r } } + } + } else if state == "suspension-pending-reconciliation" { + match hosts_decode(doc: doc, prefix: prefix) { + Absent => none + Present { value: hs } => + match member_nonempty(doc: doc, key: join([prefix, "transaction"], "")) { + Absent => none + Present { value: t } => + match member_hash(doc: doc, key: join([prefix, "authorization_binding"], "")) { + Absent => none + Present { value: ab } => + match lease_decode(doc: doc, prefix: prefix) { + Absent => none + Present { value: l } => Present { value: SuspensionPendingReconciliation { group: g, hosts: hs, transaction: t, authorization_binding: ab, lease: l } } + } + } + } + } + } else if state == "suspended-for-authorized-successor" { + match hosts_decode(doc: doc, prefix: prefix) { + Absent => none + Present { value: hs } => + match member_nonempty(doc: doc, key: join([prefix, "authorization"], "")) { + Absent => none + Present { value: au } => + match member_cleanup(doc: doc, prefix: prefix) { + Absent => none + Present { value: c } => + match realization_decode(doc: doc, prefix: join([prefix, "candidate_"], "")) { + Absent => none + Present { value: r } => + match lease_decode(doc: doc, prefix: prefix) { + Absent => none + Present { value: l } => + Present { value: SuspendedForAuthorizedSuccessor { group: g, hosts: hs, authorization: au, exact_candidate_realization: r, lease: l, cleanup: c } } + } + } + } + } + } + } else if state == "fenced-refusal" { + match hosts_decode(doc: doc, prefix: prefix) { + Absent => none + Present { value: hs } => + match member_nonempty(doc: doc, key: join([prefix, "cause"], "")) { + Absent => none + Present { value: c } => + match member_nonempty(doc: doc, key: join([prefix, "disposition_authority"], "")) { + Absent => none + Present { value: d } => Present { value: FencedRefusal { group: g, hosts: hs, cause: c, disposition_authority: d } } + } + } + } + } else if state == "released-to-fleet" { + match member_hash(doc: doc, key: join([prefix, "release_receipt"], "")) { + Absent => none + Present { value: r } => Present { value: ReleasedToFleet { group: g, release_receipt: r } } + } + } else { + none + } + } + } + } +} + +type GrantDecode + = GrantAbsent + | GrantDecoded { grant: LeaseGrant } + | GrantIncoherent + +// A grant is all six members or none; a document with some of them is incoherent, not partial. +fn grant_decode(doc: JsonValue) -> GrantDecode { + match member_nonempty(doc: doc, key: "grant_reference") { + Absent => GrantAbsent + Present { value: r } => + match member_nonempty(doc: doc, key: "grant_fence_grant") { + Absent => GrantIncoherent + Present { value: fg } => + match member_nat(doc: doc, key: "grant_fence_generation") { + Absent => GrantIncoherent + Present { value: gen } => + match member_nat(doc: doc, key: "grant_granted_at") { + Absent => GrantIncoherent + Present { value: at } => + match member_nat(doc: doc, key: "grant_maximum_duration_seconds") { + Absent => GrantIncoherent + Present { value: dur } => + match member_nat(doc: doc, key: "grant_expires_at") { + Absent => GrantIncoherent + Present { value: exp } => + GrantDecoded { grant: LeaseGrant { reference: r, fence: LeaseFence { grant: (fg as String) as GrantIdentity, generation: gen }, granted_at: at, maximum_duration_seconds: dur, expires_at: exp } } + } + } + } + } + } + } +} + +// AN OPTIONAL CONTENT-HASH MEMBER, ONCE: absent is a value; present but not a content-hash wire +// is incoherent. Both the entry-state digest and the lifecycle identity are this member under +// different keys (review 67905 found them spelled twice). +type OptionalHashMember + = HashMemberDecoded { digest: ContentHash? } + | HashMemberIncoherent + +fn optional_hash_member(doc: JsonValue, key: String) -> OptionalHashMember { + match member_string(doc: doc, key: key) { + Absent => HashMemberDecoded { digest: none } + Present { value: w } => + match parse_content_hash(wire: w) { + Absent => HashMemberIncoherent + Present { value: h } => HashMemberDecoded { digest: Present { value: h } } + } + } +} + +// A previous state is the whole `intent_prev_` authority or none of it: neither its state word +// nor its group present is "no previous state"; either present without a whole authority is +// incoherent. +type PreviousAuthorityMember + = PreviousAuthorityDecoded { previous: PairServingGroupAuthority? } + | PreviousAuthorityIncoherent + +fn previous_authority_member(doc: JsonValue) -> PreviousAuthorityMember { + let named = (match member_string(doc: doc, key: "intent_prev_state") { Absent => false Present { value: _ } => true }) + || (match member_string(doc: doc, key: "intent_prev_group") { Absent => false Present { value: _ } => true }) + if !named { + PreviousAuthorityDecoded { previous: none } + } else { + match authority_decode(doc: doc, prefix: "intent_prev_") { + Absent => PreviousAuthorityIncoherent + Present { value: p } => PreviousAuthorityDecoded { previous: Present { value: p } } + } + } +} + +fn write_intent_decode(doc: JsonValue) -> AuthorityWriteIntent? { + match member_string(doc: doc, key: "intent_group") { + Absent => none + Present { value: gw } => + match parse_fabric_group(wire: gw) { + Absent => none + Present { value: g } => + match member_string(doc: doc, key: "intent_head") { + Absent => none + Present { value: eh } => + match member_nonempty(doc: doc, key: "intent_actor") { + Absent => none + Present { value: actor } => + match member_nonempty(doc: doc, key: "intent_operation") { + Absent => none + Present { value: op } => + match previous_authority_member(doc: doc) { + PreviousAuthorityIncoherent => none + PreviousAuthorityDecoded { previous: p } => + match authority_decode(doc: doc, prefix: "intent_next_") { + Absent => none + Present { value: n } => + match optional_hash_member(doc: doc, key: "lifecycle") { + HashMemberIncoherent => none + HashMemberDecoded { digest: lc } => + match optional_hash_member(doc: doc, key: "entry_state_digest") { + HashMemberIncoherent => none + HashMemberDecoded { digest: d } => + match grant_decode(doc: doc) { + GrantIncoherent => none + GrantAbsent => Present { value: AuthorityWriteIntent { group: g, expected_head: if eh == "" { HeadAbsent } else { HeadAt { id: eh as EventId } }, actor: actor, previous: p, next: n, operation: op, lifecycle: lc, grant: none, entry_state: d } } + GrantDecoded { grant: gr } => Present { value: AuthorityWriteIntent { group: g, expected_head: if eh == "" { HeadAbsent } else { HeadAt { id: eh as EventId } }, actor: actor, previous: p, next: n, operation: op, lifecycle: lc, grant: Present { value: gr }, entry_state: d } } + } + } + } + } + } + } + } + } + } + } +} + +fn pair_authority_kind_decode(doc: JsonValue, word: String) -> PairAuthorityEvent? { + if word == "placement-prepared" { + match member_nonempty(doc: doc, key: "operation") { + Absent => none + Present { value: o } => + match member_string(doc: doc, key: "group") { + Absent => none + Present { value: gw } => + match parse_fabric_group(wire: gw) { + Absent => none + Present { value: g } => + match hosts_decode_allowing_empty(doc: doc, prefix: "previous_") { + Absent => none + Present { value: ph } => + match hosts_decode_allowing_empty(doc: doc, prefix: "next_") { + Absent => none + Present { value: nh } => Present { value: PlacementPrepared { operation: o, group: g, previous_hosts: ph, next_hosts: nh } } + } + } + } + } + } + } else if word == "placement-append-claimed" { + match member_nonempty(doc: doc, key: "preparation") { + Absent => none + Present { value: pr } => + match write_intent_decode(doc: doc) { + Absent => none + Present { value: i } => Present { value: PlacementAppendClaimed { preparation: (pr as String) as EventId, intent: i } } + } + } + } else if word == "append-cancelled" { + match member_nonempty(doc: doc, key: "preparation") { + Absent => none + Present { value: pr } => + match release_provenance_decode(doc: doc) { + Absent => none + Present { value: pv } => Present { value: AuthorityAppendCancelled { preparation: (pr as String) as EventId, provenance: pv } } + } + } + } else if word == "placement-finalized" { + match preparation_ref_decode(doc: doc) { + Absent => none + Present { value: pr } => + match member_nonempty(doc: doc, key: "authority_event") { + Absent => none + Present { value: ae } => Present { value: PlacementFinalized { preparation: pr, authority_event: (ae as String) as EventId } } + } + } + } else if word == "placement-aborted" { + match member_nonempty(doc: doc, key: "preparation") { + Absent => none + Present { value: pr } => + match member_string(doc: doc, key: "reason") { + Absent => none + Present { value: r } => + match release_provenance_decode(doc: doc) { + Absent => none + Present { value: pv } => Present { value: PlacementAborted { preparation: (pr as String) as EventId, reason: r, provenance: pv } } + } + } + } + } else if word == "established" { + match authority_decode(doc: doc, prefix: "est_") { + Absent => none + Present { value: a } => + match preparation_ref_decode(doc: doc) { + Absent => none + Present { value: pp } => + Present { value: AuthorityEstablished { authority: a, placement: pp } } + } + } + } else if word == "host-effect-admitted" { + match member_nonempty(doc: doc, key: "host") { + Absent => none + Present { value: h } => + match member_nonempty(doc: doc, key: "purpose") { + Absent => none + Present { value: p } => + match member_nonempty(doc: doc, key: "executor") { + Absent => none + Present { value: x } => + match member_nat(doc: doc, key: "term_seconds") { + Absent => none + Present { value: ts } => + match residue_spec_decode(doc: doc, prefix: "residue_") { + Absent => none + Present { value: rs } => Present { value: HostEffectAdmitted { host: (h as String) as HostIdentity, purpose: p, executor: x, term: second(count: ts), residue: rs } } + } + } + } + } + } + } else if word == "host-effect-released" { + match member_nonempty(doc: doc, key: "admitted_by") { + Absent => none + Present { value: a } => + match quiescence_evidence_decode(doc: doc) { + Absent => none + Present { value: ev } => + match release_provenance_decode(doc: doc) { + Absent => none + Present { value: pv } => Present { value: HostEffectReleased { admitted_by: (a as String) as EventId, evidence: ev, provenance: pv } } + } + } + } + } else if word == "entry-state-recorded" { + match member_nonempty(doc: doc, key: "transaction") { + Absent => none + Present { value: t } => + match member_string(doc: doc, key: "body") { + Absent => none + Present { value: b } => Present { value: EntryStateRecorded { transaction: t, body: b } } + } + } + } else if word != "transitioned" { + none + } else { + match member_nonempty(doc: doc, key: "transaction") { + Absent => none + Present { value: t } => + match authority_decode(doc: doc, prefix: "prev_") { + Absent => none + Present { value: p } => + match authority_decode(doc: doc, prefix: "next_") { + Absent => none + Present { value: n } => + match optional_hash_member(doc: doc, key: "lifecycle") { + HashMemberIncoherent => none + HashMemberDecoded { digest: lc } => + match optional_hash_member(doc: doc, key: "entry_state_digest") { + HashMemberIncoherent => none + HashMemberDecoded { digest: d } => + match preparation_ref_decode(doc: doc) { + Absent => none + Present { value: pp } => + match grant_decode(doc: doc) { + GrantIncoherent => none + GrantAbsent => Present { value: AuthorityTransitioned { previous: p, next: n, transaction: t, lifecycle: lc, grant: none, entry_state: d, placement: pp } } + GrantDecoded { grant: g } => Present { value: AuthorityTransitioned { previous: p, next: n, transaction: t, lifecycle: lc, grant: Present { value: g }, entry_state: d, placement: pp } } + } + } + } + } + } + } + } + } +} + +fn pair_authority_event_decode(text: String) -> EventDecode { + chain_event_decode(text: text, schema: pair_authority_event_schema, kind_decode: fn(doc, word) { pair_authority_kind_decode(doc: doc, word: word) }) +} + +// ── THE FOLD ─────────────────────────────────────────────────────────────────────────────────── + +// ── ONE DURABLE GENESIS PER GROUP PARTITION ──────────────────────────────────────────────────── +// +// Every group's partition begins UNESTABLISHED and commits no host: the genesis is a fact of the +// log, not of today's source roster. The source row (gunbc.spark.pair_serving_authority +// spark_pair_serving_authorities) is the DESIRED authority; it becomes the group's authority only +// when an AuthorityEstablished event lands on the partition (pair_serving_authority_establish, +// one compare-and-set at the head, preceded by its HostsCommitted record on the placement +// partition). So adding or removing a source AUTHORITY row never changes how a persisted chain +// folds: the same fold reads every group partition from the same genesis, and the authority +// roster is consulted by nothing that reads the log. ONE ROSTER DEPENDENCE REMAINS, STATED: the +// population of a PairServingActive state is the lane roster's (pair_serving_committed_hosts; +// Active carries no hosts of its own -- D0 restores the exact previous Active unchanged), so the +// preparation-copy check on an event that leaves or enters Active compares against the roster as +// read at fold time, and a lane-roster edit would make such a persisted event refuse. Every state +// D0 writes freezes its hosts on the wire; freezing them on Active too is the D1 converger's +// change to the authority model (gunbc.spark.pair_serving_authority), the trigger that retires +// this dependence. Generation counts transitions since the establishment (the +// established state is generation 0). Host placement is never on a group partition; a placement +// event here is a lying document and refuses. +type AuthorityFold + = AuthorityUnestablished { cancelled: List } + | AuthorityFolded { current: PairServingGroupAuthority, generation: Nat, grant: LeaseGrant?, entry_state: ContentHash?, previous: PairServingGroupAuthority?, admitted_by: EventId?, transitions: List, preparations_consumed: List, cancelled: List } + | AuthorityFoldRefused { at_event: EventId, reason: String } + +// A preparation an authority event consumed: the copy the event carried, the event's id and the +// write intent the event is (derived from it). The fold refuses a second consumer of one +// preparation, a consumer of a cancelled one, and a cancellation of a consumed one. +type ConsumedPreparation { + preparation: PlacementPreparationRef + authority_event: EventId + intent: AuthorityWriteIntent +} + +// A cancellation, with the provenance that wrote it, its actor and the head it was appended at: +// the joined read checks it covered the preparation's claimant and moved the very head the claim +// named. +type CancelledPreparation { + preparation: EventId + provenance: ReleaseProvenance + actor: NonEmptyStr + at_head: HeadExpectation +} + +fn cancellation_refusal(consumed: List, preparation: EventId) -> String? { + if any(consumed, c => (c.preparation.id as String) == (preparation as String)) { + Present { value: join(["the event cancels the append of preparation ", preparation as String, ", which an authority event on this partition already consumed"], "") } + } else { + none + } +} + +fn cancelled_record(env: ChainEnvelope, preparation: EventId, provenance: ReleaseProvenance) -> CancelledPreparation { + CancelledPreparation { preparation: preparation, provenance: provenance, actor: env.event.actor, at_head: match env.event.parent { Absent => HeadAbsent Present { value: p } => HeadAt { id: p } } } +} + +fn consumption_refusal(consumed: List, cancelled: List, carried: PlacementPreparationRef) -> String? { + if any(consumed, c => (c.preparation.id as String) == (carried.id as String)) { + Present { value: join(["the event consumes preparation ", carried.id as String, ", which an earlier authority event already consumed"], "") } + } else if any(cancelled, c => (c.preparation as String) == (carried.id as String)) { + Present { value: join(["the event consumes preparation ", carried.id as String, ", whose append was cancelled on this partition"], "") } + } else { + none + } +} + +// EVERY ADMITTED TRANSITION, OLDEST FIRST, BY THE TRANSACTION THAT WROTE IT. A transaction that +// recovers after a crash has to know whether IT ever moved this authority and what its last write +// landed -- a claim held with no transition means the first write never happened; a claim held +// with a settling transition means only the claim's terminal is owed. Neither is readable from +// the current state alone once other transactions have written after it, so the fold keeps the +// record. Receipt events are not transitions and are not recorded here. +type AuthorityTransitionRecord { + id: EventId + transaction: NonEmptyStr + lifecycle: ContentHash? + next: PairServingGroupAuthority + placement: PlacementPreparationRef +} + +// THE SELECTION IS BY LIFECYCLE, NOT BY THE TRANSACTION WORD: a record belongs to a lifecycle only +// when it carries that lifecycle's identity and the word agrees. A record with no lifecycle (an +// operator's write) belongs to none. +fn transitions_by(records: List, transaction: NonEmptyStr, lifecycle: ContentHash) -> List { + filter(records, r => (r.transaction as String) == (transaction as String) && (match r.lifecycle { Absent => false Present { value: l } => content_hash_equal(left: l, right: lifecycle) })) +} + + +// THE GRANT MUST BE THE LEASE'S OWN, AND THE FOLD CHECKS THE GENEALOGY. Co-location in one event +// is not provenance: a grant whose reference names another transaction, whose fence names +// another admitting event, or whose generation is not the lease epoch's would still read as a +// live term and admit a launch. So a transition that carries a grant into a leased state must +// agree on all three joins -- grant.reference = event transaction = lease key; grant.fence.grant = +// the event's parent, which is the admitting write the pending state landed by; grant.fence.generation +// = the lease epoch's generation -- or the fold refuses at that event. +type GrantGenealogy + = GrantGenealogyAgrees + | GrantGenealogyDisagrees { join: String } + | GrantWithoutLease + +fn lease_of(a: PairServingGroupAuthority) -> HeldLease? { + match a { + PairServingActive { group: _, exact_realization: _ } => none + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: l } => Present { value: l } + SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: l, cleanup: _ } => Present { value: l } + FencedRefusal { group: _, cause: _, disposition_authority: _ } => none + ReleasedToFleet { group: _, release_receipt: _ } => none + } +} + +fn grant_genealogy(gr: LeaseGrant, next: PairServingGroupAuthority, transaction: NonEmptyStr, admitted_by: EventId?) -> GrantGenealogy { + match lease_of(a: next) { + Absent => GrantWithoutLease + Present { value: l } => + if (gr.reference as String) != (transaction as String) { + GrantGenealogyDisagrees { join: join(["grant reference ", gr.reference as String, " is not the event's transaction ", transaction as String], "") } + } else if (gr.reference as String) != (l.epoch.lease_key as String) { + GrantGenealogyDisagrees { join: join(["grant reference ", gr.reference as String, " is not the lease key ", l.epoch.lease_key as String], "") } + } else if gr.fence.generation != l.epoch.generation { + GrantGenealogyDisagrees { join: join(["grant fence generation ", to_string(gr.fence.generation), " is not the lease epoch generation ", to_string(l.epoch.generation)], "") } + } else { + match admitted_by { + Absent => GrantGenealogyDisagrees { join: "a grant was carried on a transition from a state no transition admitted" } + Present { value: pid } => + if (gr.fence.grant as String) != (pid as String) { + GrantGenealogyDisagrees { join: join(["grant fence ", gr.fence.grant as String, " is not the admitting event ", pid as String], "") } + } else { + GrantGenealogyAgrees + } + } + } + } +} + +fn establishment_refusal(a: PairServingGroupAuthority) -> String? { + if pair_serving_commits_hosts(a: a) && length(pair_serving_committed_hosts(a: a)) == 0 { + Present { value: "the authority commits hosts but names none" } + } else { + match lease_of(a: a) { + Present { value: _ } => Present { value: "an authority that holds a lease cannot be established; a lease is granted by a transition" } + Absent => none + } + } +} + +fn establishment_operation(group: FabricGroup) -> NonEmptyStr { + join(["establish ", fabric_group_wire(g: group) as String], "") as NonEmptyStr +} + +// The authority event's copy of its preparation must be THIS partition's group, over exactly the +// populations the event leaves and enters, under this event's operation. +fn preparation_ref_refusal(carried: PlacementPreparationRef, group: FabricGroup, previous: List, next: List, operation: NonEmptyStr) -> String? { + if !same_group(a: carried.group, b: group) { + Present { value: join(["the event's preparation ", carried.id as String, " belongs to ", fabric_group_wire(g: carried.group) as String, ", not this partition's group"], "") } + } else if !same_host_set(a: carried.previous_hosts, b: previous) { + Present { value: join(["the event's preparation ", carried.id as String, " leaves a population that is not the state's: ", join(map(carried.previous_hosts, h => h as String), ","), " vs ", join(map(previous, h => h as String), ",")], "") } + } else if !same_host_set(a: carried.next_hosts, b: next) { + Present { value: join(["the event's preparation ", carried.id as String, " enters a population that is not the next state's: ", join(map(carried.next_hosts, h => h as String), ","), " vs ", join(map(next, h => h as String), ",")], "") } + } else if (carried.operation as String) != (operation as String) { + Present { value: join(["the event's preparation ", carried.id as String, " is for operation `", carried.operation as String, "`, not this event's `", operation as String, "`"], "") } + } else { + none + } +} + +data placement_event_on_group_partition: String = "a host-placement event on a group partition: placement lives on the host-placement partition only" + +fn authority_fold_step(acc: AuthorityFold, env: ChainEnvelope) -> AuthorityFold { + match acc { + AuthorityFoldRefused { at_event: _, reason: _ } => acc + AuthorityUnestablished { cancelled: xs } => + match env.event.payload { + AuthorityAppendCancelled { preparation: pr, provenance: pv } => AuthorityUnestablished { cancelled: concat(xs, [cancelled_record(env: env, preparation: pr, provenance: pv)]) } + AuthorityEstablished { authority: a, placement: pp } => + if (env.event.partition as String) != (pair_authority_partition(group: pair_serving_authority_group(a: a)) as String) { + AuthorityFoldRefused { at_event: env.id, reason: join(["the established authority names ", fabric_group_wire(g: pair_serving_authority_group(a: a)) as String, ", which is not this partition's group"], "") } + } else { + match establishment_refusal(a: a) { + Present { value: why } => AuthorityFoldRefused { at_event: env.id, reason: join(["the establishment is refused: ", why], "") } + Absent => + match preparation_ref_refusal(carried: pp, group: pair_serving_authority_group(a: a), previous: [] as List, next: committed_population(a: a), operation: establishment_operation(group: pair_serving_authority_group(a: a))) { + Present { value: why } => AuthorityFoldRefused { at_event: env.id, reason: why } + Absent => + match consumption_refusal(consumed: [] as List, cancelled: xs, carried: pp) { + Present { value: why } => AuthorityFoldRefused { at_event: env.id, reason: why } + Absent => AuthorityFolded { current: a, generation: 0, grant: none, entry_state: none, previous: none, admitted_by: none, transitions: [] as List, preparations_consumed: [ConsumedPreparation { preparation: pp, authority_event: env.id, intent: event_write_intent(env: env, group: pair_serving_authority_group(a: a), previous: none, next: a, operation: establishment_operation(group: pair_serving_authority_group(a: a)), lifecycle: none, grant: none, entry_state: none) }], cancelled: xs } + } + } + } + } + AuthorityTransitioned { previous: _, next: _, transaction: _, lifecycle: _, grant: _, entry_state: _, placement: _ } => + AuthorityFoldRefused { at_event: env.id, reason: "a transition was recorded before any authority was established on this partition" } + EntryStateRecorded { transaction: _, body: _ } => + AuthorityFoldRefused { at_event: env.id, reason: "an entry state was recorded before any authority was established on this partition" } + _ => AuthorityFoldRefused { at_event: env.id, reason: placement_event_on_group_partition } + } + AuthorityFolded { current: cur, generation: g, grant: cg, entry_state: ces, previous: cp, admitted_by: cab, transitions: recs, preparations_consumed: pcs, cancelled: xs } => + match env.event.payload { + AuthorityEstablished { authority: _, placement: _ } => + AuthorityFoldRefused { at_event: env.id, reason: "an authority is already established on this partition; the state moves by transition" } + EntryStateRecorded { transaction: _, body: _ } => + AuthorityFolded { current: cur, generation: g, grant: cg, entry_state: ces, previous: cp, admitted_by: cab, transitions: recs, preparations_consumed: pcs, cancelled: xs } + AuthorityAppendCancelled { preparation: pr, provenance: pv } => + match cancellation_refusal(consumed: pcs, preparation: pr) { + Present { value: why } => AuthorityFoldRefused { at_event: env.id, reason: why } + Absent => AuthorityFolded { current: cur, generation: g, grant: cg, entry_state: ces, previous: cp, admitted_by: cab, transitions: recs, preparations_consumed: pcs, cancelled: concat(xs, [cancelled_record(env: env, preparation: pr, provenance: pv)]) } + } + AuthorityTransitioned { previous: p, next: n, transaction: t, lifecycle: lc, grant: gr, entry_state: es, placement: pp } => + if !authority_eq(a: p, b: cur) { + AuthorityFoldRefused { at_event: env.id, reason: join(["the event leaves state ", authority_wire(a: p), " but the current state was ", authority_wire(a: cur)], "") } + } else if (fabric_group_wire(g: pair_serving_authority_group(a: n)) as String) != (fabric_group_wire(g: pair_serving_authority_group(a: cur)) as String) { + AuthorityFoldRefused { at_event: env.id, reason: "the event's next state names a different group than the partition's" } + } else if pair_serving_commits_hosts(a: n) && length(pair_serving_committed_hosts(a: n)) == 0 { + AuthorityFoldRefused { at_event: env.id, reason: "the event's next state commits hosts but names none: a transaction state with an empty retained population is not admitted" } + } else { + match preparation_ref_refusal(carried: pp, group: pair_serving_authority_group(a: cur), previous: committed_population(a: cur), next: committed_population(a: n), operation: t) { + Present { value: why } => AuthorityFoldRefused { at_event: env.id, reason: why } + Absent => + match consumption_refusal(consumed: pcs, cancelled: xs, carried: pp) { + Present { value: why } => AuthorityFoldRefused { at_event: env.id, reason: why } + Absent => { + let recorded = concat(recs, [AuthorityTransitionRecord { id: env.id, transaction: t, lifecycle: lc, next: n, placement: pp }]) + let consumed = concat(pcs, [ConsumedPreparation { preparation: pp, authority_event: env.id, intent: event_write_intent(env: env, group: pair_serving_authority_group(a: cur), previous: Present { value: cur }, next: n, operation: t, lifecycle: lc, grant: gr, entry_state: es) }]) + match gr { + Absent => AuthorityFolded { current: n, generation: g + 1, grant: none, entry_state: es, previous: Present { value: p }, admitted_by: Present { value: env.id }, transitions: recorded, preparations_consumed: consumed, cancelled: xs } + Present { value: grant } => + match grant_genealogy(gr: grant, next: n, transaction: t, admitted_by: cab) { + GrantGenealogyAgrees => AuthorityFolded { current: n, generation: g + 1, grant: gr, entry_state: es, previous: Present { value: p }, admitted_by: Present { value: env.id }, transitions: recorded, preparations_consumed: consumed, cancelled: xs } + GrantWithoutLease => AuthorityFoldRefused { at_event: env.id, reason: "the event carries a grant into a state that holds no lease" } + GrantGenealogyDisagrees { join: j } => AuthorityFoldRefused { at_event: env.id, reason: join(["the event's grant is not its lease's: ", j], "") } + } + } + } + } + } + } + _ => AuthorityFoldRefused { at_event: env.id, reason: placement_event_on_group_partition } + } + } +} + +// From the unestablished genesis through the establishment and every recorded transition, oldest +// first. +fn authority_fold(oldest_first: List>) -> AuthorityFold { + fold(oldest_first, init: AuthorityUnestablished { cancelled: [] as List }, f: (acc, env) => authority_fold_step(acc: acc, env: env)) +} + +// ── THE PLACEMENT FOLD ───────────────────────────────────────────────────────────────────────── +// +// Effects and preparations on one chain. A claim refuses on a host another live claim holds or +// any group fences (a live commitment, or a pending preparation's union); a preparation refuses +// while any host of its union has a live claim or is fenced by another group, when its previous +// population is not the group's live commitment, and while the group has ANOTHER pending +// preparation (one saga per group at a time, so finalizations land in authority order); an +// append-claim names a Prepared preparation; a finalization names an AppendClaimed one and its +// copy of the preparation must be the record; an abort names a pending one, with a provenance +// that is the claimant's own (an AppendClaimed preparation is aborted by its claimant, written by +// its claimant) or a recovery naming it; a release is admitted as release_admission decides. +// Anything else on this partition refuses. +type PlacementFold + = PlacementFolded { effects: List, preparations: List } + | PlacementFoldRefused { at_event: EventId, reason: String } + +fn pending_preparation_of(preps: List, preparation: EventId) -> PlacementPreparationRecord? { + first(filter(preps, r => (r.id as String) == (preparation as String) && preparation_is_pending(r: r))) +} + +fn abort_provenance_covers(r: PlacementPreparationRecord, provenance: ReleaseProvenance, actor: NonEmptyStr) -> Bool { + match r.state { + Prepared => release_provenance_covers(p: provenance, claim: r.id, claimant: r.prepared_by, actor: actor) + AppendClaimed { claimant: cl, expected_head: _, intent: _ } => release_provenance_covers(p: provenance, claim: r.id, claimant: cl, actor: actor) + _ => false + } +} + +fn placement_fold_step(acc: PlacementFold, env: ChainEnvelope) -> PlacementFold { + match acc { + PlacementFoldRefused { at_event: _, reason: _ } => acc + PlacementFolded { effects: effs, preparations: preps } => + match env.event.payload { + HostEffectAdmitted { host: h, purpose: p, executor: x, term: t, residue: rs } => + if length(live_host_effects_on(records: effs, hosts: [h], at: env.event.recorded_at)) != 0 { + PlacementFoldRefused { at_event: env.id, reason: join(["a host effect was admitted on ", h as String, " while another live host effect held it"], "") } + } else if length(groups_fencing(preps: preps, host: h)) != 0 { + PlacementFoldRefused { at_event: env.id, reason: join(["a host effect was admitted on ", h as String, " while an authority fenced it: ", join(map(groups_fencing(preps: preps, host: h), g => fabric_group_wire(g: g) as String), ", ")], "") } + } else { + PlacementFolded { effects: concat(effs, [HostEffectRecord { id: env.id, host: h, purpose: p, executor: x, recorded_at: env.event.recorded_at, term: t, residue: rs, released: false }]), preparations: preps } + } + HostEffectReleased { admitted_by: a, evidence: ev, provenance: pv } => + match release_admission(records: effs, claim: a, evidence: ev, provenance: pv, actor: env.event.actor) { + ReleaseAdmitted { record: _ } => PlacementFolded { effects: map(effs, r => if (r.id as String) == (a as String) { host_effect_released(r: r) } else { r }), preparations: preps } + other => PlacementFoldRefused { at_event: env.id, reason: join(["a host effect release is refused: ", release_refusal_reason(a: other, claim: a)], "") } + } + PlacementPrepared { operation: o, group: g, previous_hosts: ph, next_hosts: nh } => + match preparation_refusal(effects: effs, preps: preps, group: g, previous_hosts: ph, next_hosts: nh, at: env.event.recorded_at) { + Present { value: why } => PlacementFoldRefused { at_event: env.id, reason: join(["a preparation is refused: ", why], "") } + Absent => PlacementFolded { effects: effs, preparations: concat(preps, [PlacementPreparationRecord { id: env.id, operation: o, group: g, previous_hosts: ph, next_hosts: nh, recorded_at: env.event.recorded_at, prepared_by: env.event.actor, state: Prepared }]) } + } + PlacementAppendClaimed { preparation: pr, intent: i } => + if (env.event.actor as String) != (i.actor as String) { + PlacementFoldRefused { at_event: env.id, reason: join(["an append claim for a write by ", i.actor as String, " was written by ", env.event.actor as String, ": a claim is written by its claimant"], "") } + } else { + match pending_preparation_of(preps: preps, preparation: pr) { + Present { value: PlacementPreparationRecord { state: Prepared } } => { + let r = match first(filter(preps, x => (x.id as String) == (pr as String))) { Present { value: x } => Present { value: x } Absent => none } + match r { + Absent => PlacementFoldRefused { at_event: env.id, reason: join(["an append claim names no Prepared preparation ", pr as String], "") } + Present { value: record } => + match preparation_ref_refusal(carried: preparation_ref_of(r: record), group: i.group, previous: intent_previous_population(i: i), next: committed_population(a: i.next), operation: i.operation) { + Present { value: why } => PlacementFoldRefused { at_event: env.id, reason: join(["an append claim's intent is not the write its preparation is for: ", why], "") } + Absent => PlacementFolded { effects: effs, preparations: map(preps, x => if (x.id as String) == (pr as String) { preparation_with_state(r: x, state: AppendClaimed { claimant: i.actor, expected_head: i.expected_head, intent: i }) } else { x }) } + } + } + } + _ => PlacementFoldRefused { at_event: env.id, reason: join(["an append claim names no Prepared preparation ", pr as String], "") } + } + } + PlacementFinalized { preparation: pr, authority_event: ae } => + match pending_preparation_of(preps: preps, preparation: pr.id) { + Present { value: r } => + match r.state { + AppendClaimed { claimant: _, expected_head: _, intent: i } => + if !preparation_ref_eq(a: pr, b: preparation_ref_of(r: r)) { + PlacementFoldRefused { at_event: env.id, reason: join(["a finalization's copy of preparation ", pr.id as String, " is not the record: ", preparation_line(r: r)], "") } + } else { + PlacementFolded { effects: effs, preparations: map(preps, x => if (x.id as String) == (pr.id as String) { preparation_with_state(r: x, state: Finalized { authority_event: ae, finalized_by: env.id, intent: i }) } else { x }) } + } + _ => PlacementFoldRefused { at_event: env.id, reason: join(["a finalization names preparation ", pr.id as String, " whose append was never claimed"], "") } + } + Absent => PlacementFoldRefused { at_event: env.id, reason: join(["a finalization names no pending preparation ", pr.id as String], "") } + } + PlacementAborted { preparation: pr, reason: _, provenance: pv } => + match pending_preparation_of(preps: preps, preparation: pr) { + Absent => PlacementFoldRefused { at_event: env.id, reason: join(["an abort names no pending preparation ", pr as String], "") } + Present { value: r } => + if !abort_provenance_covers(r: r, provenance: pv, actor: env.event.actor) { + PlacementFoldRefused { at_event: env.id, reason: join(["an abort of preparation ", pr as String, " is not its claimant's own (its preparer's while Prepared) and names no recovery authorization for it: the provenance does not cover it"], "") } + } else { + PlacementFolded { effects: effs, preparations: map(preps, x => if (x.id as String) == (pr as String) { preparation_with_state(r: x, state: Aborted) } else { x }) } + } + } + _ => PlacementFoldRefused { at_event: env.id, reason: "an authority event on the placement partition: authority lives on the group partitions only" } + } + } +} + +// WHY A PREPARATION WOULD BE REFUSED: another pending preparation of the group, a previous +// population that is not the group's live commitment, a live effect on a host of its union, or a +// host of its union fenced by another group. +fn preparation_refusal(effects: List, preps: List, group: FabricGroup, previous_hosts: List, next_hosts: List, at: EpochSecs) -> String? { + let union = hosts_union(a: previous_hosts, b: next_hosts) + let fenced = live_host_effects_on(records: effects, hosts: union, at: at) + let pending = filter(preps, r => same_group(a: r.group, b: group) && preparation_is_pending(r: r)) + if length(pending) != 0 { + Present { value: join([fabric_group_wire(g: group) as String, " has a pending preparation; one saga at a time: ", join(map(pending, r => preparation_line(r: r)), "; ")], "") } + } else if !same_host_set(a: previous_hosts, b: group_live_commitment(preps: preps, group: group)) { + Present { value: join(["the preparation's previous population is not ", fabric_group_wire(g: group) as String, "'s live commitment (", join(map(group_live_commitment(preps: preps, group: group), h => h as String), ","), ")"], "") } + } else if length(fenced) != 0 { + Present { value: join(["the authority would commit a host under a live host effect: ", join(map(fenced, r => host_effect_line_at(r: r, at: at)), "; ")], "") } + } else { + let others = filter(union, h => any(groups_fencing(preps: preps, host: h), g => !same_group(a: g, b: group))) + if length(others) != 0 { + Present { value: join(["the authority would commit a host another group fences: ", join(map(others, h => h as String), ",")], "") } + } else { + none + } + } +} + +fn placement_fold(oldest_first: List>) -> PlacementFold { + fold(oldest_first, init: PlacementFolded { effects: [] as List, preparations: [] as List }, f: (acc, env) => placement_fold_step(acc: acc, env: env)) +} + +// ── THE READ ─────────────────────────────────────────────────────────────────────────────────── + +type CurrentAuthority + = CurrentAuthorityRead { authority: PairServingGroupAuthority, head: HeadExpectation, generation: Nat, grant: LeaseGrant?, entry_state: ContentHash?, previous: PairServingGroupAuthority?, admitted_by: EventId?, transitions: List } + | CurrentAuthorityUnread { group: FabricGroup, step: String, reason: String } + +// THE LEASE'S OBSERVED STATE IS DERIVED AT THE READ, FROM THE GRANT AND THE INSTANT. The bytes +// carry what the writer observed when it wrote; what a consumer needs is what holds NOW. A +// grant whose expiry has passed reads as LeaseRunningStale -- which pair_serving_successor_may_launch +// maps to Drifted, removing the right to START while pair_serving_may_finish keeps the right to +// FINISH -- and a leased arm with no recorded grant reads as LeaseInaccessible: its term cannot +// be established, so nothing may start under it. An arm that carries no lease is unchanged. +fn lease_observed_at(grant: LeaseGrant?, at: EpochSecs) -> HeldLeaseObservedState { + match grant { + Absent => LeaseInaccessible + Present { value: g } => if lease_grant_expired_at(grant: g, at: at) { LeaseRunningStale } else { LeaseRunningExpected } + } +} + +fn authority_with_lease_observed(a: PairServingGroupAuthority, grant: LeaseGrant?, at: EpochSecs) -> PairServingGroupAuthority { + match a { + PairServingActive { group: _, exact_realization: _ } => a + SuspensionPendingReconciliation { group: g, hosts: hs, transaction: t, authorization_binding: ab, lease: l } => + SuspensionPendingReconciliation { group: g, hosts: hs, transaction: t, authorization_binding: ab, lease: held_lease_with_observed(lease: l, observed: lease_observed_at(grant: grant, at: at)) } + SuspendedForAuthorizedSuccessor { group: g, hosts: hs, authorization: au, exact_candidate_realization: r, lease: l, cleanup: c } => + SuspendedForAuthorizedSuccessor { group: g, hosts: hs, authorization: au, exact_candidate_realization: r, lease: held_lease_with_observed(lease: l, observed: lease_observed_at(grant: grant, at: at)), cleanup: c } + FencedRefusal { group: _, cause: _, disposition_authority: _ } => a + ReleasedToFleet { group: _, release_receipt: _ } => a + } +} + + +// THE GROUP PARTITION, READ AND FOLDED. Every failure to read is Unread with its step; a refused +// fold is Unread at "fold". The two readable outcomes are the two fold states. +type GroupPartitionRead + = GroupPartitionAt { head: HeadExpectation, fold: AuthorityFold } + | GroupPartitionUnread { step: String, reason: String } + +// A REFUSED FOLD IS CARRIED AS THE FOLD IT IS, not folded into "unread" here: every consumer +// decides its own arm over it and renders it through fold_refusal_text, so the refusal is +// spelled once and no consumer carries an arm that can never fire. +fn fold_refusal_text(at_event: EventId, reason: String) -> String { + join(["the chain's fold refused at event ", at_event as String, ": ", reason], "") +} + +fn read_group_partition(store: FabricStorageBinding, group: FabricGroup) -> GroupPartitionRead { + match event_log_read_partition_with(store: store, partition: pair_authority_partition(group: group), budget: event_log_read_budget(), decode: fn(text) { pair_authority_event_decode(text: text) }) { + PartitionReadRefused { cause: c } => GroupPartitionUnread { step: "store", reason: event_log_refusal_wire(cause: c) } + PartitionReadOk { head: head, walk: walk } => + match walk { + ChainIncomplete { missing: m, newest_first_so_far: _ } => GroupPartitionUnread { step: "chain", reason: join(["partition chain is missing event ", m as String], "") } + ChainBudgetExhausted { at: a } => GroupPartitionUnread { step: "chain", reason: join(["partition chain exceeded the read budget at ", a as String], "") } + ChainWalked { oldest_first: xs } => + GroupPartitionAt { head: head, fold: authority_fold(oldest_first: xs) } + } + } +} + +// THE CURRENT AUTHORITY OF A GROUP. A group whose partition carries no establishment is not claimed +// and has no authority to read (absence is not permission); every failure to read is Unread with +// its step, and no arm answers the source row when the log could not be read. +// THE CURRENT AUTHORITY IS A JOINED READ: the group fold alone is one chain's word, so every +// preparation it consumed is looked up on the placement chain and must be there, append-claimed +// or finalized (finalized by THIS event), with a copy equal to the record. A group history that +// names a preparation the placement chain never saw, or one finalized to another event, is +// unread -- never an authority. +fn current_pair_serving_authority(store: FabricStorageBinding, group: FabricGroup, at: EpochSecs) -> CurrentAuthority { + current_authority_over(group: group, partition: read_group_partition(store: store, group: group), placement: placement_fold_read(store: store), at: at) +} + +fn current_authority_over(group: FabricGroup, partition: GroupPartitionRead, placement: PlacementRead, at: EpochSecs) -> CurrentAuthority { + match partition { + GroupPartitionUnread { step: s, reason: why } => CurrentAuthorityUnread { group: group, step: s, reason: why } + GroupPartitionAt { head: head, fold: f } => + match f { + AuthorityFoldRefused { at_event: e, reason: why } => CurrentAuthorityUnread { group: group, step: "fold", reason: fold_refusal_text(at_event: e, reason: why) } + AuthorityUnestablished { cancelled: _ } => CurrentAuthorityUnread { group: group, step: "claim", reason: "no authority has been established on this group's partition (the source row is desired, not established)" } + AuthorityFolded { current: cur, generation: g, grant: gr, entry_state: es, previous: pv, admitted_by: ab, transitions: recs, preparations_consumed: pcs, cancelled: xs } => + match placement { + PlacementUnread { step: st, reason: why } => CurrentAuthorityUnread { group: group, step: join(["placement/", st], ""), reason: why } + PlacementReadAt { head: _, effects: _, preparations: preps } => + match first(concat(flat_map(pcs, c => consumed_preparation_gap(consumed: c, preps: preps)), flat_map(xs, c => cancellation_gap(cancelled: c, preps: preps)))) { + Present { value: gap } => CurrentAuthorityUnread { group: group, step: "join", reason: gap } + Absent => CurrentAuthorityRead { authority: authority_with_lease_observed(a: cur, grant: gr, at: at), head: head, generation: g, grant: gr, entry_state: es, previous: pv, admitted_by: ab, transitions: recs } + } + } + } + } +} + +// A cancellation's word must have covered the preparation's claimant: written by the claimant, or +// a recovery naming the preparation. +fn cancellation_gap(cancelled: CancelledPreparation, preps: List) -> List { + match first(filter(preps, r => (r.id as String) == (cancelled.preparation as String))) { + Absent => [join(["a cancellation names preparation ", cancelled.preparation as String, ", which is not on the placement chain"], "")] + Present { value: r } => + match r.state { + Prepared => [join(["a cancellation names preparation ", r.id as String, " whose append was never claimed"], "")] + AppendClaimed { claimant: cl, expected_head: eh, intent: _ } => + if !release_provenance_covers(p: cancelled.provenance, claim: r.id, claimant: cl, actor: cancelled.actor) { [join(["the cancellation of preparation ", r.id as String, " was not its claimant's nor a recovery naming it"], "")] } + else if !head_expectation_eq(a: cancelled.at_head, b: eh) { [join(["the cancellation of preparation ", r.id as String, " was appended at a head other than the one its append claim named"], "")] } + else { [] as List } + Aborted => [] as List + Finalized { authority_event: _, finalized_by: _, intent: _ } => [join(["preparation ", r.id as String, " reads both cancelled and finalized"], "")] + } + } +} + +// The placement chain's word on one consumed preparation: present, claimed or finalized to this +// event with the claim's intent, and the record. +// THE ONE CROSS-CHAIN PROOF that an accepted authority event consumed a placement record: the +// event's copy is the record, the record's append was claimed (or finalized to that very event), +// and the claim's intent is the event's. Every joined read -- the current authority, the +// placement read, the consumption a finalization or abort decides on -- asks this one fn, so the +// proof cannot drift between them. Absent is the join holding. +fn consumed_join(consumed: ConsumedPreparation, r: PlacementPreparationRecord) -> String? { + if !preparation_ref_eq(a: consumed.preparation, b: preparation_ref_of(r: r)) { + Present { value: join(["authority event ", consumed.authority_event as String, " carries a copy of preparation ", r.id as String, " that is not the record"], "") } + } else { + match r.state { + AppendClaimed { claimant: _, expected_head: _, intent: i } => + if write_intent_eq(a: i, b: consumed.intent) { none } else { Present { value: join(["authority event ", consumed.authority_event as String, " is not the write preparation ", r.id as String, "'s append claim named (its intent differs: another actor, another head, or another next state)"], "") } } + Finalized { authority_event: ae, finalized_by: _, intent: i } => + if (ae as String) != (consumed.authority_event as String) { Present { value: join(["preparation ", r.id as String, " was finalized to authority event ", ae as String, ", not to ", consumed.authority_event as String], "") } } + else if !write_intent_eq(a: i, b: consumed.intent) { Present { value: join(["authority event ", consumed.authority_event as String, " is not the write preparation ", r.id as String, "'s append claim named (its intent differs)"], "") } } + else { none } + Prepared => Present { value: join(["authority event ", consumed.authority_event as String, " consumes preparation ", r.id as String, " whose append was never claimed"], "") } + Aborted => Present { value: join(["authority event ", consumed.authority_event as String, " consumes preparation ", r.id as String, ", which is aborted"], "") } + } + } +} + +fn consumed_preparation_gap(consumed: ConsumedPreparation, preps: List) -> List { + match first(filter(preps, r => (r.id as String) == (consumed.preparation.id as String))) { + Absent => [join(["authority event ", consumed.authority_event as String, " consumes preparation ", consumed.preparation.id as String, ", which is not on the placement chain"], "")] + Present { value: r } => match consumed_join(consumed: consumed, r: r) { Absent => [] as List Present { value: gap } => [gap] } + } +} +// THE PLACEMENT PARTITION, READ AND FOLDED: the head every placement writer compares-and-sets +// against, and the live effects and preparations. +type PlacementRead + = PlacementReadAt { head: HeadExpectation, effects: List, preparations: List } + | PlacementUnread { step: String, reason: String } + +// THE JOINED PLACEMENT READ: the placement fold, then every finalized preparation's authority +// event looked up on its group's chain -- it must exist and carry the record's copy. A +// finalization naming an event that is not there, or one carrying another copy, is unread. +fn placement_read(store: FabricStorageBinding) -> PlacementRead { + placement_read_over(placement: placement_fold_read(store: store), groups: map([FabricGroupA, FabricGroupB], g => GroupSnapshot { group: g, partition: read_group_partition(store: store, group: g) })) +} + +// ONE READ PER PARTITION, THEN PURE JOINS. A consumer that needs the authority of every group AND +// the placement (the host standings) reads each partition once and joins over the snapshot, so +// the whole reading is one set of generations rather than independent snapshots of a live store +// that could disagree with each other. +type GroupSnapshot { + group: FabricGroup + partition: GroupPartitionRead +} + +fn placement_read_over(placement: PlacementRead, groups: List) -> PlacementRead { + match placement { + PlacementUnread { step: st, reason: why } => PlacementUnread { step: st, reason: why } + PlacementReadAt { head: head, effects: effs, preparations: preps } => + match first(flat_map(groups, gs => finalization_gaps(group: gs.group, partition: gs.partition, preps: preps))) { + Present { value: gap } => PlacementUnread { step: "join", reason: gap } + Absent => PlacementReadAt { head: head, effects: effs, preparations: preps } + } + } +} + +// A finalized preparation's authority event must be one the group's ACCEPTED fold consumed -- +// the fold's word, never a raw event's presence: an event the fold refuses (a cancelled or +// twice-consumed preparation, a wrong previous state, a bad grant) discharges nothing. +fn finalization_gaps(group: FabricGroup, partition: GroupPartitionRead, preps: List) -> List { + let finalized = filter(preps, r => same_group(a: r.group, b: group) && (match r.state { Finalized { authority_event: _, finalized_by: _, intent: _ } => true _ => false })) + if length(finalized) == 0 { [] as List } else { + match partition { + GroupPartitionUnread { step: st, reason: why } => [join([fabric_group_wire(g: group) as String, "'s chain could not be read at ", st, ": ", why], "")] + GroupPartitionAt { head: _, fold: f } => + match f { + AuthorityFoldRefused { at_event: e, reason: why } => [join([fabric_group_wire(g: group) as String, ": ", fold_refusal_text(at_event: e, reason: why)], "")] + AuthorityUnestablished { cancelled: _ } => map(finalized, r => join(["preparation ", r.id as String, " is finalized but ", fabric_group_wire(g: group) as String, "'s accepted history consumed nothing"], "")) + AuthorityFolded { preparations_consumed: pcs } => + flat_map(finalized, r => + match r.state { + Finalized { authority_event: ae, finalized_by: _, intent: _ } => + match first(filter(pcs, c => (c.authority_event as String) == (ae as String))) { + Absent => [join(["preparation ", r.id as String, " is finalized to authority event ", ae as String, ", which ", fabric_group_wire(g: group) as String, "'s accepted history did not admit"], "")] + Present { value: c } => match consumed_join(consumed: c, r: r) { Absent => [] as List Present { value: gap } => [gap] } + } + _ => [] as List + }) + } + } + } +} +// The placement fold alone (no cross-chain join): what the joined reads start from. +fn placement_fold_read(store: FabricStorageBinding) -> PlacementRead { + match event_log_read_partition_with(store: store, partition: host_placement_partition, budget: event_log_read_budget(), decode: fn(text) { pair_authority_event_decode(text: text) }) { + PartitionReadRefused { cause: c } => PlacementUnread { step: "store", reason: event_log_refusal_wire(cause: c) } + PartitionReadOk { head: head, walk: walk } => + match walk { + ChainIncomplete { missing: m, newest_first_so_far: _ } => PlacementUnread { step: "chain", reason: join(["placement chain is missing event ", m as String], "") } + ChainBudgetExhausted { at: a } => PlacementUnread { step: "chain", reason: join(["placement chain exceeded the read budget at ", a as String], "") } + ChainWalked { oldest_first: xs } => + match placement_fold(oldest_first: xs) { + PlacementFoldRefused { at_event: e, reason: why } => PlacementUnread { step: "fold", reason: join(["event ", e as String, " refused: ", why], "") } + PlacementFolded { effects: effs, preparations: preps } => PlacementReadAt { head: head, effects: effs, preparations: preps } + } + } + } +} + +fn placement_append(store: FabricStorageBinding, head: HeadExpectation, payload: PairAuthorityEvent, actor: NonEmptyStr, at: EpochSecs) -> EventAppend { + let event = ChainEvent { + partition: host_placement_partition, + parent: match head { HeadAbsent => none HeadAt { id: h } => Present { value: h } }, + recorded_at: at, + actor: actor, + payload: payload, + } + event_log_append_with(store: store, partition: host_placement_partition, event: event, expected: head, encode: fn(e) { pair_authority_event_wire_text(event: e) }) +} + +// ── THE SAGA: PREPARE → CLAIM → AUTHORITY → FINALIZE ──────────────────────────────────────────── +// +// The preparation lands on the placement partition against the head that was read (refused, +// stale, or prepared); the writer then CLAIMS THE APPEND on the same partition (so an abort and +// an authority write contend on one head: an abort that read the preparation Prepared is stale +// once the claim landed, and a claim is stale once an abort landed); the authority event carries +// the preparation's typed copy; the finalization lands last with the same copy. What the saga +// returns beside the authority outcome is the PLACEMENT STANDING it left: finalized, aborted, or +// STILL FENCING with the preparation id and the cause -- a cleanup that did not land is never +// dropped from the result (the union stays fenced until host_placement_finalize_wet or +// host_placement_abort_wet). +type PlacementPreparation + = PlacementPreparedAt { id: EventId } + | PlacementPreparationStale + | PlacementPreparationRefused { step: String, reason: String } + +// A cleanup that could not even name its preparation -- the placement partition or the join was +// unreadable, or the named preparation is not on the chain -- is PlacementCleanupUnread: it carries +// no EventId, because there is none to carry, and nothing downstream may treat it as a preparation +// a rerun finalizes (review 69399: a minted "unknown" id flowed into D0PlacementStillFencing). +type PlacementCleanup + = PlacementFinalizedAt { preparation: EventId, id: EventId } + | PlacementAbortedAt { preparation: EventId, id: EventId } + | PlacementCleanupStillFencing { preparation: EventId, cause: String } + | PlacementCleanupUnread { cause: String } + +fn committed_population(a: PairServingGroupAuthority) -> List { + if pair_serving_commits_hosts(a: a) { pair_serving_committed_hosts(a: a) } else { [] as List } +} + +fn placement_prepare_at(store: FabricStorageBinding, read: PlacementRead, group: FabricGroup, previous_hosts: List, next_hosts: List, operation: NonEmptyStr, actor: NonEmptyStr, at: EpochSecs) -> PlacementPreparation { + match read { + PlacementUnread { step: st, reason: why } => PlacementPreparationRefused { step: join(["placement/", st], ""), reason: why } + PlacementReadAt { head: head, effects: effs, preparations: preps } => + match preparation_refusal(effects: effs, preps: preps, group: group, previous_hosts: previous_hosts, next_hosts: next_hosts, at: at) { + Present { value: why } => PlacementPreparationRefused { step: "host-effects", reason: why } + Absent => + match placement_append(store: store, head: head, payload: PlacementPrepared { operation: operation, group: group, previous_hosts: previous_hosts, next_hosts: next_hosts }, actor: actor, at: at) { + EventAppended { id: h } => PlacementPreparedAt { id: h } + EventAppendStale { expected: _, observed: _ } => PlacementPreparationStale + EventAppendRefused { cause: c } => PlacementPreparationRefused { step: "store", reason: event_log_refusal_wire(cause: c) } + } + } + } +} + +fn placement_prepare(store: FabricStorageBinding, group: FabricGroup, previous_hosts: List, next_hosts: List, operation: NonEmptyStr, actor: NonEmptyStr, at: EpochSecs) -> PlacementPreparation { + placement_prepare_at(store: store, read: placement_read(store: store), group: group, previous_hosts: previous_hosts, next_hosts: next_hosts, operation: operation, actor: actor, at: at) +} + +// THE APPEND CLAIM, against the current placement head: the writer names itself on the chain +// before touching the group partition. Stale when an abort (or anything) landed in between. +type PlacementAppendClaim + = PlacementAppendClaimedAt { id: EventId } + | PlacementAppendClaimStale + | PlacementAppendClaimRefused { step: String, reason: String } + +// The population an intent leaves: none before the establishment. +fn intent_previous_population(i: AuthorityWriteIntent) -> List { + match i.previous { Absent => [] as List Present { value: p } => committed_population(a: p) } +} + +// THE CLAIM IS ITS INTENT: the claimant is the intent's actor (the event is written by it) and the +// head it will CAS against is the intent's; the fold keeps them on the record as the intent says. +fn placement_claim_append(store: FabricStorageBinding, preparation: EventId, intent: AuthorityWriteIntent, at: EpochSecs) -> PlacementAppendClaim { + match placement_read(store: store) { + PlacementUnread { step: st, reason: why } => PlacementAppendClaimRefused { step: join(["placement/", st], ""), reason: why } + PlacementReadAt { head: head, effects: _, preparations: preps } => + match pending_preparation_of(preps: preps, preparation: preparation) { + Present { value: PlacementPreparationRecord { state: Prepared } } => + match placement_append(store: store, head: head, payload: PlacementAppendClaimed { preparation: preparation, intent: intent }, actor: intent.actor, at: at) { + EventAppended { id: h } => PlacementAppendClaimedAt { id: h } + EventAppendStale { expected: _, observed: _ } => PlacementAppendClaimStale + EventAppendRefused { cause: c } => PlacementAppendClaimRefused { step: "store", reason: event_log_refusal_wire(cause: c) } + } + _ => PlacementAppendClaimRefused { step: "preparation", reason: join(["preparation ", preparation as String, " is not Prepared (aborted, already claimed, or unknown)"], "") } + } + } +} + +// WHETHER AN AUTHORITY EVENT CONSUMED A PREPARATION, AND THE EXACT JOIN. The preparation's record +// is read from the placement chain (its group is the record's, never a caller's word); consumption +// is read from the group's ACCEPTED fold -- its `preparations_consumed` records -- never from the +// raw chain, so an event the fold refuses (cancelled or duplicate preparation, a stale previous +// state, a bad grant) consumes nothing; the consuming event's copy must be the record and its +// intent the claim's. Unread (or a refused fold) is unread -- neither a finalization nor an abort +// proceeds on it -- and a preparation no accepted event consumed is NotConsumed. +type PreparationConsumption + = PreparationConsumedBy { preparation: PlacementPreparationRef, authority_event: EventId, state: PreparationState } + | PreparationNotConsumed { record: PlacementPreparationRecord, group_head: HeadExpectation, cancelled: Bool } + | PreparationAbsent + | PreparationConsumptionUnread { step: String, reason: String } + +fn preparation_consumption(store: FabricStorageBinding, preparation: EventId) -> PreparationConsumption { + match placement_fold_read(store: store) { + PlacementUnread { step: st, reason: why } => PreparationConsumptionUnread { step: join(["placement/", st], ""), reason: why } + PlacementReadAt { head: _, effects: _, preparations: preps } => + match first(filter(preps, r => (r.id as String) == (preparation as String))) { + Absent => PreparationAbsent + Present { value: r } => { + let ref = preparation_ref_of(r: r) + match read_group_partition(store: store, group: r.group) { + GroupPartitionUnread { step: st, reason: why } => PreparationConsumptionUnread { step: st, reason: why } + GroupPartitionAt { head: ghead, fold: f } => + match f { + AuthorityFoldRefused { at_event: e, reason: why } => PreparationConsumptionUnread { step: "fold", reason: join([fabric_group_wire(g: r.group) as String, ": ", fold_refusal_text(at_event: e, reason: why)], "") } + AuthorityUnestablished { cancelled: xs } => PreparationNotConsumed { record: r, group_head: ghead, cancelled: any(xs, c => (c.preparation as String) == (preparation as String)) } + AuthorityFolded { preparations_consumed: pcs, cancelled: xs } => + match first(filter(pcs, c => (c.preparation.id as String) == (preparation as String))) { + Absent => PreparationNotConsumed { record: r, group_head: ghead, cancelled: any(xs, c => (c.preparation as String) == (preparation as String)) } + Present { value: c } => + if !preparation_ref_eq(a: c.preparation, b: ref) { PreparationConsumptionUnread { step: "join", reason: join(["authority event ", c.authority_event as String, " carries a copy of preparation ", preparation as String, " that is not the record"], "") } } + else { + match r.state { + AppendClaimed { claimant: _, expected_head: _, intent: i } => if write_intent_eq(a: i, b: c.intent) { PreparationConsumedBy { preparation: ref, authority_event: c.authority_event, state: r.state } } else { PreparationConsumptionUnread { step: "join", reason: join(["authority event ", c.authority_event as String, " is not the write preparation ", preparation as String, "'s append claim named"], "") } } + Finalized { authority_event: _, finalized_by: _, intent: i } => if write_intent_eq(a: i, b: c.intent) { PreparationConsumedBy { preparation: ref, authority_event: c.authority_event, state: r.state } } else { PreparationConsumptionUnread { step: "join", reason: join(["authority event ", c.authority_event as String, " is not the write preparation ", preparation as String, "'s append claim named"], "") } } + _ => PreparationConsumptionUnread { step: "join", reason: join(["authority event ", c.authority_event as String, " consumed preparation ", preparation as String, " whose append was never claimed"], "") } + } + } + } + } + } + } + } + } +} + +// THE FINALIZATION, from the exact join, against the current placement head; a head that moved is +// read again (bounded: the placement partition sees one write per effect or authority operation, +// so the retries are a small constant, never a spin), and a finalization that still does not land +// is STILL FENCING. +fn placement_finalize(store: FabricStorageBinding, consumed: PreparationConsumption, actor: NonEmptyStr, at: EpochSecs) -> PlacementCleanup { + match consumed { + PreparationConsumedBy { preparation: ref, authority_event: ae, state: Finalized { authority_event: fae, finalized_by: fb, intent: _ } } => + if (fae as String) == (ae as String) { PlacementFinalizedAt { preparation: ref.id, id: fb } } else { PlacementCleanupStillFencing { preparation: ref.id, cause: join(["the preparation is finalized to authority event ", fae as String, ", not to its consumer ", ae as String], "") } } + PreparationConsumedBy { preparation: ref, authority_event: ae, state: _ } => placement_finalize_attempt(store: store, preparation: ref, authority_event: ae, actor: actor, at: at, attempts_left: 3) + PreparationNotConsumed { record: rec, group_head: _, cancelled: _ } => PlacementCleanupStillFencing { preparation: rec.id, cause: "no authority event consumed this preparation; it is aborted, not finalized" } + PreparationAbsent => PlacementCleanupUnread { cause: "the preparation is not on the placement chain" } + PreparationConsumptionUnread { step: st, reason: why } => PlacementCleanupUnread { cause: join(["the join could not be read at ", st, ": ", why], "") } + } +} + +fn placement_finalize_attempt(store: FabricStorageBinding, preparation: PlacementPreparationRef, authority_event: EventId, actor: NonEmptyStr, at: EpochSecs, attempts_left: Int) -> PlacementCleanup { + match placement_read(store: store) { + PlacementUnread { step: st, reason: why } => PlacementCleanupStillFencing { preparation: preparation.id, cause: join(["the finalization could not read the placement partition at ", st, ": ", why], "") } + PlacementReadAt { head: head, effects: _, preparations: preps } => + match pending_preparation_of(preps: preps, preparation: preparation.id) { + Present { value: PlacementPreparationRecord { state: AppendClaimed { claimant: _, expected_head: _, intent: _ } } } => + match placement_append(store: store, head: head, payload: PlacementFinalized { preparation: preparation, authority_event: authority_event }, actor: actor, at: at) { + EventAppended { id: h } => PlacementFinalizedAt { preparation: preparation.id, id: h } + EventAppendStale { expected: _, observed: _ } => + if attempts_left > 1 { placement_finalize_attempt(store: store, preparation: preparation, authority_event: authority_event, actor: actor, at: at, attempts_left: attempts_left - 1) } + else { PlacementCleanupStillFencing { preparation: preparation.id, cause: "the placement partition kept moving while the finalization was being appended" } } + EventAppendRefused { cause: c } => PlacementCleanupStillFencing { preparation: preparation.id, cause: join(["the finalization was refused at ", "store", ": ", event_log_refusal_wire(cause: c)], "") } + } + _ => PlacementCleanupStillFencing { preparation: preparation.id, cause: "the finalization names no append-claimed pending preparation" } + } + } +} + +// THE ABORT, against a SUPPLIED placement read (so the contention with an append claim can be +// driven) and only when the join says no authority event consumed the preparation. The provenance +// is the writer's own (ClaimantTerminal, when its group append did not land) or an operator's +// recovery naming the preparation; the fold refuses one that does not cover the record's state. +fn placement_abort_at(store: FabricStorageBinding, read: PlacementRead, preparation: EventId, reason: String, provenance: ReleaseProvenance, actor: NonEmptyStr, at: EpochSecs) -> PlacementCleanup { + placement_abort_attempt(store: store, read: read, preparation: preparation, reason: reason, provenance: provenance, actor: actor, at: at, attempts_left: 2) +} + +fn placement_abort_attempt(store: FabricStorageBinding, read: PlacementRead, preparation: EventId, reason: String, provenance: ReleaseProvenance, actor: NonEmptyStr, at: EpochSecs, attempts_left: Int) -> PlacementCleanup { + match preparation_consumption(store: store, preparation: preparation) { + PreparationConsumptionUnread { step: st, reason: why } => PlacementCleanupStillFencing { preparation: preparation, cause: join(["the abort could not read the join at ", st, ": ", why], "") } + PreparationAbsent => PlacementCleanupStillFencing { preparation: preparation, cause: "the preparation is not on the placement chain" } + PreparationConsumedBy { preparation: _, authority_event: ae, state: _ } => PlacementCleanupStillFencing { preparation: preparation, cause: join(["the preparation was consumed by authority event ", ae as String, "; it is finalized, not aborted"], "") } + PreparationNotConsumed { record: rec, group_head: ghead, cancelled: cancelled } => + if !abort_provenance_covers(r: rec, provenance: provenance, actor: actor) { + PlacementCleanupStillFencing { preparation: preparation, cause: "the abort's provenance does not cover the preparation's state: a prepared preparation is aborted by its preparer, an append-claimed one by its claimant, or either by a recovery authorization naming it; nothing was written" } + } else { + match rec.state { + AppendClaimed { claimant: _, expected_head: eh, intent: _ } => + if !cancelled && head_expectation_eq(a: eh, b: ghead) { + if attempts_left <= 0 { + PlacementCleanupStillFencing { preparation: preparation, cause: "the group head kept moving while the claimed append was being cancelled; re-decide from a fresh read" } + } else { + match cancel_claimed_append(store: store, group: rec.group, head: ghead, preparation: preparation, provenance: provenance, actor: actor, at: at) { + CancellationRefused { cause: c } => PlacementCleanupStillFencing { preparation: preparation, cause: c } + _ => placement_abort_attempt(store: store, read: read, preparation: preparation, reason: reason, provenance: provenance, actor: actor, at: at, attempts_left: attempts_left - 1) + } + } + } else { + placement_abort_append(store: store, read: read, preparation: preparation, reason: reason, provenance: provenance, actor: actor, at: at) + } + _ => placement_abort_append(store: store, read: read, preparation: preparation, reason: reason, provenance: provenance, actor: actor, at: at) + } + } + } +} + +// THE CLAIMED APPEND COULD STILL LAND while the group head is the one the claim named: its +// compare-and-set is against that very head. So the abort of an append-claimed preparation first +// moves that head with a cancellation naming the preparation, then aborts; a cancellation that is +// stale means the head moved -- by the claimed append (consumed: finalize, never abort) or by +// something else (the claimed append can never land: the abort is safe) -- decided by a fresh +// join. The abort's own append, against the supplied placement read, follows. +fn placement_abort_append(store: FabricStorageBinding, read: PlacementRead, preparation: EventId, reason: String, provenance: ReleaseProvenance, actor: NonEmptyStr, at: EpochSecs) -> PlacementCleanup { + match read { + PlacementUnread { step: st, reason: why } => PlacementCleanupStillFencing { preparation: preparation, cause: join(["the abort could not read the placement partition at ", st, ": ", why], "") } + PlacementReadAt { head: head, effects: _, preparations: preps } => + match pending_preparation_of(preps: preps, preparation: preparation) { + Absent => PlacementCleanupStillFencing { preparation: preparation, cause: "the abort names no pending preparation" } + Present { value: r } => + if !abort_provenance_covers(r: r, provenance: provenance, actor: actor) { + PlacementCleanupStillFencing { preparation: preparation, cause: join(["the abort's provenance does not cover the preparation's state (", preparation_line(r: r), "): an append-claimed preparation is aborted by its claimant or by a recovery authorization naming it"], "") } + } else { + match placement_append(store: store, head: head, payload: PlacementAborted { preparation: preparation, reason: reason, provenance: provenance }, actor: actor, at: at) { + EventAppended { id: h } => PlacementAbortedAt { preparation: preparation, id: h } + EventAppendStale { expected: _, observed: _ } => PlacementCleanupStillFencing { preparation: preparation, cause: "the placement partition moved while the abort was being appended (an append claim may have landed); re-decide from a fresh read" } + EventAppendRefused { cause: c } => PlacementCleanupStillFencing { preparation: preparation, cause: join(["the abort was refused at ", "store", ": ", event_log_refusal_wire(cause: c)], "") } + } + } + } + } +} + +// A CANCELLATION ON THE GROUP PARTITION at the head the claimed append expects: once it lands, +// that append's compare-and-set can never succeed. Stale is not a failure -- the head moved, and +// the caller re-joins to learn by what. +type Cancellation + = CancellationLanded { id: EventId } + | CancellationStale + | CancellationRefused { cause: String } + +fn cancel_claimed_append(store: FabricStorageBinding, group: FabricGroup, head: HeadExpectation, preparation: EventId, provenance: ReleaseProvenance, actor: NonEmptyStr, at: EpochSecs) -> Cancellation { + let event = ChainEvent { + partition: pair_authority_partition(group: group), + parent: match head { HeadAbsent => none HeadAt { id: h } => Present { value: h } }, + recorded_at: at, + actor: actor, + payload: AuthorityAppendCancelled { preparation: preparation, provenance: provenance }, + } + match event_log_append_with(store: store, partition: pair_authority_partition(group: group), event: event, expected: head, encode: fn(e) { pair_authority_event_wire_text(event: e) }) { + EventAppended { id: h } => CancellationLanded { id: h } + EventAppendStale { expected: _, observed: _ } => CancellationStale + EventAppendRefused { cause: c } => CancellationRefused { cause: join(["the cancellation of the claimed append was refused at ", "store", ": ", event_log_refusal_wire(cause: c)], "") } + } +} + +fn placement_abort(store: FabricStorageBinding, preparation: EventId, reason: String, provenance: ReleaseProvenance, actor: NonEmptyStr, at: EpochSecs) -> PlacementCleanup { + placement_abort_at(store: store, read: placement_read(store: store), preparation: preparation, reason: reason, provenance: provenance, actor: actor, at: at) +} + +// A GROUP'S PENDING PREPARATION, REPAIRED WHERE THE JOIN ALLOWS: none pending is nothing to do; +// a pending one an authority event consumed is finalized (a saga that died after its group +// append); a pending one no authority event consumed is left for the operator's abort -- this +// caller cannot know the claimant's standing -- and reported as still fencing. A transaction +// that resumes on a group calls this before its own saga, because one saga is pending at a time. +fn placement_repair_pending(store: FabricStorageBinding, group: FabricGroup, actor: NonEmptyStr, at: EpochSecs) -> PlacementCleanup? { + match placement_fold_read(store: store) { + PlacementUnread { step: st, reason: why } => Present { value: PlacementCleanupUnread { cause: join(["the placement partition could not be read at ", st, ": ", why], "") } } + PlacementReadAt { head: _, effects: _, preparations: preps } => + match first(filter(preps, r => same_group(a: r.group, b: group) && preparation_is_pending(r: r))) { + Absent => none + Present { value: r } => { + let consumed = preparation_consumption(store: store, preparation: r.id) + match consumed { + PreparationConsumedBy { preparation: _, authority_event: _, state: _ } => Present { value: placement_finalize(store: store, consumed: consumed, actor: actor, at: at) } + PreparationNotConsumed { record: _, group_head: _, cancelled: _ } => Present { value: PlacementCleanupStillFencing { preparation: r.id, cause: join(["preparation ", r.id as String, " (", preparation_line(r: r), ") is pending and no authority event consumed it; an operator aborts it with host_placement_abort_wet"], "") } } + PreparationAbsent => Present { value: PlacementCleanupStillFencing { preparation: r.id, cause: "the pending preparation vanished between two reads" } } + PreparationConsumptionUnread { step: st, reason: why } => Present { value: PlacementCleanupStillFencing { preparation: r.id, cause: join(["the join could not be read at ", st, ": ", why], "") } } + } + } + } + } +} + +fn placement_cleanup_line(c: PlacementCleanup) -> String { + match c { + PlacementFinalizedAt { preparation: p, id: _ } => join(["placement finalized (preparation ", p as String, ")"], "") + PlacementAbortedAt { preparation: p, id: _ } => join(["placement aborted (preparation ", p as String, ")"], "") + PlacementCleanupStillFencing { preparation: p, cause: c } => join(["placement STILL FENCING under preparation ", p as String, ": ", c], "") + PlacementCleanupUnread { cause: c } => join(["placement cleanup UNREAD (no preparation could be named): ", c], "") + } +} + +// THE OPERATOR'S ENTRIES, for a saga that died between its steps. The group is the preparation's +// own, read from the record. Finalize when the authority event landed (the join finds it); abort +// when it did not, under a typed disposition of the writer that claimed the append: +// gunbc run --source-root dag --source-root src/v2 --entry dag/gunbc/spark/pair_serving_authority_log.dag +// --function host_placement_finalize_wet --arg preparation= +// --function host_placement_abort_wet --arg preparation= --arg reason= +// --arg claimant=terminated|abandoned --arg receipt= +fn host_placement_finalize_wet(preparation: String) -> ProcessExit + uses net: Network +{ + if preparation == "" { exit_failure(reason: "a preparation event id is required") } else { + host_placement_cleanup_wet(preparation: preparation, cleanup: fn(store, executor, at) { placement_finalize(store: store, consumed: preparation_consumption(store: store, preparation: (preparation as NonEmptyStr) as EventId), actor: executor, at: at) }) + } +} + +fn host_placement_abort_wet(preparation: String, reason: String, claimant: String, receipt: String) -> ProcessExit + uses net: Network +{ + if preparation == "" || reason == "" || receipt == "" { + exit_failure(reason: "a preparation event id, a reason and a receipt establishing the writer's standing are required") + } else { + match parse_claimant_standing(wire: claimant) { + Absent => exit_failure(reason: join(["`", claimant, "` is not a claimant standing; say terminated or abandoned"], "")) + Present { value: standing } => + host_placement_cleanup_wet(preparation: preparation, cleanup: fn(store, executor, at) { placement_abort(store: store, preparation: (preparation as NonEmptyStr) as EventId, reason: reason, provenance: RecoveryAuthorized { claim: (preparation as NonEmptyStr) as EventId, claimant: standing, authorized_by: executor, receipt: receipt as NonEmptyStr }, actor: executor, at: at) }) + } + } +} + +fn host_placement_cleanup_wet(preparation: String, cleanup: fn(FabricStorageBinding, NonEmptyStr, EpochSecs) -> PlacementCleanup) -> ProcessExit + uses net: Network +{ + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["the executor could not be identified: ", c], "")) + ExecutorReachKnown { short_hostname: executor, path: _ } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => exit_failure(reason: d) + HostStoreResolved { store: store, executor: _ } => + match now_epoch_seconds() { + Absent => exit_failure(reason: "the clock could not be read as epoch seconds") + Present { value: at } => + match cleanup(store, executor as NonEmptyStr, at) { + PlacementCleanupStillFencing { preparation: _, cause: why } => exit_failure(reason: why) + _ => ExitSuccess + } + } + } + } +} + +// ── ESTABLISHING A GROUP'S AUTHORITY ─────────────────────────────────────────────────────────── +// +// ONE COMPARE-AND-SET AT THE HEAD the establishment read. Refused when an authority is already +// established, when the authority names another group, when it would commit a host under a live +// claim -- on this partition or on any other home the host could have claimed on (its own +// partition, the other group's), read for every host it commits -- when it commits no host, or +// when it holds a lease; stale when the partition moved. +type AuthorityEstablishment + = AuthorityEstablishedAt { id: EventId, placement: PlacementCleanup } + | AuthorityEstablishmentRefused { group: FabricGroup, step: String, reason: String, placement: PlacementCleanup? } + +fn pair_serving_authority_establish(store: FabricStorageBinding, group: FabricGroup, authority: PairServingGroupAuthority, actor: NonEmptyStr, at: EpochSecs) -> AuthorityEstablishment { + if (fabric_group_wire(g: pair_serving_authority_group(a: authority)) as String) != (fabric_group_wire(g: group) as String) { + AuthorityEstablishmentRefused { group: group, step: "subject", reason: "the authority names a different group than the partition", placement: none } + } else { + match read_group_partition(store: store, group: group) { + GroupPartitionUnread { step: s, reason: why } => AuthorityEstablishmentRefused { group: group, step: s, reason: why, placement: none } + GroupPartitionAt { head: head, fold: f } => + match f { + AuthorityFoldRefused { at_event: e, reason: why } => AuthorityEstablishmentRefused { group: group, step: "fold", reason: fold_refusal_text(at_event: e, reason: why), placement: none } + AuthorityFolded { current: cur } => AuthorityEstablishmentRefused { group: group, step: "established", reason: join(["an authority is already established: ", authority_wire(a: cur)], ""), placement: none } + AuthorityUnestablished { cancelled: _ } => + match establishment_refusal(a: authority) { + Present { value: why } => AuthorityEstablishmentRefused { group: group, step: "subject", reason: why, placement: none } + Absent => { + let operation = establishment_operation(group: group) + let intent = authority_write_intent(group: group, head: head, actor: actor, previous: none, next: authority, operation: operation, lifecycle: none, grant: none, entry_state: none) + match saga_prepare_and_claim(store: store, group: group, group_head: head, previous_hosts: [] as List, next_hosts: committed_population(a: authority), operation: operation, intent: intent, actor: actor, at: at) { + SagaRefused { step: st, reason: why, placement: pl } => AuthorityEstablishmentRefused { group: group, step: st, reason: why, placement: pl } + SagaClaimed { preparation: ref, intent: _ } => { + let event = ChainEvent { + partition: pair_authority_partition(group: group), + parent: match head { HeadAbsent => none HeadAt { id: h } => Present { value: h } }, + recorded_at: at, + actor: actor, + payload: AuthorityEstablished { authority: authority, placement: ref }, + } + match event_log_append_with(store: store, partition: pair_authority_partition(group: group), event: event, expected: head, encode: fn(e) { pair_authority_event_wire_text(event: e) }) { + EventAppended { id: h } => AuthorityEstablishedAt { id: h, placement: placement_finalize_attempt(store: store, preparation: ref, authority_event: h, actor: actor, at: at, attempts_left: 3) } + EventAppendStale { expected: _, observed: _ } => AuthorityEstablishmentRefused { group: group, step: "stale", reason: "the partition moved between the read and the establishment; nothing was established", placement: Present { value: saga_abort_own(store: store, preparation: ref.id, reason: "the establishment did not land", actor: actor, at: at) } } + EventAppendRefused { cause: c } => AuthorityEstablishmentRefused { group: group, step: "store", reason: event_log_refusal_wire(cause: c), placement: Present { value: saga_abort_own(store: store, preparation: ref.id, reason: event_log_refusal_wire(cause: c), actor: actor, at: at) } } + } + } + } + } + } + } + } + } +} + +// THE FIRST TWO STEPS OF THE SAGA: prepare, then claim the append. A claim that does not land +// (stale: an abort or another write moved the partition) is followed by the writer's own abort +// of its preparation, and the refusal carries that cleanup. +type SagaEntry + = SagaClaimed { preparation: PlacementPreparationRef, intent: AuthorityWriteIntent } + | SagaRefused { step: String, reason: String, placement: PlacementCleanup? } + +fn saga_prepare_and_claim(store: FabricStorageBinding, group: FabricGroup, group_head: HeadExpectation, previous_hosts: List, next_hosts: List, operation: NonEmptyStr, intent: AuthorityWriteIntent, actor: NonEmptyStr, at: EpochSecs) -> SagaEntry { + match placement_prepare(store: store, group: group, previous_hosts: previous_hosts, next_hosts: next_hosts, operation: operation, actor: actor, at: at) { + PlacementPreparationRefused { step: st, reason: why } => SagaRefused { step: st, reason: why, placement: none } + PlacementPreparationStale => SagaRefused { step: "placement-stale", reason: "the placement partition moved between its read and the preparation; nothing was written -- re-decide from a fresh read", placement: none } + PlacementPreparedAt { id: prep } => + match placement_claim_append(store: store, preparation: prep, intent: intent, at: at) { + PlacementAppendClaimedAt { id: _ } => SagaClaimed { preparation: preparation_ref_expected(id: prep, group: group, previous: previous_hosts, next: next_hosts, operation: operation), intent: intent } + PlacementAppendClaimStale => SagaRefused { step: "placement-stale", reason: "the placement partition moved between the preparation and the append claim; nothing was written to the group", placement: Present { value: saga_abort_own(store: store, preparation: prep, reason: "the append claim did not land", actor: actor, at: at) } } + PlacementAppendClaimRefused { step: st, reason: why } => SagaRefused { step: st, reason: why, placement: Present { value: saga_abort_own(store: store, preparation: prep, reason: why, actor: actor, at: at) } } + } + } +} + +// The writer's own abort: claimant-terminal provenance naming its preparation. +fn saga_abort_own(store: FabricStorageBinding, preparation: EventId, reason: String, actor: NonEmptyStr, at: EpochSecs) -> PlacementCleanup { + placement_abort(store: store, preparation: preparation, reason: reason, provenance: ClaimantTerminal { claim: preparation, claimant_executor: actor }, actor: actor, at: at) +} + +// THE SOURCE ROW, ESTABLISHED ON THE LOG: the production route from a desired authority row to a +// group authority, run once per group by an operator on an executor with the event log placed: +// gunbc run --source-root dag --source-root src/v2 --entry dag/gunbc/spark/pair_serving_authority_log.dag +// --function pair_serving_authority_establish_wet --arg group= +// A group with no source row refuses; an already established group refuses (the state moves by +// transition, never by re-establishment). +fn pair_serving_authority_establish_wet(group: String) -> ProcessExit + uses net: Network +{ + match parse_fabric_group(wire: group) { + Absent => exit_failure(reason: join(["`", group, "` is not a fabric group"], "")) + Present { value: g } => + match pair_serving_authority_for(group: g) { + Absent => exit_failure(reason: join(["no source row declares an authority for ", group], "")) + Present { value: desired } => + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["the executor could not be identified: ", c], "")) + ExecutorReachKnown { short_hostname: executor, path: _ } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => exit_failure(reason: d) + HostStoreResolved { store: store, executor: _ } => + match now_epoch_seconds() { + Absent => exit_failure(reason: "the clock could not be read as epoch seconds") + Present { value: at } => + match pair_serving_authority_establish(store: store, group: g, authority: desired, actor: executor as NonEmptyStr, at: at) { + AuthorityEstablishedAt { id: _, placement: PlacementCleanupStillFencing { preparation: p, cause: c } } => exit_failure(reason: join(["the authority was established but its placement is still fencing under preparation ", p as String, ": ", c, "; run host_placement_finalize_wet"], "")) + AuthorityEstablishedAt { id: _, placement: _ } => ExitSuccess + AuthorityEstablishmentRefused { group: _, step: st, reason: why, placement: pl } => exit_failure(reason: join(["establishment refused at ", st, ": ", why, match pl { Absent => "" Present { value: c } => join(["; ", placement_cleanup_line(c: c)], "") }], "")) + } + } + } + } + } + } +} + +// THE ROSTER, AS AN EXECUTOR SEES IT. Every effectful consumer of the authority -- the apply seam, +// the capacity instrument, and the Cut D actuators to come -- asks this and never the source row: +// one read per claimed group, on the event log the executing host resolves. A host with no event +// log placement reads NOTHING, as one Unread per claimed group, so a consumer that folds these +// cannot mistake "could not read" for an empty roster. +fn current_pair_serving_authorities(short_hostname: String, at: EpochSecs) -> List { + match event_log_store_for_host(short_hostname: short_hostname) { + HostStoreRefused { detail: d } => + map(spark_pair_serving_authorities, a => CurrentAuthorityUnread { group: pair_serving_authority_group(a: a), step: "store", reason: d }) + HostStoreResolved { store: store, executor: _ } => + map(spark_pair_serving_authorities, a => current_pair_serving_authority(store: store, group: pair_serving_authority_group(a: a), at: at)) + } +} + +fn current_authority_for(current: List, group: FabricGroup) -> CurrentAuthority? { + first(filter(current, c => (fabric_group_wire(g: current_authority_group(c: c)) as String) == (fabric_group_wire(g: group) as String))) +} + +fn current_authority_group(c: CurrentAuthority) -> FabricGroup { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => pair_serving_authority_group(a: a) + CurrentAuthorityUnread { group: g, step: _, reason: _ } => g + } +} + +// ── THE TRANSITION ───────────────────────────────────────────────────────────────────────────── +// +// TWO COMPARE-AND-SETS, IN ORDER, NO RETRY: the hosts commitment on the placement partition first +// (the linearization point with every host effect), then the authority on the group partition; a +// group append that does not land voids the commitment it followed. A seat re-folds and tries +// again on a stale head because a seat is fungible: whichever writer wins, the loser wants the same thing from the new state. An +// authority transition is not: it was decided against ONE observed state, and if that state moved +// the decision has to be re-made by the transaction, which may now refuse. So a stale append is +// returned to the caller with the state that is now current, and nothing here re-decides. + +type AuthorityTransition + = AuthorityTransitionAppended { id: EventId, generation: Nat, next: PairServingGroupAuthority, placement: PlacementCleanup } + | AuthorityTransitionStale { expected: PairServingGroupAuthority, current: PairServingGroupAuthority, placement: PlacementCleanup? } + | AuthorityTransitionRefused { group: FabricGroup, step: String, reason: String, placement: PlacementCleanup? } + +fn pair_serving_authority_transition( + store: FabricStorageBinding, + group: FabricGroup, + expected: PairServingGroupAuthority, + next: PairServingGroupAuthority, + transaction: NonEmptyStr, + lifecycle: ContentHash?, + grant: LeaseGrant?, + entry_state: ContentHash?, + actor: NonEmptyStr, + at: EpochSecs, +) -> AuthorityTransition { + pair_serving_authority_transition_at(store: store, group: group, expected: expected, expected_head: none, next: next, transaction: transaction, lifecycle: lifecycle, grant: grant, entry_state: entry_state, actor: actor, at: at) +} + +// THE TRANSITION AGAINST AN EXACT HEAD. A caller that derived its next state from a reading -- +// D0 mints the pending state's lease epoch from the generation it read -- names the head that +// reading was taken at, and the transition is stale when the head is not that one even if the +// state VALUE is equal: a state that left and came back (Active → Released → Active by an +// operator) is the same value at another generation, and a lease epoch minted for the generation +// read would disagree with the one that lands. `expected_head: none` is the value-only CAS. +fn pair_serving_authority_transition_at( + store: FabricStorageBinding, + group: FabricGroup, + expected: PairServingGroupAuthority, + expected_head: HeadExpectation?, + next: PairServingGroupAuthority, + transaction: NonEmptyStr, + lifecycle: ContentHash?, + grant: LeaseGrant?, + entry_state: ContentHash?, + actor: NonEmptyStr, + at: EpochSecs, +) -> AuthorityTransition { + if (fabric_group_wire(g: pair_serving_authority_group(a: next)) as String) != (fabric_group_wire(g: group) as String) { + AuthorityTransitionRefused { group: group, step: "subject", reason: "the next authority names a different group than the transition", placement: none } + } else { + match current_pair_serving_authority(store: store, group: group, at: at) { + CurrentAuthorityUnread { group: g, step: s, reason: why } => AuthorityTransitionRefused { group: g, step: s, reason: why, placement: none } + CurrentAuthorityRead { authority: cur, head: head, generation: g, grant: _, entry_state: _, previous: _, admitted_by: admitted_by, transitions: _ } => + if !authority_eq(a: cur, b: expected) { + AuthorityTransitionStale { expected: expected, current: cur, placement: none } + } else if (match expected_head { Absent => false Present { value: eh } => !head_expectation_eq(a: eh, b: head) }) { + AuthorityTransitionRefused { group: group, step: "head", reason: join(["the authority's value is the expected one but its head moved since the reading the transition was decided from (generation ", to_string(g), "); re-decide from a fresh read"], ""), placement: none } + } else if pair_serving_commits_hosts(a: next) && length(pair_serving_committed_hosts(a: next)) == 0 { + AuthorityTransitionRefused { group: group, step: "subject", reason: "the next state commits hosts but names none; a transaction state must retain the population it was authorized over", placement: none } + } else { + let genealogy = match grant { + Absent => GrantGenealogyAgrees + Present { value: gr } => grant_genealogy(gr: gr, next: next, transaction: transaction, admitted_by: admitted_by) + } + match genealogy { + GrantWithoutLease => AuthorityTransitionRefused { group: group, step: "grant", reason: "a grant may only be carried into a state that holds a lease", placement: none } + GrantGenealogyDisagrees { join: j } => AuthorityTransitionRefused { group: group, step: "grant", reason: join(["the grant is not this transition's lease's: ", j], ""), placement: none } + GrantGenealogyAgrees => { + let intent = authority_write_intent(group: group, head: head, actor: actor, previous: Present { value: cur }, next: next, operation: transaction, lifecycle: lifecycle, grant: grant, entry_state: entry_state) + match saga_prepare_and_claim(store: store, group: group, group_head: head, previous_hosts: committed_population(a: cur), next_hosts: committed_population(a: next), operation: transaction, intent: intent, actor: actor, at: at) { + SagaRefused { step: st, reason: why, placement: pl } => AuthorityTransitionRefused { group: group, step: st, reason: why, placement: pl } + SagaClaimed { preparation: ref, intent: _ } => + match authority_transition_append(store: store, group: group, head: head, current: cur, next: next, transaction: transaction, lifecycle: lifecycle, grant: grant, entry_state: entry_state, preparation: ref, actor: actor, at: at) { + EventAppendRefused { cause: c } => AuthorityTransitionRefused { group: group, step: "store", reason: event_log_refusal_wire(cause: c), placement: Present { value: saga_abort_own(store: store, preparation: ref.id, reason: event_log_refusal_wire(cause: c), actor: actor, at: at) } } + EventAppendStale { expected: _, observed: _ } => { + let cleanup = saga_abort_own(store: store, preparation: ref.id, reason: "the authority moved between the read and the transition", actor: actor, at: at) + match current_pair_serving_authority(store: store, group: group, at: at) { + CurrentAuthorityUnread { group: g2, step: s, reason: why } => AuthorityTransitionRefused { group: g2, step: join(["stale/", s], ""), reason: why, placement: Present { value: cleanup } } + CurrentAuthorityRead { authority: now_cur } => AuthorityTransitionStale { expected: expected, current: now_cur, placement: Present { value: cleanup } } + } + } + EventAppended { id: h } => AuthorityTransitionAppended { id: h, generation: g + 1, next: next, placement: placement_finalize_attempt(store: store, preparation: ref, authority_event: h, actor: actor, at: at, attempts_left: 3) } + } + } + } + } + } + } + } +} + +// THE SAGA'S MIDDLE STEP, ALONE: the authority event carrying its preparation, appended against +// the group head that was read. pair_serving_authority_transition composes prepare, this, and +// finalize; the step is its own function so a saga that dies after it is a state a witness can +// drive (the preparation is consumed but not finalized, and only finalization may follow). +fn authority_transition_append(store: FabricStorageBinding, group: FabricGroup, head: HeadExpectation, current: PairServingGroupAuthority, next: PairServingGroupAuthority, transaction: NonEmptyStr, lifecycle: ContentHash?, grant: LeaseGrant?, entry_state: ContentHash?, preparation: PlacementPreparationRef, actor: NonEmptyStr, at: EpochSecs) -> EventAppend { + let event = ChainEvent { + partition: pair_authority_partition(group: group), + parent: match head { HeadAbsent => none HeadAt { id: h } => Present { value: h } }, + recorded_at: at, + actor: actor, + payload: AuthorityTransitioned { previous: current, next: next, transaction: transaction, lifecycle: lifecycle, grant: grant, entry_state: entry_state, placement: preparation }, + } + event_log_append_with(store: store, partition: pair_authority_partition(group: group), event: event, expected: head, encode: fn(e) { pair_authority_event_wire_text(event: e) }) +} + +// RECORD THE ENTRY STATE: the receipt's bytes appended as their own event, against the head the +// caller observed, so a concurrent writer is a stale refusal and never a receipt attributed to +// the wrong state. The id returned is the store's content address of the bytes; the settling +// transition names it, and a reader fetches the bytes by it from the store. +type EntryStateRecord + = EntryStateRecordedAt { id: EventId } + | EntryStateRecordStale { observed: HeadExpectation } + | EntryStateRecordRefused { step: String, reason: String } + +fn pair_serving_authority_record_entry_state(store: FabricStorageBinding, group: FabricGroup, expected_head: HeadExpectation, transaction: NonEmptyStr, body: String, actor: NonEmptyStr, at: EpochSecs) -> EntryStateRecord { + let parent = match expected_head { + HeadAbsent => none + HeadAt { id: h } => Present { value: h } + } + let event = ChainEvent { + partition: pair_authority_partition(group: group), + parent: parent, + recorded_at: at, + actor: actor, + payload: EntryStateRecorded { transaction: transaction, body: body }, + } + match event_log_append_with(store: store, partition: pair_authority_partition(group: group), event: event, expected: expected_head, encode: fn(e) { pair_authority_event_wire_text(event: e) }) { + EventAppended { id: h } => EntryStateRecordedAt { id: h } + EventAppendStale { expected: _, observed: o } => EntryStateRecordStale { observed: o } + EventAppendRefused { cause: c } => EntryStateRecordRefused { step: "store", reason: event_log_refusal_wire(cause: c) } + } +} + +fn current_authority_wire(c: CurrentAuthority) -> String { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: g, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => join(["generation ", to_string(g), ": ", authority_wire(a: a)], "") + CurrentAuthorityUnread { group: g, step: s, reason: why } => join([fabric_group_wire(g: g) as String, " authority unread at ", s, ": ", why], "") + } +} + +// ── THE HOST-EFFECT CLAIM AND ITS RELEASE ────────────────────────────────────────────────────── +// +// ONE COMPARE-AND-SET AGAINST THE PLACEMENT HEAD THE ADMISSION READ. The claim is admitted only +// when no live commitment names the host and no live claim holds it, and a write that lands +// between the read and the append -- another claim, a commitment -- makes the append stale, +// returned as such and never retried into a state that was not decided. `host_effect_admit_at` +// decides and appends against a SUPPLIED read (so an interleaving can be driven); the release +// names the admitting event and carries the quiescence evidence and provenance, refused here +// before any append when they are not for that claim, and refused again by the fold on every +// later read. +type HostEffectClaim + = HostEffectClaimed { id: EventId } + | HostEffectClaimRefused { partition: PartitionId, step: String, reason: String } + +fn host_effect_admit_at(store: FabricStorageBinding, read: PlacementRead, host: HostIdentity, purpose: NonEmptyStr, executor: NonEmptyStr, term: Second, residue: HostEffectResidueSpec, at: EpochSecs) -> HostEffectClaim { + match read { + PlacementUnread { step: st, reason: why } => HostEffectClaimRefused { partition: host_placement_partition, step: st, reason: why } + PlacementReadAt { head: head, effects: effs, preparations: preps } => + if length(groups_fencing(preps: preps, host: host)) != 0 { + HostEffectClaimRefused { partition: host_placement_partition, step: "committed", reason: join([host as String, " is fenced by an authority: ", join(map(groups_fencing(preps: preps, host: host), g => fabric_group_wire(g: g) as String), ", ")], "") } + } else if length(live_host_effects_on(records: effs, hosts: [host], at: at)) != 0 { + HostEffectClaimRefused { partition: host_placement_partition, step: "held", reason: join([host as String, " is held by another live host effect: ", join(map(live_host_effects_on(records: effs, hosts: [host], at: at), r => host_effect_line_at(r: r, at: at)), "; ")], "") } + } else { + match placement_append(store: store, head: head, payload: HostEffectAdmitted { host: host, purpose: purpose, executor: executor, term: term, residue: residue }, actor: executor, at: at) { + EventAppended { id: h } => HostEffectClaimed { id: h } + EventAppendStale { expected: _, observed: _ } => HostEffectClaimRefused { partition: host_placement_partition, step: "stale", reason: "the placement partition moved between the admission read and the claim; nothing was placed" } + EventAppendRefused { cause: c } => HostEffectClaimRefused { partition: host_placement_partition, step: "store", reason: event_log_refusal_wire(cause: c) } + } + } + } +} + +fn host_effect_admit(store: FabricStorageBinding, host: HostIdentity, purpose: NonEmptyStr, executor: NonEmptyStr, term: Second, residue: HostEffectResidueSpec, at: EpochSecs) -> HostEffectClaim { + host_effect_admit_at(store: store, read: placement_read(store: store), host: host, purpose: purpose, executor: executor, term: term, residue: residue, at: at) +} + +type HostEffectRelease + = HostEffectReleasedAt { id: EventId } + | HostEffectReleaseRefused { partition: PartitionId, step: String, reason: String } + +fn host_effect_release(store: FabricStorageBinding, claim: EventId, evidence: HostQuiescenceEvidence, provenance: ReleaseProvenance, executor: NonEmptyStr, at: EpochSecs) -> HostEffectRelease { + match placement_read(store: store) { + PlacementUnread { step: st, reason: why } => HostEffectReleaseRefused { partition: host_placement_partition, step: st, reason: why } + PlacementReadAt { head: head, effects: effs, preparations: _ } => + match release_admission(records: effs, claim: claim, evidence: evidence, provenance: provenance, actor: executor) { + ReleaseAdmitted { record: _ } => + match placement_append(store: store, head: head, payload: HostEffectReleased { admitted_by: claim, evidence: evidence, provenance: provenance }, actor: executor, at: at) { + EventAppended { id: h } => HostEffectReleasedAt { id: h } + EventAppendStale { expected: _, observed: _ } => HostEffectReleaseRefused { partition: host_placement_partition, step: "stale", reason: "the placement partition moved while the release was being appended; rerun the release" } + EventAppendRefused { cause: c } => HostEffectReleaseRefused { partition: host_placement_partition, step: "store", reason: event_log_refusal_wire(cause: c) } + } + other => HostEffectReleaseRefused { partition: host_placement_partition, step: "evidence", reason: release_refusal_reason(a: other, claim: claim) } + } + } +} + +// The live claim with this id on this host, for a recovery that names its claim exactly. +type LiveClaimLookup + = LiveClaimFound { record: HostEffectRecord } + | LiveClaimAbsent + | LiveClaimUnread { step: String, reason: String } + +fn live_claim_on(store: FabricStorageBinding, host: HostIdentity, claim: EventId) -> LiveClaimLookup { + match placement_read(store: store) { + PlacementUnread { step: st, reason: why } => LiveClaimUnread { step: st, reason: why } + PlacementReadAt { head: _, effects: effs, preparations: _ } => + match first(filter(effs, r => !r.released && (r.id as String) == (claim as String) && host_identity_eq(a: r.host, b: host))) { + Absent => LiveClaimAbsent + Present { value: r } => LiveClaimFound { record: r } + } + } +} diff --git a/dag/gunbc/spark/pair_serving_d0.dag b/dag/gunbc/spark/pair_serving_d0.dag new file mode 100644 index 00000000000..d1b67cb0343 --- /dev/null +++ b/dag/gunbc/spark/pair_serving_d0.dag @@ -0,0 +1,1507 @@ +module gunbc.spark.pair_serving_d0 + +import std.types { String, Bool, Int, NonEmptyStr, List, EpochSecs, Port } +import std.nat { Nat } +import std.measure { Second, second, second_count } +import v2.std.optional { Present, Absent } +import std.process { ProcessExit, ExitSuccess, exit_failure } +import std.resources { Network } +import std.list { list_cover } +import std.content_hash { ContentHash, content_hash_of_value, serialize_content_hash, content_hash_equal, parse_content_hash } +import std.temporal_effect { HeldLease, LeaseEpoch, held_lease, LeaseRunningExpected } +import std.scoped_authorization { + OperatorGrant, ScopedAuthorization, AuthorizationGranted, AuthorizationRequest, AuthorizationScope, AttemptIdentity, AuthorizedAction, + AuthorizationPermitted, AuthorizationRefused, AuthorizationRefusalCause, AuthorizationEscalationMismatch, authorize, authorization_scope_render, authorization_refusal_reason, + AuthorizationClaimState, Unclaimed, ClaimedBy, CompletedBy, AbortedBy, claim_authorization, claim_transition_claimed, claim_transition_completed, claim_transition_aborted, + AuthorizationClaimOutcome, ClaimHeld, ClaimLost, ClaimUndecided, authorization_claim_outcome, authorization_claim_key, parse_authorization_claim_state, +} +import std.durable_compare_and_set { + CasGeneration, ExpectSlotAbsent, ExpectSlotGeneration, CasCommitted, CasPreconditionFailed, CasStoreRefused, + CasObservedReadable, CasObservedUnreadable, CasReadableAbsent, CasReadablePresent, cas_generation_count, cas_unreadable_slot_detail, +} +import gunbc.durable_cas_file_store { + CasAttemptAdmitted, CasAttemptDigestMismatch, CasAttemptDigestIncomparable, CasAttemptKeyNotSlotAddressable, admit_cas_attempt, +} +import gunbc.durable_cas_fabric_storage { fabric_storage_compare_and_set, fabric_storage_observe_cas_slot } +import extdeps.tools.sha256sum { Sha256FileDigest, Sha256FileDigestUnavailable, sha256sum_stdin_digest_via_shell } +import extdeps.crypto.hash { sha256_digest_content_hash } +import std.effect_grant { Write, NamespacePosition, CodeNameTree } +import product.capacity.lease { LeaseGrant, LeasePolicy, ReleaseLaw, QuiescenceRequired, lease_grant, lease_fence_of } +import product.capacity.event_chain { EventId, HeadExpectation, HeadAbsent, HeadAt } +import product.placement_supply { HostIdentity, host_identity_eq } +import extdeps.filesystem.filesystem_io { Filesystem } +import gunbc.clock_read { clock_now_probed_at, probed_at_word } +import gunbc.fabric_event_log_host { HostStoreResolved, HostStoreRefused, event_log_store_for_host, now_epoch_seconds } +import gunbc.fabric_storage_client { FabricStorageBinding } +import gunbc.host_effect { FleetSsh } +import gunbc.fleet_known_hosts_anchor { FleetSshExecutionContext } +import gunbc.rung_drop { Standing, Retired, TypedDeclaration } +import gunbc.rung_drop.fabric_storage_append_principal_unrefused { fabric_storage_append_principal_unrefused } +import gunbc.fleet_ssh_locus { fleet_locus_ssh_target, FleetSshContextReady, FleetSshContextRefused, prepare_fleet_ssh_agent_context } +import gunbc.spark.fabric_reach { ExecutorReachKnown, ExecutorReachUnknown, observe_executor_reach } +import gunbc.spark.fabric_switch_observed { FabricGroup, fabric_group_wire, parse_fabric_group, fabric_group_hosts } +import gunbc.serving.serving_enrollment { ServingRouteEndpoint, ServingRouteResolved, ServingRouteUnresolved, serving_route } +import gunbc.harness.harness_backend { harness_bounded_get, harness_bounded_presence, HarnessResponded, HarnessConnectFailed, HarnessDeadline, HarnessTransportUnread, harness_metrics_url, harness_models_url, harness_launch_from_body } +import gunbc.serving.serving_enrollment { serving_route_endpoint_wire } +import gunbc.spark.vllm_endpoint_process_launch { LaunchObserved, LaunchUnobserved } +import gunbc.spark.serving_incarnation_observe { + UnitOccupancy, UnitInstalled, UnitNotInstalled, UnitOccupancyUnread, IncarnationUnitPropertyUnread, read_unit_occupancy, serving_incarnation_refusal_text, + EndpointListenerReading, EndpointNoListener, EndpointListening, EndpointListenerUnread, read_endpoint_listener, + EngineProcessScan, EngineScanLegDidNotRun, EngineScanClassified, scan_engine_processes, +} +import extdeps.procps.pgrep { PgrepMatched, PgrepNoMatch, PgrepScanFailed } +import extdeps.docker.images.sparkrun_vllm_ds4_gb10 { sparkrun_vllm_ds4_gb10_vllm_binary } +import extdeps.linux.proc_net_tcp { TcpSocketRow } +import std.algebra { trim } +import gunbc.spark.vllm_serving_launch { DigestsAgree, DigestsDiffer, DigestsIncomparable, digests_agree } +import gunbc.spark.pair_serving_desired { SparkPairServingUnitReady, SparkPairServingUnitRefused, spark_pair_serving_unit } +import gunbc.spark.pair_serving_realization { spark_pair_head_unit_name, spark_pair_worker_unit_name } +import gunbc.spark.pair_incumbent_identity { + DeclaredPairRealization, DeclaredRankUnit, DeclaredPairRealizationStanding, DeclaredPairRealizationKeyed, DeclaredPairRealizationUnavailable, declared_pair_realization, + observe_pair_realization, PairRealizationAgreement, PairRealizationEstablished, pair_realization_agreement, pair_realization_agreement_lines, +} +import gunbc.spark.released_baseline { ReleasedBaselineSpec, QuiescentReservedBaseline, released_baseline_wire } +import gunbc.spark.pair_serving_authority { + PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, SuspendedForAuthorizedSuccessor, FencedRefusal, + PairRealizationKeyed, +} +import gunbc.spark.pair_serving_authority_log { + CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authority, authority_wire, same_group, + AuthorityTransitionRecord, transitions_by, + AuthorityTransition, AuthorityTransitionAppended, AuthorityTransitionStale, AuthorityTransitionRefused, pair_serving_authority_transition, pair_serving_authority_transition_at, + PlacementCleanup, PlacementFinalizedAt, PlacementAbortedAt, PlacementCleanupStillFencing, placement_cleanup_line, placement_repair_pending, + preparation_consumption, placement_finalize, + EntryStateRecord, EntryStateRecordedAt, EntryStateRecordStale, EntryStateRecordRefused, pair_serving_authority_record_entry_state, +} + +// ── D0: TAKE THE CLAIM, RECONCILE THE OCCUPANCY, RECORD THE ENTRY STATE ───────────────────────── +// +// docs/plans/dsv41-cut-d-redesign.md §D0. The transfer is ATOMIC and lands in an intermediate +// state that the readings happen INSIDE: PairServingActive moves to +// SuspensionPendingReconciliation by one compare-and-set on the authority log +// (gunbc.spark.pair_serving_authority_log), which disables the incumbent's apply and the +// successor's launch at once; only then is the fleet read; and the answer moves the authority +// again, to exactly one of SuspendedForAuthorizedSuccessor, PairServingActive or FencedRefusal. +// An earlier draft observed first and transferred after, which left an interval in which the +// observation could go stale and two lanes could each believe they held the hosts. +// +// THE QUESTIONS ARE ORDERED. First: is a serving incumbent ANSWERING on the enrolled front door? +// If it is, this transaction is not the drain-and-preserve path and does not pretend to be: when +// the answering realization agrees with the declared one on every rank +// (gunbc.spark.pair_incumbent_identity PairRealizationEstablished) restoration is shown safe and +// the authority returns to EXACTLY the Active state it held -- unchanged, and in particular NOT +// upgraded to PairRealizationKeyed, because that agreement carries the running image as observed +// and uncompared; the incumbent's key stays minted nowhere until the image axis is compared -- +// and when the answer is anything else the group is fenced for an operator. The only Keyed +// realization D0 writes is the SUCCESSOR's, on the suspended authority, from the admitted grant's +// keyed candidate. Second, with nothing answering and the head host quiet: does the unit +// population on the four ranks reconcile with the declared realization? The head's declared unit +// is read first -- ACTIVE is OccupancyNotQuiet, a listener the route did not reach, and it fences +// as unidentified -- and with the head inactive, DeclaredOccupantDrifted (our units, not all +// running or not all our bytes -- THE EXPECTED ANSWER on the readings the redesign records, and +// the only reconciled answer, since the head was just read inactive) is eligible, the drift +// recorded and never a restoration target; OccupancyUnread fences. There is no "observed +// converged" answer here: a running declared occupant is the incumbent question's answer. +// +// WHAT THIS TRANSACTION READS AND WHAT IT DOES NOT, stated because the redesign's fourth answer is +// not produced here. ForeignOccupantObserved -- a container, process or device allocation we do +// not declare -- needs a reading of the ranks BEYOND our unit, and this transaction reads only +// the declared unit's fragment, bytes and activity. So a rank with no pair unit installed cannot +// be told apart from a rank held by something foreign, and it is reported OccupancyUnread with +// exactly that cause -- which fences, the disposition foreign occupancy would have reached. Nothing +// is fabricated; the frontier is the container and device reading, and it is named on the arm. +// +// THE ENTRY STATE IS EVIDENCE, NOT A TARGET. The receipt records what was read per rank -- the +// route's answer, each rank's unit occupancy -- and is total: an unread rank carries its cause +// rather than forcing a fabricated field. Memory used and free, absence of a serving process +// beyond the unit, and free disk are named by the redesign and are NOT read here; the receipt +// says what it holds and a later reader may not infer the rest. +// +// THE RECEIPT IS RECORDED ON THE LOG BEFORE THE SETTLING WRITE, AND THE WRITE NAMES IT. The +// entry-state receipt is rendered and appended to the authority partition as its own event before +// the second compare-and-set, and that set carries the event's content address -- so a receipt +// that cannot be recorded refuses BEFORE the authority moves, and a later reader of the suspended +// authority fetches the exact four-rank reading it was settled from by that id, from the store. +// A local copy under target/ is a convenience derived from the id, and its failure changes nothing. +// +// THE TRANSACTION IS TOTAL OVER ITS CRASH POINTS. Consent is consumed by a claim on the fleet +// event log BEFORE the first authority write, and the claim and the authority are two durable +// facts a process can die between. A rerun therefore starts from a JOIN of the two histories +// (d0_recovery): no claim means a fresh start; a claim held by this transaction with no transition +// by it means the first write never landed and is performed now (or the claim aborted when the +// authority is no longer Active); a pending state under this transaction resumes from the +// readings; a settling transition by this transaction means only the claim's terminal is owed, +// which is written with no authority write; and a claim held by another attempt, or spent, refuses. +// A rerun naming another transaction is refused at the pending state -- it is that transaction's. +// +// THE ENTRY IS NON-ADMITTING UNTIL THE AUTHORIZATION EXISTS, BY CONSTRUCTION. The dispatch +// requires an exact, time-bounded operator authorization over the candidate key, this group's +// EXACT HOST POPULATION, the cleanup baseline and the lease term (std.scoped_authorization +// OperatorGrant over D0Subject), and a successor that is KEYED (Cut 0). No producer joins an +// escalation to such a grant today and no candidate is keyed, so resolve_d0_authorization answers +// D0AuthorizationUnestablished and the entry refuses before its first write. The transaction +// itself takes the grant as a value, which is how its real-execution witness drives the whole +// two-write route with a fixture grant. +// +// TERMINAL: the authority is in exactly one of the three states, the receipt exists at the path +// the event names, and no host state was altered. This transaction starts and stops nothing. + +// ── THE AUTHORIZATION ────────────────────────────────────────────────────────────────────────── +// +// What an operator authorizes D0 over: the group AND ITS EXACT HOSTS, the KEYED successor, the +// cleanup baseline the hosts are owed afterwards, and the lease TERM the successor may start +// under. The group word alone is not a subject: FabricGroupA's membership is derived from the +// current lane observations and a later source change can move a host in or out while the wire +// still says `group-a` -- so a grant over the word would act on a host the operator never saw. +// The hosts are compared to the group's current population before the claim and before the +// first write (d0_population), and a difference refuses. +// The release law is NOT the operator's to choose: a pair-serving rank is a process resource with +// no generation fence on its touches, so an expired holder's running engine is not rejected by +// anything -- expiry removes the right to START, and capacity is free only once stale use is +// observed quiescent. That is product.capacity.lease QuiescenceRequired, derived for every D0 grant. +type D0Subject { + group: FabricGroup + hosts: List + successor: ContentHash + cleanup: ReleasedBaselineSpec + term: Second +} + +data pair_serving_release_law: ReleaseLaw = QuiescenceRequired + +fn d0_lease_policy(subject: D0Subject) -> LeasePolicy { + LeasePolicy { maximum_duration_seconds: second_count(s: subject.term), release_law: pair_serving_release_law } +} + +// THE POPULATION IS AN IDENTITY JOIN IN BOTH DIRECTIONS: every granted host is a current member +// and every current member was granted, with no duplicate on either side. A count would let a +// swap pass. +type D0Population + = D0PopulationAgrees + | D0PopulationDiffers { granted: List, current: List } + +// `current` is SUPPLIED -- the entry passes fabric_group_hosts(group) -- so the join is one +// interface over two host lists and its controls can drive a roster the source does not carry. +fn d0_population(subject: D0Subject, current: List) -> D0Population { + if length(current) == length(subject.hosts) + && list_cover(held: current, expected: subject.hosts, eq: fn(a, b) { host_identity_eq(a: a, b: b) }) + && list_cover(held: subject.hosts, expected: current, eq: fn(a, b) { host_identity_eq(a: a, b: b) }) { + D0PopulationAgrees + } else { + D0PopulationDiffers { granted: subject.hosts, current: current } + } +} + +fn hosts_wire(hs: List) -> String { + join(sort_by(map(hs, h => h as String), fn(s) { s }), ",") +} + +// THE OPERATION THE GRANT MUST COVER: a write over the pair-serving authority, action "suspend". +// The scope is the enforcing fact (std.scoped_authorization): a grant over the same subject with +// a read scope, or another action, does not authorize this transaction. +data d0_authorized_action: AuthorizedAction = "suspend-pair-serving-authority" as AuthorizedAction + +fn d0_required_scope() -> AuthorizationScope { + AuthorizationScope { verb: Write, resource: NamespacePosition { tree: CodeNameTree, path: ["gunbc", "spark", "pair_serving_authority"] }, action: d0_authorized_action } +} + +fn d0_subject_wire(subject: D0Subject) -> String { + join([ + "group=", fabric_group_wire(g: subject.group) as String, + " hosts=", hosts_wire(hs: subject.hosts), + " successor=", serialize_content_hash(hash: subject.successor) as String, + " cleanup=", released_baseline_wire(s: subject.cleanup) as String, + " term=", to_string(second_count(s: subject.term)), "s", + ], "") +} + +// THE INTENT IS THE EXACT OPERATION, AS A CRYPTOGRAPHIC IDENTITY OVER ONE CANONICAL ENCODING: +// the required scope, the group, the hosts (sorted, so order is not identity), the keyed +// successor, the cleanup, the term and the transaction. A grant whose intent was minted over +// anything else does not authorize this run. SHA-256 rather than the structural fingerprint +// content_hash_of_value mints (FNV-1a 64): the intent is the consent binding itself, and a +// collision class there is the same defect the pending-state binding was moved off it for. +// The digest is taken through the sha256sum realization, so it is an effect and can be +// unavailable -- which is a refusal, never a substituted structural hash. +data d0_intent_schema: String = "pair-serving-d0-intent/v1" + +fn d0_intent_text(subject: D0Subject, group: FabricGroup, transaction: NonEmptyStr) -> String { + join([ + d0_intent_schema, + "\nscope=", authorization_scope_render(scope: d0_required_scope()), + "\ntarget=", fabric_group_wire(g: group) as String, + "\nattempt=", transaction as String, + "\n", d0_subject_wire(subject: subject), + "\n", + ], "") +} + +fn d0_intent_hash(subject: D0Subject, group: FabricGroup, transaction: NonEmptyStr) -> ContentHash? { + sha256_of_text(text: d0_intent_text(subject: subject, group: group, transaction: transaction)) +} + +fn sha256_of_text(text: String) -> ContentHash? { + match sha256sum_stdin_digest_via_shell(content: text) { + Sha256FileDigestUnavailable { path: _, reason: _ } => none + Sha256FileDigest { digest: d } => sha256_digest_content_hash(digest: d) + } +} + +// The request D0 makes of a grant: the escalation the caller names, the required scope, the +// grant's own subject (what is being checked is that the grant agrees with itself and with this +// run), the transaction as the attempt, and the intent over the exact operation. Destructive. +fn d0_request(escalation_id: NonEmptyStr, subject: D0Subject, transaction: NonEmptyStr, intent: ContentHash) -> AuthorizationRequest { + AuthorizationRequest { + escalation_id: escalation_id, + scopes: [d0_required_scope()], + subject: subject, + purpose: "suspend the pair-serving authority for a bounded successor transaction (Cut D D0)" as NonEmptyStr, + attempt: (transaction as String) as AttemptIdentity, + intent_hash: intent, + destructive: true, + } +} + +// ── THE CLAIM: CONSENT IS CONSUMED ONCE, FLEET-WIDE, BEFORE THE FIRST WRITE ──────────────────── +// +// std.scoped_authorization decides whether a grant PERMITS an operation; it models single use as +// a compare-and-set TRANSITION of the grant's claim slot, never as a check at the reader, because +// two readers can both see "unclaimed". THE SLOT IS ON THE FABRIC EVENT LOG +// (gunbc.durable_cas_fabric_storage), not on a host-local file store: D0 may run from any executor +// that resolves a store, and two executors' local roots exclude nothing from each other -- each +// would see the grant unclaimed and both would consume the same consent. The log's one placed +// remote is the linearization point the authority already relies on, so the claim shares it, and +// exactly one ClaimedBy exists for a grant however many executors race for it. +// +// The pending state carries the claim's generation inside its binding; a resume needs the slot +// held by THIS attempt and does not re-run the permission decision (expiry removes the right to +// START -- the right to finish or clean up survives it, exactly as the successor's lease does); +// and the transaction's terminal marks the claim CompletedBy, or AbortedBy when it refused before +// its first write. + +type D0ClaimStanding + = D0ClaimHeld { generation: CasGeneration } + | D0ClaimLost { holder: NonEmptyStr } + | D0ClaimUndecided { reason: NonEmptyStr } + +fn d0_claim_grant(store: FabricStorageBinding, escalation_id: NonEmptyStr, transaction: NonEmptyStr, now: String) -> D0ClaimStanding { + let proposed = claim_transition_claimed(attempt: (transaction as String) as AttemptIdentity, at: now) + match admit_cas_attempt(attempt: claim_authorization(escalation_id: escalation_id, expected: ExpectSlotAbsent, proposed: proposed)) { + CasAttemptAdmitted { verified: v } => + match authorization_claim_outcome(proposed: proposed, outcome: fabric_storage_compare_and_set(store: store, verified: v)) { + ClaimHeld { state: _, generation: g } => D0ClaimHeld { generation: g } + ClaimLost { holder: h } => D0ClaimLost { holder: h } + ClaimUndecided { reason: r } => D0ClaimUndecided { reason: r } + } + CasAttemptDigestMismatch { claimed: _, actual: _ } => D0ClaimUndecided { reason: "claim payload and its digest disagree" as NonEmptyStr } + CasAttemptDigestIncomparable { claimed: _, actual: _ } => D0ClaimUndecided { reason: "claim digest families are incomparable" as NonEmptyStr } + CasAttemptKeyNotSlotAddressable { key: _ } => D0ClaimUndecided { reason: "authorization claim key is not slot-addressable" as NonEmptyStr } + } +} + +// THE SLOT AS A RERUN FINDS IT: decoded into the claim's own closed state, with the generation +// the terminal must name. An unreadable store, or bytes that inhabit no claim state, is Unreadable +// -- a rerun that proceeded on an unread slot would be the absorbing fallback on the one +// operation whose purpose is exclusion. +type D0ClaimReading + = D0ClaimAbsent + | D0ClaimAt { state: AuthorizationClaimState, generation: CasGeneration } + | D0ClaimUnreadable { reason: NonEmptyStr } + +fn d0_read_claim(store: FabricStorageBinding, escalation_id: NonEmptyStr) -> D0ClaimReading { + match fabric_storage_observe_cas_slot(store: store, key: authorization_claim_key(escalation_id: escalation_id)) { + CasObservedUnreadable { cause: c } => D0ClaimUnreadable { reason: join(["authorization claim slot unreadable: ", cas_unreadable_slot_detail(cause: c)], "") as NonEmptyStr } + CasObservedReadable { readable: CasReadableAbsent } => D0ClaimAbsent + CasObservedReadable { readable: CasReadablePresent { version: v } } => + match parse_authorization_claim_state(payload: v.value) { + Absent => D0ClaimUnreadable { reason: join(["the claim slot holds bytes that are no claim state: ", v.value as String], "") as NonEmptyStr } + Present { value: s } => D0ClaimAt { state: s, generation: v.generation } + } + } +} + +// The slot as this attempt must find it on resume: held by this transaction. Any other holder, +// a spent grant, an absent slot, or an unreadable store refuses. +fn d0_resume_claim(store: FabricStorageBinding, escalation_id: NonEmptyStr, transaction: NonEmptyStr) -> D0ClaimStanding { + match d0_read_claim(store: store, escalation_id: escalation_id) { + D0ClaimUnreadable { reason: r } => D0ClaimUndecided { reason: r } + D0ClaimAbsent => D0ClaimLost { holder: "no attempt holds the grant, so there is no claimed transaction to resume" as NonEmptyStr } + D0ClaimAt { state: s, generation: g } => + match s { + ClaimedBy { attempt: who, claimed_at: _ } => + if (who as String) == (transaction as String) { D0ClaimHeld { generation: g } } else { D0ClaimLost { holder: join(["claimed by attempt ", who as String], "") as NonEmptyStr } } + Unclaimed => D0ClaimLost { holder: "the slot records the grant unclaimed" as NonEmptyStr } + CompletedBy { attempt: who, completed_at: _ } => D0ClaimLost { holder: join(["completed by attempt ", who as String], "") as NonEmptyStr } + AbortedBy { attempt: who, aborted_at: _, cause: _ } => D0ClaimLost { holder: join(["aborted by attempt ", who as String], "") as NonEmptyStr } + } + } +} + +// THE TERMINAL: the claim moves to CompletedBy when the transaction settled (any of its three +// terminals) and to AbortedBy when it refused before its first write. A terminal that does not +// land is reported, never hidden: the authority already moved, and a bare success would leave an +// operator unaware that a slot inspection is owed. +type D0ClaimTerminal + = D0ClaimTerminated + | D0ClaimNotAttempted { still_held_by: NonEmptyStr } + | D0ClaimUnterminated { detail: String } + +fn d0_terminate_claim(store: FabricStorageBinding, escalation_id: NonEmptyStr, transaction: NonEmptyStr, held: CasGeneration, completed: Bool, cause: String, now: String) -> D0ClaimTerminal { + let attempt = (transaction as String) as AttemptIdentity + let terminal = if completed { claim_transition_completed(attempt: attempt, at: now) } else { claim_transition_aborted(attempt: attempt, at: now, cause: cause as NonEmptyStr) } + match admit_cas_attempt(attempt: claim_authorization(escalation_id: escalation_id, expected: ExpectSlotGeneration { generation: held }, proposed: terminal)) { + CasAttemptAdmitted { verified: v } => + match fabric_storage_compare_and_set(store: store, verified: v) { + CasCommitted { committed: _ } => D0ClaimTerminated + CasPreconditionFailed { expected: _, observed: _ } => D0ClaimUnterminated { detail: "claim slot moved under us" } + CasStoreRefused { cause: _ } => D0ClaimUnterminated { detail: "claim store refused the terminal write" } + } + CasAttemptDigestMismatch { claimed: _, actual: _ } => D0ClaimUnterminated { detail: "terminal payload and its digest disagree" } + CasAttemptDigestIncomparable { claimed: _, actual: _ } => D0ClaimUnterminated { detail: "terminal digest families are incomparable" } + CasAttemptKeyNotSlotAddressable { key: _ } => D0ClaimUnterminated { detail: "authorization claim key is not slot-addressable" } + } +} + +// ── THE ADMITTED GRANT, AND THE BINDING THE PENDING STATE CARRIES ────────────────────────────── +// +// Sole-constructed: the only ways to hold one are admit_d0_grant (start: permission decided, +// claim taken) and resume_d0_grant (resume: claim found held by this attempt). The binding is a +// SHA-256 over one canonical encoding of everything a resume must agree on -- escalation, +// attempt, the cryptographic intent, subject, expiry, and the claim's generation on the fleet +// log -- so a pending state written under one authorization cannot be settled under another, +// including a re-issued grant with a later expiry. THE EXECUTOR IS NOT IN IT: the binding is the +// AUTHORIZATION lifecycle, stable across whichever executor runs a rerun, so any executor reads +// the same history for it (and may complete a settled claim, which mutates no host). WHO may +// finish a pending transaction is a separate fact, the lease's owner fingerprint on the pending +// state, checked by the recovery join. +type AdmittedD0Grant sole_constructor { + grant: OperatorGrant + claim_generation: CasGeneration + binding: ContentHash +} + +fn d0_binding_text(grant: OperatorGrant, claim_generation: CasGeneration) -> String { + join([ + "escalation=", grant.escalation_id as String, + "\nattempt=", grant.attempt as String, + "\nintent=", serialize_content_hash(hash: grant.intent_hash) as String, + "\n", d0_subject_wire(subject: grant.subject), + "\nexpires_at=", grant.expires_at, + "\nclaim_generation=", to_string(cas_generation_count(g: claim_generation)), + "\n", + ], "") +} + +type D0GrantAdmission + = D0GrantAdmitted { admitted: AdmittedD0Grant } + | D0GrantForAnotherGroup { granted: FabricGroup } + | D0GrantForAnotherPopulation { granted: List, current: List } + | D0GrantIntentUnavailable + | D0GrantRefused { cause: AuthorizationRefusalCause } + | D0GrantClaimLost { holder: NonEmptyStr } + | D0GrantClaimUndecided { reason: NonEmptyStr } + | D0GrantBindingUnavailable { claim: D0ClaimTerminal } + +fn admitted_with_binding(grant: OperatorGrant, generation: CasGeneration) -> AdmittedD0Grant? { + match sha256_of_text(text: d0_binding_text(grant: grant, claim_generation: generation)) { + Absent => none + Present { value: b } => Present { value: AdmittedD0Grant { grant: grant, claim_generation: generation, binding: b } } + } +} + +// THE CHECKS THAT NEED NO STORE, before any claim is taken: the grant names this group, and its +// host population is the group's current one. +type D0SubjectStanding + = D0SubjectAgrees { grant: OperatorGrant } + | D0SubjectDisagrees { admission: D0GrantAdmission } + | D0SubjectNotGranted + +// THE POPULATION IS CHECKED AT THE START AND NOT ON A RESUME. Before any claim or write, the +// operator's hosts must be the group's current hosts. Once the claim is held the check moves to +// the dispatcher (d0_dispatch), where its consequence depends on how far the lifecycle got: no +// authority write yet means the claim is aborted; a pending state already written means the +// transaction must still reach a terminal under the population it was authorized for -- a source +// edit may not revoke the right to finish -- so the drift is carried into the readings and fences. +type D0PopulationCheck + = PopulationAgainst { current: List } + | PopulationNotChecked + +fn d0_subject_standing(authorization: ScopedAuthorization, group: FabricGroup, check_population: D0PopulationCheck) -> D0SubjectStanding { + match authorization { + AuthorizationGranted { grant: g, claim: _ } => + if !same_group(a: g.subject.group, b: group) { + D0SubjectDisagrees { admission: D0GrantForAnotherGroup { granted: g.subject.group } } + } else { + match check_population { + PopulationNotChecked => D0SubjectAgrees { grant: g } + PopulationAgainst { current: current } => + match d0_population(subject: g.subject, current: current) { + D0PopulationDiffers { granted: gr, current: cur } => D0SubjectDisagrees { admission: D0GrantForAnotherPopulation { granted: gr, current: cur } } + D0PopulationAgrees => D0SubjectAgrees { grant: g } + } + } + } + _ => D0SubjectNotGranted + } +} + +// START: the subject checks, the intent, the permission decision, then the claim; a binding that +// cannot be digested AFTER the claim landed aborts the claim rather than stranding it. +fn admit_d0_grant(store: FabricStorageBinding, authorization: ScopedAuthorization, escalation_id: NonEmptyStr, group: FabricGroup, current_hosts: List, transaction: NonEmptyStr, executor: NonEmptyStr, at: EpochSecs, now: String) -> D0GrantAdmission { + match d0_subject_standing(authorization: authorization, group: group, check_population: PopulationAgainst { current: current_hosts }) { + D0SubjectNotGranted => D0GrantRefused { cause: d0_not_granted_cause(authorization: authorization, escalation_id: escalation_id, group: group) } + D0SubjectDisagrees { admission: a } => a + D0SubjectAgrees { grant: g } => + match d0_intent_hash(subject: g.subject, group: group, transaction: transaction) { + Absent => D0GrantIntentUnavailable + Present { value: intent } => + match authorize(authorization: authorization, request: d0_request(escalation_id: escalation_id, subject: g.subject, transaction: transaction, intent: intent), observed_at: now) { + AuthorizationRefused { cause: c } => D0GrantRefused { cause: c } + AuthorizationPermitted { grant: permitted } => + match d0_claim_grant(store: store, escalation_id: escalation_id, transaction: transaction, now: now) { + D0ClaimLost { holder: h } => D0GrantClaimLost { holder: h } + D0ClaimUndecided { reason: r } => D0GrantClaimUndecided { reason: r } + D0ClaimHeld { generation: gen } => + match admitted_with_binding(grant: permitted, generation: gen) { + Present { value: admitted } => D0GrantAdmitted { admitted: admitted } + Absent => D0GrantBindingUnavailable { claim: d0_terminate_claim(store: store, escalation_id: escalation_id, transaction: transaction, held: gen, completed: false, cause: "the authorization binding could not be digested after the claim landed", now: now) } + } + } + } + } + } +} + +// A pending or denied authorization has no subject to check; std.scoped_authorization's own +// decision names the cause. The request handed to it carries no intent (the subject is not the +// grant's), which is fine: the decision refuses before comparing intents. +fn d0_not_granted_cause(authorization: ScopedAuthorization, escalation_id: NonEmptyStr, group: FabricGroup) -> AuthorizationRefusalCause { + let placeholder = D0Subject { group: group, hosts: [] as List, successor: content_hash_of_value(value: "unauthorized" as NonEmptyStr), cleanup: QuiescentReservedBaseline, term: second(count: 0) } + match authorize(authorization: authorization, request: d0_request(escalation_id: escalation_id, subject: placeholder, transaction: "unauthorized" as NonEmptyStr, intent: content_hash_of_value(value: "unauthorized" as NonEmptyStr)), observed_at: "") { + AuthorizationRefused { cause: c } => c + AuthorizationPermitted { grant: g } => AuthorizationEscalationMismatch { requested: escalation_id, granted: g.escalation_id } + } +} + +// RESUME: the subject checks, then the claim must already be this attempt's; permission is not +// re-decided, so an expired grant can still finish what it started. The binding is recomputed and +// compared to the pending state's by the transaction. +fn resume_d0_grant(store: FabricStorageBinding, authorization: ScopedAuthorization, escalation_id: NonEmptyStr, group: FabricGroup, transaction: NonEmptyStr, executor: NonEmptyStr) -> D0GrantAdmission { + match d0_subject_standing(authorization: authorization, group: group, check_population: PopulationNotChecked) { + D0SubjectNotGranted => D0GrantClaimLost { holder: "the authorization is not granted, so no claim of it can be resumed" as NonEmptyStr } + D0SubjectDisagrees { admission: a } => a + D0SubjectAgrees { grant: g } => + match d0_resume_claim(store: store, escalation_id: escalation_id, transaction: transaction) { + D0ClaimLost { holder: h } => D0GrantClaimLost { holder: h } + D0ClaimUndecided { reason: r } => D0GrantClaimUndecided { reason: r } + D0ClaimHeld { generation: gen } => + match admitted_with_binding(grant: g, generation: gen) { + Present { value: admitted } => D0GrantAdmitted { admitted: admitted } + Absent => D0GrantBindingUnavailable { claim: D0ClaimUnterminated { detail: "the claim stays held: a resume that cannot digest its binding leaves the transaction to a later rerun" } } + } + } + } +} + +type D0AuthorizationStanding + = D0AuthorizationResolved { authorization: ScopedAuthorization } + | D0AuthorizationUnestablished { obligation: NonEmptyStr } + +// THE PRODUCTION RESOLVER, AND WHY IT REFUSES. An authorization id is a reference to an operator +// decision; the ScopedAuthorization it names has to be produced by joining the escalation to a +// decision over THIS subject, and the subject needs a keyed candidate. Neither producer exists at +// this head, so the resolver refuses with what would ground it, and the entry refuses before its +// first write. TRIGGER: Cut 0 keys the V4.1 candidate (produced runtime image, closed weight and +// tokenizer manifests, four rank-store identities) and an escalation-to-ScopedAuthorization +// producer lands whose grant carries the write scope over the pair-serving authority, the +// transaction as attempt, the exact host population and the SHA-256 intent over the exact +// operation. SUFFICIENT FOR: this fn answers D0AuthorizationResolved for a real escalation id, +// admit_d0_grant decides it under std.scoped_authorization, and the wet entry admits exactly the +// operation that grant names. +fn resolve_d0_authorization(group: FabricGroup, authorization: String) -> D0AuthorizationStanding { + D0AuthorizationUnestablished { + obligation: join([ + "authorization `", authorization, "` over ", fabric_group_wire(g: group) as String, + " cannot be resolved: no producer joins an escalation id to a ScopedAuthorization over D0Subject, and the V4.1 candidate is not keyed (Cut 0), so the successor has no exact identity to authorize", + ], "") as NonEmptyStr, + } +} + +// The lease as the authority carries it. The epoch names the transaction, the group as the +// resource and the executor as the owner; the generation is the authority-log generation the +// pending state lands at, so a stale holder presents a generation the fence rejects. +fn d0_held_lease(group: FabricGroup, transaction: NonEmptyStr, executor: NonEmptyStr, generation: Nat) -> HeldLease { + held_lease( + epoch: LeaseEpoch { + lease_key: transaction, + resource_fingerprint: content_hash_of_value(value: fabric_group_wire(g: group)), + owner_fingerprint: content_hash_of_value(value: executor), + generation: generation, + }, + observed: LeaseRunningExpected, + ) +} + +// ── THE READINGS ─────────────────────────────────────────────────────────────────────────────── + +type RankOccupancyReading { + host: HostIdentity + unit_name: NonEmptyStr + occupancy: UnitOccupancy +} + +// FOUR ARMS, BECAUSE "NO LAUNCH OBSERVED" IS SEVERAL FACTS. A failed connect within the bound is +// the only reading under which absence can still be established -- and the decision establishes +// it on the head host (reconcile_occupancy: the head's unit first), never from the transport; the door +// answering with any HTTP status, or the metrics naming no launch, is an incumbent that IS there +// and could not be identified, and it fences; a deadline, or a transport that could not be read, +// says nothing about the door and is RouteUnread, which fences too. An earlier shape folded every +// failure into "silent" and would have suspended a group whose engine stalled. +// WHAT THE HEAD HOST SAYS ABOUT ITS OWN FRONT DOOR, read only when the door could not be connected: +// the kernel's socket tables for a LISTEN on the enrolled port (any owner), and the engine-process +// scan. Absence is established only by Quiet -- no listener on the port AND no engine process -- +// and the decision then still requires the declared unit inactive. A listener, an engine process, +// or an unread table is a possible incumbent the route did not reach, and fences. +type HeadEndpointReading + = HeadEndpointQuiet + | HeadEndpointListening { port: Port, rows: List } + | HeadEndpointEngineProcess { listing: String } + | HeadEndpointUnread { cause: NonEmptyStr } + +type IncumbentReading + = IncumbentAnswered { agreement: PairRealizationAgreement } + | IncumbentAnsweredUnread { cause: NonEmptyStr } + | IncumbentDidNotAnswer { receipt: NonEmptyStr, head: HeadEndpointReading } + | IncumbentRouteUnread { cause: NonEmptyStr } + +type D0Observation { + group: FabricGroup + incumbent: IncumbentReading + ranks: List +} + +// ── THE RECONCILIATION ───────────────────────────────────────────────────────────────────────── + +type OccupancyReconciliation + = DeclaredOccupantDrifted { drifted: List } + | OccupancyNotQuiet { live: List } + | OccupancyUnread { hosts: List, cause: NonEmptyStr } + +type RankVerdict + = RankLive { host: HostIdentity } + | RankDrifted { host: HostIdentity } + | RankUnread { host: HostIdentity, cause: NonEmptyStr } + +fn rank_verdict(declared: DeclaredRankUnit, ranks: List) -> RankVerdict { + match first(filter(ranks, r => host_identity_eq(a: r.host, b: declared.host))) { + Absent => RankUnread { host: declared.host, cause: "no occupancy reading was taken for this declared rank" as NonEmptyStr } + Present { value: r } => + match r.occupancy { + UnitOccupancyUnread { refusal: why } => RankUnread { host: declared.host, cause: serving_incarnation_refusal_text(r: why) as NonEmptyStr } + UnitNotInstalled => + RankUnread { host: declared.host, cause: "no pair unit is installed on this rank; occupancy beyond the declared unit (containers, device allocations) is not read by this transaction, so an empty rank cannot be told apart from a foreign occupant" as NonEmptyStr } + UnitInstalled { active: active, unit_digest: d } => + match digests_agree(left: d, right: declared.digest) { + DigestsIncomparable => RankUnread { host: declared.host, cause: "the installed unit's digest and the declared digest are of different hash families" as NonEmptyStr } + DigestsDiffer => RankDrifted { host: declared.host } + DigestsAgree => if active { RankLive { host: declared.host } } else { RankDrifted { host: declared.host } } + } + } + } +} + +fn verdict_unread(v: RankVerdict) -> List { + match v { + RankUnread { host: _, cause: _ } => [v] + RankLive { host: _ } => [] as List + RankDrifted { host: _ } => [] as List + } +} + +fn verdict_drifted_host(v: RankVerdict) -> List { + match v { + RankDrifted { host: h } => [h] + RankLive { host: _ } => [] as List + RankUnread { host: _, cause: _ } => [] as List + } +} + +fn live_host(v: RankVerdict) -> List { + match v { + RankLive { host: h } => [h] + RankDrifted { host: _ } => [] as List + RankUnread { host: _, cause: _ } => [] as List + } +} + +fn unread_host(v: RankVerdict) -> List { + match v { + RankUnread { host: h, cause: _ } => [h] + RankLive { host: _ } => [] as List + RankDrifted { host: _ } => [] as List + } +} + +fn unread_cause(v: RankVerdict) -> List { + match v { + RankUnread { host: h, cause: c } => [join([h as String, ": ", c as String], "")] + RankLive { host: _ } => [] as List + RankDrifted { host: _ } => [] as List + } +} + +// THE HEAD FIRST, THEN EVERY RANK. Reconciliation is asked when the front door could not be +// connected and the head host's socket table and process scan were quiet; what it establishes on +// top of that is the declared UNIT on the head: readable and not active (inactive or not +// installed) corroborates absence, ACTIVE is a listener the route did not reach (OccupancyNotQuiet +// -- an unidentified incumbent, never a reconciled answer), unread fences. With the head quiet, +// every declared rank is read against its unit: one unread rank makes the whole occupancy unread +// -- a pair three-quarters reconciled is not reconciled -- and A LIVE RANK ANYWHERE IS NOT QUIET: +// a worker whose declared unit is installed, byte-identical and active is a running occupant, so +// the answer is OccupancyNotQuiet naming every live rank, never a drifted population with the +// running rank dropped (that was the widen review 69399 found). Only with every rank read and +// none live is the drift decided; because the head was just read inactive, the head rank is +// drifted (declared bytes, not running) or unread (not installed), so the reconciled answer this +// route reaches is DeclaredOccupantDrifted: the declared occupant, observed not running, which +// is exactly the population the suspension records and D1 cleans. +fn reconcile_occupancy(declared: DeclaredPairRealization, ranks: List) -> OccupancyReconciliation { + match first(filter(ranks, r => host_identity_eq(a: r.host, b: declared.head.host))) { + Absent => OccupancyUnread { hosts: [declared.head.host], cause: "no occupancy reading was taken for the head rank" as NonEmptyStr } + Present { value: r } => + match r.occupancy { + UnitOccupancyUnread { refusal: why } => OccupancyUnread { hosts: [declared.head.host], cause: serving_incarnation_refusal_text(r: why) as NonEmptyStr } + UnitInstalled { active: true, unit_digest: _ } => OccupancyNotQuiet { live: [declared.head.host] } + _ => { + let verdicts = map(concat([declared.head], declared.workers), d => rank_verdict(declared: d, ranks: ranks)) + let unread = flat_map(verdicts, v => verdict_unread(v: v)) + let live = flat_map(verdicts, v => live_host(v: v)) + if length(unread) != 0 { + OccupancyUnread { hosts: flat_map(unread, v => unread_host(v: v)), cause: join(flat_map(unread, v => unread_cause(v: v)), "; ") as NonEmptyStr } + } else if length(live) != 0 { + OccupancyNotQuiet { live: live } + } else { + DeclaredOccupantDrifted { drifted: flat_map(verdicts, v => verdict_drifted_host(v: v)) } + } + } + } + } +} + +// ── THE DECISION ─────────────────────────────────────────────────────────────────────────────── + +type D0Cause + = D0IncumbentLive { standing: String } + | D0IncumbentUnidentified { cause: NonEmptyStr } + | D0OccupancyUnread { cause: NonEmptyStr } + | D0DeclarationUnavailable { obligation: NonEmptyStr } + +// What the second transition writes, decided from the readings alone. +type D0Decision + = D0DecidedSuspend { next: PairServingGroupAuthority, reconciliation: OccupancyReconciliation } + | D0DecidedRestore { next: PairServingGroupAuthority, key: ContentHash } + | D0DecidedFence { next: PairServingGroupAuthority, cause: D0Cause } + +data d0_disposition_authority: NonEmptyStr = "operator" as NonEmptyStr + +fn fence(group: FabricGroup, hosts: List, cause: D0Cause) -> D0Decision { + let text = match cause { + D0IncumbentLive { standing: s } => join(["a serving incumbent answers on the enrolled route and is not the declared realization: ", s], "") + D0IncumbentUnidentified { cause: c } => join(["the enrolled route answered but the answering launch could not be identified, so an incumbent may be live: ", c as String], "") + D0OccupancyUnread { cause: c } => join(["occupancy could not be reconciled: ", c as String], "") + D0DeclarationUnavailable { obligation: o } => join(["the declared pair realization could not be keyed: ", o as String], "") + } + D0DecidedFence { next: FencedRefusal { group: group, hosts: hosts, cause: text as NonEmptyStr, disposition_authority: d0_disposition_authority }, cause: cause } +} + +// THE PURE DECISION over supplied readings. The successor's lease is the one the pending state +// holds, re-read so its observed state is the read's; the candidate realization is what the +// caller was authorized for, Unestablished until Cut 0 keys it. +// THE RESTORE DOES NOT MINT A KEY. When the answering incumbent is the declared realization on +// every rank, restoration is shown safe -- but gunbc.spark.pair_incumbent_identity carries the +// running IMAGE as observed-and-uncompared (no produced-image digest exists to compare it to), so +// "established" there is rank-and-unit agreement, not an exact realization. A rebuilt image under +// the same reference with unchanged unit bytes passes it. So the authority returns to the exact +// state it was in, unchanged; PairRealizationKeyed is minted only once the image axis is compared +// (the produced-image receipt names that trigger on the agreement's own arm). +fn d0_decide( + declared: DeclaredPairRealizationStanding, + observation: D0Observation, + active: PairServingGroupAuthority, + lease: HeldLease, + grant: OperatorGrant, +) -> D0Decision { + let authorization = grant.escalation_id + let candidate = PairRealizationKeyed { key: grant.subject.successor } + let cleanup = grant.subject.cleanup + let hosts = grant.subject.hosts + let group = observation.group + match declared { + DeclaredPairRealizationUnavailable { group: _, obligation: o } => fence(group: group, hosts: hosts, cause: D0DeclarationUnavailable { obligation: o }) + DeclaredPairRealizationKeyed { declared: d } => + match observation.incumbent { + IncumbentRouteUnread { cause: c } => fence(group: group, hosts: hosts, cause: D0OccupancyUnread { cause: c }) + IncumbentAnsweredUnread { cause: c } => fence(group: group, hosts: hosts, cause: D0IncumbentUnidentified { cause: c }) + IncumbentAnswered { agreement: a } => + match a { + PairRealizationEstablished { key: k, declared: _, observed: _, image: _ } => + D0DecidedRestore { next: active, key: k } + _ => fence(group: group, hosts: hosts, cause: D0IncumbentLive { standing: join(pair_realization_agreement_lines(a: a), "; ") }) + } + IncumbentDidNotAnswer { receipt: rc, head: head } => + match head { + HeadEndpointListening { port: pt, rows: rows } => fence(group: group, hosts: hosts, cause: D0IncumbentUnidentified { cause: join([rc as String, "; and ", to_string(length(rows)), " socket(s) LISTEN on port ", to_string(pt), " on the head, so a listener the route did not reach may be the incumbent"], "") as NonEmptyStr }) + HeadEndpointEngineProcess { listing: l } => fence(group: group, hosts: hosts, cause: D0IncumbentUnidentified { cause: join([rc as String, "; and an engine process is running on the head: ", trim(s: l)], "") as NonEmptyStr }) + HeadEndpointUnread { cause: c } => fence(group: group, hosts: hosts, cause: D0OccupancyUnread { cause: join([rc as String, "; and the head's socket table or process scan could not be read: ", c as String], "") as NonEmptyStr }) + HeadEndpointQuiet => + match reconcile_occupancy(declared: d, ranks: observation.ranks) { + OccupancyUnread { hosts: _, cause: c } => fence(group: group, hosts: hosts, cause: D0OccupancyUnread { cause: join([rc as String, "; and ", c as String], "") as NonEmptyStr }) + OccupancyNotQuiet { live: lv } => fence(group: group, hosts: hosts, cause: D0IncumbentUnidentified { cause: join([rc as String, "; and the declared serving unit is ACTIVE on [", hosts_wire(hs: lv), "], so an occupant is running behind the failed connect"], "") as NonEmptyStr }) + DeclaredOccupantDrifted { drifted: ds } => + D0DecidedSuspend { + next: SuspendedForAuthorizedSuccessor { group: group, hosts: hosts, authorization: authorization, exact_candidate_realization: candidate, lease: lease, cleanup: cleanup }, + reconciliation: DeclaredOccupantDrifted { drifted: ds }, + } + } + } + } + } +} + +fn d0_decision_next(d: D0Decision) -> PairServingGroupAuthority { + match d { + D0DecidedSuspend { next: n, reconciliation: _ } => n + D0DecidedRestore { next: n, key: _ } => n + D0DecidedFence { next: n, cause: _ } => n + } +} + +fn d0_decision_is_eligible(d: D0Decision) -> Bool { + match d { + D0DecidedSuspend { next: _, reconciliation: _ } => true + D0DecidedRestore { next: _, key: _ } => false + D0DecidedFence { next: _, cause: _ } => false + } +} + +// ── THE ENTRY-STATE RECEIPT ──────────────────────────────────────────────────────────────────── + +type EntryStateReceipt { + group: FabricGroup + transaction: NonEmptyStr + at: String + observation: D0Observation + decision: D0Decision +} + +// THE RECEIPT'S IDENTITY IS THE STORE'S CONTENT ADDRESS OF ITS BYTES: the id of the +// EntryStateRecorded event the log appended (the fabric DB's object ref, a content-hash wire over +// the object's preimage), carried on the settling transition as a ContentHash so a reader fetches +// the bytes by it from the one placed store. A local convenience copy is derived from that id, +// never from the transaction's text: a NonEmptyStr admits path-significant characters and is not +// a path segment. +fn entry_state_identity(id: EventId) -> ContentHash? { + parse_content_hash(wire: id as String) +} + +fn entry_state_receipt_path(id: EventId) -> String { + join(["target/pair-serving-d0-entry-", id as String, ".txt"], "") +} + +fn occupancy_line(r: RankOccupancyReading) -> String { + let o = match r.occupancy { + UnitInstalled { active: a, unit_digest: d } => join(["installed active=", if a { "true" } else { "false" }, " unit=", serialize_content_hash(hash: d) as String], "") + UnitNotInstalled => "not-installed" + UnitOccupancyUnread { refusal: why } => join(["unread: ", serving_incarnation_refusal_text(r: why)], "") + } + join([" rank ", r.host as String, " ", r.unit_name as String, ": ", o], "") +} + +fn incumbent_lines(i: IncumbentReading) -> List { + match i { + IncumbentAnswered { agreement: a } => concat([" incumbent answered:"], map(pair_realization_agreement_lines(a: a), l => join([" ", l], ""))) + IncumbentAnsweredUnread { cause: c } => [join([" incumbent answered but unidentified: ", c as String], "")] + IncumbentDidNotAnswer { receipt: r, head: h } => [join([" incumbent did not answer: ", r as String, "; head endpoint: ", head_endpoint_line(h: h)], "")] + IncumbentRouteUnread { cause: c } => [join([" incumbent route unread: ", c as String], "")] + } +} + +fn head_endpoint_line(h: HeadEndpointReading) -> String { + match h { + HeadEndpointQuiet => "quiet (no listener on the enrolled port, no engine process)" + HeadEndpointListening { port: p, rows: rs } => join([to_string(length(rs)), " LISTEN socket(s) on port ", to_string(p)], "") + HeadEndpointEngineProcess { listing: l } => join(["engine process present: ", trim(s: l)], "") + HeadEndpointUnread { cause: c } => join(["unread: ", c as String], "") + } +} + +fn reconciliation_line(r: OccupancyReconciliation) -> String { + match r { + OccupancyNotQuiet { live: lv } => join([" reconciliation: occupancy-not-quiet, active on [", hosts_wire(hs: lv), "]"], "") + DeclaredOccupantDrifted { drifted: ds } => join([" reconciliation: declared-occupant-drifted ", join(map(ds, h => h as String), ",")], "") + OccupancyUnread { hosts: hs, cause: c } => join([" reconciliation: occupancy-unread ", join(map(hs, h => h as String), ","), ": ", c as String], "") + } +} + +fn decision_lines(d: D0Decision) -> List { + match d { + D0DecidedSuspend { next: n, reconciliation: r } => ["verdict: d0-eligible", reconciliation_line(r: r), join([" authority: ", authority_wire(a: n)], "")] + D0DecidedRestore { next: n, key: k } => ["verdict: d0-refused incumbent-live-restored", join([" key: ", serialize_content_hash(hash: k) as String], ""), join([" authority: ", authority_wire(a: n)], "")] + D0DecidedFence { next: n, cause: _ } => ["verdict: d0-refused fenced", join([" authority: ", authority_wire(a: n)], "")] + } +} + +fn entry_state_receipt_body(r: EntryStateReceipt) -> String { + join(concat( + [join(["pair serving d0 entry state at ", r.at, " transaction=", r.transaction as String, " group=", fabric_group_wire(g: r.group) as String], "")], + concat(incumbent_lines(i: r.observation.incumbent), concat(map(r.observation.ranks, o => occupancy_line(r: o)), decision_lines(d: r.decision))), + ), "\n") +} + +data pair_serving_d0_attempt_raw: String = "pair-serving-d0" + +// ── THE OBSERVATION, OVER THE FLEET ──────────────────────────────────────────────────────────── + +// A REFUSED UNIT DECLARATION IS CARRIED AS EACH DECLARED RANK'S CAUSE, not dropped: the receipt is +// evidence, and "no reading was taken" is not what happened when the desired unit could not be +// derived. Every declared rank reads Unread with the declaration's own refusal. +fn observe_ranks(context: FleetSshExecutionContext, group: FabricGroup, declared: DeclaredPairRealizationStanding) -> List { + match spark_pair_serving_unit(group: group) { + SparkPairServingUnitRefused { group: _, cause: c } => + match declared { + DeclaredPairRealizationUnavailable { group: _, obligation: _ } => [] as List + DeclaredPairRealizationKeyed { declared: d } => + map(concat([d.head], d.workers), r => RankOccupancyReading { host: r.host, unit_name: r.unit_name, occupancy: UnitOccupancyUnread { refusal: IncarnationUnitPropertyUnread { unit: r.unit_name, property: "desired-unit" as NonEmptyStr, cause: join(["the desired pair serving unit could not be derived: ", c as String], "") } } }) + } + SparkPairServingUnitReady { group: _, engine: _, head: h, workers: ws } => + concat( + [RankOccupancyReading { host: h.host, unit_name: spark_pair_head_unit_name, occupancy: read_unit_occupancy(transport: FleetSsh { target: fleet_locus_ssh_target(host: h.host), context: context }, unit: spark_pair_head_unit_name) }], + map(ws, w => RankOccupancyReading { host: w.host, unit_name: spark_pair_worker_unit_name, occupancy: read_unit_occupancy(transport: FleetSsh { target: fleet_locus_ssh_target(host: w.host), context: context }, unit: spark_pair_worker_unit_name) }), + ) + } +} + +// THE FRONT DOOR DECIDES PRESENCE; METRICS ONLY IDENTIFIES. A live incumbent can serve /v1/models +// and requests while /metrics is disabled, proxied elsewhere or malformed, so a failed metrics +// read establishes nothing about presence. The enrolled serving front door is asked for its +// STATUS within the bound (harness_bounded_presence, which does not fail on 4xx/5xx): only NO +// HTTP response is absence; a 401 or a 500 is an incumbent that answered and could not be read, +// which fences. A 2xx lets metrics name the incarnation -- and when they cannot, the incumbent is +// answering and unidentified, which fences too. +// +// The front-door reading is its own fn over the ENDPOINT so its transport can be executed against +// a local listener that answers a chosen status: the distinction between "answered 500" and "no +// response" lives in http.Client.StatusWithin's argv, and a control over a supplied reading cannot +// notice that argv regaining -f. +// "NO HTTP RESPONSE" IS NOT ABSENCE. A failed connect (curl 7) is the only failure under which +// absence can still be ESTABLISHED, and not by the transport: the reading is NoStatus and the +// decision corroborates it against the head rank's own unit occupancy (a readable, inactive or +// uninstalled declared unit on the head host); a deadline is a listener that may have accepted +// and did not answer -- a stalled engine is an incumbent -- and a transport that could not be read +// establishes nothing. Those two are Unread and fence. An earlier shape read every failure as +// silence, and a group whose engine hung would have been suspended out from under it. +type FrontDoorReading + = FrontDoorNoStatus { receipt: NonEmptyStr } + | FrontDoorAnsweredUnread { cause: NonEmptyStr } + | FrontDoorUnread { cause: NonEmptyStr } + | FrontDoorAnswered + +fn observe_front_door(endpoint: ServingRouteEndpoint) -> FrontDoorReading { + let front_door = harness_models_url(endpoint: endpoint) + match harness_bounded_presence(url: front_door) { + HarnessConnectFailed { url: u, curl_exit: code } => + FrontDoorNoStatus { receipt: join(["the serving front door at ", u, " could not be connected within the front-door bound (curl exit ", to_string(code), "); absence is established only by the head rank's unit occupancy"], "") as NonEmptyStr } + HarnessDeadline { url: u } => + FrontDoorUnread { cause: join(["the serving front door at ", u, " did not answer within the front-door bound: a listener may have accepted, and its state is unread"], "") as NonEmptyStr } + HarnessTransportUnread { url: u, curl_exit: code } => + FrontDoorUnread { cause: join(["the serving front door at ", u, " could not be read (curl exit ", to_string(code), "), so nothing is established about the incumbent"], "") as NonEmptyStr } + HarnessResponded { status: status } => + if starts_with(s: status, prefix: "2") { + FrontDoorAnswered + } else { + FrontDoorAnsweredUnread { cause: join(["the front door at ", front_door, " answered with HTTP ", status, ": an incumbent is there and its service could not be read"], "") as NonEmptyStr } + } + } +} + +// The incumbent reading a front-door answer decides on its own: Absent only when the door +// answered 2xx and identification must follow. +// The incumbent reading a front-door answer decides on its own, with the head's endpoint reading +// SUPPLIED for the no-status arm (the observer takes it only then); Absent only when the door +// answered 2xx and identification must follow. +fn front_door_as_incumbent(r: FrontDoorReading, head: fn() -> HeadEndpointReading) -> IncumbentReading? { + match r { + FrontDoorNoStatus { receipt: rc } => Present { value: IncumbentDidNotAnswer { receipt: rc, head: head() } } + FrontDoorAnsweredUnread { cause: c } => Present { value: IncumbentAnsweredUnread { cause: c } } + FrontDoorUnread { cause: c } => Present { value: IncumbentRouteUnread { cause: c } } + FrontDoorAnswered => none + } +} + +fn identify_incumbent(context: FleetSshExecutionContext, group: FabricGroup, declared: DeclaredPairRealizationStanding, endpoint: ServingRouteEndpoint) -> IncumbentReading { + let front_door = harness_models_url(endpoint: endpoint) + let url = harness_metrics_url(endpoint: endpoint) + let read = harness_bounded_get(url: url) + if !read.success { + IncumbentAnsweredUnread { cause: join(["the front door at ", front_door, " answered but the metrics endpoint at ", url, " did not, so the answering launch cannot be named"], "") as NonEmptyStr } + } else { + match harness_launch_from_body(group: group, address: serving_route_endpoint_wire(e: endpoint) as String, body: read.body) { + LaunchUnobserved { group: _, obligation: o } => IncumbentAnsweredUnread { cause: o } + LaunchObserved { launch: launch, receipt: _ } => + IncumbentAnswered { agreement: pair_realization_agreement(declared: declared, observed: observe_pair_realization(context: context, group: group, endpoint: launch)) } + } + } +} + +// THE HEAD'S OWN FRONT DOOR: the socket tables for a LISTEN on the enrolled port, then the engine +// process scan; taken over the fleet transport to the declared head host. +fn observe_head_endpoint(context: FleetSshExecutionContext, declared: DeclaredPairRealizationStanding, port: Port) -> HeadEndpointReading { + match declared { + DeclaredPairRealizationUnavailable { group: _, obligation: o } => HeadEndpointUnread { cause: join(["no declared head to read: ", o as String], "") as NonEmptyStr } + DeclaredPairRealizationKeyed { declared: d } => { + let target = fleet_locus_ssh_target(host: d.head.host) + match read_endpoint_listener(transport: FleetSsh { target: target, context: context }, port: port) { + EndpointListenerUnread { port: _, cause: c } => HeadEndpointUnread { cause: c as NonEmptyStr } + EndpointListening { port: p, rows: rs } => HeadEndpointListening { port: p, rows: rs } + EndpointNoListener { port: _ } => + match scan_engine_processes(target: target, context: context, pattern: sparkrun_vllm_ds4_gb10_vllm_binary) { + EngineScanLegDidNotRun { cause: c } => HeadEndpointUnread { cause: join(["the engine process scan did not run: ", c], "") as NonEmptyStr } + EngineScanClassified { outcome: PgrepScanFailed { exit_code: code, stderr: err } } => HeadEndpointUnread { cause: join(["the engine process scan failed (exit ", to_string(code), "): ", err], "") as NonEmptyStr } + EngineScanClassified { outcome: PgrepMatched { listing: l } } => HeadEndpointEngineProcess { listing: l } + EngineScanClassified { outcome: PgrepNoMatch } => HeadEndpointQuiet + } + } + } + } +} + +fn observe_incumbent(context: FleetSshExecutionContext, group: FabricGroup, declared: DeclaredPairRealizationStanding) -> IncumbentReading { + match serving_route(group: group) { + ServingRouteUnresolved { group: _, obligation: o } => IncumbentRouteUnread { cause: o } + ServingRouteResolved { endpoint: endpoint, ceiling: _ } => + match front_door_as_incumbent(r: observe_front_door(endpoint: endpoint), head: fn() { observe_head_endpoint(context: context, declared: declared, port: endpoint.port) }) { + Present { value: decided } => decided + Absent => identify_incumbent(context: context, group: group, declared: declared, endpoint: endpoint) + } + } +} + +fn observe_d0(context: FleetSshExecutionContext, group: FabricGroup, declared: DeclaredPairRealizationStanding) -> D0Observation { + D0Observation { group: group, incumbent: observe_incumbent(context: context, group: group, declared: declared), ranks: observe_ranks(context: context, group: group, declared: declared) } +} + +// ── THE TRANSACTION ──────────────────────────────────────────────────────────────────────────── + +fn refuse(reason: String) -> ProcessExit { + exit_failure(reason: join(["pair_serving_d0: ", reason], "")) +} + +// THE AUTHORITY MOVED BUT ITS PLACEMENT DID NOT FINALIZE is its own terminal: the group's saga is +// still pending (fencing its hosts, refusing every later saga), so the consent is NOT complete -- +// the claim stays held, and a rerun finalizes the placement from the join before it completes. +type D0Outcome + = D0Settled { decision: D0Decision, generation: Nat, receipt_path: String, entry_state: ContentHash } + | D0PlacementStillFencing { decision: D0Decision?, generation: Nat, entry_state: ContentHash?, preparation: EventId?, cause: String } + | D0Refused { cause: String } + + +fn d0_outcome_is_eligible(o: D0Outcome) -> Bool { + match o { + D0Settled { decision: d, generation: _, receipt_path: _, entry_state: _ } => d0_decision_is_eligible(d: d) + D0PlacementStillFencing { decision: _, generation: _, entry_state: _, preparation: _, cause: _ } => false + D0Refused { cause: _ } => false + } +} + +fn d0_outcome_text(o: D0Outcome) -> String { + match o { + D0Settled { decision: d, generation: g, receipt_path: p, entry_state: es } => + join(concat(decision_lines(d: d), [join([" settled at generation ", to_string(g), " entry-state ", serialize_content_hash(hash: es) as String, " receipt ", p], "")]), "\n") + D0PlacementStillFencing { decision: d, generation: g, entry_state: es, preparation: pr, cause: c } => + join(concat(match d { Absent => [" (the transfer into suspension landed; nothing was read yet)"] Present { value: dd } => decision_lines(d: dd) }, [join([" the authority moved at generation ", to_string(g), match es { Absent => "" Present { value: h } => join([" entry-state ", serialize_content_hash(hash: h) as String], "") }, " but its placement preparation ", (match pr { Absent => "(could not be named)" Present { value: id } => id as String }), " did not finalize: ", c, "; the consent is not complete -- a rerun under this transaction finalizes it from the join"], "")]), " +") + D0Refused { cause: c } => join(["pair_serving_d0 refused: ", c], "") + } +} + +fn transition_text(t: AuthorityTransition) -> String { + match t { + AuthorityTransitionAppended { id: i, generation: g, next: _, placement: pl } => join(["appended ", i as String, " at generation ", to_string(g), "; ", placement_cleanup_line(c: pl)], "") + AuthorityTransitionStale { expected: e, current: c, placement: pl } => join(["stale: expected ", authority_wire(a: e), " but the authority is ", authority_wire(a: c), match pl { Absent => "" Present { value: x } => join(["; ", placement_cleanup_line(c: x)], "") }], "") + AuthorityTransitionRefused { group: _, step: st, reason: why, placement: pl } => join(["refused at ", st, ": ", why, match pl { Absent => "" Present { value: x } => join(["; ", placement_cleanup_line(c: x)], "") }], "") + } +} + +// THE SETTLE: readings taken under the pending state, decided, the receipt written and digested, +// then ONE compare-and-set naming the receipt. The exit is decided from the transition ARM, never +// from its rendered text. +// +// THE GRANT RIDES ONLY INTO THE LEASED TERMINAL. A restore to Active and a fence hold no lease, +// and the log refuses a grant carried into a state with none -- so those two terminals write the +// transition without one, and only the suspension carries the term minted from the admitting +// write. An earlier shape carried it on every arm and could reach neither. +fn d0_settle(store: FabricStorageBinding, pending: PairServingGroupAuthority, head: HeadExpectation, active: PairServingGroupAuthority, lease: HeldLease, admitted: AdmittedD0Grant, admitting: EventId, pending_generation: Nat, transaction: NonEmptyStr, executor: NonEmptyStr, at: EpochSecs, now: String, declared: DeclaredPairRealizationStanding, observation: D0Observation) -> D0Outcome { + let group = observation.group + let decision = d0_decide(declared: declared, observation: observation, active: active, lease: lease, grant: admitted.grant) + let receipt = EntryStateReceipt { group: group, transaction: transaction, at: now, observation: observation, decision: decision } + let body = entry_state_receipt_body(r: receipt) + match pair_serving_authority_record_entry_state(store: store, group: group, expected_head: head, transaction: transaction, body: body, actor: executor, at: at) { + EntryStateRecordRefused { step: st, reason: why } => + D0Refused { cause: join(["the entry-state receipt could not be recorded on the log before settling, so the authority was not moved: ", st, ": ", why], "") } + EntryStateRecordStale { observed: _ } => + D0Refused { cause: "another writer moved the partition while the entry-state receipt was being recorded, so the authority was not moved by this run" } + EntryStateRecordedAt { id: record } => + match entry_state_identity(id: record) { + Absent => D0Refused { cause: join(["the recorded receipt's id ", record as String, " is not a content hash, so the settling write cannot name it"], "") } + Present { value: digest } => { + let path = entry_state_receipt_path(id: record) + let wr = Filesystem.Write(path: path, content: body) + let term = match decision { + D0DecidedSuspend { next: _, reconciliation: _ } => + Present { value: lease_grant(reference: transaction, fence: lease_fence_of(admitting_event: admitting, generation: pending_generation), granted_at: at, policy: d0_lease_policy(subject: admitted.grant.subject)) } + D0DecidedRestore { next: _, key: _ } => none + D0DecidedFence { next: _, cause: _ } => none + } + match pair_serving_authority_transition(store: store, group: group, expected: pending, next: d0_decision_next(d: decision), transaction: transaction, lifecycle: Present { value: admitted.binding }, grant: term, entry_state: Present { value: digest }, actor: executor, at: at) { + AuthorityTransitionAppended { id: _, generation: g, next: _, placement: PlacementFinalizedAt { preparation: _, id: _ } } => + D0Settled { decision: decision, generation: g, receipt_path: if wr.success { path } else { join(["(local copy not written: ", wr.error, ")"], "") }, entry_state: digest } + AuthorityTransitionAppended { id: _, generation: g, next: _, placement: PlacementCleanupStillFencing { preparation: pr, cause: c } } => + D0PlacementStillFencing { decision: Present { value: decision }, generation: g, entry_state: Present { value: digest }, preparation: Present { value: pr }, cause: c } + AuthorityTransitionAppended { id: _, generation: g, next: _, placement: PlacementCleanupUnread { cause: c } } => + D0PlacementStillFencing { decision: Present { value: decision }, generation: g, entry_state: Present { value: digest }, preparation: none, cause: c } + AuthorityTransitionAppended { id: _, generation: g, next: _, placement: PlacementAbortedAt { preparation: pr, id: _ } } => + D0PlacementStillFencing { decision: Present { value: decision }, generation: g, entry_state: Present { value: digest }, preparation: Present { value: pr }, cause: "the authority landed but its preparation reads aborted -- the placement and the authority disagree; operator" } + AuthorityTransitionStale { expected: e, current: c, placement: pl } => D0Refused { cause: join(["the settling write did not land; the authority is as the log says: ", transition_text(t: AuthorityTransitionStale { expected: e, current: c, placement: pl })], "") } + AuthorityTransitionRefused { group: gr, step: st, reason: why, placement: pl } => D0Refused { cause: join(["the settling write did not land: ", transition_text(t: AuthorityTransitionRefused { group: gr, step: st, reason: why, placement: pl })], "") } + } + } + } + } +} + +// THE TRANSACTION OVER A SUPPLIED OBSERVER. Takes the admitted grant and a function that reads +// the fleet under the pending state; the wet entry supplies the fleet observer, its real-execution +// witness supplies a fixture. Active takes the first write -- after re-checking that the grant's +// host population is still the group's, since this is the last point before the authority moves; +// pending under the SAME transaction AND the same authorization binding resumes from the readings +// and restores, if it restores, the EXACT Active state the first write left (the log's `previous` +// for the pending state -- never the resting row, which may not be what was suspended); anything +// else refuses without a write. THE LEASE EPOCH IS MINTED FOR THE GENERATION THE READ SAW, and the +// first write is a transition AT THAT READ'S HEAD (pair_serving_authority_transition_at): a state +// that left and came back is the same value at another generation, and a value-only compare-and-set +// would land a pending state whose lease epoch is not the generation that landed -- a grant fence +// the genealogy then refuses, stranding the group. Head-exact, the landed generation is gen + 1. +fn d0_transaction(store: FabricStorageBinding, group: FabricGroup, current_hosts: List, admitted: AdmittedD0Grant, transaction: NonEmptyStr, executor: NonEmptyStr, at: EpochSecs, now: String, observe: fn(FabricGroup) -> D0Observation) -> D0Outcome { + match current_pair_serving_authority(store: store, group: group, at: at) { + CurrentAuthorityUnread { group: _, step: st, reason: why } => D0Refused { cause: join(["authority unread at ", st, ": ", why], "") } + CurrentAuthorityRead { authority: cur, head: head, generation: gen, grant: _, entry_state: _, previous: previous, admitted_by: admitted_by, transitions: _ } => + match cur { + PairServingActive { group: _, exact_realization: _ } => + match d0_population(subject: admitted.grant.subject, current: current_hosts) { + D0PopulationDiffers { granted: gr, current: cu } => + D0Refused { cause: join(["the grant's host population [", hosts_wire(hs: gr), "] is not the group's current population [", hosts_wire(hs: cu), "], so nothing is moved"], "") } + D0PopulationAgrees => { + let lease = d0_held_lease(group: group, transaction: transaction, executor: executor, generation: gen + 1) + let pending = SuspensionPendingReconciliation { group: group, hosts: admitted.grant.subject.hosts, transaction: transaction, authorization_binding: admitted.binding, lease: lease } + match pair_serving_authority_transition_at(store: store, group: group, expected: cur, expected_head: Present { value: head }, next: pending, transaction: transaction, lifecycle: Present { value: admitted.binding }, grant: none, entry_state: none, actor: executor, at: at) { + AuthorityTransitionAppended { id: admitting, generation: pending_generation, next: _, placement: PlacementFinalizedAt { preparation: _, id: _ } } => + d0_settle(store: store, pending: pending, head: HeadAt { id: admitting }, active: cur, lease: lease, admitted: admitted, admitting: admitting, pending_generation: pending_generation, transaction: transaction, executor: executor, at: at, now: now, declared: declared_pair_realization(group: group), observation: observe(group)) + AuthorityTransitionAppended { id: _, generation: pending_generation, next: _, placement: PlacementCleanupStillFencing { preparation: pr, cause: c } } => + D0PlacementStillFencing { decision: none, generation: pending_generation, entry_state: none, preparation: Present { value: pr }, cause: c } + AuthorityTransitionAppended { id: _, generation: pending_generation, next: _, placement: PlacementCleanupUnread { cause: c } } => + D0PlacementStillFencing { decision: none, generation: pending_generation, entry_state: none, preparation: none, cause: c } + AuthorityTransitionAppended { id: _, generation: pending_generation, next: _, placement: PlacementAbortedAt { preparation: pr, id: _ } } => + D0PlacementStillFencing { decision: none, generation: pending_generation, entry_state: none, preparation: Present { value: pr }, cause: "the placement and the authority disagree; operator" } + AuthorityTransitionStale { expected: e, current: c, placement: pl } => D0Refused { cause: join(["the transfer into suspension did not land: ", transition_text(t: AuthorityTransitionStale { expected: e, current: c, placement: pl })], "") } + AuthorityTransitionRefused { group: gr, step: st, reason: why, placement: pl } => D0Refused { cause: join(["the transfer into suspension did not land: ", transition_text(t: AuthorityTransitionRefused { group: gr, step: st, reason: why, placement: pl })], "") } + } + } + } + SuspensionPendingReconciliation { group: _, transaction: pending_tx, authorization_binding: pending_binding, lease: lease } => + if (pending_tx as String) != (transaction as String) { + D0Refused { cause: join(["the group is pending under transaction ", pending_tx as String, ", which only that transaction may settle"], "") } + } else if !content_hash_equal(left: pending_binding, right: admitted.binding) { + D0Refused { cause: "the group is pending under this transaction but under a different authorization (escalation, attempt, intent, subject or executor differ), so this run may not settle it" } + } else { + match admitted_by { + Absent => D0Refused { cause: "the group is pending but no transition admitted that state, so the admitting event cannot be named" } + Present { value: admitting } => + match previous { + Absent => D0Refused { cause: "the group is pending but the log carries no previous state for it, so the exact Active state it suspended cannot be restored" } + Present { value: left } => + match left { + PairServingActive { group: _, exact_realization: _ } => + d0_settle(store: store, pending: cur, head: head, active: left, lease: lease, admitted: admitted, admitting: admitting, pending_generation: gen, transaction: transaction, executor: executor, at: at, now: now, declared: declared_pair_realization(group: group), observation: observe(group)) + _ => D0Refused { cause: join(["the pending state was not entered from an Active authority (it left ", authority_wire(a: left), "), so this transaction cannot settle it"], "") } + } + } + } + } + _ => D0Refused { cause: join(["the authority is neither active nor pending under this transaction, so there is nothing to suspend or settle: ", authority_wire(a: cur)], "") } + } + } +} + +// ── THE RECOVERY: ONE JOIN OVER THE CLAIM AND THE AUTHORITY HISTORY ──────────────────────────── +// +// The claim and the authority are two durable facts; a run is a path through both and can die at +// any point on it. So a rerun does not ask "is the authority pending?" and infer the rest: it +// reads both and decides from the JOIN, pure over supplied readings, so every crash point has +// exactly one disposition and the witness drives each one. +// +// claim absent → start (decide, claim, first write) +// ClaimedBy(T), no transition by T, authority Active → first write never landed: perform it +// ClaimedBy(T), no transition by T, authority not Active → nothing to do under this consent: abort the claim +// ClaimedBy(T), authority Pending(T) → resume the readings and settle +// ClaimedBy(T), T's last transition landed a non-pending → settled; only the claim's terminal is owed +// ClaimedBy(T), T's last transition landed pending but the +// authority is not that pending state → another writer moved a pending state +// only T may move: fenced for an operator +// ClaimedBy(T), no transition by THIS lifecycle, authority +// pending under T's word → another authorization lifecycle of the same +// word wrote it → refuse; nothing is written. +// ClaimedBy(T), Pending(T) under this lifecycle, lease owned +// by another executor → refuse; the executor that wrote it resumes. +// D0 is same-executor-only for a PENDING +// lifecycle (no handoff is modeled); any +// executor may complete or abort a claim +// whose authority is not pending, since that +// mutates no host. +// ClaimedBy(other), CompletedBy, AbortedBy, unreadable → refuse; the consent is not this run's +// +// "No transition by T" is decided from the log's own record of every admitted transition, never +// from the current state: a completed T followed by another transaction's writes would otherwise +// read as "T never ran" and be started again on a spent consent. AND THE RECORD IS SELECTED BY +// THE LIFECYCLE, NOT THE TRANSACTION WORD: the word is a branded string nothing allocates +// uniquely, so a later escalation may reuse it, and a join by the word alone would complete the +// new consent from the old transaction's terminal. The lifecycle is the authorization binding +// (escalation, attempt, intent, subject, executor, expiry, claim generation), carried on every +// D0 transition; the dispatcher digests it from the held claim before asking the join. +type D0Recovery + = D0RecoverStart + | D0RecoverFirstWrite { claim_generation: CasGeneration } + | D0RecoverResume { claim_generation: CasGeneration } + | D0RecoverComplete { claim_generation: CasGeneration, settled: AuthorityTransitionRecord } + | D0RecoverAbort { claim_generation: CasGeneration, cause: String } + | D0RecoveryRefused { cause: String } + +fn d0_recovery(claim: D0ClaimReading, transaction: NonEmptyStr, lifecycle: ContentHash?, executor: NonEmptyStr, current: PairServingGroupAuthority, transitions: List) -> D0Recovery { + match claim { + D0ClaimUnreadable { reason: r } => D0RecoveryRefused { cause: r as String } + D0ClaimAbsent => D0RecoverStart + D0ClaimAt { state: s, generation: g } => + match s { + Unclaimed => D0RecoveryRefused { cause: "the claim slot records the grant unclaimed, which no transition writes; the slot needs an operator's inspection" } + CompletedBy { attempt: who, completed_at: when } => D0RecoveryRefused { cause: join(["the grant was completed by attempt ", who as String, " at ", when, "; consent is spent"], "") } + AbortedBy { attempt: who, aborted_at: _, cause: c } => D0RecoveryRefused { cause: join(["the grant was aborted by attempt ", who as String, ": ", c as String, "; a new decision needs a new escalation"], "") } + ClaimedBy { attempt: who, claimed_at: _ } => + if (who as String) != (transaction as String) { + D0RecoveryRefused { cause: join(["the grant is held by attempt ", who as String, ", not this transaction"], "") } + } else { + match lifecycle { + Absent => D0RecoveryRefused { cause: "the grant is held by this transaction but no admitted authorization binding was supplied, so its history on the log cannot be identified" } + Present { value: binding } => + match last(transitions_by(records: transitions, transaction: transaction, lifecycle: binding)) { + Absent => + match current { + PairServingActive { group: _, exact_realization: _ } => D0RecoverFirstWrite { claim_generation: g } + SuspensionPendingReconciliation { group: _, transaction: pt, authorization_binding: _, lease: l } => + if (pt as String) == (transaction as String) { + D0RecoveryRefused { cause: join(["the authority is pending under this transaction's word but under another authorization lifecycle (lease owner ", serialize_content_hash(hash: l.epoch.owner_fingerprint) as String, "); only that lifecycle may settle it, and this run neither aborts the claim nor moves the authority"], "") } + } else { + D0RecoverAbort { claim_generation: g, cause: join(["the claim is held but no write landed and the authority is pending under another transaction (", pt as String, "), so there is nothing this consent can do"], "") } + } + _ => D0RecoverAbort { claim_generation: g, cause: join(["the claim is held but no write landed and the authority is not Active (", authority_wire(a: current), "), so there is nothing this consent can do"], "") } + } + Present { value: mine } => + match mine.next { + SuspensionPendingReconciliation { group: _, transaction: _, authorization_binding: _, lease: _ } => + match current { + SuspensionPendingReconciliation { group: _, transaction: pt, authorization_binding: pb, lease: l } => + if (pt as String) != (transaction as String) { + D0RecoveryRefused { cause: join(["the authority is pending under ", pt as String, ", not this transaction"], "") } + } else if !content_hash_equal(left: pb, right: binding) { + D0RecoveryRefused { cause: "the authority is pending under this transaction's word but another authorization's binding; only that lifecycle may settle it" } + } else if !content_hash_equal(left: l.epoch.owner_fingerprint, right: content_hash_of_value(value: executor)) { + D0RecoveryRefused { cause: join(["the pending transaction is owned by another executor (lease owner ", serialize_content_hash(hash: l.epoch.owner_fingerprint) as String, "); a resume runs there, and no handoff is modeled -- the claim and the authority are left as they are"], "") } + } else { + D0RecoverResume { claim_generation: g } + } + _ => D0RecoveryRefused { cause: join(["this transaction's pending state was moved by another writer (the authority is ", authority_wire(a: current), "); the group needs an operator's disposition"], "") } + } + _ => D0RecoverComplete { claim_generation: g, settled: mine } + } + } + } + } + } + } +} + +fn d0_recovery_text(r: D0Recovery) -> String { + match r { + D0RecoverStart => "start" + D0RecoverFirstWrite { claim_generation: _ } => "recover: the claim is held and the first write never landed; performing it" + D0RecoverResume { claim_generation: _ } => "resume: the group is pending under this transaction" + D0RecoverComplete { claim_generation: _, settled: s } => join(["recover: this transaction already settled at ", s.id as String, " (", authority_wire(a: s.next), "); completing the claim"], "") + D0RecoverAbort { claim_generation: _, cause: c } => join(["recover: aborting the claim: ", c], "") + D0RecoveryRefused { cause: c } => join(["refused: ", c], "") + } +} + +// THE ENTRY. Arguments: the group's wire word, the transaction id, the authorization id. +// gunbc run --source-root dag --source-root src/v2 --entry dag/gunbc/spark/pair_serving_d0.dag \ +// --function pair_serving_d0_wet --arg group=group-a --arg transaction= --arg authorization= +// Refuses before any write unless the authorization resolves to an admitted grant -- which no +// producer yields today (resolve_d0_authorization). Exit 0 only on D0Eligible settled. +// THE RUN AS ONE AUTHORIZATION LIFECYCLE, dispatched by d0_recovery over the claim and the +// authority as read: a start decides permission, claims and writes; a recovery under a held claim +// resumes (claim found held, no re-decision) and writes what is still owed; the claim is completed +// on any settled terminal and aborted when the run refused before its first write. A grant +// admission failure at the start takes no claim, so there is nothing to abort; a settle that did +// not land leaves the claim held for the next rerun, which the join dispatches again. +fn d0_admission_text(a: D0GrantAdmission, group: String) -> String { + match a { + D0GrantAdmitted { admitted: _ } => "admitted" + D0GrantForAnotherGroup { granted: og } => join(["the grant authorizes ", fabric_group_wire(g: og) as String, ", not ", group], "") + D0GrantForAnotherPopulation { granted: gr, current: cu } => join(["the grant authorizes hosts [", hosts_wire(hs: gr), "] but ", group, " is currently [", hosts_wire(hs: cu), "]; the operator did not see this population"], "") + D0GrantIntentUnavailable => "the intent over the exact operation could not be digested (sha256sum unavailable), so the grant cannot be compared to it" + D0GrantRefused { cause: c } => join(["the authorization does not permit this operation: ", authorization_refusal_reason(cause: c)], "") + D0GrantClaimLost { holder: h } => join(["the grant is held by another attempt: ", h as String], "") + D0GrantClaimUndecided { reason: r } => join(["the grant's claim could not be decided: ", r as String], "") + D0GrantBindingUnavailable { claim: t } => join(["the authorization binding could not be digested", claim_terminal_text(t: t)], "") + } +} + +// A DRIFTED POPULATION ON A RESUME IS CARRIED INTO THE READINGS, NOT REFUSED AT THE DOOR: the +// pending state must reach a terminal under its original authorization, and the honest terminal +// when the hosts are no longer the ones the operator saw is a fence naming both populations. +fn d0_drift_observation(group: FabricGroup, granted: List, current: List) -> D0Observation { + D0Observation { + group: group, + incumbent: IncumbentRouteUnread { cause: join(["the group's host population changed after this transaction was authorized: authorized [", hosts_wire(hs: granted), "], current [", hosts_wire(hs: current), "]; the pending transaction is settled by fencing rather than read against hosts the operator never saw"], "") as NonEmptyStr }, + ranks: [] as List, + } +} + +fn d0_run_admitted(store: FabricStorageBinding, g: FabricGroup, current_hosts: List, admitted: AdmittedD0Grant, transaction: NonEmptyStr, executor: NonEmptyStr, escalation_id: NonEmptyStr, at: EpochSecs, now: String, claimed_here: Bool) -> ProcessExit { + match d0_population(subject: admitted.grant.subject, current: current_hosts) { + D0PopulationDiffers { granted: gr, current: cu } => + d0_settle_admitted(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: transaction, executor: executor, escalation_id: escalation_id, at: at, now: now, observe: fn(og) { d0_drift_observation(group: og, granted: gr, current: cu) }) + D0PopulationAgrees => + match prepare_fleet_ssh_agent_context(attempt_raw: pair_serving_d0_attempt_raw) { + FleetSshContextRefused { cause: c } => { + let terminal = if claimed_here { d0_terminate_claim(store: store, escalation_id: escalation_id, transaction: transaction, held: admitted.claim_generation, completed: false, cause: "no fleet ssh context before the first write", now: now) } else { D0ClaimNotAttempted { still_held_by: transaction } } + refuse(reason: join(["no fleet ssh context, so nothing is read and nothing is moved: ", c, claim_terminal_text(t: terminal)], "")) + } + FleetSshContextReady { context: context, receipt: _ } => + d0_settle_admitted(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: transaction, executor: executor, escalation_id: escalation_id, at: at, now: now, observe: fn(og) { observe_d0(context: context, group: og, declared: declared_pair_realization(group: og)) }) + } + } +} + +fn d0_settle_admitted(store: FabricStorageBinding, g: FabricGroup, current_hosts: List, admitted: AdmittedD0Grant, transaction: NonEmptyStr, executor: NonEmptyStr, escalation_id: NonEmptyStr, at: EpochSecs, now: String, observe: fn(FabricGroup) -> D0Observation) -> ProcessExit { + match placement_repair_pending(store: store, group: g, actor: executor, at: at) { + Present { value: PlacementAbortedAt { preparation: pr, id: _ } } => exit_failure(reason: join(["the group's pending placement preparation ", pr as String, " was aborted by the repair, which no run of this transaction does: an operator's disposition, so this run does not proceed on it"], "")) + Present { value: PlacementCleanupStillFencing { preparation: pr, cause: c } } => exit_failure(reason: join(["the group's pending placement preparation ", pr as String, " cannot be repaired by this run: ", c, "\n the authorization claim stays held"], "")) + Present { value: PlacementCleanupUnread { cause: c } } => exit_failure(reason: join(["the group's pending placement could not be read, so this run does not proceed: ", c], "")) + Absent => d0_settle_admitted_over(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: transaction, executor: executor, escalation_id: escalation_id, at: at, now: now, observe: observe) + Present { value: PlacementFinalizedAt { preparation: _, id: _ } } => d0_settle_admitted_over(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: transaction, executor: executor, escalation_id: escalation_id, at: at, now: now, observe: observe) + } +} + +// THE SAGA'S PENDING PREPARATION IS REPAIRED BEFORE THE TRANSACTION RUNS: a previous run of this +// transaction that died after its group append left a consumed, unfinalized preparation, which +// the join finalizes here; one no authority event consumed is the operator's, and the run refuses +// rather than sitting on it. +fn d0_settle_admitted_over(store: FabricStorageBinding, g: FabricGroup, current_hosts: List, admitted: AdmittedD0Grant, transaction: NonEmptyStr, executor: NonEmptyStr, escalation_id: NonEmptyStr, at: EpochSecs, now: String, observe: fn(FabricGroup) -> D0Observation) -> ProcessExit { + let outcome = d0_transaction(store: store, group: g, current_hosts: current_hosts, admitted: admitted, transaction: transaction, executor: executor, at: at, now: now, observe: observe) + match outcome { + D0Settled { decision: _, generation: _, receipt_path: _, entry_state: _ } => { + let terminal = d0_terminate_claim(store: store, escalation_id: escalation_id, transaction: transaction, held: admitted.claim_generation, completed: true, cause: "", now: now) + if d0_outcome_is_eligible(o: outcome) { + match terminal { + D0ClaimTerminated => ExitSuccess + _ => exit_failure(reason: join([d0_outcome_text(o: outcome), "\n settled, but the claim's terminal did not land:", claim_terminal_text(t: terminal)], "")) + } + } else { + exit_failure(reason: join([d0_outcome_text(o: outcome), claim_terminal_text(t: terminal)], "")) + } + } + D0PlacementStillFencing { decision: _, generation: _, entry_state: _, preparation: _, cause: _ } => exit_failure(reason: join([d0_outcome_text(o: outcome), "\n the authorization claim stays held; a rerun under this transaction finalizes the placement from the join and then completes"], "")) + D0Refused { cause: _ } => exit_failure(reason: join([d0_outcome_text(o: outcome), "\n the authorization claim stays held; a rerun under this transaction recovers from the claim and the log"], "")) + } +} + +fn claim_terminal_text(t: D0ClaimTerminal) -> String { + match t { + D0ClaimTerminated => "" + D0ClaimNotAttempted { still_held_by: who } => join(["\n the authorization claim stays held by ", who as String, "; a rerun under it recovers from the claim and the log"], "") + D0ClaimUnterminated { detail: d } => join(["\n the authorization claim was left unterminated: ", d], "") + } +} + +// A RESUME WHOSE GRANT IS NOT ADMITTED REFUSES WITH THE ADMISSION'S OWN CAUSE -- another +// population, another group, a scoped-authorization refusal, an undecidable claim -- before the +// recovery join, which would otherwise only see "no lifecycle" and report a digest that was never +// the problem. +// THE DISPATCH. The claim is read first; when it is held by this transaction the grant is resumed +// (no re-decision) so the lifecycle binding exists BEFORE the join is asked -- the join selects +// this lifecycle's history by it. Then each recovery arm has one consequence: +// Start → decide, claim, write (admit_d0_grant, then the transaction) +// FirstWrite → the population is re-checked against the current group: drift ABORTS the claim +// (nothing was written under it); agreement performs the first write +// Resume → the pending transaction finishes under its original authorization; drift is +// carried into the readings and fences rather than stranding the group +// Complete → the claim's terminal only; the authority is not touched +// Abort → the claim's terminal only +fn d0_dispatch(store: FabricStorageBinding, g: FabricGroup, group: String, current_hosts: List, scoped: ScopedAuthorization, tx: NonEmptyStr, esc: NonEmptyStr, executor: NonEmptyStr, at: EpochSecs, now: String, current: CurrentAuthority) -> ProcessExit { + match current { + CurrentAuthorityUnread { group: _, step: st, reason: why } => refuse(reason: join(["authority unread at ", st, ": ", why], "")) + CurrentAuthorityRead { authority: cur, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: recs } => { + let claim = d0_read_claim(store: store, escalation_id: esc) + let held_here = match claim { + D0ClaimAt { state: ClaimedBy { attempt: who, claimed_at: _ }, generation: _ } => (who as String) == (tx as String) + _ => false + } + let resumed = if held_here { resume_d0_grant(store: store, authorization: scoped, escalation_id: esc, group: g, transaction: tx, executor: executor) } else { D0GrantClaimLost { holder: "no claim held by this transaction" as NonEmptyStr } } + let lifecycle = match resumed { + D0GrantAdmitted { admitted: a } => Present { value: a.binding } + _ => none + } + let resume_refusal = if held_here { (match resumed { D0GrantAdmitted { admitted: _ } => none _ => Present { value: d0_admission_text(a: resumed, group: group) } }) } else { none } + match resume_refusal { + Present { value: why } => refuse(reason: join(["the claim is held by this transaction but its grant is not admitted on this resume: ", why], "")) + Absent => { + let recovery = d0_recovery(claim: claim, transaction: tx, lifecycle: lifecycle, executor: executor, current: cur, transitions: recs) + match recovery { + D0RecoveryRefused { cause: c } => refuse(reason: c) + D0RecoverStart => { + let admission = admit_d0_grant(store: store, authorization: scoped, escalation_id: esc, group: g, current_hosts: current_hosts, transaction: tx, executor: executor, at: at, now: now) + match admission { + D0GrantAdmitted { admitted: admitted } => d0_run_admitted(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: tx, executor: executor, escalation_id: esc, at: at, now: now, claimed_here: true) + _ => refuse(reason: d0_admission_text(a: admission, group: group)) + } + } + D0RecoverFirstWrite { claim_generation: cg } => + match resumed { + D0GrantAdmitted { admitted: admitted } => + match d0_population(subject: admitted.grant.subject, current: current_hosts) { + D0PopulationAgrees => d0_run_admitted(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: tx, executor: executor, escalation_id: esc, at: at, now: now, claimed_here: false) + D0PopulationDiffers { granted: gr, current: cu } => { + let cause = join(["the group's host population changed after the claim and before any write: authorized [", hosts_wire(hs: gr), "], current [", hosts_wire(hs: cu), "]"], "") + match d0_terminate_claim(store: store, escalation_id: esc, transaction: tx, held: cg, completed: false, cause: cause, now: now) { + D0ClaimTerminated => refuse(reason: join([cause, "; the claim is aborted and the authority was not moved"], "")) + other => refuse(reason: join([cause, claim_terminal_text(t: other)], "")) + } + } + } + _ => refuse(reason: join([d0_recovery_text(r: recovery), "; ", d0_admission_text(a: resumed, group: group)], "")) + } + D0RecoverResume { claim_generation: _ } => + match resumed { + D0GrantAdmitted { admitted: admitted } => d0_run_admitted(store: store, g: g, current_hosts: current_hosts, admitted: admitted, transaction: tx, executor: executor, escalation_id: esc, at: at, now: now, claimed_here: false) + _ => refuse(reason: join([d0_recovery_text(r: recovery), "; ", d0_admission_text(a: resumed, group: group)], "")) + } + D0RecoverComplete { claim_generation: cg, settled: st } => + match placement_finalize(store: store, consumed: preparation_consumption(store: store, preparation: st.placement.id), actor: executor, at: at) { + PlacementCleanupStillFencing { preparation: pr, cause: c } => exit_failure(reason: join([d0_recovery_text(r: recovery), "; the settling write's placement preparation ", pr as String, " is not finalized: ", c, "; the claim stays held until it is"], "")) + PlacementCleanupUnread { cause: c } => exit_failure(reason: join([d0_recovery_text(r: recovery), "; the settling write's placement preparation ", st.placement.id as String, " could not be read: ", c, "; the claim is left held for a rerun"], "")) + PlacementAbortedAt { preparation: pr, id: _ } => exit_failure(reason: join([d0_recovery_text(r: recovery), "; the settling write's placement preparation ", pr as String, " reads aborted -- the placement and the authority disagree; operator"], "")) + PlacementFinalizedAt { preparation: _, id: _ } => + match d0_terminate_claim(store: store, escalation_id: esc, transaction: tx, held: cg, completed: true, cause: "", now: now) { + D0ClaimTerminated => + match st.next { + SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => ExitSuccess + _ => exit_failure(reason: join([d0_recovery_text(r: recovery), "; the claim is completed and the authority was not moved by this run"], "")) + } + other => exit_failure(reason: join([d0_recovery_text(r: recovery), claim_terminal_text(t: other)], "")) + } + } + D0RecoverAbort { claim_generation: cg, cause: c } => + match d0_terminate_claim(store: store, escalation_id: esc, transaction: tx, held: cg, completed: false, cause: c, now: now) { + D0ClaimTerminated => refuse(reason: join([d0_recovery_text(r: recovery), "; the claim is aborted"], "")) + other => refuse(reason: join([d0_recovery_text(r: recovery), claim_terminal_text(t: other)], "")) + } + } + } + } + } + } +} + +// THE STORE'S WRITE WALL IS AN INPUT TO THIS DOOR. Every fact D0 writes -- the group's authority, +// host placement, the operator-consent slot -- lives behind the fabric DB's write door, and while +// gunbc.rung_drop fabric_storage_append_principal_unrefused stands that door admits any tailnet member, +// so a writer that never passed this door could move the same facts. The door therefore reads the +// drop's standing and refuses while it stands: Cut 0 and the escalation -> authorization producer +// cannot make D0 executable before the wall is restored, by construction rather than by sequence. +// +// TWO WALLS, NOT ONE. Retiring that drop restores OUTSIDER exclusion (a roster of fleet-writer +// principals); it does not make the store refuse a rostered writer that holds no D0 authorization +// for the operation, because the served door carries no partition, action, subject or grant. That +// second wall never existed under the git realization either, so it is not a rung drop but a +// missing construction, declared here on the door that needs it: the door refuses while EITHER +// stands, and the second is retired only by editing this row when the store admits writes per +// operation (the wall's trigger). +type StoreOperationWall + = StoreOperationWallMissing { trigger: String } + | StoreOperationWallRestored { by: String } + +data d0_store_operation_wall: StoreOperationWall = StoreOperationWallMissing { + trigger: "the fabric DB's served door admits a put/advance on a pair-serving-authority partition, host-placement or a D0 consent slot only with a D0 authorization the store verifies for that operation (subject, grant, attempt) -- SUFFICIENT FOR a rostered fleet writer without that authorization to be refused at the store", +} + +fn d0_store_write_wall_standing() -> String? { + match fabric_storage_append_principal_unrefused.standing { + Standing => Present { value: join(["the fabric DB's write door admits any tailnet member (gunbc.rung_drop fabric_storage_append_principal_unrefused stands; its trigger: ", fabric_storage_append_principal_unrefused_trigger(), "), so a writer that never passed this door could move the facts this transaction writes; D0 does not run until that drop is retired"], "") } + Retired { trigger_fired: _ } => + match d0_store_operation_wall { + StoreOperationWallRestored { by: _ } => none + StoreOperationWallMissing { trigger: t } => Present { value: join(["the fabric DB's write door admits any rostered fleet writer on every head without a D0 authorization for the operation (d0_store_operation_wall is missing; its trigger: ", t, "); D0 does not run until that wall is restored"], "") } + } + } +} + +fn fabric_storage_append_principal_unrefused_trigger() -> String { + match fabric_storage_append_principal_unrefused.declaration { + TypedDeclaration { previous: _, temporary: _, reason: _, population: _, restoration_trigger: t } => t + _ => "(see the drop's declaration)" + } +} + +fn pair_serving_d0_wet(group: String, transaction: String, authorization: String) -> ProcessExit + uses net: Network +{ + let now = probed_at_word(reading: clock_now_probed_at()) as String + match parse_fabric_group(wire: group) { + Absent => refuse(reason: join(["`", group, "` names no fabric group"], "")) + Present { value: g } => + if transaction == "" { + refuse(reason: "a transaction must be named") + } else if authorization == "" { + refuse(reason: "an authorization must be named") + } else { + match d0_store_write_wall_standing() { + Present { value: wall } => refuse(reason: wall) + Absent => + match resolve_d0_authorization(group: g, authorization: authorization) { + D0AuthorizationUnestablished { obligation: o } => refuse(reason: o as String) + D0AuthorizationResolved { authorization: scoped } => + match observe_executor_reach() { + ExecutorReachUnknown { cause: c } => refuse(reason: c) + ExecutorReachKnown { short_hostname: executor, path: _ } => + match event_log_store_for_host(short_hostname: executor) { + HostStoreRefused { detail: d } => refuse(reason: d) + HostStoreResolved { store: store, executor: _ } => + match now_epoch_seconds() { + Absent => refuse(reason: "the clock could not be read as epoch seconds") + Present { value: at } => + d0_dispatch(store: store, g: g, group: group, current_hosts: fabric_group_hosts(g: g), scoped: scoped, tx: transaction as NonEmptyStr, esc: authorization as NonEmptyStr, executor: executor as NonEmptyStr, at: at, now: now, current: current_pair_serving_authority(store: store, group: g, at: at)) + } + } + } + } + } + } + } +} diff --git a/dag/gunbc/spark/pair_serving_desired.dag b/dag/gunbc/spark/pair_serving_desired.dag index f9fd46a2ba3..83f183c2d74 100644 --- a/dag/gunbc/spark/pair_serving_desired.dag +++ b/dag/gunbc/spark/pair_serving_desired.dag @@ -234,7 +234,7 @@ data spark_pair_serving_declared_enforcers: List = [KernelOo // that reads as freedom. So those two are not freed, they are REFUSED, including to the very // PairServingUnitOn FabricGroupA subject that wanted them. srv7/srv8 meanwhile lose their only cause and // become uncommitted -- admissible to any serving subject and -// eligible as admit_unplaced_host effect targets, which gunbc.spark.v41_runtime_image_probe consumes to +// eligible as admit_unplaced_host_live effect targets, which gunbc.spark.v41_runtime_image_probe consumes to // pull an image and start a container. A lane that empties this row to "release" Group A for its own use // removes the modeled commitment that serving selection, build admission and probe admission consult to // avoid conflicting placement -- a PairServingUnitOn FabricGroupA subject derives its host population diff --git a/dag/gunbc/spark/released_baseline.dag b/dag/gunbc/spark/released_baseline.dag new file mode 100644 index 00000000000..ea1333cd56f --- /dev/null +++ b/dag/gunbc/spark/released_baseline.dag @@ -0,0 +1,47 @@ +module gunbc.spark.released_baseline + +import std.types { String, NonEmptyStr } +import v2.std.optional { Present, Absent } + +// ── THE STATE A TRANSACTION RETURNS THE HOSTS TO IS DECLARED BEFORE IT STARTS ────────────────── +// +// docs/plans/dsv41-cut-d-redesign.md §2 names three subjects that one word "baseline" had been +// standing for: the ENTRY STATE D0 records (never a return target), the RELEASED BASELINE SPEC a +// transaction declares before D1 runs (the target), and the RELEASED BASELINE RECEIPT D1 reads +// back (the evidence). This is the second. It is carried on the suspended authority as its +// cleanup contract -- gunbc.spark.pair_serving_authority SuspendedForAuthorizedSuccessor.cleanup +// -- so the state the hosts are owed is a fact of the authorization, not a word in a script. +// +// THE SPEC MUST CHOOSE RESERVED OR FREE, and an earlier draft claimed both. Under +// QuiescentReservedBaseline no rank process, engine, offer, seat or live device allocation +// survives, immutable verified artifacts may remain on local storage, and the exact four-host +// reservation REMAINS -- so an unrelated build or probe cannot occupy the hosts before D2. +// FleetReleasedBaseline gives the hosts back: D2 must reacquire them and may not assume the same +// four are available. They are two terminals with two names because they license different +// things, and a converger asked to reach "the baseline" without saying which would be choosing. +type ReleasedBaselineSpec + = QuiescentReservedBaseline + | FleetReleasedBaseline + +fn released_baseline_wire(s: ReleasedBaselineSpec) -> NonEmptyStr { + match s { + QuiescentReservedBaseline => "quiescent-reserved-baseline" as NonEmptyStr + FleetReleasedBaseline => "fleet-released-baseline" as NonEmptyStr + } +} + +fn parse_released_baseline(wire: String) -> ReleasedBaselineSpec? { + if wire == "quiescent-reserved-baseline" { Present { value: QuiescentReservedBaseline } } + else if wire == "fleet-released-baseline" { Present { value: FleetReleasedBaseline } } + else { none } +} + +// Whether the four-host reservation survives the transaction's cleanup. This is the projection +// gunbc.spark.host_commitment will consume when the D1 converger lands: under the reserved +// baseline the hosts stay committed after cleanup; under the fleet-released one they do not. +fn released_baseline_retains_reservation(s: ReleasedBaselineSpec) -> Bool { + match s { + QuiescentReservedBaseline => true + FleetReleasedBaseline => false + } +} diff --git a/dag/gunbc/spark/serving_incarnation_observe.dag b/dag/gunbc/spark/serving_incarnation_observe.dag index 9ddfd9fe3d7..957b1da90ac 100644 --- a/dag/gunbc/spark/serving_incarnation_observe.dag +++ b/dag/gunbc/spark/serving_incarnation_observe.dag @@ -1,6 +1,6 @@ module gunbc.spark.serving_incarnation_observe -import std.types { String, NonEmptyStr, Bool, Int, List, EpochSecs } +import std.types { String, NonEmptyStr, Bool, Int, List, EpochSecs, Port } import std.algebra { trim } import std.decimal { ExactDecimal, decimal_pow10 } import std.checked_arithmetic { checked_int_multiply, checked_int_add, CheckedIntReady, CheckedIntOverflow } @@ -47,8 +47,9 @@ import gunbc.systemctl_show_read { SystemdPropertyCaptured, SystemdPropertyBlank import gunbc.host_effect { HostEffectTransport, FleetSsh } import gunbc.host_operation_exec { HostOperation, HostOperationObserved, HostOperationRefused, host_operation_exec, - ProcfsReadStat, ProcfsReadPidStat, ProcfsReadPidCgroup, GetconfClockTicksPerSecond, Sha256SumFile, + ProcfsReadStat, ProcfsReadPidStat, ProcfsReadPidCgroup, ProcfsReadNetTcp, ProcfsReadNetTcp6, GetconfClockTicksPerSecond, Sha256SumFile, } +import extdeps.linux.proc_net_tcp { TcpSocketRow, ProcNetTcpParsed, ProcNetTcpUnparseable, proc_net_tcp_table, tcp_rows_listening_on } import gunbc.fleet_known_hosts_anchor { FleetSshExecutionContext, SshTarget } import gunbc.fleet_ssh_locus { fleet_locus_ssh_target } import gunbc.typed_argv_exec { typed_argv_exec_over_fleet_ssh, TypedArgvExecConverged, TypedArgvExecRefused } @@ -566,6 +567,48 @@ fn read_unit_identity(transport: HostEffectTransport, unit: NonEmptyStr) -> Unit } } +// ── ONE UNIT'S OCCUPANCY, WHETHER OR NOT IT RUNS ──────────────────────────────────────────────── +// +// read_unit_identity asks about a RUNNING unit and refuses an inactive one, because an +// incarnation is a launch. The Cut D D0 reconciliation asks a different question of the same +// three reads: is our unit INSTALLED on this rank at all, what bytes is it, and is it running -- +// where "installed and not running" is the expected answer on the readings the redesign records, +// not a refusal. So the fragment path decides installed, its digest decides which bytes, and the +// invocation decides running; only a read that could not be taken is unread. +type UnitOccupancy + = UnitInstalled { active: Bool, unit_digest: ContentHash } + | UnitNotInstalled + | UnitOccupancyUnread { refusal: ServingIncarnationRefusal } + +fn read_unit_occupancy(transport: HostEffectTransport, unit: NonEmptyStr) -> UnitOccupancy { + match read_unit_property(transport: transport, unit: unit, property: FragmentPathProperty) { + UnitTextRefused { cause: c } => + UnitOccupancyUnread { refusal: IncarnationUnitPropertyUnread { unit: unit, property: systemd_unit_property_wire(property: FragmentPathProperty), cause: c } } + UnitTextObserved { value: fragment_path } => + if fragment_path == "" { + UnitNotInstalled + } else { + match host_read_text(transport: transport, operation: Sha256SumFile { path: fragment_path as NonEmptyStr }) { + HostTextRefused { cause: c } => UnitOccupancyUnread { refusal: IncarnationUnitDigestUnread { unit: unit, cause: c } } + HostTextObserved { stdout: line } => + match sha256sum_line_digest(line: line) { + Absent => UnitOccupancyUnread { refusal: IncarnationUnitDigestUnread { unit: unit, cause: "sha256sum printed no 64-hex digest line" } } + Present { value: d } => + match sha256_digest_content_hash(digest: d) { + Absent => UnitOccupancyUnread { refusal: IncarnationUnitDigestUnread { unit: unit, cause: "the sha256sum output did not read as a sha256 content hash" } } + Present { value: digest } => + match read_unit_property(transport: transport, unit: unit, property: InvocationIDProperty) { + UnitTextRefused { cause: c } => + UnitOccupancyUnread { refusal: IncarnationUnitPropertyUnread { unit: unit, property: systemd_unit_property_wire(property: InvocationIDProperty), cause: c } } + UnitTextObserved { value: invocation_text } => UnitInstalled { active: invocation_text != "", unit_digest: digest } + } + } + } + } + } + } +} + // THE ENTRY. One head host, one group, the endpoint launch the harness just read from that group's // route, and the sealed fleet context; everything else is derived. The unit, container and engine // binary are the GROUP'S, read from gunbc.spark.serving_group_launch: group A's launch is the @@ -591,3 +634,41 @@ fn observe_endpoint_incarnation( ) } } + +// ── DOES ANYTHING LISTEN ON THE ENROLLED PORT, READ ON THE HOST ──────────────────────────────── +// +// The kernel's socket tables (/proc/net/tcp and /proc/net/tcp6, extdeps.linux.proc_net_tcp) are +// read on the host and every LISTEN row on the port is returned, whatever bound it: our unit, an +// orphan engine, a foreign process. This is the host-side half of an absence reading -- a front +// door that could not be connected says nothing about a listener the route did not reach, and the +// declared unit's invocation id says nothing about a process that is not the unit. Both tables +// are read because a v6 wildcard bind serves v4 too; one table unreadable is the reading unread. +// A listener in another network namespace (a container with its own netns) is not in these +// tables; the enrolled address is reached through the host's namespace, and a port published from +// a container appears there as the proxy's listener. +type EndpointListenerReading + = EndpointNoListener { port: Port } + | EndpointListening { port: Port, rows: List } + | EndpointListenerUnread { port: Port, cause: String } + +fn read_endpoint_listener(transport: HostEffectTransport, port: Port) -> EndpointListenerReading { + match host_read_text(transport: transport, operation: ProcfsReadNetTcp) { + HostTextRefused { cause: c } => EndpointListenerUnread { port: port, cause: join(["/proc/net/tcp: ", c], "") } + HostTextObserved { stdout: v4 } => + match proc_net_tcp_table(text: v4) { + ProcNetTcpUnparseable { line: l } => EndpointListenerUnread { port: port, cause: join(["/proc/net/tcp row could not be read: ", l], "") } + ProcNetTcpParsed { rows: rows4 } => + match host_read_text(transport: transport, operation: ProcfsReadNetTcp6) { + HostTextRefused { cause: c } => EndpointListenerUnread { port: port, cause: join(["/proc/net/tcp6: ", c], "") } + HostTextObserved { stdout: v6 } => + match proc_net_tcp_table(text: v6) { + ProcNetTcpUnparseable { line: l } => EndpointListenerUnread { port: port, cause: join(["/proc/net/tcp6 row could not be read: ", l], "") } + ProcNetTcpParsed { rows: rows6 } => { + let listening = tcp_rows_listening_on(rows: concat(rows4, rows6), port: port) + if length(listening) == 0 { EndpointNoListener { port: port } } else { EndpointListening { port: port, rows: listening } } + } + } + } + } + } +} diff --git a/dag/gunbc/spark/v41_runtime_image_probe.dag b/dag/gunbc/spark/v41_runtime_image_probe.dag index 28023e6b0c4..98f61788769 100644 --- a/dag/gunbc/spark/v41_runtime_image_probe.dag +++ b/dag/gunbc/spark/v41_runtime_image_probe.dag @@ -9,7 +9,7 @@ import std.content_hash { serialize_content_hash } import extdeps.filesystem.filesystem_io { Filesystem } import extdeps.exec.command { ArgvCommand, argv_words } import extdeps.docker.cli { - docker_run_ephemeral_command, docker_image_pull_command, docker_image_inspect_command, + docker_run_ephemeral_command, docker_image_pull_command, docker_image_inspect_command, DockerPsByAncestor, docker_image_inspect_id_from_stdout, DockerImageInspectId, DockerImageInspectIdUnreadable, docker_image_inspect_names_no_such_image, } @@ -57,7 +57,8 @@ import gunbc.spark.serving_runtime_capability_probe { runtime_kernel_capability_label, } import gunbc.spark.vllm_runtime_image_build { v41_gb10_recipe } -import gunbc.spark.host_commitment { SparkHostAdmitted, SparkHostRefused, admit_unplaced_host } +import gunbc.spark.host_commitment { SparkHostAdmitted, SparkHostRefused, admit_unplaced_host_live, release_host_effect_live } +import gunbc.spark.host_effect_quiescence { HostEffectResidueSpec } // ASKING THE BUILT IMAGE WHAT IT CONTAINS, INSIDE IT, ON THE ACCELERATOR IT WILL RUN ON. // @@ -700,6 +701,18 @@ fn v41_probe_ci_body(at: String, host: NonEmptyStr, reference: NonEmptyStr, outc // is the part that is genuinely this lane's. data v41_probe_host_purpose: NonEmptyStr = "a published-image probe, which pulls a 9.6 GB image and starts a container" as NonEmptyStr +// THE BOUND ON THE PROBE'S HOST-EFFECT CLAIM: a pull and a container start, bounded by the CI +// step's own hour; an unreleased claim reads OVERDUE after it and still fences until the host is +// observed quiet. +data v41_probe_host_effect_term: Second = second(count: 3600) + +// WHAT THE PROBE LEAVES IF IT DID NOT FINISH: the pull, the inspect and the run all carry the image +// reference on their command line, and the run starts an unnamed container FROM that reference, +// which `docker ps --filter ancestor=` finds while it runs. +fn v41_probe_host_residue(reference: NonEmptyStr) -> HostEffectResidueSpec { + HostEffectResidueSpec { process_pattern: reference, container: Present { value: DockerPsByAncestor { image: reference } } } +} + fn v41_published_image_probe_ci_wet() -> ProcessExit uses net: Network { @@ -709,9 +722,10 @@ fn v41_published_image_probe_ci_wet() -> ProcessExit SparkConvergeTargetAllObserveOnly => exit_failure(reason: "SPARK_CONVERGE_TARGET is unset: this mode pulls an image and starts a container on ONE host, so it refuses rather than choosing one") SparkConvergeTargetSingle { host: target_host } => - match admit_unplaced_host(raw: target_host as String, purpose: v41_probe_host_purpose) { + match admit_unplaced_host_live(term: v41_probe_host_effect_term, raw: target_host as String, purpose: v41_probe_host_purpose, residue: v41_probe_host_residue(reference: v41_published_arm64_reference())) { SparkHostRefused { cause: why } => exit_failure(reason: why as String) - SparkHostAdmitted { host: _ } => + SparkHostAdmitted { host: _, claim: claim } => + release_host_effect_live(handle: claim, exit: { match observe_executor_reach() { ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["v41_published_image_probe_ci: ", c], "")) ExecutorReachKnown { short_hostname: executor, path: path } => @@ -763,7 +777,7 @@ fn v41_published_image_probe_ci_wet() -> ProcessExit } } } - } + }}) } } } diff --git a/dag/gunbc/spark/v41_source_patch_converge_wet.dag b/dag/gunbc/spark/v41_source_patch_converge_wet.dag index d0a90aa6bfc..4d61edba935 100644 --- a/dag/gunbc/spark/v41_source_patch_converge_wet.dag +++ b/dag/gunbc/spark/v41_source_patch_converge_wet.dag @@ -18,7 +18,7 @@ import gunbc.fleet_ssh_credential_verify { fleet_ssh_credential_verification_refusal_reason, } import gunbc.fleet_multi_principal_probe { probe_agent_credential_verification } -import gunbc.spark.host_commitment { SparkHostAdmitted, SparkHostRefused, admit_unplaced_host } +import gunbc.spark.host_commitment { SparkHostAdmitted, SparkHostRefused, admit_unplaced_host_live, release_host_effect_live } import gunbc.spark.managed_access_bootstrap { spark_managed_executor_login } import gunbc.spark.fabric_reach { ReachManagementLan, spark_reach_endpoint, spark_reach_path_wire } import gunbc.spark.v41_runtime_candidate { @@ -26,7 +26,7 @@ import gunbc.spark.v41_runtime_candidate { v41_storage_backed_patch_population, v41_candidate_recipe, } import gunbc.spark.vllm_runtime_image_build { - vllm_build_host_purpose, vllm_source_dir, vllm_build_root, + vllm_build_host_purpose, vllm_source_dir, vllm_build_root, vllm_build_host_residue, } import gunbc.spark.v41_source_patch_converge { converge_vllm_source_with_patches, @@ -54,9 +54,10 @@ fn v41_patched_source_acquire_wet(host: String) -> ProcessExit V41PatchesUnestablished { obligation: o } => exit_failure(reason: o as String) V41PatchesObserved { patches: ps, partition: _ } => { let now = clock_now_probed_at() - match admit_unplaced_host(raw: host, purpose: vllm_build_host_purpose) { + match admit_unplaced_host_live(term: vllm_build_host_effect_term, raw: host, purpose: vllm_build_host_purpose, residue: vllm_build_host_residue()) { SparkHostRefused { cause: why } => exit_failure(reason: why as String) - SparkHostAdmitted { host: build_host } => + SparkHostAdmitted { host: build_host, claim: claim } => + release_host_effect_live(handle: claim, exit: { match fleet_ssh_attempt_identity(raw: v41_patched_source_acquire_attempt_raw) { Absent => exit_failure(reason: "attempt identity is not a safe path segment") Present { value: attempt } => @@ -95,7 +96,7 @@ fn v41_patched_source_acquire_wet(host: String) -> ProcessExit } } } - } + }}) } } } diff --git a/dag/gunbc/spark/vllm_runtime_image_build.dag b/dag/gunbc/spark/vllm_runtime_image_build.dag index 91989904747..6f54fd368fc 100644 --- a/dag/gunbc/spark/vllm_runtime_image_build.dag +++ b/dag/gunbc/spark/vllm_runtime_image_build.dag @@ -5,12 +5,13 @@ import std.process { ProcessExit, ExitSuccess, exit_failure } import std.resources { Network } import std.algebra { trim } import std.nat { Nat } -import std.measure { HardwareThreadCount, hardware_thread_count, hardware_thread_count_value } +import std.measure { HardwareThreadCount, hardware_thread_count, hardware_thread_count_value, Second, second } import std.content_hash { ContentHash, content_hash_atom, serialize_content_hash } import std.artifact_store { ArtifactRealizationInput, artifact_realization_digest } import product.placement_supply { HostIdentity, host_identity_eq } import gunbc.spark.fabric_switch_observed { fabric_lane_for_host } -import gunbc.spark.host_commitment { SparkHostAdmitted, SparkHostRefused, admit_unplaced_host, spark_unplaced_hosts } +import gunbc.spark.host_commitment { SparkHostAdmitted, SparkHostRefused, admit_unplaced_host_live, release_host_effect_live, spark_unplaced_hosts_of, spark_host_standings_live, CurrentStandingsRead, CurrentStandingsUnread } +import gunbc.spark.host_effect_quiescence { HostEffectResidueSpec } import extdeps.filesystem.filesystem_io { Filesystem } import extdeps.exec.command { ArgvCommand, argv_command, argv_words } import extdeps.docker { DockerMount, BindMount } @@ -208,6 +209,19 @@ import gunbc.typed_argv_exec { // which is the one thing the commitment authority cannot know and the refusal has to say. data vllm_build_host_purpose: NonEmptyStr = "a vLLM source build, which occupies the host for the better part of an hour" as NonEmptyStr +// THE BOUND ON THE BUILD'S HOST-EFFECT CLAIM: the one executed cold build took about 75 minutes; +// three hours is the term after which an unreleased claim (a crashed lane) reads OVERDUE. It +// still fences until released: the release is the observation that the host is quiet. +data vllm_build_host_effect_term: Second = second(count: 10800) + +// WHAT THE BUILD LEAVES IF IT DID NOT FINISH: every leg -- the clone, the fetch, the checkout, the +// docker build with its --file and context under the source dir -- carries the source directory +// on its command line, so a process still holding it is the residue; the build starts no +// container of its own. +fn vllm_build_host_residue() -> HostEffectResidueSpec { + HostEffectResidueSpec { process_pattern: vllm_source_dir(), container: none } +} + // THE BUILD ROOT IS DERIVED FROM THE PRINCIPAL THAT OWNS IT, NOT SPELLED. An earlier revision put it // under the OPERATOR's home while the legs authenticate as the managed executor -- a path belonging to // one account written into work performed by another, which is a permission error waiting at the far @@ -951,9 +965,10 @@ fn vllm_runtime_image_build_entry(r: VllmContainerBuildRecipe) -> ProcessExit SparkConvergeTargetAllObserveOnly => exit_failure(reason: "SPARK_CONVERGE_TARGET is unset: a build occupies one host for the better part of an hour, so it refuses rather than choosing one") SparkConvergeTargetSingle { host: build_host } => - match admit_unplaced_host(raw: build_host as String, purpose: vllm_build_host_purpose) { + match admit_unplaced_host_live(term: vllm_build_host_effect_term, raw: build_host as String, purpose: vllm_build_host_purpose, residue: vllm_build_host_residue()) { SparkHostRefused { cause: why } => exit_failure(reason: why as String) - SparkHostAdmitted { host: _ } => + SparkHostAdmitted { host: _, claim: claim } => + release_host_effect_live(handle: claim, exit: { match observe_executor_reach() { ExecutorReachUnknown { cause: c } => exit_failure(reason: join(["vllm_runtime_image_build_ci: ", c], "")) ExecutorReachKnown { short_hostname: executor, path: path } => @@ -1023,7 +1038,7 @@ fn vllm_runtime_image_build_entry(r: VllmContainerBuildRecipe) -> ProcessExit } } } - } + }}) } } } @@ -1074,9 +1089,10 @@ fn vllm_source_acquire_wet(host: String) -> ProcessExit uses net: Network { let now = clock_now_probed_at() - match admit_unplaced_host(raw: host, purpose: vllm_build_host_purpose) { + match admit_unplaced_host_live(term: vllm_build_host_effect_term, raw: host, purpose: vllm_build_host_purpose, residue: vllm_build_host_residue()) { SparkHostRefused { cause: why } => exit_failure(reason: why as String) - SparkHostAdmitted { host: build_host } => + SparkHostAdmitted { host: build_host, claim: claim } => + release_host_effect_live(handle: claim, exit: { match fleet_ssh_attempt_identity(raw: vllm_source_acquire_attempt_raw) { Absent => exit_failure(reason: "attempt identity is not a safe path segment") Present { value: attempt } => @@ -1114,7 +1130,7 @@ fn vllm_source_acquire_wet(host: String) -> ProcessExit } } } - } + }}) } } @@ -1655,8 +1671,11 @@ fn spark_build_host_reachability_wet() -> ProcessExit let materialization = materialize_fleet_known_hosts_anchor(attempt: attempt) match fleet_ssh_execution_context_of(outcome: materialization, credential: binding) { Absent => exit_failure(reason: join(["trust could not be established, so no leg ran: ", known_hosts_materialization_receipt(outcome: materialization)], "")) - Present { value: context } => { - let readings = map(spark_unplaced_hosts, h => probe_spark_host_reachability(context: context, host: h)) + Present { value: context } => + match spark_host_standings_live() { + CurrentStandingsUnread { cause: c } => exit_failure(reason: join(["spark reachability: the admissible build hosts cannot be derived: ", c as String], "")) + CurrentStandingsRead { standings: standings } => { + let readings = map(spark_unplaced_hosts_of(standings: standings), h => probe_spark_host_reachability(context: context, host: h)) let body = join([ join(["spark build-host reachability probed_at=", probed_at_word(reading: now) as String, " question=short hostname only\n"], ""), join(["credential ", fleet_ssh_credential_locus_label(binding: binding), " public_material_identity=", serialize_content_hash(hash: fleet_ssh_credential_identity(binding: binding)) as String, "\n"], ""), @@ -1674,7 +1693,8 @@ fn spark_build_host_reachability_wet() -> ProcessExit exit_failure(reason: join(["not every admissible build host answered:\n", body], "")) } } - } + } + } } } } diff --git a/dag/std/content_hash.dag b/dag/std/content_hash.dag index c13792c9195..11215e47c51 100644 --- a/dag/std/content_hash.dag +++ b/dag/std/content_hash.dag @@ -198,6 +198,21 @@ type ContentHashComparison // (membership value_eq) match explicitly on the outcome; integrity admission maps // CrossFamilyIncomparable to typed refusal causes rather than encoding dispatch locally. +// THE ONE BOOL OVER THE UNION, IN THE REFUSING DIRECTION. compare_content_hash keeps the +// cross-family arm distinct because it is a decision; this is that decision made once, for the +// consumers whose question is "may these be treated as the same digest" -- two digests from +// different families are never the same, so incomparable answers false. A consumer that must +// tell incomparable from different (an integrity admission that reports which) matches +// compare_content_hash itself. It was authored in gunbc.machine_intake.subject and then re-minted +// twice in gunbc.spark (review 67905); one home, here beside the comparison it folds. +fn content_hash_equal(left: ContentHash, right: ContentHash) -> Bool { + match compare_content_hash(left: left, right: right) { + ContentHashEqual => true + ContentHashDifferent => false + ContentHashCrossFamilyIncomparable => false + } +} + fn compare_content_hash(left: ContentHash, right: ContentHash) -> ContentHashComparison { match left { Fnv1a64(left_structural) => @@ -260,6 +275,56 @@ fn serialize_content_hash(hash: ContentHash) -> NonEmptyStr { } } +// THE INVERSE OF serialize_content_hash, AT THE SAME HOME. It stood in std.materialization_object +// as parse_recorded_store_key, named for one consumer; the pair-serving authority log is a second +// consumer of the same wire, so the inverse lives beside the grammar it inverts (DESIGN §4: one +// grammar read in both directions). The round-trip check is what makes it exact: a candidate that +// does not re-serialize to the text it was read from is not that text's hash. +fn parse_content_hash_candidate(wire: String) -> ContentHash? { + if starts_with(s: wire, prefix: "sha256:") { + match get(xs: split(s: wire, delimiter: "sha256:"), index: 1) { + Absent => none + Present { value: hex } => + match sha256_hex_digest(hex: hex) { + Absent => none + Present { value: d } => Present { value: as_content_hash_cryptographic(digest: d) } + } + } + } else if starts_with(s: wire, prefix: "sha512:") { + match get(xs: split(s: wire, delimiter: "sha512:"), index: 1) { + Absent => none + Present { value: hex } => + match sha512_hex_digest(hex: hex) { + Absent => none + Present { value: d } => Present { value: as_content_hash_sha512(digest: d) } + } + } + } else if content_hash_validate_lower_hex_length(text: wire, expected_hex_digits: 40) { + match sha1_hex_digest(hex: wire) { + Absent => none + Present { value: d } => Present { value: as_content_hash_sha1(digest: d) } + } + } else { + content_hash_from_structural_digest(digest: wire) + } +} + +// RAW HEX IS TWO FAMILIES ON THE WIRE, TOLD APART BY LENGTH. serialize_content_hash writes a +// Sha1Hash and a Fnv1a64 both as bare hex -- forty digits and sixteen -- so the inverse reads +// forty as SHA-1 (a git object id is one) and sixteen as the structural fingerprint; the +// round-trip check below is what makes a wrong length a refusal rather than a guess. +fn parse_content_hash(wire: String) -> ContentHash? { + match parse_content_hash_candidate(wire: wire) { + Absent => none + Present { value: parsed } => + if (serialize_content_hash(hash: parsed) as String) == wire { + Present { value: parsed } + } else { + none + } + } +} + fn sha256_digest_wire_form(digest: Sha256Digest) -> NonEmptyStr { join(["sha256:", digest.hex as String], "") as NonEmptyStr } diff --git a/dag/std/materialization_object.dag b/dag/std/materialization_object.dag index 10e7ac031f2..d28e969d502 100644 --- a/dag/std/materialization_object.dag +++ b/dag/std/materialization_object.dag @@ -3,6 +3,7 @@ module std.materialization_object import std.types { Bool, Int, List, String, NonEmptyStr } import std.content_hash { ContentHash, + parse_content_hash, Fnv1a64Structural, content_hash_atom, content_hash_from_structural_digest, @@ -271,42 +272,6 @@ fn store_object_line(line: String) -> StoreObjectLine { } } -fn parse_recorded_store_key_candidate(key_text: String) -> ContentHash? { - if starts_with(s: key_text, prefix: "sha256:") { - match get(xs: split(s: key_text, delimiter: "sha256:"), index: 1) { - Absent => none - Present { value: hex } => - match sha256_hex_digest(hex: hex) { - Absent => none - Present { value: d } => Present { value: as_content_hash_cryptographic(digest: d) } - } - } - } else if starts_with(s: key_text, prefix: "sha512:") { - match get(xs: split(s: key_text, delimiter: "sha512:"), index: 1) { - Absent => none - Present { value: hex } => - match sha512_hex_digest(hex: hex) { - Absent => none - Present { value: d } => Present { value: as_content_hash_sha512(digest: d) } - } - } - } else { - content_hash_from_structural_digest(digest: key_text) - } -} - -fn parse_recorded_store_key(key_text: String) -> ContentHash? { - match parse_recorded_store_key_candidate(key_text: key_text) { - Absent => none - Present { value: parsed } => - if (serialize_content_hash(hash: parsed) as String) == key_text { - Present { value: parsed } - } else { - none - } - } -} - type StoreObjectDecode = StoreObjectDecoded { manifest: ArtifactManifest, @@ -408,7 +373,7 @@ fn decode_store_object(content: String) -> StoreObjectDecode { match store_object_header_value(lines: lines, index: 2, tag: "request_key") { Absent => StoreObjectMalformed { cause: "line 3 is not a `request_key ` header" } Present { value: key_text } => - match parse_recorded_store_key(key_text: key_text) { + match parse_content_hash(wire: key_text) { Absent => StoreObjectMalformed { cause: concat("the recorded request key is not a ContentHash wire form: ", key_text) } Present { value: key } => match store_object_header_value(lines: lines, index: 3, tag: "evaluation_preimage") { diff --git a/dag/std/materialization_provider.dag b/dag/std/materialization_provider.dag index 9f8aa3caefb..5c1938ee73d 100644 --- a/dag/std/materialization_provider.dag +++ b/dag/std/materialization_provider.dag @@ -354,7 +354,7 @@ type EvaluationIdentityCollisionDisposition = FnvBucketPreimageVerified { restoration_trigger: NonEmptyStr } data evaluation_identity_fnv_store_preimage_exception: EvaluationIdentityCollisionDisposition = FnvBucketPreimageVerified { - restoration_trigger: "parse_recorded_store_key (algorithm-qualified ContentHash wire, inverse of serialize_content_hash) round-trips Sha256Family and Sha512Family object headers, and every persistent consumer of EvaluationStoreAddress — store_object_name, store_lookup_decide, store_commit_prepare, local_store_lookup, provider_serve PersistedProbe — derives EvaluationIdentity from evaluation_bucket under that family for PersistentStoreScope and CrossRunScope" + restoration_trigger: "std.content_hash parse_content_hash (algorithm-qualified ContentHash wire, inverse of serialize_content_hash) round-trips Sha256Family and Sha512Family object headers, and every persistent consumer of EvaluationStoreAddress — store_object_name, store_lookup_decide, store_commit_prepare, local_store_lookup, provider_serve PersistedProbe — derives EvaluationIdentity from evaluation_bucket under that family for PersistentStoreScope and CrossRunScope" } type EvaluationStoreAddressStanding diff --git a/dag/std/scoped_authorization.dag b/dag/std/scoped_authorization.dag index 6adbc87f3a1..93cebefc32e 100644 --- a/dag/std/scoped_authorization.dag +++ b/dag/std/scoped_authorization.dag @@ -1,6 +1,7 @@ module std.scoped_authorization -import std.types { NonEmptyStr, String, Bool, Timestamp } +import std.types { NonEmptyStr, String, Bool, Timestamp, List } +import v2.std.optional { Present, Absent } import std.content_hash { ContentHash, ContentHashComparison, @@ -470,6 +471,65 @@ type AuthorizationClaimOutcome | ClaimLost { holder: NonEmptyStr } | ClaimUndecided { reason: NonEmptyStr } +// THE EXACT INVERSE OF THE SERIALIZER, for a reader that must know WHO holds a slot rather than +// only that it lost to someone. A transaction recovering after a crash reads its grant's slot and +// has to decide from the state found there -- claimed by this attempt, claimed by another, +// completed, aborted -- and each answer is a different remedy, so the bytes are decoded into +// the same closed type they were written from and a payload that inhabits no arm is Absent, +// never a guessed state. The LOST arm of authorization_claim_outcome still carries the raw +// winner's payload: a loser needs no decision from it, only the report. +fn parse_authorization_claim_state(payload: NonEmptyStr) -> AuthorizationClaimState? { + let parts = split(s: payload as String, delimiter: "\t") + match get(xs: parts, index: 0) { + Absent => none + Present { value: word } => + if word == "unclaimed" { + if length(parts) == 1 { Present { value: Unclaimed } } else { none } + } else if word == "claimed-by" { + claim_state_two_fields(parts: parts, completed: false) + } else if word == "completed-by" { + claim_state_two_fields(parts: parts, completed: true) + } else if word == "aborted-by" { + if length(parts) != 4 { none } else { + match get(xs: parts, index: 1) { + Absent => none + Present { value: who } => + match get(xs: parts, index: 2) { + Absent => none + Present { value: when } => + match get(xs: parts, index: 3) { + Absent => none + Present { value: why } => + if who == "" || why == "" { none } else { + Present { value: AbortedBy { attempt: (who as NonEmptyStr) as AttemptIdentity, aborted_at: when, cause: why as NonEmptyStr } } + } + } + } + } + } + } else { + none + } + } +} + +fn claim_state_two_fields(parts: List, completed: Bool) -> AuthorizationClaimState? { + if length(parts) != 3 { none } else { + match get(xs: parts, index: 1) { + Absent => none + Present { value: who } => + match get(xs: parts, index: 2) { + Absent => none + Present { value: when } => + if who == "" { none } else { + let attempt = (who as NonEmptyStr) as AttemptIdentity + Present { value: if completed { CompletedBy { attempt: attempt, completed_at: when } } else { ClaimedBy { attempt: attempt, claimed_at: when } } } + } + } + } + } +} + // THE HELD STATE IS THE ONE WE PROPOSED, not one decoded back out of the slot. A committed write // is proof the store took our bytes, so re-deriving the state from them would be a round trip // proving the serializer agrees with itself. The LOST arm carries the winner's raw payload rather diff --git a/dag/test/claim/capacity_lease_chain_witness_test.dag b/dag/test/claim/capacity_lease_chain_witness_test.dag index 80d34320318..825874a4632 100644 --- a/dag/test/claim/capacity_lease_chain_witness_test.dag +++ b/dag/test/claim/capacity_lease_chain_witness_test.dag @@ -7,12 +7,13 @@ import product.capacity.pool { Pool, NoReplenishment, pool_of, pool_reading_at, import product.capacity.event_chain { PartitionId, EventId, ChainEvent, ChainEnvelope, HeadExpectation, HeadAbsent, HeadAt, AppendDecision, AppendAdmitted, AppendStale, decide_append, chain_from_head, ChainWalk, ChainWalked, ChainIncomplete, ChainBudgetExhausted, + EventDecoded, EventUndecodable, } import product.capacity.lease { LeaseFence, lease_fence_of, fence_verdict, FenceVerdict, FenceCurrent, FenceObsolete, FenceForeignGrant, LeasePolicy, LeaseGrant, lease_grant, ReleaseLaw, FencedResource, QuiescenceRequired, QuiescenceFact, Quiescent, QuiescenceUnobserved, CapacityAfterExpiry, LeaseNotExpired, CapacityFreed, CapacityHeldPendingQuiescence, capacity_after_expiry } import product.capacity.pool_events { PoolEvent, PoolAcquired, PoolSettled, PoolReleased, PoolLapsed, PoolUpstreamObserved, PoolResetObserved, PoolFold, PoolFolded, PoolFoldRefused, pool_fold, - SeatRequest, SeatProposal, SeatProposed, SeatRefused, propose_acquire, pool_event_wire_text, pool_event_decode, PoolEventDecode, PoolEventDecoded, PoolEventUndecodable, + SeatRequest, SeatProposal, SeatProposed, SeatRefused, propose_acquire, pool_event_wire_text, pool_event_decode, grant_from_admission, } @@ -169,11 +170,11 @@ test fn every_pool_event_kind_round_trips_through_its_wire_and_junk_is_refused() all(kinds, k => { let env = w_env(id: "x", parent: "p", payload: k, at: 42) match pool_event_decode(text: pool_event_wire_text(event: env.event)) { - PoolEventDecoded { event: e } => pool_event_wire_text(event: e) == pool_event_wire_text(event: env.event) && e.recorded_at == 42 - PoolEventUndecodable { reason: _ } => false + EventDecoded { event: e } => pool_event_wire_text(event: e) == pool_event_wire_text(event: env.event) && e.recorded_at == 42 + EventUndecodable { reason: _ } => false } }) - && (match pool_event_decode(text: "{\"schema\":\"nope\"}") { PoolEventUndecodable { reason: _ } => true PoolEventDecoded { event: _ } => false }) + && (match pool_event_decode(text: "{\"schema\":\"nope\"}") { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) } // A LOWERED CEILING DOES NOT INVALIDATE HISTORY THAT WAS ADMITTED UNDER THE OLD ONE. diff --git a/dag/test/claim/durable_cas_fabric_storage_real_execution_witness_test.dag b/dag/test/claim/durable_cas_fabric_storage_real_execution_witness_test.dag new file mode 100644 index 00000000000..4a9a7625810 --- /dev/null +++ b/dag/test/claim/durable_cas_fabric_storage_real_execution_witness_test.dag @@ -0,0 +1,151 @@ +module test.claim.durable_cas_fabric_storage_real_execution + +import std.logic { Bool } +import std.types { String, NonEmptyStr, FilePath } +import v2.std.optional { Present, Absent } +import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } +import extdeps.shell +import std.content_hash { content_hash_of_value } +import std.durable_compare_and_set { + ExpectSlotAbsent, ExpectSlotGeneration, CasCommitted, CasPreconditionFailed, CasStoreRefused, + CasObservedReadable, CasObservedUnreadable, CasReadableAbsent, CasReadablePresent, cas_attempt, cas_generation_count, +} +import gunbc.durable_cas_file_store { CasAttemptAdmitted, admit_cas_attempt, VerifiedCasAttempt } +import gunbc.fabric_storage_client { FabricStorageBinding, FabricStorageLocalFiles } +import gunbc.fabric_storage_file_store { fabric_storage_file_root, fabric_storage_file_root_ensure, FabricStorageRootReady } +import gunbc.durable_cas_fabric_storage { fabric_storage_compare_and_set, fabric_storage_observe_cas_slot, cas_slot_head } +import gunbc.fabric_storage_client { fabric_storage_head, fabric_storage_put, fabric_storage_advance } +import std.fabric_storage { FabricObject, FabricObjectRef, fabric_object_ref_eq, FabricHeadObserved, FabricHeadAt, ExpectHeadAt, FabricObjectStored, FabricHeadMoved } +import std.types { List } + +data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly + +// THE COMPARE-AND-SET OVER A FABRIC-DB HEAD, BY REAL EXECUTION against a file-backed store in a +// temporary directory, through TWO bindings to it -- the shape two executors have (one placed +// store, every executor reaching the same instance). Established here and nowhere hermetic: an +// absent slot reads absent through both; a create-if-absent commits generation 1 and the other +// binding reads it back at that generation with those bytes; a second create-if-absent through the +// other binding is a precondition failure carrying the committed version, not a second +// generation 1; an update expecting generation 1 commits generation 2; an update expecting the +// stale generation 1 after that fails naming generation 2. + +fn dcel_with_temp_dir(scenario: fn(FilePath) -> Bool) -> Bool { + let dir = shell.Mktemp.Dir() + let observed = scenario(dir.path) + let cleaned = shell.Remove.RecursiveForce(path: dir.path) + dir.success && observed && cleaned.success +} + +fn dcel_store(dir: FilePath) -> FabricStorageBinding { + FabricStorageLocalFiles { root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr) } +} + +fn dcel_store_ready(dir: FilePath) -> Bool { + match fabric_storage_file_root_ensure(root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr)) { FabricStorageRootReady => true _ => false } +} + +fn dcel_verified(key: NonEmptyStr, expected_generation: Int, value: NonEmptyStr) -> VerifiedCasAttempt? { + let expected = if expected_generation == 0 { ExpectSlotAbsent } else { ExpectSlotGeneration { generation: expected_generation } } + match admit_cas_attempt(attempt: cas_attempt(key: key, expected: expected, proposed: value, proposed_content: content_hash_of_value(value: value))) { + CasAttemptAdmitted { verified: v } => Present { value: v } + _ => none + } +} + +fn dcel_commits_at(store: FabricStorageBinding, key: NonEmptyStr, expected_generation: Int, value: NonEmptyStr, generation: Int) -> Bool { + match dcel_verified(key: key, expected_generation: expected_generation, value: value) { + Absent => false + Present { value: v } => + match fabric_storage_compare_and_set(store: store, verified: v) { + CasCommitted { committed: c } => cas_generation_count(g: c.generation) == generation && (c.value as String) == (value as String) + _ => false + } + } +} + +fn dcel_fails_seeing(store: FabricStorageBinding, key: NonEmptyStr, expected_generation: Int, value: NonEmptyStr, observed_generation: Int) -> Bool { + match dcel_verified(key: key, expected_generation: expected_generation, value: value) { + Absent => false + Present { value: v } => + match fabric_storage_compare_and_set(store: store, verified: v) { + CasPreconditionFailed { expected: _, observed: CasReadablePresent { version: o } } => cas_generation_count(g: o.generation) == observed_generation + _ => false + } + } +} + +fn dcel_reads(store: FabricStorageBinding, key: NonEmptyStr, generation: Int, value: String) -> Bool { + match fabric_storage_observe_cas_slot(store: store, key: key) { + CasObservedReadable { readable: CasReadablePresent { version: v } } => cas_generation_count(g: v.generation) == generation && (v.value as String) == value + _ => false + } +} + +fn dcel_reads_absent(store: FabricStorageBinding, key: NonEmptyStr) -> Bool { + match fabric_storage_observe_cas_slot(store: store, key: key) { + CasObservedReadable { readable: CasReadableAbsent } => true + _ => false + } +} + +test fn one_slot_is_one_head_and_two_bindings_see_one_generation_by_real_execution() -> Bool { + dcel_with_temp_dir(scenario: fn(dir) { + let e1 = dcel_store(dir: dir) + let e2 = dcel_store(dir: dir) + let key = "witness-slot" as NonEmptyStr + dcel_store_ready(dir: dir) + && dcel_reads_absent(store: e1, key: key) + && dcel_reads_absent(store: e2, key: key) + && dcel_commits_at(store: e1, key: key, expected_generation: 0, value: "first" as NonEmptyStr, generation: 1) + && dcel_reads(store: e2, key: key, generation: 1, value: "first") + && dcel_fails_seeing(store: e2, key: key, expected_generation: 0, value: "second-create" as NonEmptyStr, observed_generation: 1) + && dcel_commits_at(store: e2, key: key, expected_generation: 1, value: "second" as NonEmptyStr, generation: 2) + && dcel_fails_seeing(store: e1, key: key, expected_generation: 1, value: "stale" as NonEmptyStr, observed_generation: 2) + && dcel_reads(store: e1, key: key, generation: 2, value: "second") + && dcel_reads_absent(store: e1, key: "another-slot" as NonEmptyStr) + }) +} + +// A VALUE THAT CYCLES DOES NOT COLLAPSE THE GENERATION, AT THE STORE BOUNDARY THE REPAIR LIVES +// AT. W1 observes the slot at A@1 and keeps the head's exact object ref O1; the slot moves to B@2 +// and back to A@3 (O3). The repair is that O3 != O1 -- each generation object carries the +// generation it lands, so the same value at another generation is another object -- and W1's +// advance expecting O1 is FabricHeadMoved carrying the A@3 head. Without the generation in the +// object O3 == O1 and that stale advance would pass the head's object comparison; a fresh +// compare-and-set after the cycle would refuse on either implementation (it re-observes +// generation 3), so this drives fabric_storage_advance directly with W1's stale expectation. +test fn a_cycled_value_does_not_let_a_stale_writer_advance_by_real_execution() -> Bool { + dcel_with_temp_dir(scenario: fn(dir) { + let store = dcel_store(dir: dir) + let key = "cycling-slot" as NonEmptyStr + dcel_store_ready(dir: dir) + && dcel_commits_at(store: store, key: key, expected_generation: 0, value: "first" as NonEmptyStr, generation: 1) + && (match dcel_head_object(store: store, key: key) { + Absent => false + Present { value: o1 } => + dcel_commits_at(store: store, key: key, expected_generation: 1, value: "second" as NonEmptyStr, generation: 2) + && dcel_commits_at(store: store, key: key, expected_generation: 2, value: "first" as NonEmptyStr, generation: 3) + && (match dcel_head_object(store: store, key: key) { + Absent => false + Present { value: o3 } => + !fabric_object_ref_eq(a: o1, b: o3) + && (match fabric_storage_put(binding: store, object: FabricObject { links: [] as List, body: "{\"value\":\"stale\"}" as NonEmptyStr }) { + FabricObjectStored { object: target } => + (match fabric_storage_advance(binding: store, name: cas_slot_head(key: key), expected: ExpectHeadAt { object: o1 }, target: target) { + FabricHeadMoved { expected: _, observed: FabricHeadAt { object: now, generation: g } } => fabric_object_ref_eq(a: now, b: o3) && cas_generation_count(g: g) == 3 + _ => false + }) + _ => false + }) + && dcel_reads(store: store, key: key, generation: 3, value: "first") + }) + }) + }) +} + +fn dcel_head_object(store: FabricStorageBinding, key: NonEmptyStr) -> FabricObjectRef? { + match fabric_storage_head(binding: store, name: cas_slot_head(key: key)) { + FabricHeadObserved { reading: FabricHeadAt { object: o, generation: _ } } => Present { value: o } + _ => none + } +} diff --git a/dag/test/claim/machine_intake/machine_intake_bmc_secure_witness_test.dag b/dag/test/claim/machine_intake/machine_intake_bmc_secure_witness_test.dag index 42182cbf5fb..57bb9ba8741 100644 --- a/dag/test/claim/machine_intake/machine_intake_bmc_secure_witness_test.dag +++ b/dag/test/claim/machine_intake/machine_intake_bmc_secure_witness_test.dag @@ -2,7 +2,7 @@ module test.claim.machine_intake_bmc_secure_witness_test import std.types { Bool, EpochMs, Int, List, NonEmptyStr, String, list_length } import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } -import std.content_hash { ContentHash, content_hash_of_value } +import std.content_hash { ContentHash, content_hash_of_value, content_hash_equal } import extdeps.bmc.endpoint { BmcControllerEndpoint } import extdeps.bmc.types { AccountRoleAdministrator, RedfishAccountRole } import extdeps.bmc.ipmi_channel { @@ -33,7 +33,6 @@ import gunbc.machine_intake_subject { MachineIntakeSubject, QualificationSubject, UnitKey, - content_hash_equal, qualification_subject_of, } import gunbc.machine_intake_phase { diff --git a/dag/test/claim/machine_intake/machine_intake_disposition_witness_test.dag b/dag/test/claim/machine_intake/machine_intake_disposition_witness_test.dag index 30366e8db27..3c8d6b63454 100644 --- a/dag/test/claim/machine_intake/machine_intake_disposition_witness_test.dag +++ b/dag/test/claim/machine_intake/machine_intake_disposition_witness_test.dag @@ -2,11 +2,10 @@ module test.claim.machine_intake_disposition_witness_test import std.types { Bool, EpochMs, Int, List, NonEmptyStr, String, list_length } import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } -import std.content_hash { ContentHash, ContentHashEqual, ContentHashDifferent, ContentHashCrossFamilyIncomparable, compare_content_hash, content_hash_of_value } +import std.content_hash { ContentHash, ContentHashEqual, ContentHashDifferent, ContentHashCrossFamilyIncomparable, compare_content_hash, content_hash_of_value, content_hash_equal } import extdeps.toolchain.types { Aarch64, Architecture, X86_64 } import gunbc.boot_artifact { BootArtifact, IntakeLinuxEnvironment, IsoImage } import gunbc.machine_intake_subject { - content_hash_equal, SubjectCurrent, SubjectStale, StaleAxis, diff --git a/dag/test/claim/machine_intake/mtcollins1_census_image_local_wet_test.dag b/dag/test/claim/machine_intake/mtcollins1_census_image_local_wet_test.dag index 5d6a77199b1..6973854675b 100644 --- a/dag/test/claim/machine_intake/mtcollins1_census_image_local_wet_test.dag +++ b/dag/test/claim/machine_intake/mtcollins1_census_image_local_wet_test.dag @@ -233,6 +233,12 @@ test fn the_workload_emits_its_token_only_on_a_complete_agreeing_pass_by_real_ex workload_emitted_the_token(o: complete) && complete.exit_code == 0 } +// (Hoisted to module grain from the body of the item below: a source annotation inside a +// declaration body is not modeled and refuses to parse.) +// ABSENCE IS ASSERTED AS A SUBSTRING, not as a whole line: substring-absence is the STRONGER +// claim -- it also rules out the token appearing with anything appended -- and a RED may not be +// weakened to share a helper with the positive case. +// // THE THREE REDS. Each would be called STABLE by a body that only compares the two manifests: both // empty agree, both partial agree, and the disagreeing case is the ordinary instability. None may // emit the token, and each must leave the stage nonzero. @@ -242,9 +248,6 @@ test fn a_failed_or_partial_or_disagreeing_workload_emits_no_token_by_real_execu let partial = workload_case(stubs: concat(workload_stub_dd, workload_stub_partial)) let cleared_again = shell.Remove.FileForce(path: "/tmp/gunbc-wl-counter") let disagree = workload_case(stubs: concat(workload_stub_dd, workload_stub_disagree)) - // ABSENCE IS ASSERTED AS A SUBSTRING, not as a whole line: substring-absence is the STRONGER - // claim -- it also rules out the token appearing with anything appended -- and a RED may not be - // weakened to share a helper with the positive case. string_contains(s: empty.stdout, pattern: "workload-digest-stable=yes") == false && empty.exit_code != 0 && string_contains(s: partial.stdout, pattern: "workload-digest-stable=yes") == false diff --git a/dag/test/claim/materialization_store_witness_test.dag b/dag/test/claim/materialization_store_witness_test.dag index 2b996c3b11d..a767a81c699 100644 --- a/dag/test/claim/materialization_store_witness_test.dag +++ b/dag/test/claim/materialization_store_witness_test.dag @@ -2,7 +2,7 @@ module test.claim.materialization_store_witness import std.types { Bool, Int, List, String, NonEmptyStr } import std.content_hash { - ContentHash, Fnv1a64Structural, content_hash_atom, content_hash_of_value, serialize_content_hash, + ContentHash, Fnv1a64Structural, content_hash_atom, content_hash_of_value, serialize_content_hash, parse_content_hash, sha256_hex_digest, sha512_hex_digest, as_content_hash_cryptographic, as_content_hash_sha512, content_hash_family, Sha256Family, Sha512Family, Sha1Family, Fnv1a64StructuralFamily } @@ -37,7 +37,6 @@ import std.materialization_object { store_lookup_is_hit, store_lookup_is_miss, store_lookup_payload, store_lookup_refusal_tag, store_commit_refusal_tag, store_lookup_fold, encode_store_object, decode_store_object, percent_encode_payload, percent_decode_payload, - parse_recorded_store_key, result_content_canonical, store_object_name, store_object_magic, store_object_format_version, StoreObjectDecoded, StoreObjectMalformed, store_observed_digest, artifact_from_manifest, ManifestDecoded, @@ -647,12 +646,12 @@ test fn recorded_store_key_round_trips_sha256_and_sha512_wire() -> Bool { Sha1Family => false Sha256Family => false } - && match parse_recorded_store_key(key_text: serialize_content_hash(hash: k256) as String) { + && match parse_content_hash(wire: serialize_content_hash(hash: k256) as String) { Absent => false Present { value: p } => (p == k256) && ((serialize_content_hash(hash: p) as String) == (serialize_content_hash(hash: k256) as String)) } - && match parse_recorded_store_key(key_text: serialize_content_hash(hash: k512) as String) { + && match parse_content_hash(wire: serialize_content_hash(hash: k512) as String) { Absent => false Present { value: p } => (p == k512) && ((serialize_content_hash(hash: p) as String) == (serialize_content_hash(hash: k512) as String)) @@ -664,17 +663,17 @@ test fn recorded_store_key_round_trips_sha256_and_sha512_wire() -> Bool { } } -test fn parse_recorded_store_key_refuses_trailing_repeated_prefix() -> Bool { +test fn parse_content_hash_refuses_trailing_repeated_prefix() -> Bool { match sha256_hex_digest(hex: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") { Absent => false Present { value: sha } => { let h = as_content_hash_cryptographic(digest: sha) let canon = serialize_content_hash(hash: h) as String let alias = concat(canon, "sha256:ignored") - match parse_recorded_store_key(key_text: alias) { + match parse_content_hash(wire: alias) { Present { value: _ } => false Absent => - match parse_recorded_store_key(key_text: canon) { + match parse_content_hash(wire: canon) { Absent => false Present { value: p } => (p == h) && ((serialize_content_hash(hash: p) as String) == canon) diff --git a/dag/test/claim/network_boot_delivery_join_witness_test.dag b/dag/test/claim/network_boot_delivery_join_witness_test.dag index f857db30318..87ef6bd5098 100644 --- a/dag/test/claim/network_boot_delivery_join_witness_test.dag +++ b/dag/test/claim/network_boot_delivery_join_witness_test.dag @@ -1,7 +1,7 @@ module test.claim.network_boot_delivery_join_witness import std.types { Bool, EpochMs, NonEmptyStr, String } -import std.content_hash { ContentHash, content_hash_of_value } +import std.content_hash { ContentHash, content_hash_of_value, content_hash_equal } import std.decl_ref { DeclarationRef, WholeDeclaration } import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } import extdeps.crypto.mac { MacKeyId } @@ -20,7 +20,6 @@ import gunbc.machine_intake_subject { IntakeAttemptId, MachineIntakeSubject, UnitKey, - content_hash_equal, qualification_subject_of, } import gunbc.machine_intake_access { BootDeliveryTarget } diff --git a/dag/test/claim/scoped_authorization_claim_state_witness_test.dag b/dag/test/claim/scoped_authorization_claim_state_witness_test.dag new file mode 100644 index 00000000000..dc7e9d16dc2 --- /dev/null +++ b/dag/test/claim/scoped_authorization_claim_state_witness_test.dag @@ -0,0 +1,44 @@ +module test.claim.scoped_authorization_claim_state + +import std.logic { Bool } +import std.types { String, NonEmptyStr } +import v2.std.optional { Present, Absent } +import std.scoped_authorization { + AuthorizationClaimState, Unclaimed, ClaimedBy, CompletedBy, AbortedBy, AttemptIdentity, + serialize_authorization_claim_state, parse_authorization_claim_state, +} + +// THE PARSER IS THE SERIALIZER'S EXACT INVERSE, at the same home, so a consumer that must know +// WHO holds a slot (gunbc.spark.pair_serving_d0's recovery join) reads the closed state rather +// than re-spelling the tag word and the tab layout. Every arm round-trips; bytes that inhabit no +// arm -- an unknown tag, a missing field, an empty attempt -- are Absent, never a guessed state. + +fn sacs_attempt(s: String) -> AttemptIdentity { + (s as NonEmptyStr) as AttemptIdentity +} + +fn sacs_round_trips(state: AuthorizationClaimState) -> Bool { + match parse_authorization_claim_state(payload: serialize_authorization_claim_state(state: state)) { + Absent => false + Present { value: back } => serialize_authorization_claim_state(state: back) == serialize_authorization_claim_state(state: state) + } +} + +test fn every_claim_state_round_trips_through_its_wire_form() -> Bool { + sacs_round_trips(state: Unclaimed) + && sacs_round_trips(state: ClaimedBy { attempt: sacs_attempt(s: "tx-1"), claimed_at: "2026-09-18T00:00:00Z" }) + && sacs_round_trips(state: CompletedBy { attempt: sacs_attempt(s: "tx-1"), completed_at: "2026-09-18T00:00:01Z" }) + && sacs_round_trips(state: AbortedBy { attempt: sacs_attempt(s: "tx-1"), aborted_at: "2026-09-18T00:00:02Z", cause: "no fleet ssh context" as NonEmptyStr }) + && (match parse_authorization_claim_state(payload: serialize_authorization_claim_state(state: ClaimedBy { attempt: sacs_attempt(s: "tx-9"), claimed_at: "t" })) { + Present { value: ClaimedBy { attempt: who, claimed_at: when } } => (who as String) == "tx-9" && when == "t" + _ => false + }) +} + +test fn bytes_that_inhabit_no_claim_state_parse_to_nothing() -> Bool { + (match parse_authorization_claim_state(payload: "held-by\ttx-1\tt" as NonEmptyStr) { Absent => true Present { value: _ } => false }) + && (match parse_authorization_claim_state(payload: "claimed-by\ttx-1" as NonEmptyStr) { Absent => true Present { value: _ } => false }) + && (match parse_authorization_claim_state(payload: "claimed-by\t\tt" as NonEmptyStr) { Absent => true Present { value: _ } => false }) + && (match parse_authorization_claim_state(payload: "aborted-by\ttx-1\tt" as NonEmptyStr) { Absent => true Present { value: _ } => false }) + && (match parse_authorization_claim_state(payload: "unclaimed\textra" as NonEmptyStr) { Absent => true Present { value: _ } => false }) +} diff --git a/dag/test/claim/spark/fabric_capacity_standing_wet_witness_test.dag b/dag/test/claim/spark/fabric_capacity_standing_wet_witness_test.dag new file mode 100644 index 00000000000..f0c203c6be5 --- /dev/null +++ b/dag/test/claim/spark/fabric_capacity_standing_wet_witness_test.dag @@ -0,0 +1,27 @@ +module test.claim.spark.fabric_capacity_standing_wet_witness + +import std.logic { Bool } +import std.types { String } +import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } +import std.process { ExitSuccess, ExitFailure } +import gunbc.instruments.fabric_capacity_standing { fabric_capacity_standing } + +data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly + +// THE LIVE ENTRY, EXECUTED. test.claim.spark.fabric_capacity_standing_witness drives the verdict +// over a SUPPLIED roster so the real group_report producer and its binding run hermetically; this +// file is the other half of DESIGN §3's pairing obligation, the one that cannot be hermetic: the +// entry reads the executor's hostname (shell.Exec.RunArgv, no mock_response), resolves its event +// log layout and reads every claimed group's authority from the log before any standing is +// judged. On the wet lane's runner that read does not reach a log -- the runner is no dashboard +// host, or its remote is unreachable -- so the honest verdict is a refusal that NAMES THE READ: +// every claimed group is reported "serving authority could not be read at ", and no +// group's capacity is stood for. That is the route: deleting the live binding, or letting an +// unreadable log fall back to the source row, changes this text. +test fn the_live_entry_reads_the_authority_before_judging_and_refuses_off_fleet() -> Bool { + match fabric_capacity_standing() { + ExitSuccess => false + ExitFailure { code: _, reason: reason } => + string_contains(s: reason, pattern: "group-a: serving authority could not be read at ") + } +} diff --git a/dag/test/claim/spark/fabric_capacity_standing_witness_test.dag b/dag/test/claim/spark/fabric_capacity_standing_witness_test.dag index 879dfd0622e..d906547e8b7 100644 --- a/dag/test/claim/spark/fabric_capacity_standing_witness_test.dag +++ b/dag/test/claim/spark/fabric_capacity_standing_witness_test.dag @@ -1,8 +1,9 @@ module test.claim.spark.fabric_capacity_standing_witness -import std.types { Bool, String, NonEmptyStr } +import std.types { Bool, String, NonEmptyStr, List } import std.process { ProcessExit, ExitSuccess, ExitFailure } import v2.std.optional { Present, Absent } +import gunbc.spark.fabric_switch_observed { fabric_group_hosts } import std.content_hash { ContentHash, Sha256Hash, Sha256Digest, Sha256DigestHex } import std.measure { concurrent_request_hundredths_count, token_count } import extdeps.vllm.server { VllmSourceRevision, vllm_source_revision, vllm_source_revision_read } @@ -17,13 +18,16 @@ import gunbc.instruments.fabric_capacity_standing { ProbeFieldName, Invocation, UnitSha, Container, ImageSha, StartedAt, PoolLine, ConcurrencyLine, AllocatorBlocks, BlockSize, MaxModelLen, LaunchRead, LaunchIdentified, LaunchUnidentified, launch_from_fields, - GroupStanding, capacity_standing_verdict, group_report, fabric_capacity_standing, + GroupStanding, capacity_standing_verdict, group_report, fabric_capacity_standing_current, standing_for_authority, } +import gunbc.spark.pair_serving_authority_log { CurrentAuthority, CurrentAuthorityRead, AuthorityTransitionRecord } +import product.capacity.event_chain { HeadAbsent } import std.temporal_effect { held_lease, LeaseEpoch, LeaseRunningExpected } import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest } import gunbc.spark.pair_serving_authority { PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, ReleasedToFleet, PairRealizationUnestablished, + spark_pair_serving_authorities, } fn w_cell(name: ProbeFieldName) -> String { @@ -308,9 +312,18 @@ test fn one_refusing_member_refuses_the_whole_standing() -> Bool { // the entry refuses until a modeled host-effect realizes remote inventory, so refusal IS today's // correct answer on the live path. When that frontier fires this claim flips, and it should -- it is // the control that notices, rather than a green that would have said nothing either way. -test fn the_live_entry_point_runs_its_real_producer_and_refuses_today() -> Bool { +// THE ROSTER IS SUPPLIED AT GENERATION 0 -- every claimed group's resting row, which is what the +// authority log folds to when no transition is recorded -- so the claim stays hermetic while the +// real group_report producer and the binding that maps it over the roster both execute. The +// live entry, which reads that roster from the event log first, is executed by +// test.claim.spark.fabric_capacity_standing_wet_witness on the local-repo wet lane. +fn w_resting_current() -> List { + map(spark_pair_serving_authorities, a => CurrentAuthorityRead { authority: a, head: HeadAbsent, generation: 0, grant: none, entry_state: none, previous: none, admitted_by: none, transitions: [] as List }) +} + +test fn the_verdict_over_the_resting_roster_runs_its_real_producer_and_refuses_today() -> Bool { let live = group_report(group: FabricGroupA) - match fabric_capacity_standing() { + match fabric_capacity_standing_current(current: w_resting_current()) { ExitSuccess => false ExitFailure { code: _, reason: reason } => (live.admitting == false) @@ -337,8 +350,9 @@ fn with_capacity_hash(scenario: fn(ContentHash) -> Bool) -> Bool { fn w_suspended_authority(h: ContentHash) -> PairServingGroupAuthority { SuspensionPendingReconciliation { - group: FabricGroupA, + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "witness-transaction" as NonEmptyStr, + authorization_binding: h, lease: held_lease( epoch: LeaseEpoch { lease_key: "witness-lease" as NonEmptyStr, diff --git a/dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag b/dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag new file mode 100644 index 00000000000..a239cd8013f --- /dev/null +++ b/dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag @@ -0,0 +1,541 @@ +module test.claim.spark.pair_serving_authority_log_real_execution + +import std.logic { Bool } +import std.types { String, NonEmptyStr, FilePath, List } +import v2.std.optional { Present, Absent } +import gunbc.spark.fabric_switch_observed { fabric_group_hosts } +import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } +import extdeps.shell +import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest } +import std.temporal_effect { held_lease, LeaseEpoch, LeaseRunningExpected } +import product.capacity.lease { LeaseGrant, LeasePolicy, QuiescenceRequired, lease_grant, lease_fence_of } +import gunbc.spark.released_baseline { QuiescentReservedBaseline } +import product.capacity.event_chain { EventId, HeadAbsent, HeadAt, head_expectation_eq } +import gunbc.fabric_storage_client { FabricStorageBinding, FabricStorageLocalFiles } +import gunbc.fabric_storage_file_store { fabric_storage_file_root, fabric_storage_file_root_ensure, FabricStorageRootReady } +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, FabricGroupB } +import gunbc.spark.pair_serving_authority { PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, SuspendedForAuthorizedSuccessor, ReleasedToFleet, PairRealizationKeyed, PairRealizationUnestablished, pair_serving_authority_for, pair_serving_successor_may_launch } +import gunbc.spark.host_commitment { + SparkHostAdmitted, SparkHostRefused, HostEffectHandle, admit_unplaced_host_over, release_host_effect_over, + HostEffectSettled, HostEffectStillHeld, settle_host_effect_over, spark_host_standings_over, CurrentStandingsRead, CurrentStandingsUnread, spark_standing_of, SparkHostCommitted, SparkHostUncommitted, spark_commitment_causes, AuthorityHeldCommitment, OccupiedByHostEffect, +} +import product.placement_supply { HostIdentity, host_identity_eq } +import std.measure { second } +import extdeps.shell +import extdeps.python +import gunbc.spark.host_effect_quiescence { + HostEffectResidueSpec, HostQuiescenceEvidence, HostQuiet, ArgvRun, ArgvRan, observe_host_effect_quiescence, ReleaseProvenance, ClaimantTerminal, RecoveryAuthorized, ClaimantTerminated, +} +import gunbc.spark.pair_serving_authority_log { + HostEffectClaim, HostEffectClaimed, HostEffectClaimRefused, HostEffectRelease, HostEffectReleasedAt, HostEffectReleaseRefused, + host_effect_admit, host_effect_admit_at, host_effect_release, + PlacementRead, PlacementReadAt, PlacementUnread, placement_read, group_live_commitment, group_fenced_hosts, + PlacementPreparation, PlacementPreparedAt, PlacementPreparationStale, PlacementPreparationRefused, placement_prepare, placement_prepare_at, + PlacementCleanup, PlacementFinalizedAt, PlacementAbortedAt, PlacementCleanupStillFencing, placement_abort, placement_abort_at, placement_finalize, authority_transition_append, + PlacementAppendClaim, PlacementAppendClaimedAt, PlacementAppendClaimStale, PlacementAppendClaimRefused, placement_claim_append, + PreparationConsumption, PreparationConsumedBy, PreparationNotConsumed, PreparationAbsent, PreparationConsumptionUnread, preparation_consumption, + PlacementPreparationRef, preparation_ref_expected, authority_write_intent, AuthorityWriteIntent, + CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authority, + AuthorityEstablishment, AuthorityEstablishedAt, AuthorityEstablishmentRefused, pair_serving_authority_establish, + AuthorityTransition, AuthorityTransitionAppended, AuthorityTransitionStale, AuthorityTransitionRefused, pair_serving_authority_transition, + authority_eq, +} + +data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly + +// THE TRANSITION IS A COMPARE-AND-SET ON A REAL PARTITION HEAD, by real execution against a +// file-backed fabric DB in a temporary directory -- the same store test.claim.fabric.fabric_event_log_wet_witness +// exercises for the seat ledger, because it is the same log. What is established here and nowhere +// hermetic: an empty partition folds to the resting row; one admitted transition is read back as +// the current authority at generation 1; a second transition decided against the state the first +// one LEFT is stale, carries the state now current, and appends nothing. + +fn alr_with_temp_dir(scenario: fn(FilePath) -> Bool) -> Bool { + let dir = shell.Mktemp.Dir() + let observed = scenario(dir.path) + let cleaned = shell.Remove.RecursiveForce(path: dir.path) + dir.success && observed && cleaned.success +} + +fn alr_witness_layout(dir: FilePath) -> FabricStorageBinding { + FabricStorageLocalFiles { root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr) } +} + +// A ready remote has group A's source row ESTABLISHED on its partition (the durable genesis the +// fold reads from), at 990, before any scenario event; group B stays unestablished. +fn alr_witness_remote_ready(dir: FilePath) -> Bool { + (match fabric_storage_file_root_ensure(root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr)) { FabricStorageRootReady => true _ => false }) + && alr_established(store: alr_witness_layout(dir: dir), group: FabricGroupA) +} + +fn alr_established(store: FabricStorageBinding, group: FabricGroup) -> Bool { + match pair_serving_authority_for(group: group) { + Absent => false + Present { value: desired } => + match pair_serving_authority_establish(store: store, group: group, authority: desired, actor: "witness" as NonEmptyStr, at: 990) { AuthorityEstablishedAt { id: _ } => true AuthorityEstablishmentRefused { group: _, step: _, reason: _ } => false } + } +} + +fn alr_with_hash(scenario: fn(ContentHash) -> Bool) -> Bool { + match sha256_hex_digest(hex: "2222222222222222222222222222222222222222222222222222222222222222") { + Absent => false + Present { value: d } => scenario(Sha256Hash(d)) + } +} + +fn alr_w_pending(h: ContentHash) -> PairServingGroupAuthority { + SuspensionPendingReconciliation { + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-real" as NonEmptyStr, authorization_binding: h, + lease: held_lease(epoch: LeaseEpoch { lease_key: "tx-real" as NonEmptyStr, resource_fingerprint: h, owner_fingerprint: h, generation: 1 }, observed: LeaseRunningExpected), + } +} + +// THE GRANT IS MINTED FROM THE ADMITTING WRITE, as the D0 transaction mints it: reference = the +// transaction, fence = (the event that landed the pending state, the pending generation), a +// 1000-second term from 1000 -- so a read at 1000 sees the lease running and a read at 2500 +// sees it stale. A grant whose fence names another event, or another transaction, is refused +// BEFORE anything is appended. +fn alr_w_grant(admitting: EventId, reference: NonEmptyStr, generation: Int) -> LeaseGrant { + lease_grant(reference: reference, fence: lease_fence_of(admitting_event: admitting, generation: generation), granted_at: 1000, policy: LeasePolicy { maximum_duration_seconds: 1000, release_law: QuiescenceRequired }) +} + +fn alr_w_suspended(h: ContentHash) -> PairServingGroupAuthority { + SuspendedForAuthorizedSuccessor { + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), authorization: "esc-real" as NonEmptyStr, exact_candidate_realization: PairRealizationKeyed { key: h }, + lease: held_lease(epoch: LeaseEpoch { lease_key: "tx-real" as NonEmptyStr, resource_fingerprint: h, owner_fingerprint: h, generation: 1 }, observed: LeaseRunningExpected), + cleanup: QuiescentReservedBaseline, + } +} + +fn alr_current_is(c: CurrentAuthority, expected: PairServingGroupAuthority, generation: Int) -> Bool { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: g, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => authority_eq(a: a, b: expected) && g == generation + CurrentAuthorityUnread { group: _, step: _, reason: _ } => false + } +} + +fn alr_current_may_launch(c: CurrentAuthority) -> Bool { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => pair_serving_successor_may_launch(a: a) + CurrentAuthorityUnread { group: _, step: _, reason: _ } => false + } +} + +// The whole route: empty log reads the resting row at 0; the pending transition appends without a +// grant and reads back at 1; the settling transition carries the grant minted from that admitting +// write and reads back at 2 with the lease running, and stale after its expiry; a grant fenced on +// another event, or naming another transaction, is refused before any append; a transition still +// expecting the resting row is stale and names the current. +test fn a_transition_is_read_back_and_a_stale_one_names_the_current_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + alr_witness_remote_ready(dir: dir) + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1000), expected: resting, generation: 0) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: alr_w_pending(h: h), transaction: "tx-real" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { + AuthorityTransitionAppended { id: admitting, generation: g1, next: _ } => + g1 == 1 + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1000), expected: alr_w_pending(h: h), generation: 1) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: alr_w_pending(h: h), next: alr_w_suspended(h: h), transaction: "tx-real" as NonEmptyStr, lifecycle: none, grant: Present { value: alr_w_grant(admitting: "not-the-admitting-event" as EventId, reference: "tx-real" as NonEmptyStr, generation: 1) }, entry_state: none, actor: "witness" as NonEmptyStr, at: 1001) { + AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "grant" + _ => false + }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: alr_w_pending(h: h), next: alr_w_suspended(h: h), transaction: "tx-real" as NonEmptyStr, lifecycle: none, grant: Present { value: alr_w_grant(admitting: admitting, reference: "tx-other" as NonEmptyStr, generation: 1) }, entry_state: none, actor: "witness" as NonEmptyStr, at: 1001) { + AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "grant" + _ => false + }) + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001), expected: alr_w_pending(h: h), generation: 1) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: alr_w_pending(h: h), next: alr_w_suspended(h: h), transaction: "tx-real" as NonEmptyStr, lifecycle: none, grant: Present { value: alr_w_grant(admitting: admitting, reference: "tx-real" as NonEmptyStr, generation: 1) }, entry_state: none, actor: "witness" as NonEmptyStr, at: 1001) { + AuthorityTransitionAppended { id: _, generation: g2, next: _ } => g2 == 2 + _ => false + }) + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001), expected: alr_w_suspended(h: h), generation: 2) + && alr_current_may_launch(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001)) + && !alr_current_may_launch(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 2500)) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: alr_w_pending(h: h), transaction: "tx-late" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 2600) { + AuthorityTransitionStale { expected: _, current: c } => authority_eq(a: c, b: alr_w_suspended(h: h)) + _ => false + }) + _ => false + }) + } + }) + }) +} + +// RED -- an unclaimed group has no authority to read or move: both refuse at the claim, and the +// log is never touched. +test fn an_unclaimed_group_is_refused_at_the_claim_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + alr_witness_remote_ready(dir: dir) + && (match current_pair_serving_authority(store: store, group: FabricGroupB, at: 1000) { + CurrentAuthorityUnread { group: _, step: s, reason: _ } => s == "claim" + CurrentAuthorityRead { authority: _, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => false + }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupB, expected: PairServingActive { group: FabricGroupB, exact_realization: PairRealizationUnestablished { obligation: "w" as NonEmptyStr } }, next: PairServingActive { group: FabricGroupB, exact_realization: PairRealizationUnestablished { obligation: "w" as NonEmptyStr } }, transaction: "tx" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { + AuthorityTransitionRefused { group: _, step: s, reason: _ } => s == "claim" + _ => false + }) + }) + }) +} + +// ── THE HOST-EFFECT CLAIM IS A FENCE, BY REAL EXECUTION ──────────────────────────────────────── +// +// Group A is released to the fleet (generation 1). The production effect admission over the +// current authority (admit_unplaced_host_over, the fold admit_unplaced_host_live runs on the +// executor's store) admits srv5 and lands a CLAIM on the group's partition; the operator's +// re-claim (Released → Active) is then REFUSED at the head -- the same head lease the claim +// landed under -- until the claim is released; after the release it lands. +fn alr_srv(s: String) -> HostIdentity { + s as HostIdentity +} + +fn alr_residue(pattern: String) -> HostEffectResidueSpec { + HostEffectResidueSpec { process_pattern: pattern as NonEmptyStr, container: none } +} + +// THE QUIET LEG: answers "no match" to pgrep. The only route to evidence is the observer, so a +// witness that wants to release mints its evidence through it over a supplied leg. +fn alr_quiet_leg(argv: List) -> ArgvRun { + ArgvRan { exit_code: 1, stdout: "", stderr: "" } +} + +fn alr_evidence_for(claim: EventId, host: HostIdentity, spec: HostEffectResidueSpec, at: Int) -> HostQuiescenceEvidence? { + match observe_host_effect_quiescence(run: fn(argv) { alr_quiet_leg(argv: argv) }, claim: claim, host: host, spec: spec, observer: "witness" as NonEmptyStr, at: at) { + HostQuiet { evidence: e } => Present { value: e } + _ => none + } +} + +// THE LOCAL LEG, REAL: the argv runs on this host with no shell in between (so pgrep cannot match +// the leg that runs it), and its exit code, stdout and stderr are handed to the observer as they +// came. +fn alr_local_leg(argv: List) -> ArgvRun { + match get(xs: argv, index: 0) { + Absent => ArgvRan { exit_code: 2, stdout: "", stderr: "empty argv" } + Present { value: program } => { + let run = shell.Exec.RunArgv(program: program, arguments: skip(argv, n: 1)) + ArgvRan { exit_code: run.exit_code, stdout: run.stdout, stderr: run.stderr } + } + } +} + +fn alr_occupied(store: FabricStorageBinding, host: HostIdentity, at: Int) -> Bool { + match spark_host_standings_over(store: store, at: at) { + CurrentStandingsUnread { cause: _ } => false + CurrentStandingsRead { standings: st } => + match spark_standing_of(standings: st, host: host) { + Present { value: SparkHostCommitted { host: _, commitments: c } } => any(spark_commitment_causes(c: c), o => match o { AuthorityHeldCommitment { cause: OccupiedByHostEffect { purpose: _, executor: _ } } => true _ => false }) + _ => false + } + } +} + +// The witness's intent for `cur → released` under `operation` at `head`. +fn alr_intent(head: HeadExpectation, cur: PairServingGroupAuthority, released: PairServingGroupAuthority, operation: String) -> AuthorityWriteIntent { + authority_write_intent(group: FabricGroupA, head: head, actor: "witness" as NonEmptyStr, previous: Present { value: cur }, next: released, operation: operation as NonEmptyStr, lifecycle: none, grant: none, entry_state: none) +} + +fn alr_recovery(preparation: EventId) -> ReleaseProvenance { + RecoveryAuthorized { claim: preparation, claimant: ClaimantTerminated, authorized_by: "operator" as NonEmptyStr, receipt: "the lane's run was cancelled" as NonEmptyStr } +} + +fn alr_release_with_quiet(handle: HostEffectHandle, store: FabricStorageBinding, at: Int) -> Bool { + match alr_evidence_for(claim: handle.claim, host: handle.host, spec: handle.residue, at: at) { + Absent => false + Present { value: e } => match release_host_effect_over(handle: handle, evidence: e, provenance: ClaimantTerminal { claim: handle.claim, claimant_executor: "executor-1" as NonEmptyStr }, store: store, executor: "executor-1" as NonEmptyStr, at: at) { Absent => true Present { value: _ } => false } + } +} + +test fn a_live_host_effect_claim_fences_the_reclaim_until_released_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let srv5 = match get(xs: fabric_group_hosts(g: FabricGroupA), index: 1) { Present { value: x } => x Absent => alr_srv(s: "srv8") } + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + alr_witness_remote_ready(dir: dir) + && (match admit_unplaced_host_over(raw: srv5 as String, purpose: "a witness build" as NonEmptyStr, term: second(count: 100), residue: alr_residue(pattern: "/srv/build"), store: store, executor: "executor-1" as NonEmptyStr, at: 1000) { SparkHostRefused { cause: _ } => true SparkHostAdmitted { host: _, claim: _ } => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: released, transaction: "tx-release" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 _ => false }) + && (match admit_unplaced_host_over(raw: srv5 as String, purpose: "a witness build" as NonEmptyStr, term: second(count: 100), residue: alr_residue(pattern: "/srv/build"), store: store, executor: "executor-1" as NonEmptyStr, at: 1001) { + SparkHostAdmitted { host: _, claim: Present { value: handle } } => + (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1002) { + AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "host-effects" + _ => false + }) + && alr_release_with_quiet(handle: handle, store: store, at: 1003) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1004) { + AuthorityTransitionAppended { id: _, generation: g2, next: _ } => g2 == 2 + _ => false + }) + _ => false + }) + } + }) + }) +} + +// AN OVERDUE CLAIM STILL FENCES. A term-10 claim at 1001 refuses the re-claim at 1005 AND at 1012 +// (the claim is old, the effect is not known gone); a release with quiescence evidence lands the +// re-claim; a claim on a host the Active state commits refuses at `committed`; a second claim on a +// host another live claim holds refuses at `held`. +test fn an_overdue_host_effect_claim_fences_until_released_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let srv = match get(xs: fabric_group_hosts(g: FabricGroupA), index: 1) { Present { value: x } => x Absent => alr_srv(s: "srv8") } + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + alr_witness_remote_ready(dir: dir) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: released, transaction: "tx-release" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { AuthorityTransitionAppended { id: _, generation: _, next: _ } => true _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a crashed lane" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 10), residue: alr_residue(pattern: "/srv/build"), at: 1001) { + HostEffectClaimed { id: claim } => + (match host_effect_admit(store: store, host: srv, purpose: "a second lane" as NonEmptyStr, executor: "executor-2" as NonEmptyStr, term: second(count: 10), residue: alr_residue(pattern: "/srv/build"), at: 1002) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "held" _ => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1005) { AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "host-effects" _ => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1012) { AuthorityTransitionRefused { group: _, step: st, reason: r } => st == "host-effects" && r.contains("OVERDUE") _ => false }) + && (match alr_evidence_for(claim: claim, host: srv, spec: alr_residue(pattern: "/srv/other"), at: 1013) { Present { value: e } => (match host_effect_release(store: store, claim: claim, evidence: e, provenance: RecoveryAuthorized { claim: claim, claimant: ClaimantTerminated, authorized_by: "operator" as NonEmptyStr, receipt: "lane run 7 cancelled" as NonEmptyStr }, executor: "operator" as NonEmptyStr, at: 1013) { HostEffectReleaseRefused { partition: _, step: st, reason: _ } => st == "evidence" _ => false }) Absent => false }) + && (match alr_evidence_for(claim: claim, host: srv, spec: alr_residue(pattern: "/srv/build"), at: 1013) { Present { value: e } => (match host_effect_release(store: store, claim: claim, evidence: e, provenance: RecoveryAuthorized { claim: claim, claimant: ClaimantTerminated, authorized_by: "operator" as NonEmptyStr, receipt: "lane run 7 cancelled" as NonEmptyStr }, executor: "operator" as NonEmptyStr, at: 1013) { HostEffectReleasedAt { id: _ } => true _ => false }) Absent => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1014) { AuthorityTransitionAppended { id: _, generation: g2, next: _ } => g2 == 2 _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 10), residue: alr_residue(pattern: "/srv/build"), at: 1015) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + _ => false + }) + } + }) + }) +} + +// A HOST OF NO FABRIC GROUP CLAIMS ON THE SAME PLACEMENT PARTITION AS EVERY OTHER HOST. Every +// production unit is in a group, so this is driven with a fixture identity: the first claim +// lands, a second is refused as held, the release lands, a second release of the same claim is +// refused at `evidence` (it is not live), and a third claim is admitted again. +test fn an_ungrouped_host_is_claimed_on_the_placement_partition_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let host = alr_srv(s: "witness-ungrouped-host") + let spec = alr_residue(pattern: "/srv/build") + alr_witness_remote_ready(dir: dir) + && (match host_effect_admit(store: store, host: host, purpose: "a witness build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1000) { + HostEffectClaimed { id: c } => + (match host_effect_admit(store: store, host: host, purpose: "a second build" as NonEmptyStr, executor: "executor-2" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1001) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "held" _ => false }) + && (match placement_read(store: store) { PlacementReadAt { head: _, effects: effs, preparations: _ } => length(filter(effs, r => !r.released)) == 1 _ => false }) + && alr_release_with_quiet(handle: HostEffectHandle { host: host, residue: spec, claim: c }, store: store, at: 1002) + && (match alr_evidence_for(claim: c, host: host, spec: spec, at: 1003) { Present { value: e } => (match host_effect_release(store: store, claim: c, evidence: e, provenance: ClaimantTerminal { claim: c, claimant_executor: "executor-1" as NonEmptyStr }, executor: "executor-1" as NonEmptyStr, at: 1003) { HostEffectReleaseRefused { partition: _, step: st, reason: _ } => st == "evidence" _ => false }) Absent => false }) + && (match host_effect_admit(store: store, host: host, purpose: "a third build" as NonEmptyStr, executor: "executor-2" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1004) { HostEffectClaimed { id: _ } => true _ => false }) + _ => false + }) + }) + }) +} + +// AN UNESTABLISHED GROUP'S HOST: a claim on a group-B host lands on the placement partition while +// group B has no authority; the standings read it as OccupiedByHostEffect; establishing an +// Active(B) over it refuses at `host-effects` (its commitment cannot land); a non-committing +// ReleasedToFleet(B) is established at generation 0; the committing transition refuses under the +// claim; the claim's own release lands; the same transition then succeeds at generation 1 and the +// placement partition shows group B's live commitment naming the host. +test fn a_claim_on_an_unestablished_groups_host_fences_its_establishment_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let host = match get(xs: fabric_group_hosts(g: FabricGroupB), index: 1) { Present { value: x } => x Absent => alr_srv(s: "srv14") } + let spec = alr_residue(pattern: "/srv/build") + let active_b = PairServingActive { group: FabricGroupB, exact_realization: PairRealizationUnestablished { obligation: "w" as NonEmptyStr } } + let released_b = ReleasedToFleet { group: FabricGroupB, release_receipt: h } + alr_witness_remote_ready(dir: dir) + && (match pair_serving_authority_for(group: FabricGroupB) { Absent => true Present { value: _ } => false }) + && !alr_occupied(store: store, host: host, at: 999) + && (match host_effect_admit(store: store, host: host, purpose: "a witness build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1000) { + HostEffectClaimed { id: c } => + alr_occupied(store: store, host: host, at: 1001) + && (match pair_serving_authority_establish(store: store, group: FabricGroupB, authority: active_b, actor: "operator" as NonEmptyStr, at: 1002) { AuthorityEstablishmentRefused { group: _, step: st, reason: _ } => st == "host-effects" _ => false }) + && (match pair_serving_authority_establish(store: store, group: FabricGroupB, authority: released_b, actor: "operator" as NonEmptyStr, at: 1003) { AuthorityEstablishedAt { id: _ } => true _ => false }) + && (match current_pair_serving_authority(store: store, group: FabricGroupB, at: 1004) { CurrentAuthorityRead { authority: a, generation: g } => g == 0 && authority_eq(a: a, b: released_b) _ => false }) + && (match pair_serving_authority_establish(store: store, group: FabricGroupB, authority: released_b, actor: "operator" as NonEmptyStr, at: 1004) { AuthorityEstablishmentRefused { group: _, step: st, reason: _ } => st == "established" _ => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupB, expected: released_b, next: active_b, transaction: "tx-claim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "operator" as NonEmptyStr, at: 1005) { AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "host-effects" _ => false }) + && alr_occupied(store: store, host: host, at: 1006) + && alr_release_with_quiet(handle: HostEffectHandle { host: host, residue: spec, claim: c }, store: store, at: 1007) + && (match pair_serving_authority_transition(store: store, group: FabricGroupB, expected: released_b, next: active_b, transaction: "tx-claim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "operator" as NonEmptyStr, at: 1008) { AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 _ => false }) + && (match placement_read(store: store) { PlacementReadAt { head: _, effects: _, preparations: preps } => any(group_live_commitment(preps: preps, group: FabricGroupB), x => host_identity_eq(a: x, b: host)) _ => false }) + && (match host_effect_admit(store: store, host: host, purpose: "a build under the authority" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1009) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + _ => false + }) + }) + }) +} + +// THE RELEASE NEEDS THE HOST OBSERVED QUIET, BY REAL EXECUTION: with a process still carrying the +// claim's residue pattern on its command line (a daemonized sleeper under the temp dir), the +// settlement over the real local leg reports the host still held and the re-claim stays fenced; +// after the process is gone the same settlement observes quiet, the release lands, and the +// re-claim proceeds. No arm of this route takes text or an exit code as a receipt. +data alr_sleeper_fixture: String = "import os, sys, time\npid = os.fork()\nif pid:\n sys.stdout.write('%d\\n' % pid)\n sys.stdout.flush()\n os._exit(0)\nos.setsid()\nnull = os.open(os.devnull, os.O_RDWR)\nos.dup2(null, 0)\nos.dup2(null, 1)\nos.dup2(null, 2)\ntime.sleep(float(sys.argv[1]))\nos._exit(0)\n" + +test fn a_release_needs_the_host_observed_quiet_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let srv = match get(xs: fabric_group_hosts(g: FabricGroupA), index: 1) { Present { value: x } => x Absent => alr_srv(s: "srv8") } + let script = join([dir as String, "/sleeper.py"], "") + let spec = alr_residue(pattern: script) + let w = Filesystem.Write(path: script, content: alr_sleeper_fixture) + let spawned = python.Interpreter.RunFile(workdir: dir, script_path: script as FilePath, args: ["60"]) + let pid = trim(s: spawned.stdout) + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => { + let observed = w.success && spawned.success && pid != "" + && alr_witness_remote_ready(dir: dir) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: released, transaction: "tx-release" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { AuthorityTransitionAppended { id: _, generation: _, next: _ } => true _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a witness build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1001) { + HostEffectClaimed { id: c } => { + let handle = HostEffectHandle { host: srv, residue: spec, claim: c } + (match settle_host_effect_over(handle: handle, provenance: ClaimantTerminal { claim: handle.claim, claimant_executor: "executor-1" as NonEmptyStr }, run: fn(argv) { alr_local_leg(argv: argv) }, store: store, executor: "executor-1" as NonEmptyStr, at: 1002) { HostEffectStillHeld { cause: why } => why.contains("still carries the effect") HostEffectSettled => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1003) { AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "host-effects" _ => false }) + && shell.Exec.RunArgv(program: "kill", arguments: ["-9", pid]).exit_code == 0 + && (match settle_host_effect_over(handle: handle, provenance: ClaimantTerminal { claim: handle.claim, claimant_executor: "executor-1" as NonEmptyStr }, run: fn(argv) { alr_local_leg(argv: argv) }, store: store, executor: "executor-1" as NonEmptyStr, at: 1004) { HostEffectSettled => true HostEffectStillHeld { cause: _ } => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1005) { AuthorityTransitionAppended { id: _, generation: g2, next: _ } => g2 == 2 _ => false }) + } + _ => false + }) + let cleaned = shell.Exec.RunArgv(program: "kill", arguments: ["-9", pid]) + observed + } + } + }) + }) +} + +// THE PLACEMENT HEAD IS THE FENCE, NOT THE READ: an effect that read the partition quiet, then +// an authority transition committed its host, appends against the head it read -- STALE, nothing +// placed; and an authority that read the partition quiet, then a claim lands, prepares against +// the head it read -- STALE, and its fresh re-decision refuses at `host-effects`. A preparation +// whose group append never happens fences the host until it is aborted; the abort of a +// preparation no authority event consumed lands, and the host is claimable again. +test fn a_stale_placement_read_cannot_place_or_commit_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let srv = match get(xs: fabric_group_hosts(g: FabricGroupA), index: 1) { Present { value: x } => x Absent => alr_srv(s: "srv8") } + let spec = alr_residue(pattern: "/srv/build") + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + alr_witness_remote_ready(dir: dir) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: released, transaction: "tx-release" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { AuthorityTransitionAppended { id: _, generation: _, next: _, placement: PlacementFinalizedAt { preparation: _, id: _ } } => true _ => false }) + && { + let effect_read = placement_read(store: store) + (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1001) { AuthorityTransitionAppended { id: _, generation: g2, next: _ } => g2 == 2 _ => false }) + && (match host_effect_admit_at(store: store, read: effect_read, host: srv, purpose: "a build that read quiet" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1002) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "stale" _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a build that read fresh" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1003) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + } + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: released, transaction: "tx-release-2" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1004) { AuthorityTransitionAppended { id: _, generation: g3, next: _ } => g3 == 3 _ => false }) + && { + let authority_read = placement_read(store: store) + match host_effect_admit(store: store, host: srv, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1005) { + HostEffectClaimed { id: c } => + (match placement_prepare_at(store: store, read: authority_read, group: FabricGroupA, previous_hosts: [] as List, next_hosts: [srv], operation: "tx-reclaim-2" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1006) { PlacementPreparationStale => true _ => false }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: released, next: resting, transaction: "tx-reclaim-2" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1007) { AuthorityTransitionRefused { group: _, step: st, reason: _ } => st == "host-effects" _ => false }) + && alr_release_with_quiet(handle: HostEffectHandle { host: srv, residue: spec, claim: c }, store: store, at: 1008) + && (match placement_prepare(store: store, group: FabricGroupA, previous_hosts: [] as List, next_hosts: [srv], operation: "stranded" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1009) { + PlacementPreparedAt { id: prep } => + (match host_effect_admit(store: store, host: srv, purpose: "a build under a stranded preparation" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1010) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + && (match placement_abort(store: store, preparation: prep, reason: "the witness's group append never happened", provenance: alr_recovery(preparation: prep), actor: "operator" as NonEmptyStr, at: 1011) { PlacementAbortedAt { preparation: _, id: _ } => true _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a build after the abort" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1012) { HostEffectClaimed { id: _ } => true _ => false }) + _ => false + }) + _ => false + } + } + } + }) + }) +} + +// THE SAGA NEVER FREES A HOST EARLY, ABORT AND APPEND CONTEND ON ONE HEAD (the placement head +// while the preparation is Prepared; the GROUP head once its append is claimed -- a recovery +// abort of a claimed preparation first lands a cancellation at the head the claim named, so the +// claimed append can never land: driven both ways below), AND A CONSUMED PREPARATION CANNOT BE +// ABORTED. Group A is Active over its four hosts. (1) A release (Active → +// ReleasedToFleet) that dies after its preparation: the old hosts stay fenced -- group B cannot +// prepare over one, an effect cannot claim one -- the authority still reads Active, a second +// preparation for A refuses (one saga at a time), and the operator's abort (a recovery naming the +// preparation; the group is the record's, there is no group to supply wrongly) lands and frees +// nothing wrongly (the live commitment is still Active's). (2) The abort/append race: an abort +// that read the preparation Prepared, then the writer's append claim landed, appends STALE -- +// and once claimed, an abort by anyone but the claimant refuses at the fold unless it is a +// recovery. (3) A release that dies after its group append, before finalization: the authority +// reads Released, the old hosts are STILL fenced, the abort REFUSES because the join finds the +// consuming authority event, and the finalization from that join lands and frees them. +test fn a_saga_never_frees_a_host_early_and_a_consumed_preparation_cannot_be_aborted_by_real_execution() -> Bool { + alr_with_hash(scenario: fn(h) { + alr_with_temp_dir(scenario: fn(dir) { + let store = alr_witness_layout(dir: dir) + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let srv = match get(xs: fabric_group_hosts(g: FabricGroupA), index: 1) { Present { value: x } => x Absent => alr_srv(s: "srv8") } + let spec = alr_residue(pattern: "/srv/build") + let a_hosts = fabric_group_hosts(g: FabricGroupA) + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + alr_witness_remote_ready(dir: dir) + && (match placement_prepare(store: store, group: FabricGroupA, previous_hosts: a_hosts, next_hosts: [] as List, operation: "tx-release-crashed" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1000) { + PlacementPreparedAt { id: prep } => + (match placement_prepare(store: store, group: FabricGroupB, previous_hosts: [] as List, next_hosts: [srv], operation: "tx-b" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1001) { PlacementPreparationRefused { step: st, reason: r } => st == "host-effects" && r.contains("another group") _ => false }) + && (match placement_prepare(store: store, group: FabricGroupA, previous_hosts: a_hosts, next_hosts: [] as List, operation: "tx-release-again" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1001) { PlacementPreparationRefused { step: _, reason: r } => r.contains("one saga at a time") _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1002) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1003), expected: resting, generation: 0) + && (match preparation_consumption(store: store, preparation: prep) { PreparationNotConsumed { record: _ } => true _ => false }) + && (match placement_abort(store: store, preparation: prep, reason: "the lane died before its group append", provenance: alr_recovery(preparation: prep), actor: "operator" as NonEmptyStr, at: 1004) { PlacementAbortedAt { preparation: _, id: _ } => true _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1005) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + _ => false + }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1006) { + CurrentAuthorityRead { authority: cur, head: ghead } => + (match placement_prepare(store: store, group: FabricGroupA, previous_hosts: a_hosts, next_hosts: [] as List, operation: "tx-race" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1006) { + PlacementPreparedAt { id: prep } => { + let abort_read = placement_read(store: store) + (match placement_claim_append(store: store, preparation: prep, intent: alr_intent(head: ghead, cur: cur, released: released, operation: "tx-race"), at: 1007) { PlacementAppendClaimedAt { id: _ } => true _ => false }) + && (match placement_abort(store: store, preparation: prep, reason: "not mine" as String, provenance: ClaimantTerminal { claim: prep, claimant_executor: "someone-else" as NonEmptyStr }, actor: "someone-else" as NonEmptyStr, at: 1009) { PlacementCleanupStillFencing { preparation: _, cause: c } => c.contains("does not cover") && c.contains("nothing was written") _ => false }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1009) { CurrentAuthorityRead { authority: _, head: h2 } => head_expectation_eq(a: h2, b: ghead) _ => false }) + && (match placement_abort_at(store: store, read: abort_read, preparation: prep, reason: "read it Prepared" as String, provenance: alr_recovery(preparation: prep), actor: "operator" as NonEmptyStr, at: 1008) { PlacementCleanupStillFencing { preparation: _, cause: c } => c.contains("moved") _ => false }) + && (match placement_abort(store: store, preparation: prep, reason: "the writer died after its claim", provenance: alr_recovery(preparation: prep), actor: "operator" as NonEmptyStr, at: 1010) { PlacementAbortedAt { preparation: _, id: _ } => true _ => false }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1010) { CurrentAuthorityRead { authority: _, head: h2, generation: g0 } => g0 == 0 && !head_expectation_eq(a: h2, b: ghead) _ => false }) + && (match authority_transition_append(store: store, group: FabricGroupA, head: ghead, current: cur, next: released, transaction: "tx-race" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, preparation: preparation_ref_expected(id: prep, group: FabricGroupA, previous: a_hosts, next: [] as List, operation: "tx-race" as NonEmptyStr), actor: "witness" as NonEmptyStr, at: 1010) { EventAppendStale { expected: _, observed: _ } => true _ => false }) + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1010), expected: resting, generation: 0) + } + _ => false + }) + _ => false + }) + && (match placement_prepare(store: store, group: FabricGroupA, previous_hosts: a_hosts, next_hosts: [] as List, operation: "tx-release" as NonEmptyStr, actor: "witness" as NonEmptyStr, at: 1011) { + PlacementPreparedAt { id: prep } => + (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1012) { + CurrentAuthorityRead { authority: cur, head: head } => + (match placement_claim_append(store: store, preparation: prep, intent: alr_intent(head: head, cur: cur, released: released, operation: "tx-release"), at: 1011) { PlacementAppendClaimedAt { id: _ } => true _ => false }) + && (match authority_transition_append(store: store, group: FabricGroupA, head: head, current: cur, next: released, transaction: "tx-release" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, preparation: preparation_ref_expected(id: prep, group: FabricGroupA, previous: a_hosts, next: [] as List, operation: "tx-release" as NonEmptyStr), actor: "witness" as NonEmptyStr, at: 1012) { EventAppended { id: _ } => true _ => false }) + _ => false + }) + && alr_current_is(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1013), expected: released, generation: 1) + && (match host_effect_admit(store: store, host: srv, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1013) { HostEffectClaimRefused { partition: _, step: st, reason: _ } => st == "committed" _ => false }) + && (match placement_abort(store: store, preparation: prep, reason: "wrongly assumed stranded" as String, provenance: alr_recovery(preparation: prep), actor: "operator" as NonEmptyStr, at: 1014) { PlacementCleanupStillFencing { preparation: _, cause: c } => c.contains("consumed by authority event") _ => false }) + && (match placement_finalize(store: store, consumed: preparation_consumption(store: store, preparation: prep), actor: "operator" as NonEmptyStr, at: 1015) { PlacementFinalizedAt { preparation: _, id: _ } => true _ => false }) + && (match host_effect_admit(store: store, host: srv, purpose: "a build after the finalization" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: spec, at: 1016) { HostEffectClaimed { id: _ } => true _ => false }) + _ => false + }) + } + }) + }) +} diff --git a/dag/test/claim/spark/pair_serving_authority_log_witness_test.dag b/dag/test/claim/spark/pair_serving_authority_log_witness_test.dag new file mode 100644 index 00000000000..47396ba6317 --- /dev/null +++ b/dag/test/claim/spark/pair_serving_authority_log_witness_test.dag @@ -0,0 +1,806 @@ +module test.claim.spark.pair_serving_authority_log_witness + +import std.logic { Bool } +import std.types { String, NonEmptyStr, List } +import v2.std.optional { Present, Absent } +import product.placement_supply { HostIdentity } +import gunbc.spark.fabric_switch_observed { fabric_group_hosts } +import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest, compare_content_hash, ContentHashEqual, ContentHashDifferent, ContentHashCrossFamilyIncomparable, content_hash_equal } +import product.capacity.lease { LeaseGrant } +import std.temporal_effect { HeldLease, held_lease, LeaseEpoch, LeaseRunningExpected, LeaseInaccessible } +import product.capacity.event_chain { ChainEvent, ChainEnvelope, EventId, EventDecoded, EventUndecodable, HeadAbsent, HeadAt, HeadExpectation } +import gunbc.spark.fabric_switch_observed { FabricGroupA, FabricGroupB } +import gunbc.spark.released_baseline { QuiescentReservedBaseline } +import std.measure { second, second_count } +import extdeps.docker.cli { DockerPsByName, DockerPsByAncestor } +import gunbc.spark.host_effect_quiescence { + HostEffectResidueSpec, HostQuiescenceEvidence, HostQuiescence, HostQuiet, HostResidue, HostQuiescenceUnread, ArgvRan, ArgvLegDidNotRun, observe_host_effect_quiescence, + ReleaseProvenance, ClaimantTerminal, RecoveryAuthorized, ClaimantTerminated, +} +import gunbc.spark.pair_serving_authority { + PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, SuspendedForAuthorizedSuccessor, FencedRefusal, ReleasedToFleet, + PairRealizationKeyed, PairRealizationUnestablished, +} +import gunbc.spark.pair_serving_authority_log { + PairAuthorityEvent, AuthorityEstablished, AuthorityTransitioned, EntryStateRecorded, HostEffectAdmitted, HostEffectReleased, PlacementPrepared, PlacementAppendClaimed, PlacementFinalized, PlacementAborted, AuthorityAppendCancelled, + PlacementPreparationRef, committed_population, + pair_authority_partition, host_placement_partition, + pair_authority_event_wire_text, pair_authority_event_decode, authority_eq, authority_wire, + AuthorityFold, AuthorityUnestablished, AuthorityFolded, AuthorityFoldRefused, authority_fold, AuthorityTransitionRecord, transitions_by, authority_write_intent, AuthorityWriteIntent, event_write_intent, write_intent_eq, CancelledPreparation, cancellation_gap, + PlacementFold, PlacementFolded, PlacementFoldRefused, placement_fold, PlacementPreparationRecord, AppendClaimed, Prepared, Aborted, group_live_commitment, group_fenced_hosts, same_host_set, +} + +// THE CODEC AND THE FOLD, OVER SUPPLIED VALUES. Every arm of the authority round-trips through +// the wire, the fold walks recorded transitions from a resting row and refuses one that leaves a +// state that was not current. The linearization itself -- append under a leased head, read back -- +// executes for real in test.claim.spark.pair_serving_authority_log_real_execution on the wet lane. + +fn alw_with_hash(scenario: fn(ContentHash) -> Bool) -> Bool { + match sha256_hex_digest(hex: "1111111111111111111111111111111111111111111111111111111111111111") { + Absent => false + Present { value: d } => scenario(Sha256Hash(d)) + } +} + +fn alw_w_lease(h: ContentHash, inaccessible: Bool) -> HeldLease { + held_lease( + epoch: LeaseEpoch { lease_key: "witness-lease" as NonEmptyStr, resource_fingerprint: h, owner_fingerprint: h, generation: 3 }, + observed: if inaccessible { LeaseInaccessible } else { LeaseRunningExpected }, + ) +} + +fn alw_w_active_unestablished() -> PairServingGroupAuthority { + PairServingActive { group: FabricGroupA, exact_realization: PairRealizationUnestablished { obligation: "witness" as NonEmptyStr } } +} + +fn alw_w_all_arms(h: ContentHash) -> List { + [ + alw_w_active_unestablished(), + PairServingActive { group: FabricGroupA, exact_realization: PairRealizationKeyed { key: h } }, + SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) }, + SuspendedForAuthorizedSuccessor { + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), authorization: "auth-1" as NonEmptyStr, exact_candidate_realization: PairRealizationKeyed { key: h }, + lease: alw_w_lease(h: h, inaccessible: true), cleanup: QuiescentReservedBaseline, + }, + FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "foreign occupant" as NonEmptyStr, disposition_authority: "operator" as NonEmptyStr }, + ReleasedToFleet { group: FabricGroupA, release_receipt: h }, + ] +} + +fn alw_w_event(previous: PairServingGroupAuthority, next: PairServingGroupAuthority, parent: EventId?) -> ChainEvent { + ChainEvent { + partition: pair_authority_partition(group: FabricGroupA), + parent: parent, + recorded_at: 1000, + actor: "witness" as NonEmptyStr, + payload: AuthorityTransitioned { previous: previous, next: next, transaction: "tx-1" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: previous, next: next, operation: "tx-1") }, + } +} + +fn alw_round_trips(previous: PairServingGroupAuthority, next: PairServingGroupAuthority) -> Bool { + let e = alw_w_event(previous: previous, next: next, parent: none) + match pair_authority_event_decode(text: pair_authority_event_wire_text(event: e)) { + EventUndecodable { reason: _ } => false + EventDecoded { event: back } => + match back.payload { + AuthorityTransitioned { previous: p, next: n, transaction: t, lifecycle: _, grant: _, entry_state: _ } => + authority_eq(a: p, b: previous) && authority_eq(a: n, b: next) && (t as String) == "tx-1" + && back.recorded_at == 1000 && (back.partition as String) == (pair_authority_partition(group: FabricGroupA) as String) + _ => false + } + } +} + +// ── THE CODEC ────────────────────────────────────────────────────────────────────────────────── + +// EVERY ARM, WITH EVERY FIELD IT CARRIES, SURVIVES THE WIRE -- one claim per arm, because the +// fold that renders and parses an authority is the whole cost of each and a conjunction over six +// arms pays it six times under one budget (DESIGN §3: split independent cases when splitting +// loses no coverage). The `previous` position runs the same members fold, so it is covered once. +fn alw_arm_round_trips(index: Int) -> Bool { + alw_with_hash(scenario: fn(h) { + match get(xs: alw_w_all_arms(h: h), index: index) { + Absent => false + Present { value: a } => alw_round_trips(previous: alw_w_active_unestablished(), next: a) + } + }) +} + +test fn an_unestablished_active_arm_round_trips_through_the_wire() -> Bool { + alw_arm_round_trips(index: 0) +} + +test fn a_keyed_active_arm_round_trips_through_the_wire() -> Bool { + alw_arm_round_trips(index: 1) +} + +test fn a_pending_arm_round_trips_through_the_wire() -> Bool { + alw_arm_round_trips(index: 2) +} + +test fn a_suspended_for_successor_arm_round_trips_through_the_wire() -> Bool { + alw_arm_round_trips(index: 3) +} + +test fn a_fenced_arm_round_trips_through_the_wire() -> Bool { + alw_arm_round_trips(index: 4) +} + +test fn a_released_arm_round_trips_through_the_wire() -> Bool { + alw_arm_round_trips(index: 5) +} + +// The `previous` position, with a leased arm, so both sides of one event are read back. +test fn a_leased_previous_arm_round_trips_through_the_wire() -> Bool { + alw_with_hash(scenario: fn(h) { + match get(xs: alw_w_all_arms(h: h), index: 3) { + Absent => false + Present { value: a } => alw_round_trips(previous: a, next: alw_w_active_unestablished()) + } + }) +} + +// The wire distinguishes every arm and every identity field -- and does NOT distinguish the +// lease's observed state, which is a fact about the reading instant rather than about the +// authority: one pending state read while its lease runs and again after it expires is ONE state, +// or the settling transition could never name the state it leaves. +test fn the_wire_distinguishes_arms_and_fields_but_not_the_lease_observation() -> Bool { + alw_with_hash(scenario: fn(h) { + let arms = alw_w_all_arms(h: h) + let live = alw_w_lease(h: h, inaccessible: false) + let dead = alw_w_lease(h: h, inaccessible: true) + length(arms) == 6 + && !any(arms, a => any(arms, b => (authority_wire(a: a) != authority_wire(a: b)) && authority_eq(a: a, b: b))) + && authority_eq( + a: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: live }, + b: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: dead }) + && !authority_eq( + a: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: live }, + b: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-2" as NonEmptyStr, authorization_binding: h, lease: live }) + && !authority_eq(a: alw_w_active_unestablished(), b: PairServingActive { group: FabricGroupB, exact_realization: PairRealizationUnestablished { obligation: "witness" as NonEmptyStr } }) + }) +} + +// A document of another schema, or one whose state word names an arm its fields do not inhabit, +// is a typed refusal -- never an authority. +test fn a_foreign_or_incoherent_document_does_not_decode() -> Bool { + match pair_authority_event_decode(text: "{\"schema\":\"capacity-pool-event/v1\"}") { + EventUndecodable { reason: _ } => + match pair_authority_event_decode(text: "{\"schema\":\"pair-serving-authority-event/v1\",\"partition\":\"p\",\"parent\":\"\",\"recorded_at\":1,\"actor\":\"w\",\"kind\":\"transitioned\",\"transaction\":\"t\",\"prev_state\":\"suspension-pending-reconciliation\",\"prev_group\":\"group-a\",\"next_state\":\"active\",\"next_group\":\"group-a\",\"next_realization_obligation\":\"o\"}") { + EventUndecodable { reason: _ } => true + EventDecoded { event: _ } => false + } + EventDecoded { event: _ } => false + } +} + +// ── THE FOLD ─────────────────────────────────────────────────────────────────────────────────── + +// The fold from the durable genesis with `resting` established as event e0: what the resting row +// used to be to the fold is now the first event on the chain. +// The preparation an authority event must carry: the populations of the states it leaves and +// enters, under its operation, with an id unique to that write (a preparation is consumed once). +// The intent of a write the witness authors at `parent`, as an append claim carries it. +fn alw_intent(parent: EventId?, previous: PairServingGroupAuthority?, next: PairServingGroupAuthority, operation: String, lifecycle: ContentHash?, grant: LeaseGrant?, entry_state: ContentHash?) -> AuthorityWriteIntent { + authority_write_intent(group: FabricGroupA, head: match parent { Absent => HeadAbsent Present { value: h } => HeadAt { id: h } }, actor: "witness" as NonEmptyStr, previous: previous, next: next, operation: operation as NonEmptyStr, lifecycle: lifecycle, grant: grant, entry_state: entry_state) +} + +// The intent a witness-shaped transition event proves: group A, operation tx-op, no lifecycle, +// grant or entry state. +fn write_intent_of_witness(env: ChainEnvelope, previous: PairServingGroupAuthority, next: PairServingGroupAuthority) -> AuthorityWriteIntent { + event_write_intent(env: env, group: FabricGroupA, previous: Present { value: previous }, next: next, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none) +} + +fn alw_ref(previous: PairServingGroupAuthority, next: PairServingGroupAuthority, operation: String) -> PlacementPreparationRef { + PlacementPreparationRef { id: (join(["p-", operation, "-", authority_wire(a: previous), ">", authority_wire(a: next)], "") as NonEmptyStr) as EventId, group: FabricGroupA, previous_hosts: committed_population(a: previous), next_hosts: committed_population(a: next), operation: operation as NonEmptyStr } +} + +fn alw_establish_ref(a: PairServingGroupAuthority) -> PlacementPreparationRef { + PlacementPreparationRef { id: "p0" as EventId, group: FabricGroupA, previous_hosts: [] as List, next_hosts: committed_population(a: a), operation: "establish group-a" as NonEmptyStr } +} + +fn alw_establish(a: PairServingGroupAuthority) -> ChainEvent { + ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 990, actor: "witness" as NonEmptyStr, payload: AuthorityEstablished { authority: a, placement: alw_establish_ref(a: a) } } +} + +fn alw_fold(resting: PairServingGroupAuthority, oldest_first: List>) -> AuthorityFold { + authority_fold(oldest_first: concat([alw_env(id: "e0", e: alw_establish(a: resting))], oldest_first)) +} + +fn alw_env(id: String, e: ChainEvent) -> ChainEnvelope { + ChainEnvelope { id: id as EventId, event: e } +} + +// No transitions: the resting row at generation 0. Two chained transitions: the last next, at 2. +test fn the_fold_walks_from_the_resting_row_through_each_transition() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + let fenced = FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "foreign occupant" as NonEmptyStr, disposition_authority: "operator" as NonEmptyStr } + let chain = [alw_env(id: "e1", e: alw_w_event(previous: resting, next: pending, parent: none)), alw_env(id: "e2", e: alw_w_event(previous: pending, next: fenced, parent: Present { value: "e1" as EventId }))] + (match alw_fold(resting: resting, oldest_first: [] as List>) { + AuthorityFolded { current: c, generation: g, grant: _, entry_state: _ } => g == 0 && authority_eq(a: c, b: resting) + _ => false + }) + && (match alw_fold(resting: resting, oldest_first: chain) { + AuthorityFolded { current: c, generation: g, grant: _, entry_state: _ } => g == 2 && authority_eq(a: c, b: fenced) + _ => false + }) + }) +} + +// RED -- an event that claims to leave a state that was not current is refused AT THAT EVENT, and +// the fold does not continue past it. +test fn a_transition_from_a_state_that_was_not_current_refuses_at_that_event() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + let fenced = FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "foreign occupant" as NonEmptyStr, disposition_authority: "operator" as NonEmptyStr } + let chain = [alw_env(id: "e1", e: alw_w_event(previous: pending, next: fenced, parent: none))] + match alw_fold(resting: resting, oldest_first: chain) { + AuthorityFoldRefused { at_event: e, reason: _ } => (e as String) == "e1" + _ => false + } + }) +} + +// RED -- a transition whose next state names another group than the partition's is refused. +test fn a_transition_to_another_groups_authority_refuses() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let elsewhere = ReleasedToFleet { group: FabricGroupB, release_receipt: h } + match alw_fold(resting: resting, oldest_first: [alw_env(id: "e1", e: alw_w_event(previous: resting, next: elsewhere, parent: none))]) { + AuthorityFoldRefused { at_event: e, reason: _ } => (e as String) == "e1" + _ => false + } + }) +} + + +// THE LIFECYCLE RIDES THE WIRE: a transition carrying a lifecycle identity decodes to the same hash. +fn alw_w_with_lifecycle(h: ContentHash) -> ChainEvent { + let resting = alw_w_active_unestablished() + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: pending, transaction: "tx-1" as NonEmptyStr, lifecycle: Present { value: h }, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: pending, operation: "tx-1") } } +} + +test fn the_lifecycle_identity_round_trips_through_the_wire() -> Bool { + alw_with_hash(scenario: fn(h) { + match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_w_with_lifecycle(h: h))) { + EventDecoded { event: back } => + match back.payload { + AuthorityTransitioned { previous: _, next: _, transaction: _, lifecycle: Present { value: l }, grant: _, entry_state: _ } => content_hash_equal(left: l, right: h) + _ => false + } + EventUndecodable { reason: _ } => false + } + }) +} + +// A transition without a lifecycle decodes to none. +test fn a_transition_without_a_lifecycle_decodes_to_none() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_w_event(previous: resting, next: pending, parent: none))) { + EventDecoded { event: back } => match back.payload { AuthorityTransitioned { previous: _, next: _, transaction: _, lifecycle: Absent, grant: _, entry_state: _ } => true _ => false } + EventUndecodable { reason: _ } => false + } + }) +} + +// RED -- a lifecycle member that is not a content-hash wire refuses the document. +test fn an_incoherent_lifecycle_member_refuses_the_document() -> Bool { + alw_with_hash(scenario: fn(h) { + match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_w_with_lifecycle(h: h)).replace(from: "sha256:", to: "sha256x:")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false } + }) +} + +// AND THE FOLD KEEPS IT: the record of transitions carries each event's lifecycle so a recovery +// can select by it rather than by the transaction word; an operator's write carries none. +test fn the_fold_records_each_transitions_lifecycle() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + let with = ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: pending, transaction: "tx-1" as NonEmptyStr, lifecycle: Present { value: h }, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: pending, operation: "tx-1") } } + match alw_fold(resting: resting, oldest_first: [alw_env(id: "e1", e: with), alw_env(id: "e2", e: alw_w_event(previous: pending, next: resting, parent: Present { value: "e1" as EventId }))]) { + AuthorityFolded { current: _, generation: g, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: recs } => + g == 2 && length(transitions_by(records: recs, transaction: "tx-1" as NonEmptyStr, lifecycle: h)) == 1 + && (match first(recs) { Present { value: r } => (r.id as String) == "e1" && (match r.lifecycle { Present { value: _ } => true Absent => false }) Absent => false }) + && (match last(recs) { Present { value: r } => (r.id as String) == "e2" && (match r.lifecycle { Absent => true Present { value: _ } => false }) Absent => false }) + _ => false + } + }) +} + +// ── THE HOSTS WIRE IS INJECTIVE ───────────────────────────────────────────────────────────────── +// +// HostIdentity admits any non-empty string, so the population's wire must be one that every +// distinct list decodes back from: a JSON array. ["a,b","c"] and ["a","b","c"] are different +// authorities; an element that is empty, or not a string, refuses the document. +fn alw_fenced_over(hs: List) -> PairServingGroupAuthority { + FencedRefusal { group: FabricGroupA, hosts: hs, cause: "w" as NonEmptyStr, disposition_authority: "w" as NonEmptyStr } +} + +fn alw_host(s: String) -> HostIdentity { + s as HostIdentity +} + +test fn the_hosts_wire_distinguishes_every_population() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let joined = alw_fenced_over(hs: [alw_host(s: "a,b"), alw_host(s: "c")]) + let flat = alw_fenced_over(hs: [alw_host(s: "a"), alw_host(s: "b"), alw_host(s: "c")]) + let wire_joined = pair_authority_event_wire_text(event: alw_w_event(previous: resting, next: joined, parent: none)) + let wire_flat = pair_authority_event_wire_text(event: alw_w_event(previous: resting, next: flat, parent: none)) + wire_joined != wire_flat + && !authority_eq(a: joined, b: flat) + && (match pair_authority_event_decode(text: wire_joined) { + EventDecoded { event: back } => match back.payload { AuthorityTransitioned { previous: _, next: n, transaction: _, lifecycle: _, grant: _, entry_state: _ } => authority_eq(a: n, b: joined) && !authority_eq(a: n, b: flat) _ => false } + EventUndecodable { reason: _ } => false + }) + }) +} + +// The hosts wire under one malformed member: the fold's decoder refuses `hosts_wire_replaced` +// where a lenient decoder would normalize. Two claims, two shapes each, so neither reaches past +// the floor's per-claim budget (each decode is a full envelope parse). +fn alw_hosts_wire_refuses(replaced: String) -> Bool { + let resting = alw_w_active_unestablished() + let wire = pair_authority_event_wire_text(event: alw_w_event(previous: resting, next: alw_fenced_over(hs: [alw_host(s: "srv5"), alw_host(s: "srv6")]), parent: none)) + wire.contains("[\"srv5\", \"srv6\"]") + && (match pair_authority_event_decode(text: wire.replace(from: "[\"srv5\", \"srv6\"]", to: replaced)) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) +} + +test fn a_malformed_hosts_member_refuses_rather_than_normalizing() -> Bool { + alw_with_hash(scenario: fn(h) { + alw_hosts_wire_refuses(replaced: "[\"srv5\", \"\", \"srv6\"]") && alw_hosts_wire_refuses(replaced: "[\"srv5\", 6]") + }) +} + +test fn a_hosts_wire_that_is_not_an_array_or_is_empty_refuses() -> Bool { + alw_with_hash(scenario: fn(h) { + alw_hosts_wire_refuses(replaced: "\"srv5,srv6\"") && alw_hosts_wire_refuses(replaced: "[]") + }) +} + +// ── HOST PLACEMENT: EFFECTS AND COMMITMENTS ON ONE PARTITION ─────────────────────────────────── + +fn alw_prepare(parent: EventId?, group: FabricGroup, previous: List, next: List, at: Int) -> ChainEvent { + let ph = previous + let nh = next + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: "witness" as NonEmptyStr, payload: PlacementPrepared { operation: "tx-op" as NonEmptyStr, group: group, previous_hosts: ph, next_hosts: nh } } +} + +// A finalization of preparation `preparation` (prepared by alw_prepare over group A, previous → +// next, operation tx-op), carrying the copy the fold checks against the record. +fn alw_finalize_of(parent: EventId?, preparation: String, previous: List, next: List, at: Int) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: "witness" as NonEmptyStr, payload: PlacementFinalized { preparation: PlacementPreparationRef { id: (preparation as NonEmptyStr) as EventId, group: FabricGroupA, previous_hosts: previous, next_hosts: next, operation: "tx-op" as NonEmptyStr }, authority_event: "g1" as EventId } } +} + +fn alw_finalize(parent: EventId?, preparation: String, at: Int) -> ChainEvent { + alw_finalize_of(parent: parent, preparation: preparation, previous: [] as List, next: [alw_host(s: "srv5")], at: at) +} + +// The intent every hermetic claim here is for: the [] -> [srv5] tx-op preparation, written by +// `actor` at an absent group head, entering a fenced state over srv5 (the one arm that commits +// exactly the hosts it names). +fn alw_srv5_intent(actor: String) -> AuthorityWriteIntent { + authority_write_intent(group: FabricGroupA, head: HeadAbsent, actor: actor as NonEmptyStr, previous: none, next: alw_fenced_over(hs: [alw_host(s: "srv5")]), operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none) +} + +// The claim for a release preparation ([srv5] -> [], tx-op): the intent leaves the fenced state +// over srv5 for a state that commits no host. +fn alw_release_claim(parent: EventId?, preparation: String, at: Int) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: "witness" as NonEmptyStr, payload: PlacementAppendClaimed { preparation: (preparation as NonEmptyStr) as EventId, intent: authority_write_intent(group: FabricGroupA, head: HeadAbsent, actor: "witness" as NonEmptyStr, previous: Present { value: alw_fenced_over(hs: [alw_host(s: "srv5")]) }, next: alw_fenced_over(hs: [] as List), operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none) } } +} + +// The writer's append claim (actor witness) and its own abort. +fn alw_claim(parent: EventId?, preparation: String, at: Int) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: "witness" as NonEmptyStr, payload: PlacementAppendClaimed { preparation: (preparation as NonEmptyStr) as EventId, intent: alw_srv5_intent(actor: "witness") } } +} + +fn alw_abort_by(parent: EventId?, preparation: String, at: Int, actor: String) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: actor as NonEmptyStr, payload: PlacementAborted { preparation: (preparation as NonEmptyStr) as EventId, reason: "the group append did not land", provenance: ClaimantTerminal { claim: (preparation as NonEmptyStr) as EventId, claimant_executor: "witness" as NonEmptyStr } } } +} + +fn alw_abort(parent: EventId?, preparation: String, at: Int) -> ChainEvent { + alw_abort_by(parent: parent, preparation: preparation, at: at, actor: "witness") +} + +// A prepared-and-finalized commitment of group A over srv5: events e3 (prepare) and e4 (finalize). +fn alw_a_over_srv5(parent: EventId?, at: Int) -> List> { + [alw_env(id: "e3", e: alw_prepare(parent: parent, group: FabricGroupA, previous: [] as List, next: [alw_host(s: "srv5")], at: at)), alw_env(id: "e3c", e: alw_claim(parent: Present { value: "e3" as EventId }, preparation: "e3", at: at)), alw_env(id: "e4", e: alw_finalize(parent: Present { value: "e3c" as EventId }, preparation: "e3", at: at + 1))] +} + +// Group A's live commitment after a placement fold names exactly these hosts, and it fences them. +fn alw_a_commits(f: PlacementFold, hosts: List) -> Bool { + match f { + PlacementFolded { effects: _, preparations: preps } => { + same_host_set(a: group_live_commitment(preps: preps, group: FabricGroupA), b: map(hosts, h => alw_host(s: h))) + && all(hosts, h => any(group_fenced_hosts(preps: preps, group: FabricGroupA), x => (x as String) == h)) + } + PlacementFoldRefused { at_event: _, reason: _ } => false + } +} + +fn alw_residue(pattern: String) -> HostEffectResidueSpec { + HostEffectResidueSpec { process_pattern: pattern as NonEmptyStr, container: none } +} + +fn alw_effect(parent: EventId?, host: HostIdentity, at: Int) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: "witness" as NonEmptyStr, payload: HostEffectAdmitted { host: host, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: alw_residue(pattern: "/srv/build") } } +} + +// THE ONLY WAY TO EVIDENCE: the observer over a supplied leg. A leg that answers "no match" to +// pgrep (exit 1) and nothing to docker ps mints the quiet reading; the witness cannot construct +// one otherwise, which is the point of the sole constructor. +fn alw_quiet_leg(argv: List) -> ArgvRun { + match get(xs: argv, index: 0) { + Present { value: "pgrep" } => ArgvRan { exit_code: 1, stdout: "", stderr: "" } + _ => ArgvRan { exit_code: 0, stdout: "", stderr: "" } + } +} + +fn alw_observe(leg: fn(List) -> ArgvRun, spec: HostEffectResidueSpec) -> HostQuiescence { + observe_host_effect_quiescence(run: leg, claim: "e2" as EventId, host: alw_host(s: "srv5"), spec: spec, observer: "witness" as NonEmptyStr, at: 1005) +} + +fn alw_evidence_at(claim: String, host: HostIdentity, spec: HostEffectResidueSpec, at: Int) -> HostQuiescenceEvidence? { + match observe_host_effect_quiescence(run: fn(argv) { alw_quiet_leg(argv: argv) }, claim: claim as EventId, host: host, spec: spec, observer: "witness" as NonEmptyStr, at: at) { + HostQuiet { evidence: e } => Present { value: e } + _ => none + } +} + +// Evidence for srv5's claim e2 (the fold fixtures admit it at 1000), taken at 1005. +fn alw_evidence(host: HostIdentity, spec: HostEffectResidueSpec) -> HostQuiescenceEvidence? { + match observe_host_effect_quiescence(run: fn(argv) { alw_quiet_leg(argv: argv) }, claim: "e2" as EventId, host: host, spec: spec, observer: "witness" as NonEmptyStr, at: 1005) { + HostQuiet { evidence: e } => Present { value: e } + _ => none + } +} + +fn alw_release_with(parent: EventId?, admitted_by: String, at: Int, evidence: HostQuiescenceEvidence) -> ChainEvent { + alw_release_by(parent: parent, admitted_by: admitted_by, at: at, evidence: evidence, actor: "executor-1") +} + +// A claimant-terminal release written by `actor`, naming the claim and executor-1 as claimant (the +// executor every alw_effect claim is acquired by). +fn alw_release_by(parent: EventId?, admitted_by: String, at: Int, evidence: HostQuiescenceEvidence, actor: String) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: actor as NonEmptyStr, payload: HostEffectReleased { admitted_by: (admitted_by as NonEmptyStr) as EventId, evidence: evidence, provenance: ClaimantTerminal { claim: (admitted_by as NonEmptyStr) as EventId, claimant_executor: "executor-1" as NonEmptyStr } } } +} + +// A release for srv5's build claim e2: the evidence names the claim, srv5 and the claim's own residue. +fn alw_release(parent: EventId?, admitted_by: String, at: Int) -> ChainEvent { + match alw_evidence(host: alw_host(s: "srv5"), spec: alw_residue(pattern: "/srv/build")) { + Present { value: e } => alw_release_with(parent: parent, admitted_by: admitted_by, at: at, evidence: e) + Absent => alw_effect(parent: parent, host: alw_host(s: "srv5"), at: at) + } +} + +// Both host-effect events round-trip, the admitted one with its term and residue (a named +// container and an image ancestor are distinct on the wire), the released one with its evidence. +test fn host_effect_events_round_trip_through_the_wire() -> Bool { + let named = HostEffectResidueSpec { process_pattern: "/srv/build" as NonEmptyStr, container: Present { value: DockerPsByName { name: "vllm-a" as NonEmptyStr } } } + let by_image = HostEffectResidueSpec { process_pattern: "/srv/build" as NonEmptyStr, container: Present { value: DockerPsByAncestor { image: "vllm-a" as NonEmptyStr } } } + let named_event = ChainEvent { partition: host_placement_partition, parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: HostEffectAdmitted { host: alw_host(s: "srv10"), purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: named } } + let named_wire = pair_authority_event_wire_text(event: named_event) + (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_effect(parent: none, host: alw_host(s: "srv10"), at: 1000))) { + EventDecoded { event: back } => match back.payload { HostEffectAdmitted { host: hs, purpose: p, executor: x, term: t, residue: rs } => (hs as String) == "srv10" && (p as String) == "a build" && (x as String) == "executor-1" && second_count(s: t) == 100 && (rs.process_pattern as String) == "/srv/build" && (match rs.container { Absent => true Present { value: _ } => false }) _ => false } + EventUndecodable { reason: _ } => false + }) + && (match pair_authority_event_decode(text: named_wire) { + EventDecoded { event: back } => match back.payload { HostEffectAdmitted { residue: rs } => (match rs.container { Present { value: DockerPsByName { name: n } } => (n as String) == "vllm-a" _ => false }) _ => false } + EventUndecodable { reason: _ } => false + }) + && named_wire.contains("\"residue_container_name\"") + && (pair_authority_event_wire_text(event: named_event) != pair_authority_event_wire_text(event: ChainEvent { partition: host_placement_partition, parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: HostEffectAdmitted { host: alw_host(s: "srv10"), purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: by_image } })) + && (match pair_authority_event_decode(text: named_wire.replace(from: "\"residue_container_name\"", to: "\"residue_container_image\"")) { + EventDecoded { event: back } => match back.payload { HostEffectAdmitted { residue: rs } => (match rs.container { Present { value: DockerPsByAncestor { image: i } } => (i as String) == "vllm-a" _ => false }) _ => false } + EventUndecodable { reason: _ } => false + }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_release(parent: none, admitted_by: "e1", at: 1001))) { + EventDecoded { event: back } => match back.payload { HostEffectReleased { admitted_by: a, evidence: ev, provenance: ClaimantTerminal { claim: pc, claimant_executor: px } } => (a as String) == "e1" && (pc as String) == "e1" && (px as String) == "executor-1" && (ev.claim as String) == "e2" && (ev.host as String) == "srv5" && (ev.process_pattern as String) == "/srv/build" && (ev.observer as String) == "witness" && ev.observed_at == 1005 _ => false } + EventUndecodable { reason: _ } => false + }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_release(parent: none, admitted_by: "e1", at: 1001)).replace(from: "\"evidence_host\"", to: "\"evidence_hots\"")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) +} + +// The preparation, finalization and abort events round-trip; a preparation of nothing (an empty +// next population) is a valid record on this wire while a committing AUTHORITY with no hosts is +// not; a mis-keyed member refuses. +test fn placement_events_round_trip_through_the_wire() -> Bool { + let prepared = alw_prepare(parent: none, group: FabricGroupB, previous: [alw_host(s: "srv12")], next: [alw_host(s: "srv12"), alw_host(s: "srv13")], at: 1000) + let releasing = alw_prepare(parent: none, group: FabricGroupB, previous: [alw_host(s: "srv12")], next: [] as List, at: 1001) + (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: prepared)) { EventDecoded { event: back } => match back.payload { PlacementPrepared { operation: o, group: FabricGroupB, previous_hosts: ph, next_hosts: nh } => (o as String) == "tx-op" && length(ph) == 1 && length(nh) == 2 _ => false } EventUndecodable { reason: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: releasing)) { EventDecoded { event: back } => match back.payload { PlacementPrepared { operation: _, group: FabricGroupB, previous_hosts: ph, next_hosts: nh } => length(ph) == 1 && length(nh) == 0 _ => false } EventUndecodable { reason: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: prepared).replace(from: "\"group\"", to: "\"grop\"")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_finalize(parent: none, preparation: "p1", at: 1002))) { EventDecoded { event: back } => match back.payload { PlacementFinalized { preparation: pr, authority_event: ae } => (pr.id as String) == "p1" && (ae as String) == "g1" && length(pr.next_hosts) == 1 _ => false } EventUndecodable { reason: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_abort(parent: none, preparation: "p1", at: 1002))) { EventDecoded { event: back } => match back.payload { PlacementAborted { preparation: pr, reason: r, provenance: ClaimantTerminal { claim: c, claimant_executor: _ } } => (pr as String) == "p1" && (c as String) == "p1" && r.contains("did not land") _ => false } EventUndecodable { reason: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_establish(a: alw_w_active_unestablished())).replace(from: "\"prep_id\"", to: "\"prep_di\"")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_claim(parent: none, preparation: "p1", at: 1002))) { EventDecoded { event: back } => match back.payload { PlacementAppendClaimed { preparation: pr, intent: i } => (pr as String) == "p1" && (i.actor as String) == "witness" && write_intent_eq(a: i, b: alw_srv5_intent(actor: "witness")) _ => false } EventUndecodable { reason: _ } => false }) +} + +// THE OBSERVER: quiet is minted only from a process scan that matched nothing and, when a +// container is declared, a docker ps that listed nothing. A listing is residue; a scan that did +// not reach a verdict (pgrep exit 2, docker ps non-zero, a leg that did not run) is unread; and +// the evidence carries exactly the host and residue it was taken against. +test fn the_observer_mints_quiet_only_from_a_scan_that_found_nothing() -> Bool { + let srv5 = alw_host(s: "srv5") + let plain = alw_residue(pattern: "/srv/build") + let with_container = HostEffectResidueSpec { process_pattern: "/srv/build" as NonEmptyStr, container: Present { value: DockerPsByName { name: "vllm-a" as NonEmptyStr } } } + (match alw_observe(leg: fn(argv) { alw_quiet_leg(argv: argv) }, spec: plain) { HostQuiet { evidence: e } => (e.host as String) == "srv5" && (e.claim as String) == "e2" && (match e.container { Absent => true Present { value: _ } => false }) _ => false }) + && (match alw_observe(leg: fn(argv) { ArgvRan { exit_code: 0, stdout: "4242\n", stderr: "" } }, spec: plain) { HostResidue { host: _, detail: d } => d.contains("4242") _ => false }) + && (match alw_observe(leg: fn(argv) { ArgvRan { exit_code: 2, stdout: "", stderr: "pgrep: bad pattern" } }, spec: plain) { HostQuiescenceUnread { host: _, cause: c } => c.contains("bad pattern") _ => false }) + && (match alw_observe(leg: fn(argv) { ArgvLegDidNotRun { cause: "no route" } }, spec: plain) { HostQuiescenceUnread { host: _, cause: c } => c.contains("no route") _ => false }) + && (match alw_observe(leg: fn(argv) { alw_quiet_leg(argv: argv) }, spec: with_container) { HostQuiet { evidence: e } => (match e.container { Present { value: DockerPsByName { name: n } } => (n as String) == "vllm-a" _ => false }) _ => false }) + && (match alw_observe(leg: fn(argv) { match get(xs: argv, index: 0) { Present { value: "pgrep" } => ArgvRan { exit_code: 1, stdout: "", stderr: "" } _ => ArgvRan { exit_code: 0, stdout: "0a1b2c\n", stderr: "" } } }, spec: with_container) { HostResidue { host: _, detail: d } => d.contains("0a1b2c") _ => false }) + && (match alw_observe(leg: fn(argv) { match get(xs: argv, index: 0) { Present { value: "pgrep" } => ArgvRan { exit_code: 1, stdout: "", stderr: "" } _ => ArgvRan { exit_code: 1, stdout: "", stderr: "Cannot connect to the Docker daemon" } } }, spec: with_container) { HostQuiescenceUnread { host: _, cause: c } => c.contains("Docker daemon") _ => false }) +} + +// THE EVIDENCE IS FOR ONE CLAIM: a release carrying a quiet reading of another host, of another +// residue on the same host, taken for another claim, or taken before this claim was admitted, +// refuses at that event and the claim stays live (a commitment of the host after it still +// refuses); the reading for this claim, its host and residue, after its admission, releases it +// and the commitment then lands. +test fn a_release_whose_evidence_is_for_another_host_or_effect_refuses_at_that_event() -> Bool { + let srv5 = alw_host(s: "srv5") + let claim = alw_env(id: "e2", e: alw_effect(parent: none, host: srv5, at: 1000)) + let commit = alw_prepare(parent: Present { value: "e3" as EventId }, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1050) + alw_refuses_at_e3(prefix: [claim], after: commit, evidence: alw_evidence(host: alw_host(s: "srv6"), spec: alw_residue(pattern: "/srv/build"))) + && alw_refuses_at_e3(prefix: [claim], after: commit, evidence: alw_evidence(host: srv5, spec: alw_residue(pattern: "/srv/other"))) + && alw_refuses_at_e3(prefix: [claim], after: commit, evidence: alw_evidence(host: srv5, spec: HostEffectResidueSpec { process_pattern: "/srv/build" as NonEmptyStr, container: Present { value: DockerPsByName { name: "x" as NonEmptyStr } } })) + && alw_refuses_at_e3(prefix: [claim], after: commit, evidence: alw_evidence_at(claim: "e9", host: srv5, spec: alw_residue(pattern: "/srv/build"), at: 1005)) + && alw_refuses_at_e3(prefix: [claim], after: commit, evidence: alw_evidence_at(claim: "e2", host: srv5, spec: alw_residue(pattern: "/srv/build"), at: 999)) + && (match alw_evidence(host: srv5, spec: alw_residue(pattern: "/srv/build")) { + Absent => false + Present { value: e } => alw_a_commits(f: placement_fold(oldest_first: concat(alw_chain_releasing(prefix: [claim], after: commit, evidence: e), [alw_env(id: "e4c", e: alw_claim(parent: Present { value: "e4" as EventId }, preparation: "e4", at: 1050)), alw_env(id: "e5", e: alw_finalize(parent: Present { value: "e4c" as EventId }, preparation: "e4", at: 1051))])), hosts: ["srv5"]) + }) +} + +fn alw_chain_releasing(prefix: List>, after: ChainEvent, evidence: HostQuiescenceEvidence) -> List> { + concat(prefix, [alw_env(id: "e3", e: alw_release_with(parent: Present { value: "e2" as EventId }, admitted_by: "e2", at: 1010, evidence: evidence)), alw_env(id: "e4", e: after)]) +} + +fn alw_refuses_at_e3(prefix: List>, after: ChainEvent, evidence: HostQuiescenceEvidence?) -> Bool { + match evidence { + Absent => false + Present { value: e } => + match placement_fold(oldest_first: alw_chain_releasing(prefix: prefix, after: after, evidence: e)) { + PlacementFoldRefused { at_event: at, reason: r } => (at as String) == "e3" && r.contains("not for claim e2") + _ => false + } + } +} + +// THE PLACEMENT FOLD: a preparation that would commit a host under a live claim refuses at that +// event -- STILL after the claim's term, since liveness is quiescence-required, not timed -- and +// after the release the same preparation folds and, finalized, is the live commitment; a claim on +// a host a pending preparation or a live commitment fences refuses; a second claim on a held host +// refuses; a release of a claim that is not live refuses; two groups cannot fence one host, but an +// aborted preparation frees it; a preparation whose previous population is not the group's live +// commitment refuses; a finalization or an abort of a non-pending preparation refuses; a release +// preparation (next = []) keeps the OLD hosts fenced until finalized; an authority event on this +// partition refuses; a placement event on a GROUP partition refuses in the authority fold. +test fn the_placement_fold_fences_preparations_under_a_live_host_effect() -> Bool { + let srv5 = alw_host(s: "srv5") + let claim = alw_env(id: "e2", e: alw_effect(parent: none, host: srv5, at: 1000)) + let prep = alw_prepare(parent: Present { value: "e2" as EventId }, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1050) + let prep_late = alw_prepare(parent: Present { value: "e2" as EventId }, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1200) + let prep_b = alw_prepare(parent: none, group: FabricGroupB, previous: [] as List, next: [srv5], at: 1060) + let committed = alw_a_over_srv5(parent: none, at: 1050) + (match placement_fold(oldest_first: [claim, alw_env(id: "e3", e: prep)]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3" && r.contains("live host effect") _ => false }) + && (match placement_fold(oldest_first: [claim, alw_env(id: "e3", e: prep_late)]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3" && r.contains("OVERDUE") _ => false }) + && alw_a_commits(f: placement_fold(oldest_first: concat([claim, alw_env(id: "r", e: alw_release(parent: Present { value: "e2" as EventId }, admitted_by: "e2", at: 1010))], committed)), hosts: ["srv5"]) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e9", e: alw_effect(parent: Present { value: "e3" as EventId }, host: srv5, at: 1051))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e9" && r.contains("fenced it") _ => false }) + && (match placement_fold(oldest_first: concat(committed, [alw_env(id: "e9", e: alw_effect(parent: Present { value: "e4" as EventId }, host: srv5, at: 1052))])) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e9" && r.contains("fenced it") _ => false }) + && (match placement_fold(oldest_first: [claim, alw_env(id: "e3", e: alw_effect(parent: Present { value: "e2" as EventId }, host: srv5, at: 1001))]) { PlacementFoldRefused { at_event: e, reason: _ } => (e as String) == "e3" _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: alw_release(parent: none, admitted_by: "nope", at: 1001))]) { PlacementFoldRefused { at_event: e, reason: _ } => (e as String) == "e3" _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e5", e: prep_b)]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e5" && r.contains("another group") _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e5", e: alw_abort(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1055)), alw_env(id: "e6", e: prep_b)]) { PlacementFolded { effects: _, preparations: preps } => length(group_fenced_hosts(preps: preps, group: FabricGroupB)) == 1 && length(group_fenced_hosts(preps: preps, group: FabricGroupA)) == 0 _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e5", e: alw_abort(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1055)), alw_env(id: "e6", e: alw_effect(parent: Present { value: "e5" as EventId }, host: srv5, at: 1056))]) { PlacementFolded { effects: effs, preparations: _ } => length(effs) == 1 _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: alw_prepare(parent: none, group: FabricGroupA, previous: [srv5], next: [srv5], at: 1000))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3" && r.contains("previous population") _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e1", e: alw_abort(parent: none, preparation: "nope", at: 1055))]) { PlacementFoldRefused { at_event: e, reason: _ } => (e as String) == "e1" _ => false }) + && (match placement_fold(oldest_first: concat(committed, [alw_env(id: "e5", e: alw_finalize(parent: Present { value: "e4" as EventId }, preparation: "e3", at: 1052))])) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e5" && r.contains("no pending preparation") _ => false }) + && (match placement_fold(oldest_first: concat(committed, [alw_env(id: "e5", e: alw_abort(parent: Present { value: "e4" as EventId }, preparation: "e3", at: 1052))])) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e5" && r.contains("no pending preparation") _ => false }) + && (match placement_fold(oldest_first: concat(committed, [alw_env(id: "e5", e: alw_prepare(parent: Present { value: "e4" as EventId }, group: FabricGroupA, previous: [srv5], next: [] as List, at: 1060)), alw_env(id: "e6", e: alw_effect(parent: Present { value: "e5" as EventId }, host: srv5, at: 1061))])) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e6" && r.contains("fenced it") _ => false }) + && (match placement_fold(oldest_first: concat(committed, [alw_env(id: "e5", e: alw_prepare(parent: Present { value: "e4" as EventId }, group: FabricGroupA, previous: [srv5], next: [] as List, at: 1060)), alw_env(id: "e5c", e: alw_release_claim(parent: Present { value: "e5" as EventId }, preparation: "e5", at: 1060)), alw_env(id: "e6", e: alw_finalize_of(parent: Present { value: "e5c" as EventId }, preparation: "e5", previous: [srv5], next: [] as List, at: 1061)), alw_env(id: "e7", e: alw_effect(parent: Present { value: "e6" as EventId }, host: srv5, at: 1062))])) { PlacementFolded { effects: effs, preparations: preps } => length(effs) == 1 && length(group_fenced_hosts(preps: preps, group: FabricGroupA)) == 0 _ => false }) + && (match placement_fold(oldest_first: concat(committed, [alw_env(id: "e5", e: alw_prepare(parent: Present { value: "e4" as EventId }, group: FabricGroupA, previous: [srv5], next: [] as List, at: 1060)), alw_env(id: "e5c", e: alw_release_claim(parent: Present { value: "e5" as EventId }, preparation: "e5", at: 1060)), alw_env(id: "e6", e: alw_finalize_of(parent: Present { value: "e5c" as EventId }, preparation: "e5", previous: [srv5], next: [srv5], at: 1061))])) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e6" && r.contains("not the record") _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e4", e: alw_finalize(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1051))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e4" && r.contains("never claimed") _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e4", e: alw_prepare(parent: Present { value: "e3" as EventId }, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1051))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e4" && r.contains("one saga at a time") _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e3c", e: alw_claim(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1050)), alw_env(id: "e4", e: alw_abort_by(parent: Present { value: "e3c" as EventId }, preparation: "e3", at: 1051, actor: "someone-else"))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e4" && r.contains("not its claimant's own") _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e3c", e: alw_claim(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1050)), alw_env(id: "e4", e: alw_abort(parent: Present { value: "e3c" as EventId }, preparation: "e3", at: 1051))]) { PlacementFolded { effects: _, preparations: preps } => length(group_fenced_hosts(preps: preps, group: FabricGroupA)) == 0 _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e3", e: prep), alw_env(id: "e3c", e: alw_claim(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1050)), alw_env(id: "e4", e: alw_claim(parent: Present { value: "e3c" as EventId }, preparation: "e3", at: 1051))]) { PlacementFoldRefused { at_event: e, reason: _ } => (e as String) == "e4" _ => false }) + && (match placement_fold(oldest_first: [alw_env(id: "e1", e: ChainEvent { partition: host_placement_partition, parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityEstablished { authority: alw_w_active_unestablished(), placement: alw_establish_ref(a: alw_w_active_unestablished()) } })]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e1" && r.contains("authority event") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e0", e: alw_establish(a: alw_w_active_unestablished())), alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: HostEffectAdmitted { host: srv5, purpose: "a build" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, term: second(count: 100), residue: alw_residue(pattern: "/srv/build") } })]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e1" && r.contains("host-placement partition only") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: PlacementPrepared { operation: "x" as NonEmptyStr, group: FabricGroupA, previous_hosts: [] as List, next_hosts: [srv5] } })]) { AuthorityFoldRefused { at_event: e, reason: _ } => (e as String) == "e1" _ => false }) +} + +// THE ESTABLISHMENT IS ONE EVENT FROM THE DURABLE GENESIS: an empty chain is unestablished (no +// authority, no generation); establishing folds to generation 0 with the row as current; a second +// establishment refuses; a transition or an entry state before any establishment refuses; a +// leased state cannot be established; an authority naming another group than the partition's +// refuses; the established event round-trips. +test fn the_partition_folds_from_an_unestablished_genesis_through_the_establishment() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + let elsewhere = PairServingActive { group: FabricGroupB, exact_realization: PairRealizationUnestablished { obligation: "w" as NonEmptyStr } } + let to_active = ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1050, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: released, next: resting, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: released, next: resting, operation: "tx-op") } } + (match authority_fold(oldest_first: [] as List>) { AuthorityUnestablished { cancelled: _ } => true _ => false }) + && (match alw_fold(resting: resting, oldest_first: [] as List>) { AuthorityFolded { current: c, generation: g, preparations_consumed: pcs } => g == 0 && authority_eq(a: c, b: resting) && length(pcs) == 1 && (match first(pcs) { Present { value: pp } => (pp.preparation.id as String) == "p0" Absent => false }) _ => false }) + && (match alw_fold(resting: resting, oldest_first: [alw_env(id: "e1", e: alw_establish(a: released))]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e1" && r.contains("already established") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e1", e: to_active)]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e1" && r.contains("before any authority") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e0", e: alw_establish(a: pending))]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e0" && r.contains("lease") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e0", e: alw_establish(a: elsewhere))]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e0" && r.contains("not this partition's group") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e0", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 990, actor: "witness" as NonEmptyStr, payload: AuthorityEstablished { authority: resting, placement: PlacementPreparationRef { id: "p0" as EventId, group: FabricGroupB, previous_hosts: [] as List, next_hosts: committed_population(a: resting), operation: "establish group-a" as NonEmptyStr } } })]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e0" && r.contains("belongs to group-b") _ => false }) + && (match authority_fold(oldest_first: [alw_env(id: "e0", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 990, actor: "witness" as NonEmptyStr, payload: AuthorityEstablished { authority: resting, placement: PlacementPreparationRef { id: "p0" as EventId, group: FabricGroupA, previous_hosts: [] as List, next_hosts: [alw_host(s: "srv9")], operation: "establish group-a" as NonEmptyStr } } })]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e0" && r.contains("not the next state's") _ => false }) + && (match alw_fold(resting: resting, oldest_first: [alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: released, operation: "tx-other") } })]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e1" && r.contains("not this event's") _ => false }) + && (match alw_fold(resting: resting, oldest_first: [alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: resting, operation: "tx-op") } })]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e1" && r.contains("not the next state's") + && (match write_intent_of_witness(env: alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "someone-else" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: released, operation: "tx-op") } }), previous: resting, next: released) { derived => !write_intent_eq(a: derived, b: alw_intent(parent: none, previous: Present { value: resting }, next: released, operation: "tx-op", lifecycle: none, grant: none, entry_state: none)) }) + && (match write_intent_of_witness(env: alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: Present { value: "e0" as EventId }, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: released, operation: "tx-op") } }), previous: resting, next: released) { derived => !write_intent_eq(a: derived, b: alw_intent(parent: none, previous: Present { value: resting }, next: released, operation: "tx-op", lifecycle: none, grant: none, entry_state: none)) }) + && (match write_intent_of_witness(env: alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: resting, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: resting, operation: "tx-op") } }), previous: resting, next: resting) { derived => !write_intent_eq(a: derived, b: alw_intent(parent: none, previous: Present { value: resting }, next: released, operation: "tx-op", lifecycle: none, grant: none, entry_state: none)) }) + && (match write_intent_of_witness(env: alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: none, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: released, operation: "tx-op") } }), previous: resting, next: released) { derived => write_intent_eq(a: derived, b: alw_intent(parent: none, previous: Present { value: resting }, next: released, operation: "tx-op", lifecycle: none, grant: none, entry_state: none)) }) _ => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_establish(a: resting))) { EventDecoded { event: back } => match back.payload { AuthorityEstablished { authority: a, placement: _ } => authority_eq(a: a, b: resting) _ => false } EventUndecodable { reason: _ } => false }) + }) +} + +// EVIDENCE IS NOT REPLAYABLE ACROSS CLAIMS, AND A RECOVERY NAMES ITS CLAIM: the quiet reading that +// released C1 cannot release a later C2 on the same host with the same residue (the reading names +// C1); a recovery provenance naming another claim refuses; one naming this claim, with this +// claim's evidence, releases. Both round-trip through the wire. +test fn a_claims_evidence_cannot_release_a_later_identical_claim_and_a_recovery_names_its_claim() -> Bool { + let srv5 = alw_host(s: "srv5") + let c1 = alw_env(id: "e2", e: alw_effect(parent: none, host: srv5, at: 1000)) + let c2 = alw_env(id: "e4", e: alw_effect(parent: Present { value: "e3" as EventId }, host: srv5, at: 1020)) + match alw_evidence_at(claim: "e2", host: srv5, spec: alw_residue(pattern: "/srv/build"), at: 1010) { + Absent => false + Present { value: ev1 } => + match alw_evidence_at(claim: "e4", host: srv5, spec: alw_residue(pattern: "/srv/build"), at: 1030) { + Absent => false + Present { value: ev2 } => { + let r1 = alw_env(id: "e3", e: alw_release_with(parent: Present { value: "e2" as EventId }, admitted_by: "e2", at: 1010, evidence: ev1)) + let replay = alw_env(id: "e5", e: alw_release_with(parent: Present { value: "e4" as EventId }, admitted_by: "e4", at: 1030, evidence: ev1)) + let recovery_other = ChainEvent { partition: host_placement_partition, parent: Present { value: "e4" as EventId }, recorded_at: 1030, actor: "operator" as NonEmptyStr, payload: HostEffectReleased { admitted_by: "e4" as EventId, evidence: ev2, provenance: alw_recovery_for(claim: "e2") } } + let recovery_own = ChainEvent { partition: host_placement_partition, parent: Present { value: "e4" as EventId }, recorded_at: 1030, actor: "operator" as NonEmptyStr, payload: HostEffectReleased { admitted_by: "e4" as EventId, evidence: ev2, provenance: alw_recovery_for(claim: "e4") } } + (match placement_fold(oldest_first: [c1, r1, c2, replay]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e5" && r.contains("not for claim e4") _ => false }) + && (match placement_fold(oldest_first: [c1, r1, c2, alw_env(id: "e5", e: recovery_other)]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e5" && r.contains("release provenance is not for claim e4") _ => false }) + && (match placement_fold(oldest_first: [c1, r1, c2, alw_env(id: "e5", e: recovery_own)]) { PlacementFolded { effects: effs, preparations: _ } => length(filter(effs, r => !r.released)) == 0 _ => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: recovery_own)) { EventDecoded { event: back } => match back.payload { HostEffectReleased { admitted_by: _, evidence: _, provenance: RecoveryAuthorized { claim: c, claimant: ClaimantTerminated, authorized_by: by, receipt: rc } } => (c as String) == "e4" && (by as String) == "operator" && (rc as String) == "run 42 cancelled" _ => false } EventUndecodable { reason: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: recovery_own).replace(from: "\"terminated\"", to: "\"tired\"")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) + } + } + } +} + +fn alw_recovery_for(claim: String) -> ReleaseProvenance { + RecoveryAuthorized { claim: claim as EventId, claimant: ClaimantTerminated, authorized_by: "operator" as NonEmptyStr, receipt: "run 42 cancelled" as NonEmptyStr } +} + +// A CLAIMANT-TERMINAL RELEASE IS THE CLAIMANT'S: written by another executor (E2) against E1's +// claim, with evidence correctly bound to the claim, it refuses at that event and the claim stays +// live; written by E1 it releases; naming another claim in its provenance, it refuses. A recovery +// by an operator, naming the claim, is the other executor's only route. +test fn a_terminal_release_by_another_executor_refuses() -> Bool { + let srv5 = alw_host(s: "srv5") + let claim = alw_env(id: "e2", e: alw_effect(parent: none, host: srv5, at: 1000)) + let commit = alw_env(id: "e4", e: alw_prepare(parent: Present { value: "e3" as EventId }, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1050)) + let claimed = alw_env(id: "e4c", e: alw_claim(parent: Present { value: "e4" as EventId }, preparation: "e4", at: 1050)) + let fin = alw_env(id: "e5", e: alw_finalize(parent: Present { value: "e4c" as EventId }, preparation: "e4", at: 1051)) + match alw_evidence_at(claim: "e2", host: srv5, spec: alw_residue(pattern: "/srv/build"), at: 1010) { + Absent => false + Present { value: ev } => { + let by_e2 = alw_env(id: "e3", e: alw_release_by(parent: Present { value: "e2" as EventId }, admitted_by: "e2", at: 1010, evidence: ev, actor: "executor-2")) + let by_e1 = alw_env(id: "e3", e: alw_release_by(parent: Present { value: "e2" as EventId }, admitted_by: "e2", at: 1010, evidence: ev, actor: "executor-1")) + let names_other = alw_env(id: "e3", e: ChainEvent { partition: host_placement_partition, parent: Present { value: "e2" as EventId }, recorded_at: 1010, actor: "executor-1" as NonEmptyStr, payload: HostEffectReleased { admitted_by: "e2" as EventId, evidence: ev, provenance: ClaimantTerminal { claim: "e9" as EventId, claimant_executor: "executor-1" as NonEmptyStr } } }) + let by_operator = alw_env(id: "e3", e: ChainEvent { partition: host_placement_partition, parent: Present { value: "e2" as EventId }, recorded_at: 1010, actor: "operator" as NonEmptyStr, payload: HostEffectReleased { admitted_by: "e2" as EventId, evidence: ev, provenance: alw_recovery_for(claim: "e2") } }) + (match placement_fold(oldest_first: [claim, by_e2, commit]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3" && r.contains("written by its claimant executor-1") _ => false }) + && (match placement_fold(oldest_first: [claim, names_other, commit]) { PlacementFoldRefused { at_event: e, reason: _ } => (e as String) == "e3" _ => false }) + && alw_a_commits(f: placement_fold(oldest_first: [claim, by_e1, commit, claimed, fin]), hosts: ["srv5"]) + && alw_a_commits(f: placement_fold(oldest_first: [claim, by_operator, commit, claimed, fin]), hosts: ["srv5"]) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: alw_release_by(parent: none, admitted_by: "e2", at: 1010, evidence: ev, actor: "executor-1")).replace(from: "\"terminal_claimant\"", to: "\"terminal_claimnt\"")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) + } + } +} + +// ── ROUND 18: THE CLAIM IS ITS CLAIMANT'S, CONSUMED AND CANCELLED EXCLUDE EACH OTHER, THE +// CANCELLATION MOVES THE HEAD THE CLAIM NAMED, AND THE INTENT IS COMPARED FIELD FOR FIELD ───── + +// A claim over the [] -> [srv5] preparation written by `actor` for a write by `claimant` (the +// intent's actor). +fn alw_claim_written_by(parent: EventId?, preparation: String, actor: String, claimant: String, at: Int) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: actor as NonEmptyStr, payload: PlacementAppendClaimed { preparation: (preparation as NonEmptyStr) as EventId, intent: alw_srv5_intent(actor: claimant) } } +} + +// A claim by the witness whose intent is for another write than its preparation: another group, +// another population, another operation. +fn alw_claim_for(parent: EventId?, preparation: String, intent: AuthorityWriteIntent, at: Int) -> ChainEvent { + ChainEvent { partition: host_placement_partition, parent: parent, recorded_at: at, actor: "witness" as NonEmptyStr, payload: PlacementAppendClaimed { preparation: (preparation as NonEmptyStr) as EventId, intent: intent } } +} + +// A cancellation of `preparation` on group A's partition by an operator recovery, at `parent`. +fn alw_cancel(parent: EventId?, preparation: String, at: Int) -> ChainEvent { + ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: parent, recorded_at: at, actor: "operator" as NonEmptyStr, payload: AuthorityAppendCancelled { preparation: (preparation as NonEmptyStr) as EventId, provenance: alw_recovery_for(claim: preparation) } } +} + +// A claim written by E2 naming E1 as claimant refuses in the placement fold; the same claim +// written by E1 lands as AppendClaimed by E1. +test fn an_append_claim_is_written_by_its_claimant_and_is_the_write_its_preparation_is_for() -> Bool { + let srv5 = alw_host(s: "srv5") + let prep = alw_env(id: "e3", e: alw_prepare(parent: none, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1000)) + (match placement_fold(oldest_first: [prep, alw_env(id: "e3c", e: alw_claim_written_by(parent: Present { value: "e3" as EventId }, preparation: "e3", actor: "executor-2", claimant: "executor-1", at: 1001))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3c" && r.contains("written by its claimant") _ => false }) + && (match placement_fold(oldest_first: [prep, alw_env(id: "e3c", e: alw_claim_written_by(parent: Present { value: "e3" as EventId }, preparation: "e3", actor: "executor-1", claimant: "executor-1", at: 1001))]) { PlacementFolded { effects: _, preparations: ps } => any(ps, r => (r.id as String) == "e3" && (match r.state { AppendClaimed { claimant: c, expected_head: _, intent: _ } => (c as String) == "executor-1" _ => false })) _ => false }) + && (match placement_fold(oldest_first: [prep, alw_env(id: "e3c", e: alw_claim_for(parent: Present { value: "e3" as EventId }, preparation: "e3", intent: authority_write_intent(group: FabricGroupB, head: HeadAbsent, actor: "witness" as NonEmptyStr, previous: none, next: FencedRefusal { group: FabricGroupB, hosts: [srv5], cause: "w" as NonEmptyStr, disposition_authority: "w" as NonEmptyStr }, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none), at: 1001))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3c" && r.contains("not the write its preparation is for") _ => false }) + && (match placement_fold(oldest_first: [prep, alw_env(id: "e3c", e: alw_claim_for(parent: Present { value: "e3" as EventId }, preparation: "e3", intent: authority_write_intent(group: FabricGroupA, head: HeadAbsent, actor: "witness" as NonEmptyStr, previous: none, next: alw_fenced_over(hs: [alw_host(s: "srv6")]), operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none), at: 1001))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3c" && r.contains("not the next state's") _ => false }) + && (match placement_fold(oldest_first: [prep, alw_env(id: "e3c", e: alw_claim_for(parent: Present { value: "e3" as EventId }, preparation: "e3", intent: authority_write_intent(group: FabricGroupA, head: HeadAbsent, actor: "witness" as NonEmptyStr, previous: none, next: alw_fenced_over(hs: [srv5]), operation: "tx-other" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none), at: 1001))]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e3c" && r.contains("not this event's") _ => false }) +} + +// On the group partition, a cancellation of a preparation an accepted authority event already +// consumed refuses at the cancellation (the authority stays readable); a cancellation of a +// preparation nothing consumed lands and a later consumer of it refuses (round 16). +test fn a_consumed_preparation_cannot_be_cancelled_and_a_cancelled_one_cannot_be_consumed() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let est = alw_env(id: "e0", e: alw_establish(a: resting)) + let tx = alw_env(id: "e1", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: Present { value: "e0" as EventId }, recorded_at: 1000, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: released, operation: "tx-op") } }) + let pid = (alw_ref(previous: resting, next: released, operation: "tx-op").id as String) + (match authority_fold(oldest_first: [est, tx, alw_env(id: "e2", e: alw_cancel(parent: Present { value: "e1" as EventId }, preparation: pid, at: 1001))]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e2" && r.contains("already consumed") _ => false }) + && (match authority_fold(oldest_first: [est, tx]) { AuthorityFolded { generation: g } => g == 1 _ => false }) + && (match authority_fold(oldest_first: [est, alw_env(id: "e2", e: alw_cancel(parent: Present { value: "e0" as EventId }, preparation: pid, at: 1001)), alw_env(id: "e3", e: ChainEvent { partition: pair_authority_partition(group: FabricGroupA), parent: Present { value: "e2" as EventId }, recorded_at: 1002, actor: "witness" as NonEmptyStr, payload: AuthorityTransitioned { previous: resting, next: released, transaction: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, placement: alw_ref(previous: resting, next: released, operation: "tx-op") } })]) { AuthorityFoldRefused { at_event: e, reason: r } => (e as String) == "e3" && r.contains("cancelled") _ => false }) + && (match authority_fold(oldest_first: [est, alw_env(id: "e2", e: alw_cancel(parent: Present { value: "e0" as EventId }, preparation: pid, at: 1001))]) { AuthorityFolded { generation: g, cancelled: xs } => g == 0 && length(xs) == 1 && any(xs, c => (c.preparation as String) == pid && (match c.at_head { HeadAt { id: hd } => (hd as String) == "e0" HeadAbsent => false })) _ => false }) + }) +} + +// The joined read's word on a cancellation: covered by a recovery naming the preparation AND +// appended at the head the claim named; a cancellation at another head is a gap. +test fn a_cancellation_must_move_the_head_its_claim_named() -> Bool { + let srv5 = alw_host(s: "srv5") + let claimed_at_e0 = PlacementPreparationRecord { id: "p1" as EventId, operation: "tx-op" as NonEmptyStr, group: FabricGroupA, previous_hosts: [] as List, next_hosts: [srv5], recorded_at: 1000, prepared_by: "witness" as NonEmptyStr, state: AppendClaimed { claimant: "witness" as NonEmptyStr, expected_head: HeadAt { id: "e0" as EventId }, intent: alw_intent(parent: Present { value: "e0" as EventId }, previous: none, next: alw_w_active_unestablished(), operation: "tx-op", lifecycle: none, grant: none, entry_state: none) } } + let at_e0 = CancelledPreparation { preparation: "p1" as EventId, provenance: alw_recovery_for(claim: "p1"), actor: "operator" as NonEmptyStr, at_head: HeadAt { id: "e0" as EventId } } + let at_e9 = CancelledPreparation { preparation: "p1" as EventId, provenance: alw_recovery_for(claim: "p1"), actor: "operator" as NonEmptyStr, at_head: HeadAt { id: "e9" as EventId } } + let stranger = CancelledPreparation { preparation: "p1" as EventId, provenance: ClaimantTerminal { claim: "p1" as EventId, claimant_executor: "someone-else" as NonEmptyStr }, actor: "someone-else" as NonEmptyStr, at_head: HeadAt { id: "e0" as EventId } } + length(cancellation_gap(cancelled: at_e0, preps: [claimed_at_e0])) == 0 + && any(cancellation_gap(cancelled: at_e9, preps: [claimed_at_e0]), g => g.contains("other than the one its append claim named")) + && any(cancellation_gap(cancelled: stranger, preps: [claimed_at_e0]), g => g.contains("not its claimant's")) + && any(cancellation_gap(cancelled: at_e0, preps: [] as List), g => g.contains("not on the placement chain")) +} + +// The intent is a typed value compared field for field -- each field alone distinguishes two +// writes (no digest stands between the claim and the event) -- and a claim carrying a previous +// state, a lifecycle, a grant and an entry state round-trips through the wire to the same value. +test fn the_write_intent_is_compared_field_for_field_and_round_trips() -> Bool { + alw_with_hash(scenario: fn(h) { + let resting = alw_w_active_unestablished() + let released = ReleasedToFleet { group: FabricGroupA, release_receipt: h } + let pending = SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-1" as NonEmptyStr, authorization_binding: h, lease: alw_w_lease(h: h, inaccessible: false) } + let base = authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: released, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none) + let full = authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: pending, operation: "tx-1" as NonEmptyStr, lifecycle: Present { value: h }, grant: none, entry_state: Present { value: h }) + let claim = ChainEvent { partition: host_placement_partition, parent: none, recorded_at: 1001, actor: "witness" as NonEmptyStr, payload: PlacementAppendClaimed { preparation: "p1" as EventId, intent: full } } + write_intent_eq(a: base, b: base) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAbsent, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: released, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none)) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "someone-else" as NonEmptyStr, previous: Present { value: resting }, next: released, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none)) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: none, next: released, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none)) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: pending, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none)) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: released, operation: "tx-other" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none)) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: released, operation: "tx-op" as NonEmptyStr, lifecycle: Present { value: h }, grant: none, entry_state: none)) + && !write_intent_eq(a: base, b: authority_write_intent(group: FabricGroupA, head: HeadAt { id: "e0" as EventId }, actor: "witness" as NonEmptyStr, previous: Present { value: resting }, next: released, operation: "tx-op" as NonEmptyStr, lifecycle: none, grant: none, entry_state: Present { value: h })) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: claim)) { EventDecoded { event: back } => match back.payload { PlacementAppendClaimed { preparation: _, intent: i } => write_intent_eq(a: i, b: full) && (match i.expected_head { HeadAt { id: hd } => (hd as String) == "e0" HeadAbsent => false }) _ => false } EventUndecodable { reason: _ } => false }) + && (match pair_authority_event_decode(text: pair_authority_event_wire_text(event: claim).replace(from: "\"intent_prev_state\"", to: "\"intent_prev_stale\"")) { EventUndecodable { reason: _ } => true EventDecoded { event: _ } => false }) + }) +} + +// A PREPARED PREPARATION'S FENCE IS ITS PREPARER'S: a stranger's claimant-terminal abort of it +// refuses in the placement fold (the fence stays), the preparer's own lands, and an operator's +// recovery naming it lands. +test fn a_prepared_preparation_is_aborted_only_by_its_preparer_or_a_recovery() -> Bool { + let srv5 = alw_host(s: "srv5") + let prep = alw_env(id: "e3", e: alw_prepare(parent: none, group: FabricGroupA, previous: [] as List, next: [srv5], at: 1000)) + let stranger = ChainEvent { partition: host_placement_partition, parent: Present { value: "e3" as EventId }, recorded_at: 1001, actor: "someone-else" as NonEmptyStr, payload: PlacementAborted { preparation: "e3" as EventId, reason: "not mine", provenance: ClaimantTerminal { claim: "e3" as EventId, claimant_executor: "someone-else" as NonEmptyStr } } } + let recovery = ChainEvent { partition: host_placement_partition, parent: Present { value: "e3" as EventId }, recorded_at: 1001, actor: "operator" as NonEmptyStr, payload: PlacementAborted { preparation: "e3" as EventId, reason: "the lane died", provenance: alw_recovery_for(claim: "e3") } } + (match placement_fold(oldest_first: [prep, alw_env(id: "e4", e: stranger)]) { PlacementFoldRefused { at_event: e, reason: r } => (e as String) == "e4" && r.contains("not its claimant's own") _ => false }) + && (match placement_fold(oldest_first: [prep, alw_env(id: "e4", e: alw_abort(parent: Present { value: "e3" as EventId }, preparation: "e3", at: 1001))]) { PlacementFolded { effects: _, preparations: ps } => any(ps, r => (r.id as String) == "e3" && (match r.state { Aborted => true _ => false })) _ => false }) + && (match placement_fold(oldest_first: [prep, alw_env(id: "e4", e: recovery)]) { PlacementFolded { effects: _, preparations: ps } => any(ps, r => (r.id as String) == "e3" && (match r.state { Aborted => true _ => false })) _ => false }) +} diff --git a/dag/test/claim/spark/pair_serving_authority_witness_test.dag b/dag/test/claim/spark/pair_serving_authority_witness_test.dag index 73e0596fd47..9354f94af8f 100644 --- a/dag/test/claim/spark/pair_serving_authority_witness_test.dag +++ b/dag/test/claim/spark/pair_serving_authority_witness_test.dag @@ -6,6 +6,7 @@ import v2.std.optional { Present, Absent } import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest } import std.temporal_effect { HeldLease, held_lease, LeaseEpoch, LeaseRunningExpected, LeaseRunningStale, LeaseAbsent, LeasePortForeign, LeaseInaccessible } import gunbc.spark.fabric_switch_observed { FabricGroupA, FabricGroupB, fabric_group_hosts, fabric_group_wire } +import gunbc.spark.released_baseline { QuiescentReservedBaseline } import gunbc.spark.pair_serving_desired { spark_pair_serving_unit } import gunbc.spark.pair_serving_apply { spark_pair_apply_plan_under, SparkPairApplyPlan, SparkPairApplyPlanned, SparkPairApplyRefused } import gunbc.spark.host_commitment { spark_claimed_serving_group_members, spark_claimed_members_under } @@ -59,25 +60,26 @@ fn w_active() -> PairServingGroupAuthority { fn w_pending(h: ContentHash, stale: Bool) -> PairServingGroupAuthority { SuspensionPendingReconciliation { - group: FabricGroupA, + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "witness-transaction" as NonEmptyStr, + authorization_binding: h, lease: w_lease(h: h, observed_stale: stale), } } fn w_suspended(h: ContentHash, stale: Bool) -> PairServingGroupAuthority { SuspendedForAuthorizedSuccessor { - group: FabricGroupA, + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), authorization: "witness-authorization" as NonEmptyStr, exact_candidate_realization: PairRealizationUnestablished { obligation: "witness" as NonEmptyStr }, lease: w_lease(h: h, observed_stale: stale), - cleanup: "witness-cleanup" as NonEmptyStr, + cleanup: QuiescentReservedBaseline, } } fn w_fenced() -> PairServingGroupAuthority { FencedRefusal { - group: FabricGroupA, + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "witness-foreign-occupant" as NonEmptyStr, disposition_authority: "witness-operator" as NonEmptyStr, } diff --git a/dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag b/dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag new file mode 100644 index 00000000000..e57d506794d --- /dev/null +++ b/dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag @@ -0,0 +1,423 @@ +module test.claim.spark.pair_serving_d0_front_door_real_execution + +import std.logic { Bool } +import std.types { String, NonEmptyStr, FilePath, List, Int, Port } +import v2.std.optional { Present, Absent } +import std.measure { second } +import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } +import extdeps.shell +import extdeps.python +import extdeps.filesystem.filesystem_io { Filesystem } +import std.algebra { trim } +import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest } + +import gunbc.fabric_storage_client { FabricStorageBinding, FabricStorageLocalFiles } +import gunbc.fabric_storage_file_store { fabric_storage_file_root, fabric_storage_file_root_ensure, FabricStorageRootReady } +import gunbc.serving.serving_enrollment { ServingRouteEndpoint } +import gunbc.harness.harness_backend { HarnessBoundedPresence, HarnessResponded, HarnessConnectFailed, HarnessDeadline, HarnessTransportUnread, harness_bounded_presence, harness_models_url } +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA } +import gunbc.spark.pair_serving_authority { FencedRefusal, pair_serving_authority_for } +import gunbc.spark.serving_incarnation_observe { UnitInstalled, EndpointListenerReading, EndpointNoListener, EndpointListening, EndpointListenerUnread, read_endpoint_listener } +import gunbc.host_effect { LocalShell } +import gunbc.spark.pair_incumbent_identity { DeclaredPairRealizationKeyed, DeclaredPairRealizationUnavailable, declared_pair_realization } +import gunbc.spark.pair_serving_authority_log { CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authority, AuthorityTransitionAppended, pair_serving_authority_transition, AuthorityEstablishedAt, AuthorityEstablishmentRefused, pair_serving_authority_establish } +import gunbc.spark.pair_serving_d0 { + FrontDoorReading, FrontDoorNoStatus, FrontDoorAnsweredUnread, FrontDoorUnread, FrontDoorAnswered, observe_front_door, front_door_as_incumbent, + IncumbentReading, IncumbentAnsweredUnread, IncumbentDidNotAnswer, IncumbentRouteUnread, D0Observation, RankOccupancyReading, + HeadEndpointReading, HeadEndpointQuiet, HeadEndpointListening, HeadEndpointUnread, + D0Outcome, D0Settled, D0Decision, D0DecidedFence, D0DecidedSuspend, D0Cause, D0IncumbentUnidentified, D0OccupancyUnread, DeclaredOccupantDrifted, d0_transaction, + AdmittedD0Grant, D0GrantAdmitted, admit_d0_grant, D0Subject, d0_intent_hash, d0_authorized_action, +} +import std.scoped_authorization { OperatorGrant, ScopedAuthorization, AuthorizationGranted, Unclaimed, AuthorizationScope, AttemptIdentity } +import std.effect_grant { Write, NamespacePosition, CodeNameTree } +import gunbc.spark.released_baseline { QuiescentReservedBaseline } +import gunbc.spark.fabric_switch_observed { fabric_group_hosts } + +data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly + +// THE HTTP-STATUS DISTINCTION AT ITS PRODUCTION INTERFACE, BY REAL EXECUTION. The D0 decision has +// hermetic controls over a SUPPLIED IncumbentAnsweredUnread; what none of them can notice is the +// transport regaining curl's -f, or the %{http_code} binding drifting, after which a 500 would read +// as silence again and the supplied-arm claims would stay green. So a local listener that answers +// one chosen status is started, the real http.Client.StatusWithin is run against it through +// harness_bounded_presence and observe_front_door, and the reading is carried through the real +// transaction to FencedRefusal on the log. The listener is CPython's http.server, forked into the +// background by the fixture below, which prints the port and pid; it is killed at the end of the +// scenario and exits by itself after a bounded lifetime if the kill never comes. + +data d0fd_listener_fixture: String = "import http.server, os, socketserver, sys, threading, time\nstatus = int(sys.argv[1]) if sys.argv[1] != 'hang' else 0\nlifetime = float(sys.argv[2])\nclass H(http.server.BaseHTTPRequestHandler):\n def do_GET(self):\n if status == 0:\n time.sleep(lifetime)\n return\n self.send_response(status)\n self.send_header('Content-Length', '0')\n self.end_headers()\n def log_message(self, *a):\n pass\nsocketserver.TCPServer.allow_reuse_address = True\nsrv = socketserver.TCPServer(('127.0.0.1', 0), H)\nport = srv.server_address[1]\npid = os.fork()\nif pid:\n sys.stdout.write('%d %d\\n' % (port, pid))\n sys.stdout.flush()\n os._exit(0)\nos.setsid()\nnull = os.open(os.devnull, os.O_RDWR)\nos.dup2(null, 0)\nos.dup2(null, 1)\nos.dup2(null, 2)\nthreading.Thread(target=srv.serve_forever, daemon=True).start()\ntime.sleep(lifetime)\nos._exit(0)\n" + +data d0fd_kill_fixture: String = "import os, signal, sys\nos.kill(int(sys.argv[1]), signal.SIGTERM)\n" + +type D0fdListener { + port: Int + pid: Int +} + +fn d0fd_with_temp_dir(scenario: fn(FilePath) -> Bool) -> Bool { + let dir = shell.Mktemp.Dir() + let observed = scenario(dir.path) + let cleaned = shell.Remove.RecursiveForce(path: dir.path) + dir.success && observed && cleaned.success +} + +fn d0fd_parse_listener(out: String) -> D0fdListener? { + let parts = split(s: trim(s: out), delimiter: " ") + match get(xs: parts, index: 0) { + Absent => none + Present { value: p } => + match get(xs: parts, index: 1) { + Absent => none + Present { value: q } => + match parse_int(s: p) { + Absent => none + Present { value: port } => + match parse_int(s: q) { + Absent => none + Present { value: pid } => Present { value: D0fdListener { port: port, pid: pid } } + } + } + } + } +} + +// A listener answering `status` for the scenario, killed afterwards whatever the scenario says. +fn d0fd_with_listener(dir: FilePath, status: String, scenario: fn(D0fdListener) -> Bool) -> Bool { + let script = join([dir as String, "/listener.py"], "") + let kill = join([dir as String, "/kill.py"], "") + let w1 = Filesystem.Write(path: script, content: d0fd_listener_fixture) + let w2 = Filesystem.Write(path: kill, content: d0fd_kill_fixture) + if !w1.success || !w2.success { false } else { + let run = python.Interpreter.RunFile(workdir: dir, script_path: script as FilePath, args: [status, "60"]) + if !run.success { false } else { + match d0fd_parse_listener(out: run.stdout) { + Absent => false + Present { value: l } => { + let observed = scenario(l) + let killed = python.Interpreter.RunFile(workdir: dir, script_path: kill as FilePath, args: [to_string(l.pid)]) + observed && killed.success + } + } + } + } +} + +fn d0fd_port(n: Int) -> Port { + n +} + +fn d0fd_endpoint(port: Int) -> ServingRouteEndpoint { + d0fd_endpoint_at(address: "127.0.0.1", port: port) +} + +fn d0fd_endpoint_at(address: String, port: Int) -> ServingRouteEndpoint { + ServingRouteEndpoint { group: FabricGroupA, address: address as NonEmptyStr, port: d0fd_port(n: port) } +} + +// The head's endpoint reading over the LOCAL host: the kernel's socket tables for a LISTEN on +// the port. (The engine-process scan half runs over the fleet ssh leg and is not driven here.) +fn d0fd_head_endpoint(port: Int) -> HeadEndpointReading { + match read_endpoint_listener(transport: LocalShell, port: d0fd_port(n: port)) { + EndpointNoListener { port: _ } => HeadEndpointQuiet + EndpointListening { port: p, rows: rs } => HeadEndpointListening { port: p, rows: rs } + EndpointListenerUnread { port: _, cause: c } => HeadEndpointUnread { cause: c as NonEmptyStr } + } +} + +fn d0fd_status_is(p: HarnessBoundedPresence, status: String) -> Bool { + match p { + HarnessResponded { status: s } => s == status + _ => false + } +} + +// 500 and 401 are ANSWERS -- the transport reports the status, the front door reads as answered +// and unread, and the incumbent reading fences. No listener (the port just vacated) is SILENCE: +// no HTTP response, the front door reads silent, and the incumbent reading is absence. +test fn an_error_status_is_an_answer_and_no_listener_is_silence_by_real_execution() -> Bool { + d0fd_with_temp_dir(scenario: fn(dir) { + d0fd_with_listener(dir: dir, status: "500", scenario: fn(l) { + d0fd_status_is(p: harness_bounded_presence(url: harness_models_url(endpoint: d0fd_endpoint(port: l.port))), status: "500") + && (match observe_front_door(endpoint: d0fd_endpoint(port: l.port)) { FrontDoorAnsweredUnread { cause: c } => (c as String).contains("HTTP 500") _ => false }) + && (match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint(port: l.port)), head: fn() { HeadEndpointQuiet }) { Present { value: IncumbentAnsweredUnread { cause: _ } } => true _ => false }) + }) + && d0fd_with_listener(dir: dir, status: "401", scenario: fn(l) { + d0fd_status_is(p: harness_bounded_presence(url: harness_models_url(endpoint: d0fd_endpoint(port: l.port))), status: "401") + && (match observe_front_door(endpoint: d0fd_endpoint(port: l.port)) { FrontDoorAnsweredUnread { cause: _ } => true _ => false }) + }) + && d0fd_with_listener(dir: dir, status: "200", scenario: fn(l) { + d0fd_status_is(p: harness_bounded_presence(url: harness_models_url(endpoint: d0fd_endpoint(port: l.port))), status: "200") + && (match observe_front_door(endpoint: d0fd_endpoint(port: l.port)) { FrontDoorAnswered => true _ => false }) + && (match front_door_as_incumbent(r: FrontDoorAnswered, head: fn() { HeadEndpointQuiet }) { Absent => true Present { value: _ } => false }) + }) + }) +} + +// The port a killed listener vacated fails to connect (curl 7): the same transport, the same bound, +// and the one failure arm under which absence can still be established -- on the head, below. +test fn a_vacated_port_reads_as_no_response_by_real_execution() -> Bool { + d0fd_with_temp_dir(scenario: fn(dir) { + match d0fd_vacated_port(dir: dir) { + Absent => false + Present { value: port } => + (match harness_bounded_presence(url: harness_models_url(endpoint: d0fd_endpoint(port: port))) { HarnessConnectFailed { url: _, curl_exit: code } => code == 7 _ => false }) + && (match observe_front_door(endpoint: d0fd_endpoint(port: port)) { FrontDoorNoStatus { receipt: _ } => true _ => false }) + && (match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint(port: port)), head: fn() { HeadEndpointQuiet }) { Present { value: IncumbentDidNotAnswer { receipt: _, head: HeadEndpointQuiet } } => true _ => false }) + } + }) +} + +// A LISTENER THAT ACCEPTS AND NEVER ANSWERS IS NOT ABSENCE. The fixture in `hang` mode accepts the +// connection and writes no status line; after the front-door bound curl exits 28, the presence +// reading is AcceptedNoAnswer, the front door is Unread, the incumbent reading is RouteUnread -- +// and through the transaction the group is FENCED at generation 2, never suspended as absent. +test fn a_listener_that_accepts_and_never_answers_fences_rather_than_reading_as_absent_by_real_execution() -> Bool { + d0fd_with_hash(scenario: fn(h) { + d0fd_with_temp_dir(scenario: fn(dir) { + let store = d0fd_witness_layout(dir: dir) + d0fd_witness_remote_ready(dir: dir) + && d0fd_with_listener(dir: dir, status: "hang", scenario: fn(l) { + (match harness_bounded_presence(url: harness_models_url(endpoint: d0fd_endpoint(port: l.port))) { HarnessDeadline { url: _ } => true _ => false }) + && (match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint(port: l.port)), head: fn() { HeadEndpointQuiet }) { Present { value: IncumbentRouteUnread { cause: _ } } => true _ => false }) + && (match d0fd_w_admitted(store: store, successor: h) { + Absent => false + Present { value: admitted } => + (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: admitted, transaction: "tx-front-door" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", + observe: fn(g) { + D0Observation { + group: g, + incumbent: match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint(port: l.port)), head: fn() { HeadEndpointQuiet }) { + Present { value: r } => r + Absent => IncumbentDidNotAnswer { receipt: "witness: the front door answered 2xx, which this fixture does not" as NonEmptyStr, head: HeadEndpointQuiet } + }, + ranks: [] as List, + } + }) { + D0Settled { decision: D0DecidedFence { next: _, cause: D0OccupancyUnread { cause: _ } }, generation: g2, receipt_path: _, entry_state: _ } => g2 == 2 + _ => false + }) + }) + }) + }) + }) +} + +// Start a listener, kill it, and hand back its port -- vacated, since the kill's success is +// checked before the port is returned. +fn d0fd_vacated_port(dir: FilePath) -> Int? { + let script = join([dir as String, "/listener.py"], "") + let kill = join([dir as String, "/kill.py"], "") + let w1 = Filesystem.Write(path: script, content: d0fd_listener_fixture) + let w2 = Filesystem.Write(path: kill, content: d0fd_kill_fixture) + if !w1.success || !w2.success { none } else { + let run = python.Interpreter.RunFile(workdir: dir, script_path: script as FilePath, args: ["204", "60"]) + if !run.success { none } else { + match d0fd_parse_listener(out: run.stdout) { + Absent => none + Present { value: l } => { + let killed = python.Interpreter.RunFile(workdir: dir, script_path: kill as FilePath, args: [to_string(l.pid)]) + if killed.success { Present { value: l.port } } else { none } + } + } + } + } +} + +// ── THROUGH THE TRANSACTION TO THE LOG ───────────────────────────────────────────────────────── + +fn d0fd_witness_layout(dir: FilePath) -> FabricStorageBinding { + FabricStorageLocalFiles { root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr) } +} + +// A ready remote has group A's source row ESTABLISHED on its partition (the log's durable +// genesis) at 990, before any scenario event. +fn d0fd_witness_remote_ready(dir: FilePath) -> Bool { + (match fabric_storage_file_root_ensure(root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr)) { FabricStorageRootReady => true _ => false }) + && (match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: desired } => + match pair_serving_authority_establish(store: d0fd_witness_layout(dir: dir), group: FabricGroupA, authority: desired, actor: "witness" as NonEmptyStr, at: 990) { AuthorityEstablishedAt { id: _ } => true AuthorityEstablishmentRefused { group: _, step: _, reason: _ } => false } + }) +} + +fn d0fd_with_hash(scenario: fn(ContentHash) -> Bool) -> Bool { + match sha256_hex_digest(hex: "4444444444444444444444444444444444444444444444444444444444444444") { + Absent => false + Present { value: d } => scenario(Sha256Hash(d)) + } +} + +fn d0fd_w_admitted(store: FabricStorageBinding, successor: ContentHash) -> AdmittedD0Grant? { + d0fd_w_admitted_as(store: store, successor: successor, tx: "tx-front-door" as NonEmptyStr, esc: "esc-front-door" as NonEmptyStr) +} + +fn d0fd_w_admitted_as(store: FabricStorageBinding, successor: ContentHash, tx: NonEmptyStr, esc: NonEmptyStr) -> AdmittedD0Grant? { + let subject = D0Subject { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), successor: successor, cleanup: QuiescentReservedBaseline, term: second(count: 1000) } + match d0_intent_hash(subject: subject, group: FabricGroupA, transaction: tx) { + Absent => none + Present { value: intent } => { + let auth = AuthorizationGranted { + grant: OperatorGrant { + escalation_id: esc, + subject: subject, + scopes: [AuthorizationScope { verb: Write, resource: NamespacePosition { tree: CodeNameTree, path: ["gunbc", "spark", "pair_serving_authority"] }, action: d0_authorized_action }], + purpose: "witness d0 front door" as NonEmptyStr, + attempt: (tx as String) as AttemptIdentity, + intent_hash: intent, + granted_by: "witness" as NonEmptyStr, + granted_at: "2026-09-18T00:00:00Z", + expires_at: "2026-09-19T00:00:00Z", + }, + claim: Unclaimed, + } + match admit_d0_grant(store: store, authorization: auth, escalation_id: esc, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), transaction: tx, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z") { + D0GrantAdmitted { admitted: a } => Present { value: a } + _ => none + } + } + } +} + +// A front door answering 500, read by the real transport INSIDE the transaction's observer (under +// the pending state, as production reads it), reaches D0DecidedFence with the unidentified- +// incumbent cause and FencedRefusal on the log at generation 2 -- the ranks never consulted. +test fn a_front_door_answering_500_fences_the_group_through_the_transaction_by_real_execution() -> Bool { + d0fd_with_hash(scenario: fn(h) { + d0fd_with_temp_dir(scenario: fn(dir) { + let store = d0fd_witness_layout(dir: dir) + d0fd_witness_remote_ready(dir: dir) + && d0fd_with_listener(dir: dir, status: "500", scenario: fn(l) { + match d0fd_w_admitted(store: store, successor: h) { + Absent => false + Present { value: admitted } => + (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: admitted, transaction: "tx-front-door" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", + observe: fn(g) { + D0Observation { + group: g, + incumbent: match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint(port: l.port)), head: fn() { HeadEndpointQuiet }) { + Present { value: r } => r + Absent => IncumbentDidNotAnswer { receipt: "witness: the front door answered 2xx, which this fixture does not" as NonEmptyStr, head: HeadEndpointQuiet } + }, + ranks: [] as List, + } + }) { + D0Settled { decision: D0DecidedFence { next: _, cause: D0IncumbentUnidentified { cause: _ } }, generation: g2, receipt_path: _, entry_state: _ } => g2 == 2 + _ => false + }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001) { + CurrentAuthorityRead { authority: FencedRefusal { group: _, cause: c, disposition_authority: _ }, head: _, generation: g, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => g == 2 && (c as String).contains("HTTP 500") + _ => false + }) + } + }) + }) + }) +} + +// A FAILED CONNECT IS NOT ABSENCE BY ITSELF. The same vacated port, read inside the transaction: +// with the head's declared unit ACTIVE the group is fenced (a listener the route did not reach); +// with every declared unit installed and inactive, absence is corroborated on the head and the +// group is suspended drifted-eligible. +fn d0fd_ranks(head_active: Bool) -> List { + match declared_pair_realization(group: FabricGroupA) { + DeclaredPairRealizationUnavailable { group: _, obligation: _ } => [] as List + DeclaredPairRealizationKeyed { declared: d } => + concat( + [RankOccupancyReading { host: d.head.host, unit_name: d.head.unit_name, occupancy: UnitInstalled { active: head_active, unit_digest: d.head.digest } }], + map(d.workers, w => RankOccupancyReading { host: w.host, unit_name: w.unit_name, occupancy: UnitInstalled { active: false, unit_digest: w.digest } }), + ) + } +} + +fn d0fd_observe_vacated(port: Int, head_active: Bool, g: FabricGroup) -> D0Observation { + D0Observation { + group: g, + incumbent: match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint(port: port)), head: fn() { d0fd_head_endpoint(port: port) }) { + Present { value: r } => r + Absent => IncumbentAnsweredUnread { cause: "witness: a vacated port answered" as NonEmptyStr } + }, + ranks: d0fd_ranks(head_active: head_active), + } +} + +test fn a_failed_connect_fences_under_an_active_head_unit_and_suspends_only_under_a_quiet_one_by_real_execution() -> Bool { + d0fd_with_hash(scenario: fn(h) { + d0fd_with_temp_dir(scenario: fn(dir) { + let store = d0fd_witness_layout(dir: dir) + d0fd_witness_remote_ready(dir: dir) + && (match d0fd_vacated_port(dir: dir) { + Absent => false + Present { value: port } => + match d0fd_w_admitted(store: store, successor: h) { + Absent => false + Present { value: admitted } => + (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: admitted, transaction: "tx-front-door" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", observe: fn(g) { d0fd_observe_vacated(port: port, head_active: true, g: g) }) { + D0Settled { decision: D0DecidedFence { next: _, cause: D0IncumbentUnidentified { cause: _ } }, generation: g2, receipt_path: _, entry_state: _ } => g2 == 2 + _ => false + }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001) { + CurrentAuthorityRead { authority: fenced, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: fenced, next: resting, transaction: "tx-operator-restore" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1001) { + AuthorityTransitionAppended { id: _, generation: g3, next: _ } => g3 == 3 + _ => false + } + } + _ => false + }) + && (match d0fd_w_admitted_as(store: store, successor: h, tx: "tx-front-door-2" as NonEmptyStr, esc: "esc-front-door-2" as NonEmptyStr) { + Absent => false + Present { value: second } => + match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: second, transaction: "tx-front-door-2" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1002, now: "2026-09-18T00:00:02Z", observe: fn(g) { d0fd_observe_vacated(port: port, head_active: false, g: g) }) { + D0Settled { decision: D0DecidedSuspend { next: _, reconciliation: DeclaredOccupantDrifted { drifted: ds } }, generation: g5, receipt_path: _, entry_state: _ } => g5 == 5 && length(ds) == 4 + _ => false + } + }) + } + }) + }) + }) +} + +// A LISTENER THE ROUTE DID NOT REACH. The fixture listens on 127.0.0.1:P; the door is asked at +// 127.0.0.2:P, where nothing accepts, so curl exits 7 (a failed connect, exactly what a broken +// route produces). The head's socket tables, read on this host through the real procfs +// operation, show the LISTEN on P: the head endpoint reads Listening, and through the transaction +// the group is FENCED as an unidentified incumbent -- never suspended as absent. Once the listener +// is killed the same tables read no listener on P. +test fn a_listener_behind_a_failed_connect_is_read_on_the_host_and_fences_by_real_execution() -> Bool { + d0fd_with_hash(scenario: fn(h) { + d0fd_with_temp_dir(scenario: fn(dir) { + let store = d0fd_witness_layout(dir: dir) + d0fd_witness_remote_ready(dir: dir) + && d0fd_with_listener(dir: dir, status: "200", scenario: fn(l) { + (match read_endpoint_listener(transport: LocalShell, port: d0fd_port(n: l.port)) { EndpointListening { port: p, rows: rs } => (p as Int) == l.port && length(rs) >= 1 _ => false }) + && (match harness_bounded_presence(url: harness_models_url(endpoint: d0fd_endpoint_at(address: "127.0.0.2", port: l.port))) { HarnessConnectFailed { url: _, curl_exit: code } => code == 7 _ => false }) + && (match d0fd_w_admitted(store: store, successor: h) { + Absent => false + Present { value: admitted } => + match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: admitted, transaction: "tx-front-door" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", + observe: fn(g) { + D0Observation { + group: g, + incumbent: match front_door_as_incumbent(r: observe_front_door(endpoint: d0fd_endpoint_at(address: "127.0.0.2", port: l.port)), head: fn() { d0fd_head_endpoint(port: l.port) }) { + Present { value: r } => r + Absent => IncumbentAnsweredUnread { cause: "witness: 127.0.0.2 answered" as NonEmptyStr } + }, + ranks: d0fd_ranks(head_active: false), + } + }) { + D0Settled { decision: D0DecidedFence { next: _, cause: D0IncumbentUnidentified { cause: c } }, generation: g2, receipt_path: _, entry_state: _ } => g2 == 2 && (c as String).contains("LISTEN") + _ => false + } + }) + }) + && (match d0fd_vacated_port(dir: dir) { + Absent => false + Present { value: port } => match read_endpoint_listener(transport: LocalShell, port: d0fd_port(n: port)) { EndpointNoListener { port: _ } => true _ => false } + }) + }) + }) +} diff --git a/dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag b/dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag new file mode 100644 index 00000000000..2dd30b7abaf --- /dev/null +++ b/dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag @@ -0,0 +1,897 @@ +module test.claim.spark.pair_serving_d0_real_execution + +import std.logic { Bool } +import std.types { String, NonEmptyStr, FilePath, List, Int } +import v2.std.optional { Present, Absent } +import std.measure { second } +import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } +import extdeps.shell +import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest, compare_content_hash, ContentHashEqual, ContentHashDifferent, ContentHashCrossFamilyIncomparable, serialize_content_hash, content_hash_equal } +import std.scoped_authorization { OperatorGrant, ScopedAuthorization, AuthorizationGranted, Unclaimed, AuthorizationScope, AttemptIdentity } +import std.effect_grant { Write, NamespacePosition, CodeNameTree } +import gunbc.harness.harness_seat { + SeatAuthorityGate, SeatAuthorityUngoverned, SeatAuthorityAdmits, SeatAuthorityRefuses, harness_seat_authority_gate, + SeatAfterGrant, SeatStillAdmitted, SeatReleasedAuthorityMoved, SeatUnreleasedAuthorityMoved, harness_seat_after_grant_authority, harness_seat_pool_root, +} +import product.capacity.lease { LeasePolicy, QuiescenceRequired } +import product.capacity.pool_events { SeatRequest } +import gunbc.fabric_storage_client { FabricStorageBinding, FabricStorageLocalFiles } +import gunbc.fabric_storage_file_store { fabric_storage_file_root, fabric_storage_file_root_ensure, FabricStorageRootReady } +import gunbc.fabric_event_log { + SeatGranted, fabric_seat_acquire, SeatRoomObserved, fabric_seat_observe, + PartitionReadOk, event_log_read_partition_with, +} +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, FabricGroupB } +import product.fabric.demand { ObservationReceiptRef } +import gunbc.spark.serving_incarnation_observe { UnitInstalled, UnitNotInstalled } +import gunbc.spark.vllm_endpoint_process_launch { vllm_endpoint_process_launch } +import gunbc.spark.vllm_serving_launch { vllm_serving_launch, endpoint_incarnation_observation } +import gunbc.spark.pair_incumbent_identity { + DeclaredPairRealizationKeyed, DeclaredPairRealizationUnavailable, declared_pair_realization, + ObservedRankUnit, observed_pair_realization_of, ObservedPairRealizationRead, pair_realization_agreement, +} +import gunbc.spark.released_baseline { QuiescentReservedBaseline } +import gunbc.spark.pair_serving_authority { + PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, SuspendedForAuthorizedSuccessor, FencedRefusal, PairRealizationKeyed, + pair_serving_authority_for, pair_serving_successor_may_launch, pair_serving_may_finish, +} +import gunbc.spark.pair_serving_authority_log { + CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, current_pair_serving_authority, HostEffectRecord, + AuthorityTransitionAppended, pair_serving_authority_transition, authority_eq, + AuthorityEstablishedAt, AuthorityEstablishmentRefused, pair_serving_authority_establish, + PairAuthorityEvent, EntryStateRecorded, pair_authority_partition, pair_authority_event_decode, +} +import product.capacity.event_chain { PartitionId, ChainWalked } +import std.process { ProcessExit, ExitSuccess, ExitFailure } +import gunbc.spark.pair_serving_d0 { + d0_dispatch, + D0Subject, RankOccupancyReading, IncumbentDidNotAnswer, IncumbentAnswered, D0Observation, HeadEndpointQuiet, + D0Outcome, D0Settled, D0Refused, D0Decision, D0DecidedSuspend, D0DecidedRestore, D0DecidedFence, DeclaredOccupantDrifted, d0_transaction, d0_held_lease, + D0GrantAdmission, + AdmittedD0Grant, D0GrantAdmitted, D0GrantClaimLost, D0GrantForAnotherPopulation, admit_d0_grant, resume_d0_grant, d0_intent_hash, d0_authorized_action, + D0ClaimReading, D0ClaimAbsent, D0ClaimAt, D0ClaimUnreadable, d0_read_claim, d0_terminate_claim, D0ClaimTerminated, + D0Recovery, D0RecoverStart, D0RecoverFirstWrite, D0RecoverResume, D0RecoverComplete, D0RecoverAbort, D0RecoveryRefused, d0_recovery, +} +import std.scoped_authorization { ClaimedBy, CompletedBy, AbortedBy } +import product.placement_supply { HostIdentity } +import gunbc.fleet_intent_network { operator_host_srv1 } +import gunbc.spark.fabric_switch_observed { fabric_group_hosts } +import gunbc.spark.host_commitment { + spark_claimed_members_under, spark_host_standings_current, CurrentStandings, CurrentStandingsRead, CurrentStandingsUnread, admit_unplaced_host_in, SparkHostAdmission, SparkHostAdmitted, SparkHostRefused, + SparkHostStanding, SparkHostUncommitted, SparkHostCommitted, spark_standing_of, spark_authority_held_causes_of, MemberOfClaimedServingGroup, +} +import gunbc.fleet_intent_network { operator_host_srv10 } +import product.placement_supply { host_identity_eq } + +data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly + +// THE EXACT PRODUCTION TWO-WRITE ROUTE, BY REAL EXECUTION against a bare remote in a temporary +// directory. The pure decision has its own witness and the generic compare-and-set its own; what +// neither establishes is their COMPOSITION, which is where the first shape of this transaction +// failed: it wrote the pending state without a grant, read it back with a derived observation, +// and could never settle. So this file drives d0_transaction itself -- the same fn the wet entry +// calls -- with a fixture grant and a supplied observer, and asserts what the redesign requires: +// resting Active → first write → pending read back → readings → second write +// → SuspendedForAuthorizedSuccessor at generation 2, naming the entry-state receipt +// crash after the first write → a rerun under the SAME transaction settles from pending +// a rerun under ANOTHER transaction is refused and moves nothing +// the lease expires → the successor may not start, may finish, and a cleanup transition +// still appends against the state as read, so the chain folds to generation 3 +// the claim is on the same store: two executors (two bindings to the one placed store) race for one grant +// and exactly one holds it +// crash after the claim and before the first write → the join performs the first write +// crash after the settling write and before CompletedBy → the join completes the claim only +// a grant over another host population is refused before any claim or write + +fn d0r_with_temp_dir(scenario: fn(FilePath) -> Bool) -> Bool { + let dir = shell.Mktemp.Dir() + let observed = scenario(dir.path) + let cleaned = shell.Remove.RecursiveForce(path: dir.path) + dir.success && observed && cleaned.success +} + +fn d0r_witness_layout(dir: FilePath) -> FabricStorageBinding { + FabricStorageLocalFiles { root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr) } +} + +// A ready remote has group A's source row ESTABLISHED on its partition (the log's durable +// genesis) at 990, before any scenario event. +fn d0r_witness_remote_ready(dir: FilePath) -> Bool { + (match fabric_storage_file_root_ensure(root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr)) { FabricStorageRootReady => true _ => false }) + && (match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: desired } => + match pair_serving_authority_establish(store: d0r_witness_layout(dir: dir), group: FabricGroupA, authority: desired, actor: "witness" as NonEmptyStr, at: 990) { AuthorityEstablishedAt { id: _ } => true AuthorityEstablishmentRefused { group: _, step: _, reason: _ } => false } + }) +} + +// A scenario over a ready remote: the bare repository exists BEFORE the first claim, since the +// claim is the first thing written to the log now. +fn d0r_with_remote(scenario: fn(FilePath, FabricStorageBinding) -> Bool) -> Bool { + d0r_with_temp_dir(scenario: fn(dir) { d0r_witness_remote_ready(dir: dir) && scenario(dir, d0r_witness_layout(dir: dir)) }) +} + +// A second executor's binding to the SAME store: one placed fabric DB, reached by every executor. +fn d0r_second_executor_layout(dir: FilePath) -> FabricStorageBinding { + FabricStorageLocalFiles { root: fabric_storage_file_root(root: join([dir as String, "/fabric-storage"], "") as NonEmptyStr) } +} + +fn d0r_with_hash(scenario: fn(ContentHash) -> Bool) -> Bool { + match sha256_hex_digest(hex: "3333333333333333333333333333333333333333333333333333333333333333") { + Absent => false + Present { value: d } => scenario(Sha256Hash(d)) + } +} + +// A fixture authorization over Group A for transaction `tx`: a keyed successor, the reserved +// baseline, a 1000-second term, the write scope over the authority, the transaction as attempt, +// the intent over the exact operation -- everything admit_d0_grant decides on. +fn d0r_w_authorization_over(hosts: List, successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr) -> ScopedAuthorization? { + let subject = D0Subject { group: FabricGroupA, hosts: hosts, successor: successor, cleanup: QuiescentReservedBaseline, term: second(count: 1000) } + match d0_intent_hash(subject: subject, group: FabricGroupA, transaction: tx) { + Absent => none + Present { value: intent } => + Present { value: AuthorizationGranted { + grant: OperatorGrant { + escalation_id: escalation, + subject: subject, + scopes: [AuthorizationScope { verb: Write, resource: NamespacePosition { tree: CodeNameTree, path: ["gunbc", "spark", "pair_serving_authority"] }, action: d0_authorized_action }], + purpose: "witness d0" as NonEmptyStr, + attempt: (tx as String) as AttemptIdentity, + intent_hash: intent, + granted_by: "witness" as NonEmptyStr, + granted_at: "2026-09-18T00:00:00Z", + expires_at: "2026-09-19T00:00:00Z", + }, + claim: Unclaimed, + } } + } +} + +fn d0r_w_authorization(successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr) -> ScopedAuthorization? { + d0r_w_authorization_over(hosts: fabric_group_hosts(g: FabricGroupA), successor: successor, tx: tx, escalation: escalation) +} + +// START: permission decided and the claim TAKEN, on the scenario's event log -- the same log the +// authority partition lives on, through the store the "executor" resolves. +fn d0r_w_admit(store: FabricStorageBinding, successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr, executor: NonEmptyStr) -> D0GrantAdmission { + match d0r_w_authorization(successor: successor, tx: tx, escalation: escalation) { + Absent => D0GrantClaimLost { holder: "witness: the fixture intent could not be digested" as NonEmptyStr } + Present { value: auth } => admit_d0_grant(store: store, authorization: auth, escalation_id: escalation, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), transaction: tx, executor: executor, at: 1000, now: "2026-09-18T00:00:00Z") + } +} + +fn d0r_w_admitted(store: FabricStorageBinding, successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr, executor: NonEmptyStr) -> AdmittedD0Grant? { + match d0r_w_admit(store: store, successor: successor, tx: tx, escalation: escalation, executor: executor) { + D0GrantAdmitted { admitted: a } => Present { value: a } + _ => none + } +} + +// RESUME: the claim found held by this attempt; permission is not re-decided. +fn d0r_w_resumed(store: FabricStorageBinding, successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr, executor: NonEmptyStr) -> AdmittedD0Grant? { + match d0r_w_authorization(successor: successor, tx: tx, escalation: escalation) { + Absent => none + Present { value: auth } => + match resume_d0_grant(store: store, authorization: auth, escalation_id: escalation, group: FabricGroupA, transaction: tx, executor: executor) { + D0GrantAdmitted { admitted: a } => Present { value: a } + _ => none + } + } +} + +// The claim slot as the log holds it: ClaimedBy / CompletedBy the named attempt. +fn d0r_claim_is(store: FabricStorageBinding, escalation: NonEmptyStr, by: String, completed: Bool) -> Bool { + match d0_read_claim(store: store, escalation_id: escalation) { + D0ClaimAt { state: ClaimedBy { attempt: who, claimed_at: _ }, generation: _ } => !completed && (who as String) == by + D0ClaimAt { state: CompletedBy { attempt: who, completed_at: _ }, generation: _ } => completed && (who as String) == by + _ => false + } +} + +fn d0r_claim_is_lost(a: D0GrantAdmission) -> Bool { + match a { + D0GrantClaimLost { holder: _ } => true + _ => false + } +} + +// The expected readings: nothing answers on the route, every declared rank has our unit +// installed and inactive -- the drifted-eligible case. Digests are the PRODUCTION declaration's, +// so reconciliation is against what Group A really declares. +fn d0r_w_observe(group: FabricGroup) -> D0Observation { + let ranks = match declared_pair_realization(group: group) { + DeclaredPairRealizationUnavailable { group: _, obligation: _ } => [] as List + DeclaredPairRealizationKeyed { declared: d } => + concat( + [RankOccupancyReading { host: d.head.host, unit_name: d.head.unit_name, occupancy: UnitInstalled { active: false, unit_digest: d.head.digest } }], + map(d.workers, w => RankOccupancyReading { host: w.host, unit_name: w.unit_name, occupancy: UnitInstalled { active: false, unit_digest: w.digest } }), + ) + } + D0Observation { group: group, incumbent: IncumbentDidNotAnswer { receipt: "witness: the route did not answer" as NonEmptyStr, head: HeadEndpointQuiet }, ranks: ranks } +} + + +fn d0r_seat_gate_is(g: SeatAuthorityGate, admits: Bool) -> Bool { + match g { + SeatAuthorityAdmits => admits + SeatAuthorityRefuses { cause: _ } => !admits + SeatAuthorityUngoverned => false + } +} + +fn d0r_read_is_suspended_at(c: CurrentAuthority, generation: Int, entry_state: ContentHash, launch: Bool) -> Bool { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: g, grant: gr, entry_state: es, previous: _, admitted_by: _, transitions: _ } => + g == generation + && (match a { SuspendedForAuthorizedSuccessor { group: _, authorization: au, exact_candidate_realization: _, lease: _, cleanup: QuiescentReservedBaseline } => (au as String) == "esc-d0-witness" _ => false }) + && (match gr { Present { value: _ } => true Absent => false }) + && (match es { Present { value: h } => content_hash_equal(left: h, right: entry_state) Absent => false }) + && pair_serving_successor_may_launch(a: a) == launch + && pair_serving_may_finish(a: a) + CurrentAuthorityUnread { group: _, step: _, reason: _ } => false + } +} + +// THE RECEIPT IS ON THE LOG: the settling event names a content hash that is the id of an +// EntryStateRecorded event on the same partition, and that event carries the receipt's bytes. +fn d0r_entry_state_is_on_the_log(store: FabricStorageBinding, entry_state: ContentHash) -> Bool { + match event_log_read_partition_with(store: store, partition: pair_authority_partition(group: FabricGroupA), budget: 64, decode: fn(text) { pair_authority_event_decode(text: text) }) { + PartitionReadOk { head: _, walk: ChainWalked { oldest_first: envs } } => + any(envs, env => (env.id as String) == (serialize_content_hash(hash: entry_state) as String) + && (match env.event.payload { EntryStateRecorded { transaction: t, body: b } => (t as String) == "tx-d0" && b != "" _ => false })) + _ => false + } +} + +fn d0r_settled_eligible(o: D0Outcome) -> ContentHash? { + match o { + D0Settled { decision: D0DecidedSuspend { next: _, reconciliation: DeclaredOccupantDrifted { drifted: ds } }, generation: g, receipt_path: _, entry_state: es } => + if length(ds) == 4 && g == 2 { Present { value: es } } else { none } + _ => none + } +} + +// THE WHOLE ROUTE, then expiry, then cleanup. +test fn the_production_route_settles_and_the_lease_governs_launch_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + match d0r_w_admitted(store: store, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: "esc-d0-witness" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: admitted } => + d0r_seat_gate_is(g: harness_seat_authority_gate(store: store, group: FabricGroupA, now: 1000), admits: true) + && (match harness_seat_authority_gate(store: store, group: FabricGroupB, now: 1000) { SeatAuthorityUngoverned => true _ => false }) + && (match d0r_settled_eligible(o: d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: admitted, transaction: "tx-d0" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", observe: fn(g) { d0r_w_observe(group: g) })) { + Absent => false + Present { value: es } => + d0r_read_is_suspended_at(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1000), generation: 2, entry_state: es, launch: true) + && d0r_seat_gate_is(g: harness_seat_authority_gate(store: store, group: FabricGroupA, now: 1000), admits: false) + && d0r_read_is_suspended_at(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 2500), generation: 2, entry_state: es, launch: false) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 2500) { + CurrentAuthorityRead { authority: expired, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: expired, next: FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "witness cleanup" as NonEmptyStr, disposition_authority: "witness" as NonEmptyStr }, transaction: "tx-d0" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 2500) { + AuthorityTransitionAppended { id: _, generation: g3, next: _ } => + g3 == 3 && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 2600) { + CurrentAuthorityRead { authority: FencedRefusal { group: _, cause: _, disposition_authority: _ }, head: _, generation: g4, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => g4 == 3 + _ => false + }) + _ => false + } + CurrentAuthorityUnread { group: _, step: _, reason: _ } => false + }) + }) + } + }) + }) +} + +// CRASH AFTER THE FIRST WRITE, FROM A NON-RESTING ACTIVE. The authority is first moved to +// Active(Keyed K) -- not the resting row -- then the pending state is written by the production +// transition under tx-d0 and this grant's binding (exactly what d0_transaction writes first), and +// the process is gone. Then: a rerun under tx-other is refused; a rerun under tx-d0 but ANOTHER +// admitted grant (a different escalation, so a different binding) is refused; a second START under +// the same escalation and transaction finds the grant already claimed (the consent was consumed +// once); nothing moved; and a RESUME under tx-d0 -- the claim found held, permission not +// re-decided -- finds a live declared incumbent and restores EXACTLY Active(Keyed K), not the +// resting row. +test fn a_rerun_resumes_its_own_pending_state_and_restores_the_exact_active_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => { + let keyed = PairServingActive { group: FabricGroupA, exact_realization: PairRealizationKeyed { key: h } } + match d0r_w_admitted(store: store, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: "esc-d0-witness" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: admitted } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: "esc-another-decision" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: other_grant_same_tx } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-other" as NonEmptyStr, escalation: "esc-third-decision" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: other_tx } => { + let pending = SuspensionPendingReconciliation { + group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-d0" as NonEmptyStr, authorization_binding: admitted.binding, + lease: d0_held_lease(group: FabricGroupA, transaction: "tx-d0" as NonEmptyStr, executor: "witness" as NonEmptyStr, generation: 2), + } + (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: keyed, transaction: "tx-operator-keyed" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 999) { + AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 + _ => false + }) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: keyed, next: pending, transaction: "tx-d0" as NonEmptyStr, lifecycle: Present { value: admitted.binding }, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { + AuthorityTransitionAppended { id: _, generation: g2, next: _ } => g2 == 2 + _ => false + }) + && (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: other_tx, transaction: "tx-other" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1001, now: "2026-09-18T00:00:01Z", observe: fn(g) { d0r_w_observe(group: g) }) { + D0Refused { cause: _ } => true + _ => false + }) + && (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: other_grant_same_tx, transaction: "tx-d0" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1001, now: "2026-09-18T00:00:01Z", observe: fn(g) { d0r_w_observe(group: g) }) { + D0Refused { cause: _ } => true + _ => false + }) + && d0r_claim_is_lost(a: d0r_w_admit(store: store, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: "esc-d0-witness" as NonEmptyStr, executor: "witness" as NonEmptyStr)) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001) { + CurrentAuthorityRead { authority: SuspensionPendingReconciliation { group: _, transaction: t, authorization_binding: _, lease: _ }, head: _, generation: g1, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => (t as String) == "tx-d0" && g1 == 2 + _ => false + }) + && (match d0r_w_resumed(store: store, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: "esc-d0-witness" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: resumed } => + (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: resumed, transaction: "tx-d0" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1002, now: "2026-09-18T00:00:02Z", observe: fn(g) { d0r_w_observe_live_declared(group: g) }) { + D0Settled { decision: D0DecidedRestore { next: _, key: _ }, generation: g3, receipt_path: _, entry_state: _ } => g3 == 3 + _ => false + }) + && d0r_read_authority_at(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1002), expected: keyed, generation: 3) + }) + } + } + } + } + } + } + }) + }) +} + +// THE SEAT WALL AT ITS LINEARIZATION POINT. A seat attempt reads the authority (Active: admits), +// the D0 transaction lands its first write, and THEN the seat's compare-and-set runs on its own +// partition and is granted -- the interleaving the pre-check cannot settle. The post-grant read +// (harness_seat_after_grant_authority, the seam harness_fence_grant runs first) finds the +// authority moved, releases the seat on its partition, and no binding is returned; the seat +// partition folds back to its full room. No unaccounted seat remains. +test fn a_seat_granted_after_the_suspension_landed_is_released_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let partition = "witness-group-a-seat" as PartitionId + let ceiling = 1 + let root = harness_seat_pool_root(partition: partition, ceiling: ceiling) + let policy = LeasePolicy { maximum_duration_seconds: 300, release_law: QuiescenceRequired } + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: "esc-d0-witness" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: admitted } => + d0r_seat_gate_is(g: harness_seat_authority_gate(store: store, group: FabricGroupA, now: 1000), admits: true) + && (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-d0" as NonEmptyStr, authorization_binding: admitted.binding, lease: d0_held_lease(group: FabricGroupA, transaction: "tx-d0" as NonEmptyStr, executor: "witness" as NonEmptyStr, generation: 1) }, transaction: "tx-d0" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { + AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 + _ => false + }) + && (match fabric_seat_acquire(store: store, partition: partition, root: root, actor: "witness" as NonEmptyStr, request: SeatRequest { reference: "witness-seat-1" as NonEmptyStr, amount: 1, at: 1001, term_seconds: 300 }, policy: policy, attempts: 3, budget: 64) { + SeatGranted { grant: g, generation: _, attempts_used: _ } => + (match harness_seat_after_grant_authority(store: store, group: FabricGroupA, partition: partition, ceiling: ceiling, grant: g, now: 1001) { + SeatReleasedAuthorityMoved { cause: _ } => true + _ => false + }) + && (match fabric_seat_observe(store: store, partition: partition, root: root, at: 1002, budget: 64) { + SeatRoomObserved { headroom: room, generation: _ } => room == ceiling + _ => false + }) + _ => false + }) + } + } + }) + }) +} + +// An observation whose head rank has NO pair unit installed: unread, so the transaction fences. +fn d0r_w_observe_uninstalled(group: FabricGroup) -> D0Observation { + let base = d0r_w_observe(group: group) + D0Observation { + group: group, incumbent: base.incumbent, + ranks: map(base.ranks, r => if (r.unit_name as String) == "gunbc-spark-pair-head.service" { RankOccupancyReading { host: r.host, unit_name: r.unit_name, occupancy: UnitNotInstalled } } else { r }), + } +} + +// An observation where the enrolled route ANSWERS with exactly the declared realization on every +// rank: restoration is shown safe, and the authority returns to the resting Active state unchanged. +fn d0r_w_observe_live_declared(group: FabricGroup) -> D0Observation { + match declared_pair_realization(group: group) { + DeclaredPairRealizationUnavailable { group: _, obligation: _ } => d0r_w_observe(group: group) + DeclaredPairRealizationKeyed { declared: d } => { + let head = endpoint_incarnation_observation( + endpoint: vllm_endpoint_process_launch(group: group, endpoint: "http://10.0.0.5:8000" as NonEmptyStr, process_start_field: "1" as NonEmptyStr), + launch: vllm_serving_launch(group: group, head: d.head.host, systemd_invocation: "inv" as NonEmptyStr, container_id: "c" as NonEmptyStr, executable_digest: d.head.digest, unit_digest: d.head.digest, started_at: 1), + container_env: [] as List, + receipt: "test.claim.spark.pair_serving_d0_real_execution fixture" as ObservationReceiptRef, + ) + let workers = map(d.workers, w => ObservedRankUnit { host: w.host, unit_name: w.unit_name, systemd_invocation: "inv" as NonEmptyStr, unit_digest: w.digest }) + let agreement = pair_realization_agreement( + declared: DeclaredPairRealizationKeyed { declared: d }, + observed: ObservedPairRealizationRead { observed: observed_pair_realization_of(group: group, head: head, workers: workers, receipt: "test.claim.spark.pair_serving_d0_real_execution fixture" as ObservationReceiptRef) }, + ) + D0Observation { group: group, incumbent: IncumbentAnswered { agreement: agreement }, ranks: d0r_w_observe(group: group).ranks } + } + } +} + +fn d0r_read_authority_at(c: CurrentAuthority, expected: PairServingGroupAuthority, generation: Int) -> Bool { + match c { + CurrentAuthorityRead { authority: a, head: _, generation: g, grant: gr, entry_state: es, previous: _, admitted_by: _, transitions: _ } => + g == generation && authority_eq(a: a, b: expected) + && (match gr { Absent => true Present { value: _ } => false }) + && (match es { Present { value: _ } => true Absent => false }) + CurrentAuthorityUnread { group: _, step: _, reason: _ } => false + } +} + +// THE OTHER TWO TERMINALS ARE REACHED. Two operator decisions, one per transaction -- a claimed +// grant is consumed by its transaction, so the restore needs its own. An unread rank fences: the authority lands on FencedRefusal +// at generation 2 with no grant and the receipt digest on the event, and the seat gate refuses. A +// live declared incumbent restores: the authority lands back on the resting Active state, +// unchanged (no keyed mint), at generation 2, and the seat gate admits again. +test fn a_fence_and_a_restore_each_reach_their_terminal_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-fence" as NonEmptyStr, escalation: "esc-d0-witness" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: fence_grant } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-restore" as NonEmptyStr, escalation: "esc-d0-restore" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: restore_grant } => + (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: fence_grant, transaction: "tx-fence" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", observe: fn(g) { d0r_w_observe_uninstalled(group: g) }) { + D0Settled { decision: D0DecidedFence { next: _, cause: _ }, generation: g2, receipt_path: _, entry_state: _ } => g2 == 2 + _ => false + }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1000) { + CurrentAuthorityRead { authority: FencedRefusal { group: _, cause: _, disposition_authority: _ }, head: _, generation: g, grant: Absent, entry_state: Present { value: _ }, previous: _, admitted_by: _, transitions: _ } => g == 2 + _ => false + }) + && d0r_seat_gate_is(g: harness_seat_authority_gate(store: store, group: FabricGroupA, now: 1000), admits: false) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1000) { + CurrentAuthorityRead { authority: fenced, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: fenced, next: resting, transaction: "tx-operator-restore" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1001) { + AuthorityTransitionAppended { id: _, generation: g3, next: _ } => g3 == 3 + _ => false + } + _ => false + }) + && (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: restore_grant, transaction: "tx-restore" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1002, now: "2026-09-18T00:00:02Z", observe: fn(g) { d0r_w_observe_live_declared(group: g) }) { + D0Settled { decision: D0DecidedRestore { next: _, key: _ }, generation: g5, receipt_path: _, entry_state: _ } => g5 == 5 + _ => false + }) + && d0r_read_authority_at(c: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1002), expected: resting, generation: 5) + && d0r_seat_gate_is(g: harness_seat_authority_gate(store: store, group: FabricGroupA, now: 1002), admits: true) + } + } + } + }) + }) +} + +// ── THE CLAIM IS FLEET-GLOBAL, AND THE RUN IS TOTAL OVER ITS CRASH POINTS ────────────────────── + +// The join as the dispatcher asks it: the lifecycle is the resumed grant's binding when the claim +// is held by this transaction, Absent otherwise. +fn d0r_recovery_of(store: FabricStorageBinding, successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr, at: Int) -> D0Recovery? { + match current_pair_serving_authority(store: store, group: FabricGroupA, at: at) { + CurrentAuthorityUnread { group: _, step: _, reason: _ } => none + CurrentAuthorityRead { authority: cur, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: recs } => { + let lifecycle = match d0r_w_resumed(store: store, successor: successor, tx: tx, escalation: escalation, executor: "witness" as NonEmptyStr) { + Present { value: a } => Present { value: a.binding } + Absent => none + } + Present { value: d0_recovery(claim: d0_read_claim(store: store, escalation_id: escalation), transaction: tx, lifecycle: lifecycle, executor: "witness" as NonEmptyStr, current: cur, transitions: recs) } + } + } +} + +fn d0r_generation_is(store: FabricStorageBinding, generation: Int, at: Int) -> Bool { + match current_pair_serving_authority(store: store, group: FabricGroupA, at: at) { + CurrentAuthorityRead { authority: _, head: _, generation: g, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => g == generation + CurrentAuthorityUnread { group: _, step: _, reason: _ } => false + } +} + +// TWO EXECUTORS, ONE GRANT. Each executor has its own binding to the one placed fabric DB -- the +// shape two hosts have. The first start claims the grant; the second start, from the other +// executor, finds it held and takes nothing; the slot read through either binding names the one +// holder; and once the first executor completes the claim +// the second reads it completed and a third start is refused as spent. The host-local file store +// could not establish any of this: its two roots would each have admitted a claim. +test fn the_same_grant_claimed_from_two_executors_is_held_once_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, e1) { + let e2 = d0r_second_executor_layout(dir: dir) + let esc = "esc-two-executors" as NonEmptyStr + let tx = "tx-d0" as NonEmptyStr + match d0r_w_admitted(store: e1, successor: h, tx: tx, escalation: esc, executor: "executor-1" as NonEmptyStr) { + Absent => false + Present { value: admitted } => + d0r_claim_is_lost(a: d0r_w_admit(store: e2, successor: h, tx: tx, escalation: esc, executor: "executor-2" as NonEmptyStr)) + && d0r_claim_is(store: e2, escalation: esc, by: "tx-d0", completed: false) + && d0r_claim_is(store: e1, escalation: esc, by: "tx-d0", completed: false) + && (match d0_terminate_claim(store: e1, escalation_id: esc, transaction: tx, held: admitted.claim_generation, completed: true, cause: "", now: "2026-09-18T00:00:01Z") { D0ClaimTerminated => true _ => false }) + && d0r_claim_is(store: e2, escalation: esc, by: "tx-d0", completed: true) + && d0r_claim_is_lost(a: d0r_w_admit(store: e2, successor: h, tx: tx, escalation: esc, executor: "executor-2" as NonEmptyStr)) + && (match d0r_recovery_of(store: e2, successor: h, tx: tx, escalation: esc, at: 1002) { Present { value: D0RecoveryRefused { cause: _ } } => true _ => false }) + } + }) + }) +} + +// CRASH AFTER THE CLAIM, BEFORE THE FIRST WRITE. The claim lands on the log and the process is +// gone before Active moved. A rerun's join reads ClaimedBy(tx-d0), an Active authority and no +// transition by tx-d0, and dispatches the first write: the resumed grant (no re-decision) drives +// the whole route from Active, settling at generation 2; then the claim completes. A second +// consent whose first write never landed against an authority that is no longer Active is +// ABORTED by the join, not stranded. +test fn a_crash_after_the_claim_and_before_the_first_write_is_recovered_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let esc = "esc-d0-witness" as NonEmptyStr + let tx = "tx-d0" as NonEmptyStr + match d0r_w_admitted(store: store, successor: h, tx: tx, escalation: esc, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: _ } => + d0r_generation_is(store: store, generation: 0, at: 1000) + && (match d0r_recovery_of(store: store, successor: h, tx: tx, escalation: esc, at: 1000) { Present { value: D0RecoverFirstWrite { claim_generation: _ } } => true _ => false }) + && (match d0r_w_resumed(store: store, successor: h, tx: tx, escalation: esc, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: resumed } => + (match d0r_settled_eligible(o: d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: resumed, transaction: tx, executor: "witness" as NonEmptyStr, at: 1001, now: "2026-09-18T00:00:01Z", observe: fn(g) { d0r_w_observe(group: g) })) { Present { value: _ } => true Absent => false }) + && (match d0_terminate_claim(store: store, escalation_id: esc, transaction: tx, held: resumed.claim_generation, completed: true, cause: "", now: "2026-09-18T00:00:02Z") { D0ClaimTerminated => true _ => false }) + && d0r_claim_is(store: store, escalation: esc, by: "tx-d0", completed: true) + }) + && (match d0r_w_admitted(store: store, successor: h, tx: "tx-second" as NonEmptyStr, escalation: "esc-second" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: second } => + match d0r_recovery_of(store: store, successor: h, tx: "tx-second" as NonEmptyStr, escalation: "esc-second" as NonEmptyStr, at: 1003) { + Present { value: D0RecoverAbort { claim_generation: cg, cause: _ } } => + cg == second.claim_generation + && (match d0_terminate_claim(store: store, escalation_id: "esc-second" as NonEmptyStr, transaction: "tx-second" as NonEmptyStr, held: cg, completed: false, cause: "witness: nothing to do", now: "2026-09-18T00:00:03Z") { D0ClaimTerminated => true _ => false }) + && (match d0_read_claim(store: store, escalation_id: "esc-second" as NonEmptyStr) { D0ClaimAt { state: AbortedBy { attempt: who, aborted_at: _, cause: _ }, generation: _ } => (who as String) == "tx-second" _ => false }) + _ => false + } + }) + && d0r_generation_is(store: store, generation: 2, at: 1004) + } + }) + }) +} + +// CRASH AFTER THE SETTLING WRITE, BEFORE CompletedBy. The transaction settled at generation 2 and +// the process died with the claim still ClaimedBy(tx-d0). A rerun's join finds tx-d0's last +// transition landed the suspended state and dispatches COMPLETE: the claim is completed and the +// authority is not moved. The join reads the log's record, not the current state: after an +// operator moves the group on (a later transaction fences it), the join still answers Complete +// for tx-d0 rather than mistaking the group for one tx-d0 never touched. +test fn a_crash_after_the_settling_write_and_before_completion_completes_the_claim_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let esc = "esc-d0-witness" as NonEmptyStr + let tx = "tx-d0" as NonEmptyStr + match d0r_w_admitted(store: store, successor: h, tx: tx, escalation: esc, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: admitted } => + (match d0r_settled_eligible(o: d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: admitted, transaction: tx, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", observe: fn(g) { d0r_w_observe(group: g) })) { Present { value: _ } => true Absent => false }) + && d0r_claim_is(store: store, escalation: esc, by: "tx-d0", completed: false) + && (match d0r_recovery_of(store: store, successor: h, tx: tx, escalation: esc, at: 1001) { + Present { value: D0RecoverComplete { claim_generation: cg, settled: s } } => + cg == admitted.claim_generation + && (match s.next { SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => true _ => false }) + _ => false + }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001) { + CurrentAuthorityRead { authority: suspended, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: suspended, next: FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "witness operator" as NonEmptyStr, disposition_authority: "witness" as NonEmptyStr }, transaction: "tx-operator" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1002) { + AuthorityTransitionAppended { id: _, generation: g3, next: _ } => g3 == 3 + _ => false + } + _ => false + }) + && (match d0r_recovery_of(store: store, successor: h, tx: tx, escalation: esc, at: 1003) { Present { value: D0RecoverComplete { claim_generation: _, settled: _ } } => true _ => false }) + && (match d0_terminate_claim(store: store, escalation_id: esc, transaction: tx, held: admitted.claim_generation, completed: true, cause: "", now: "2026-09-18T00:00:03Z") { D0ClaimTerminated => true _ => false }) + && d0r_claim_is(store: store, escalation: esc, by: "tx-d0", completed: true) + && (match d0r_recovery_of(store: store, successor: h, tx: tx, escalation: esc, at: 1004) { Present { value: D0RecoveryRefused { cause: _ } } => true _ => false }) + && d0r_generation_is(store: store, generation: 3, at: 1004) + } + }) + }) +} + +// GROUP A'S MEMBERSHIP IS NOT WHAT THE OPERATOR SAW. A grant over four hosts with srv1 in place +// of one Group A member is refused at admission -- before any claim (the slot stays absent) and +// before any write (generation 0) -- naming both populations. +test fn a_grant_over_another_host_population_is_refused_before_any_claim_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let esc = "esc-other-hosts" as NonEmptyStr + let current = fabric_group_hosts(g: FabricGroupA) + let swapped = concat(current.skip(n: 1), [operator_host_srv1]) + match d0r_w_authorization_over(hosts: swapped, successor: h, tx: "tx-d0" as NonEmptyStr, escalation: esc) { + Absent => false + Present { value: auth } => + (match admit_d0_grant(store: store, authorization: auth, escalation_id: esc, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-d0" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z") { + D0GrantForAnotherPopulation { granted: gr, current: cu } => length(gr) == 4 && length(cu) == 4 + _ => false + }) + && (match d0_read_claim(store: store, escalation_id: esc) { D0ClaimAbsent => true _ => false }) + && d0r_generation_is(store: store, generation: 0, at: 1000) + } + }) + }) +} + +// ── THE LIFECYCLE, NOT THE WORD; AND DRIFT AFTER THE CLAIM ──────────────────────────────────── + +fn d0r_exit_failed(e: ProcessExit) -> Bool { + match e { + ExitSuccess => false + ExitFailure { code: _, reason: _ } => true + } +} + +// E1 under word tx-d0 settles and completes; an operator restores the group; E2 -- a fresh +// decision under the SAME word -- claims and dies before its first write. E2's join must answer +// FirstWrite from E2's (empty) history, never Complete from E1's terminal; and driving E2 from +// there performs E2's own first write and settles at generation 4. +test fn a_fresh_authorization_under_a_reused_word_is_not_completed_from_the_old_history_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let tx = "tx-d0" as NonEmptyStr + match d0r_w_admitted(store: store, successor: h, tx: tx, escalation: "esc-e1" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: e1 } => + (match d0r_settled_eligible(o: d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: e1, transaction: tx, executor: "witness" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", observe: fn(g) { d0r_w_observe(group: g) })) { Present { value: _ } => true Absent => false }) + && (match d0_terminate_claim(store: store, escalation_id: "esc-e1" as NonEmptyStr, transaction: tx, held: e1.claim_generation, completed: true, cause: "", now: "2026-09-18T00:00:01Z") { D0ClaimTerminated => true _ => false }) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1002) { + CurrentAuthorityRead { authority: suspended, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: suspended, next: resting, transaction: "tx-operator-restore" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1002) { + AuthorityTransitionAppended { id: _, generation: g3, next: _ } => g3 == 3 + _ => false + } + } + _ => false + }) + && (match d0r_w_admitted(store: store, successor: h, tx: tx, escalation: "esc-e2" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: e2 } => + (match d0r_recovery_of(store: store, successor: h, tx: tx, escalation: "esc-e2" as NonEmptyStr, at: 1003) { Present { value: D0RecoverFirstWrite { claim_generation: _ } } => true _ => false }) + && (match d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: e2, transaction: tx, executor: "witness" as NonEmptyStr, at: 1003, now: "2026-09-18T00:00:03Z", observe: fn(g) { d0r_w_observe(group: g) }) { + D0Settled { decision: D0DecidedSuspend { next: _, reconciliation: _ }, generation: g5, receipt_path: _, entry_state: _ } => g5 == 5 + _ => false + }) + && (match d0r_recovery_of(store: store, successor: h, tx: tx, escalation: "esc-e2" as NonEmptyStr, at: 1004) { Present { value: D0RecoverComplete { claim_generation: _, settled: st } } => (match st.next { SuspendedForAuthorizedSuccessor { group: _, authorization: _, exact_candidate_realization: _, lease: _, cleanup: _ } => true _ => false }) _ => false }) + }) + } + }) + }) +} + +fn d0r_drifted_hosts() -> List { + concat(fabric_group_hosts(g: FabricGroupA).skip(n: 1), [operator_host_srv1]) +} + +fn d0r_w_scoped(successor: ContentHash, tx: NonEmptyStr, escalation: NonEmptyStr) -> ScopedAuthorization? { + d0r_w_authorization(successor: successor, tx: tx, escalation: escalation) +} + +// THE ROSTER CHANGES AFTER THE CLAIM. Two lifecycles, one supplied drifted roster (srv1 in place +// of one member) handed to the dispatcher as the current population: +// claim landed, no write → the dispatcher aborts the claim (AbortedBy) and the authority stays +// at generation 0; +// pending landed → the dispatcher still drives the transaction to a terminal under the original +// authorization: FencedRefusal at generation 2 naming both populations, the claim completed, +// nothing stranded. +test fn a_population_change_after_the_claim_aborts_before_a_write_and_fences_after_one_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let drifted = d0r_drifted_hosts() + match d0r_w_scoped(successor: h, tx: "tx-a" as NonEmptyStr, escalation: "esc-a" as NonEmptyStr) { + Absent => false + Present { value: scoped_a } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-a" as NonEmptyStr, escalation: "esc-a" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: _ } => + d0r_exit_failed(e: d0_dispatch(store: store, g: FabricGroupA, group: "group-a", current_hosts: drifted, scoped: scoped_a, tx: "tx-a" as NonEmptyStr, esc: "esc-a" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1001, now: "2026-09-18T00:00:01Z", current: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001))) + && (match d0_read_claim(store: store, escalation_id: "esc-a" as NonEmptyStr) { D0ClaimAt { state: AbortedBy { attempt: who, aborted_at: _, cause: _ }, generation: _ } => (who as String) == "tx-a" _ => false }) + && d0r_generation_is(store: store, generation: 0, at: 1001) + && (match d0r_w_scoped(successor: h, tx: "tx-b" as NonEmptyStr, escalation: "esc-b" as NonEmptyStr) { + Absent => false + Present { value: scoped_b } => + match d0r_w_admitted(store: store, successor: h, tx: "tx-b" as NonEmptyStr, escalation: "esc-b" as NonEmptyStr, executor: "witness" as NonEmptyStr) { + Absent => false + Present { value: b } => + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: "tx-b" as NonEmptyStr, authorization_binding: b.binding, lease: d0_held_lease(group: FabricGroupA, transaction: "tx-b" as NonEmptyStr, executor: "witness" as NonEmptyStr, generation: 1) }, transaction: "tx-b" as NonEmptyStr, lifecycle: Present { value: b.binding }, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1002) { + AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 + _ => false + }) + && d0r_exit_failed(e: d0_dispatch(store: store, g: FabricGroupA, group: "group-a", current_hosts: drifted, scoped: scoped_b, tx: "tx-b" as NonEmptyStr, esc: "esc-b" as NonEmptyStr, executor: "witness" as NonEmptyStr, at: 1003, now: "2026-09-18T00:00:03Z", current: current_pair_serving_authority(store: store, group: FabricGroupA, at: 1003))) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1004) { + CurrentAuthorityRead { authority: FencedRefusal { group: _, cause: c, disposition_authority: _ }, head: _, generation: g, grant: _, entry_state: Present { value: _ }, previous: _, admitted_by: _, transitions: _ } => g == 2 && (c as String).contains("host population changed") + _ => false + }) + && d0r_claim_is(store: store, escalation: "esc-b" as NonEmptyStr, by: "tx-b", completed: true) + && (match current_pair_serving_authority(store: store, group: FabricGroupA, at: 1005) { + CurrentAuthorityRead { authority: fenced, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => { + let committed = map(spark_claimed_members_under(authorities: [fenced]), m => m.host) + length(committed) == 4 + && all(fabric_group_hosts(g: FabricGroupA), h => any(committed, c => host_identity_eq(a: c, b: h))) + && !any(committed, c => host_identity_eq(a: c, b: operator_host_srv1)) + } + _ => false + }) + } + } + }) + } + } + }) + }) +} + +// TWO EXECUTORS, ONE PENDING LIFECYCLE. Executor 1 claims and writes the pending state under its +// lifecycle (the binding names the executor) and dies. Executor 2, reading the same fleet-global +// claim and the same transaction word, computes another lifecycle, finds no history for it and a +// pending authority: it is REFUSED -- the claim stays ClaimedBy(tx), the authority stays pending at +// generation 1 -- and executor 1's rerun still resumes and settles. A foreign executor never +// terminalizes a claim it did not write under. +test fn a_second_executor_cannot_abort_a_pending_lifecycle_it_did_not_write_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + let e2 = d0r_second_executor_layout(dir: dir) + let tx = "tx-d0" as NonEmptyStr + let esc = "esc-two-lifecycles" as NonEmptyStr + match d0r_w_scoped(successor: h, tx: tx, escalation: esc) { + Absent => false + Present { value: scoped } => + match d0r_w_admitted(store: store, successor: h, tx: tx, escalation: esc, executor: "executor-1" as NonEmptyStr) { + Absent => false + Present { value: one } => + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: tx, authorization_binding: one.binding, lease: d0_held_lease(group: FabricGroupA, transaction: tx, executor: "executor-1" as NonEmptyStr, generation: 1) }, transaction: tx, lifecycle: Present { value: one.binding }, grant: none, entry_state: none, actor: "executor-1" as NonEmptyStr, at: 1000) { + AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 + _ => false + }) + && d0r_exit_failed(e: d0_dispatch(store: e2, g: FabricGroupA, group: "group-a", current_hosts: fabric_group_hosts(g: FabricGroupA), scoped: scoped, tx: tx, esc: esc, executor: "executor-2" as NonEmptyStr, at: 1001, now: "2026-09-18T00:00:01Z", current: current_pair_serving_authority(store: e2, group: FabricGroupA, at: 1001))) + && d0r_claim_is(store: e2, escalation: esc, by: "tx-d0", completed: false) + && (match current_pair_serving_authority(store: e2, group: FabricGroupA, at: 1001) { + CurrentAuthorityRead { authority: SuspensionPendingReconciliation { group: _, transaction: t, authorization_binding: _, lease: _ }, head: _, generation: g, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => g == 1 && (t as String) == "tx-d0" + _ => false + }) + && (match d0r_w_resumed(store: store, successor: h, tx: tx, escalation: esc, executor: "executor-1" as NonEmptyStr) { + Absent => false + Present { value: resumed } => + (match d0r_settled_eligible(o: d0_transaction(store: store, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: resumed, transaction: tx, executor: "executor-1" as NonEmptyStr, at: 1002, now: "2026-09-18T00:00:02Z", observe: fn(g) { d0r_w_observe(group: g) })) { Present { value: _ } => true Absent => false }) + }) + } + } + } + }) + }) +} + +// ── THE LIFECYCLE IS NOT THE EXECUTOR: ANY EXECUTOR COMPLETES A SETTLED CLAIM ───────────────── + +// E1 settles Suspended and dies before CompletedBy. E2 (its own binding, its own executor name), +// reading the same grant and word, computes the SAME lifecycle -- the binding carries no executor +// -- finds E1's settling transition, and COMPLETES the claim (exit 0: the settled state is the +// eligible one) without touching the authority. Then, after an operator restores Active and a +// third lifecycle settles by restoring Active and dies, E2 completes that one too rather than +// starting a second first write: the claim is CompletedBy and the generation is unchanged. +test fn any_executor_completes_a_settled_lifecycle_and_never_restarts_it_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, e1) { + let e2 = d0r_second_executor_layout(dir: dir) + let tx = "tx-d0" as NonEmptyStr + match d0r_w_scoped(successor: h, tx: tx, escalation: "esc-settled" as NonEmptyStr) { + Absent => false + Present { value: scoped_a } => + match d0r_w_admitted(store: e1, successor: h, tx: tx, escalation: "esc-settled" as NonEmptyStr, executor: "executor-1" as NonEmptyStr) { + Absent => false + Present { value: one } => + (match d0r_settled_eligible(o: d0_transaction(store: e1, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: one, transaction: tx, executor: "executor-1" as NonEmptyStr, at: 1000, now: "2026-09-18T00:00:00Z", observe: fn(g) { d0r_w_observe(group: g) })) { Present { value: _ } => true Absent => false }) + && (match d0_dispatch(store: e2, g: FabricGroupA, group: "group-a", current_hosts: fabric_group_hosts(g: FabricGroupA), scoped: scoped_a, tx: tx, esc: "esc-settled" as NonEmptyStr, executor: "executor-2" as NonEmptyStr, at: 1001, now: "2026-09-18T00:00:01Z", current: current_pair_serving_authority(store: e2, group: FabricGroupA, at: 1001)) { ExitSuccess => true ExitFailure { code: _, reason: _ } => false }) + && d0r_claim_is(store: e2, escalation: "esc-settled" as NonEmptyStr, by: "tx-d0", completed: true) + && d0r_generation_is(store: e2, generation: 2, at: 1001) + && (match current_pair_serving_authority(store: e1, group: FabricGroupA, at: 1002) { + CurrentAuthorityRead { authority: suspended, head: _, generation: _, grant: _, entry_state: _, previous: _, admitted_by: _, transitions: _ } => + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => + match pair_serving_authority_transition(store: e1, group: FabricGroupA, expected: suspended, next: resting, transaction: "tx-operator-restore" as NonEmptyStr, lifecycle: none, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1002) { + AuthorityTransitionAppended { id: _, generation: g3, next: _ } => g3 == 3 + _ => false + } + } + _ => false + }) + && (match d0r_w_scoped(successor: h, tx: "tx-restore" as NonEmptyStr, escalation: "esc-restored" as NonEmptyStr) { + Absent => false + Present { value: scoped_b } => + match d0r_w_admitted(store: e1, successor: h, tx: "tx-restore" as NonEmptyStr, escalation: "esc-restored" as NonEmptyStr, executor: "executor-1" as NonEmptyStr) { + Absent => false + Present { value: two } => + (match d0_transaction(store: e1, group: FabricGroupA, current_hosts: fabric_group_hosts(g: FabricGroupA), admitted: two, transaction: "tx-restore" as NonEmptyStr, executor: "executor-1" as NonEmptyStr, at: 1003, now: "2026-09-18T00:00:03Z", observe: fn(g) { d0r_w_observe_live_declared(group: g) }) { + D0Settled { decision: D0DecidedRestore { next: _, key: _ }, generation: g5, receipt_path: _, entry_state: _ } => g5 == 5 + _ => false + }) + && (match d0_dispatch(store: e2, g: FabricGroupA, group: "group-a", current_hosts: fabric_group_hosts(g: FabricGroupA), scoped: scoped_b, tx: "tx-restore" as NonEmptyStr, esc: "esc-restored" as NonEmptyStr, executor: "executor-2" as NonEmptyStr, at: 1004, now: "2026-09-18T00:00:04Z", current: current_pair_serving_authority(store: e2, group: FabricGroupA, at: 1004)) { ExitFailure { code: _, reason: _ } => true ExitSuccess => false }) + && d0r_claim_is(store: e2, escalation: "esc-restored" as NonEmptyStr, by: "tx-restore", completed: true) + && d0r_generation_is(store: e2, generation: 5, at: 1004) + } + }) + } + } + }) + }) +} + +// ── THE FROZEN RESERVATION REACHES THE PRODUCTION ADMISSION ─────────────────────────────────── + +// A fenced lifecycle on the log froze [srv5, srv6, srv7, srv10] (the operator's population, +// differing from today's roster by one host). The production standings fold over the CURRENT +// authorities -- the same fold admit_unplaced_host_live runs after reading the executor's log -- +// commits srv10 (refused for a build, as a member of the claimed serving group) and does not +// commit the roster's fourth host AS A GROUP MEMBER (it may still carry its cell-role commitment, +// which is another authority's), while the resting projection over the source row says the +// opposite. The reservation is the log's, not the roster's. The reservation is the log's, not the roster's. +test fn the_production_admission_commits_the_frozen_hosts_of_a_fenced_lifecycle_by_real_execution() -> Bool { + d0r_with_hash(scenario: fn(h) { + d0r_with_remote(scenario: fn(dir, store) { + match pair_serving_authority_for(group: FabricGroupA) { + Absent => false + Present { value: resting } => { + let roster = fabric_group_hosts(g: FabricGroupA) + let frozen = concat(roster.take(n: 3), [operator_host_srv10]) + let dropped = match get(xs: roster, index: 3) { Present { value: d } => d Absent => operator_host_srv10 } + (match pair_serving_authority_transition(store: store, group: FabricGroupA, expected: resting, next: FencedRefusal { group: FabricGroupA, hosts: frozen, cause: "witness: a fenced lifecycle over a frozen population" as NonEmptyStr, disposition_authority: "witness" as NonEmptyStr }, transaction: "tx-frozen" as NonEmptyStr, lifecycle: Present { value: h }, grant: none, entry_state: none, actor: "witness" as NonEmptyStr, at: 1000) { + AuthorityTransitionAppended { id: _, generation: g1, next: _ } => g1 == 1 + _ => false + }) + && (match spark_host_standings_current(current: [current_pair_serving_authority(store: store, group: FabricGroupA, at: 1001)], host_claims: [] as List) { + CurrentStandingsUnread { cause: _ } => false + CurrentStandingsRead { standings: st } => + (match admit_unplaced_host_in(raw: "srv10", purpose: "witness build" as NonEmptyStr, standings: st) { SparkHostRefused { cause: _ } => true SparkHostAdmitted { host: _ } => false }) + && d0r_committed_to_group_a(standings: st, host: operator_host_srv10) + && !d0r_committed_to_group_a(standings: st, host: dropped) + }) + && (match spark_host_standings_current(current: [CurrentAuthorityUnread { group: FabricGroupA, step: "witness" as String, reason: "unread on purpose" as String }], host_claims: [] as List) { CurrentStandingsUnread { cause: _ } => true CurrentStandingsRead { standings: _ } => false }) + } + } + }) + }) +} + +fn d0r_committed_to_group_a(standings: List, host: HostIdentity) -> Bool { + match spark_standing_of(standings: standings, host: host) { + Absent => false + Present { value: SparkHostUncommitted { host: _ } } => false + Present { value: SparkHostCommitted { host: _, commitments: c } } => + any(spark_authority_held_causes_of(c: c), o => match o { MemberOfClaimedServingGroup { group: FabricGroupA } => true _ => false }) + } +} diff --git a/dag/test/claim/spark/pair_serving_d0_witness_test.dag b/dag/test/claim/spark/pair_serving_d0_witness_test.dag new file mode 100644 index 00000000000..519b90c085c --- /dev/null +++ b/dag/test/claim/spark/pair_serving_d0_witness_test.dag @@ -0,0 +1,608 @@ +module test.claim.spark.pair_serving_d0_witness + +import std.logic { Bool } +import std.types { String, NonEmptyStr, List, Int, Port } +import std.nat { Nat } +import v2.std.optional { Present, Absent } +import std.measure { second } +import std.content_hash { ContentHash, Sha256Hash, sha256_hex_digest, compare_content_hash, ContentHashEqual, ContentHashDifferent, ContentHashCrossFamilyIncomparable, content_hash_equal } +import std.temporal_effect { HeldLease, held_lease, LeaseEpoch, LeaseRunningExpected } +import product.placement_supply { HostIdentity } +import product.fabric.demand { ObservationReceiptRef } +import gunbc.fleet_intent_network { operator_host_srv1, operator_host_srv5, operator_host_srv6, operator_host_srv7, operator_host_srv8 } +import gunbc.spark.fabric_switch_observed { FabricGroup, FabricGroupA, FabricGroupB } +import gunbc.spark.serving_incarnation_observe { UnitOccupancy, UnitInstalled, UnitNotInstalled, UnitOccupancyUnread, IncarnationUnitNotActive } +import gunbc.spark.vllm_endpoint_process_launch { vllm_endpoint_process_launch } +import gunbc.spark.vllm_serving_launch { vllm_serving_launch, endpoint_incarnation_observation } +import gunbc.spark.pair_serving_realization { spark_pair_head_unit_name, spark_pair_worker_unit_name } +import gunbc.spark.pair_incumbent_identity { + DeclaredRankUnit, DeclaredPairRealization, declared_pair_realization_of, DeclaredPairRealizationKeyed, DeclaredPairRealizationUnavailable, + ObservedRankUnit, observed_pair_realization_of, ObservedPairRealizationRead, pair_realization_agreement, +} +import gunbc.spark.released_baseline { QuiescentReservedBaseline } +import gunbc.spark.pair_serving_authority { + pair_serving_committed_hosts, + PairServingGroupAuthority, PairServingActive, SuspensionPendingReconciliation, SuspendedForAuthorizedSuccessor, FencedRefusal, + PairRealizationKeyed, PairRealizationUnestablished, pair_serving_apply_admits, pair_serving_successor_may_launch, +} +import std.scoped_authorization { + OperatorGrant, ScopedAuthorization, AuthorizationGranted, Unclaimed, AuthorizationScope, AttemptIdentity, AuthorizedAction, + AuthorizationClaimState, ClaimedBy, CompletedBy, AbortedBy, +} +import std.effect_grant { Read, Write, NamespacePosition, CodeNameTree } +import gunbc.rung_drop { Standing, Retired } +import gunbc.rung_drop.fabric_storage_append_principal_unrefused { fabric_storage_append_principal_unrefused } +import gunbc.spark.pair_serving_d0 { + d0_store_write_wall_standing, d0_store_operation_wall, StoreOperationWallMissing, StoreOperationWallRestored, + D0Subject, RankOccupancyReading, IncumbentReading, IncumbentAnswered, IncumbentAnsweredUnread, IncumbentDidNotAnswer, IncumbentRouteUnread, D0Observation, + HeadEndpointReading, HeadEndpointQuiet, HeadEndpointListening, HeadEndpointEngineProcess, HeadEndpointUnread, d0_held_lease, + OccupancyReconciliation, OccupancyNotQuiet, DeclaredOccupantDrifted, OccupancyUnread, reconcile_occupancy, + D0Decision, D0DecidedSuspend, D0DecidedRestore, D0DecidedFence, D0Cause, D0IncumbentLive, D0IncumbentUnidentified, D0OccupancyUnread, D0DeclarationUnavailable, + d0_decide, d0_decision_is_eligible, d0_authorized_action, d0_intent_text, + D0Population, D0PopulationAgrees, D0PopulationDiffers, d0_population, + D0AuthorizationStanding, D0AuthorizationResolved, D0AuthorizationUnestablished, resolve_d0_authorization, + D0ClaimReading, D0ClaimAbsent, D0ClaimAt, D0ClaimUnreadable, + D0Recovery, D0RecoverStart, D0RecoverFirstWrite, D0RecoverResume, D0RecoverComplete, D0RecoverAbort, D0RecoveryRefused, d0_recovery, +} +import gunbc.spark.fabric_switch_observed { fabric_group_hosts } +import gunbc.spark.pair_serving_authority_log { AuthorityTransitionRecord, PlacementPreparationRef, committed_population, +} +import product.capacity.event_chain { EventId } +import extdeps.linux.proc_net_tcp { TcpSocketRow, TcpPortBound } + +// THE DECISION AND THE RECONCILIATION, OVER SUPPLIED READINGS. Every branch of the redesign's D0 +// table is driven here with a constructed observation; the transfer and the readings themselves +// execute in the wet entry (pair_serving_d0_wet) against the fleet, and the compare-and-set it +// relies on has its own real-execution witness on the authority log. + +fn d0w_t_sha(hex: String) -> ContentHash? { + match sha256_hex_digest(hex: hex) { + Absent => none + Present { value: d } => Present { value: Sha256Hash(d) } + } +} + +fn d0w_with_digests(scenario: fn(ContentHash, ContentHash, ContentHash) -> Bool) -> Bool { + match d0w_t_sha(hex: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa") { + Absent => false + Present { value: a } => + match d0w_t_sha(hex: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb") { + Absent => false + Present { value: b } => + match d0w_t_sha(hex: "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc") { + Absent => false + Present { value: c } => scenario(a, b, c) + } + } + } +} + +fn d0w_t_declared(head: ContentHash, worker: ContentHash) -> DeclaredPairRealization { + declared_pair_realization_of( + group: FabricGroupA, + head: DeclaredRankUnit { host: operator_host_srv5, unit_name: spark_pair_head_unit_name, digest: head }, + workers: [ + DeclaredRankUnit { host: operator_host_srv6, unit_name: spark_pair_worker_unit_name, digest: worker }, + DeclaredRankUnit { host: operator_host_srv7, unit_name: spark_pair_worker_unit_name, digest: worker }, + DeclaredRankUnit { host: operator_host_srv8, unit_name: spark_pair_worker_unit_name, digest: worker }, + ], + ) +} + +fn d0w_t_rank(host: HostIdentity, unit: NonEmptyStr, occupancy: UnitOccupancy) -> RankOccupancyReading { + RankOccupancyReading { host: host, unit_name: unit, occupancy: occupancy } +} + +fn d0w_t_ranks(head: UnitOccupancy, w6: UnitOccupancy, w7: UnitOccupancy, w8: UnitOccupancy) -> List { + [ + d0w_t_rank(host: operator_host_srv5, unit: spark_pair_head_unit_name, occupancy: head), + d0w_t_rank(host: operator_host_srv6, unit: spark_pair_worker_unit_name, occupancy: w6), + d0w_t_rank(host: operator_host_srv7, unit: spark_pair_worker_unit_name, occupancy: w7), + d0w_t_rank(host: operator_host_srv8, unit: spark_pair_worker_unit_name, occupancy: w8), + ] +} + +fn d0w_t_lease(h: ContentHash) -> HeldLease { + held_lease(epoch: LeaseEpoch { lease_key: "tx-w" as NonEmptyStr, resource_fingerprint: h, owner_fingerprint: h, generation: 1 }, observed: LeaseRunningExpected) +} + +data d0w_t_tx: NonEmptyStr = "tx-w" as NonEmptyStr + +fn d0w_t_subject(successor: ContentHash) -> D0Subject { + D0Subject { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), successor: successor, cleanup: QuiescentReservedBaseline, term: second(count: 1000) } +} + +// A grant over the exact operation: the write scope over the authority, the transaction as the +// attempt, and an intent hash SUPPLIED as a fixture -- the real intent is a SHA-256 taken through +// the sha256sum realization, an effect the wet witness executes; the decision here consumes only +// the grant's subject and escalation. The knobs let the reds vary one binding fact at a time. +fn d0w_t_authorization(successor: ContentHash, scope_verb_write: Bool, attempt: NonEmptyStr, intent: ContentHash, expires_at: String) -> ScopedAuthorization { + AuthorizationGranted { + grant: OperatorGrant { + escalation_id: "auth-w" as NonEmptyStr, + subject: d0w_t_subject(successor: successor), + scopes: [AuthorizationScope { verb: if scope_verb_write { Write } else { Read }, resource: NamespacePosition { tree: CodeNameTree, path: ["gunbc", "spark", "pair_serving_authority"] }, action: d0_authorized_action }], + purpose: "witness d0" as NonEmptyStr, + attempt: (attempt as String) as AttemptIdentity, + intent_hash: intent, + granted_by: "witness" as NonEmptyStr, + granted_at: "2026-09-18T00:00:00Z", + expires_at: expires_at, + }, + claim: Unclaimed, + } +} + +fn d0w_t_good_authorization(successor: ContentHash) -> ScopedAuthorization { + d0w_t_authorization(successor: successor, scope_verb_write: true, attempt: d0w_t_tx, intent: successor, expires_at: "2026-09-19T00:00:00Z") +} + +// The grant the decision is judged under: the fixture authorization's own grant. The DECISION +// consumes the grant's fields (successor, cleanup, escalation); the TRANSACTION consumes an +// AdmittedD0Grant, which only admit_d0_grant mints and which the real-execution witness drives. +fn d0w_t_grant(successor: ContentHash) -> OperatorGrant? { + match d0w_t_good_authorization(successor: successor) { + AuthorizationGranted { grant: g, claim: _ } => Present { value: g } + _ => none + } +} + +fn d0w_t_active() -> PairServingGroupAuthority { + PairServingActive { group: FabricGroupA, exact_realization: PairRealizationUnestablished { obligation: "witness" as NonEmptyStr } } +} + +fn d0w_t_decide(declared: DeclaredPairRealization, incumbent: IncumbentReading, ranks: List, h: ContentHash) -> D0Decision { + match d0w_t_grant(successor: h) { + Absent => D0DecidedFence { next: FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "witness: the fixture authorization carries no grant" as NonEmptyStr, disposition_authority: "witness" as NonEmptyStr }, cause: D0DeclarationUnavailable { obligation: "witness fixture" as NonEmptyStr } } + Present { value: grant } => + d0_decide( + declared: DeclaredPairRealizationKeyed { declared: declared }, + observation: D0Observation { group: FabricGroupA, incumbent: incumbent, ranks: ranks }, + active: d0w_t_active(), + lease: d0w_t_lease(h: h), + grant: grant, + ) + } +} + +fn d0w_silent() -> IncumbentReading { + IncumbentDidNotAnswer { receipt: "the front door could not be connected within the bound" as NonEmptyStr, head: HeadEndpointQuiet } +} + + +// ── RECONCILIATION ───────────────────────────────────────────────────────────────────────────── + +// Our units, our bytes, all running: NOT a reconciled answer. That reading is the running declared +// occupant, which the incumbent question owns; reconciliation is asked only after the head was +// corroborated quiet, so it refuses as OccupancyNotQuiet naming the ranks -- and handed to the +// decision behind a silent front door it fences as unidentified, never suspends. +test fn all_ranks_installed_running_with_declared_bytes_is_not_quiet_and_fences() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let ranks = d0w_t_ranks(head: UnitInstalled { active: true, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: b }, w7: UnitInstalled { active: true, unit_digest: b }, w8: UnitInstalled { active: true, unit_digest: b }) + (match reconcile_occupancy(declared: d0w_t_declared(head: a, worker: b), ranks: ranks) { + OccupancyNotQuiet { live: lv } => length(lv) == 1 && any(lv, h => (h as String) == (operator_host_srv5 as String)) + _ => false + }) + && (match d0w_t_decide(declared: d0w_t_declared(head: a, worker: b), incumbent: d0w_silent(), ranks: ranks, h: c) { + D0DecidedFence { next: _, cause: D0IncumbentUnidentified { cause: _ } } => true + _ => false + }) + }) +} + +// THE EXPECTED ANSWER: our units installed, none running. Drifted, naming every inactive rank -- +// and eligible, not refused (the redesign's correction of an earlier draft). +test fn installed_but_inactive_ranks_are_drifted_and_eligible() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let ranks = d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b }) + (match reconcile_occupancy(declared: d0w_t_declared(head: a, worker: b), ranks: ranks) { + DeclaredOccupantDrifted { drifted: ds } => length(ds) == 4 + _ => false + }) + && (match d0w_t_decide(declared: d0w_t_declared(head: a, worker: b), incumbent: d0w_silent(), ranks: ranks, h: c) { + D0DecidedSuspend { next: SuspendedForAuthorizedSuccessor { group: _, authorization: au, exact_candidate_realization: PairRealizationKeyed { key: k }, lease: _, cleanup: QuiescentReservedBaseline }, reconciliation: DeclaredOccupantDrifted { drifted: ds } } => + (au as String) == "auth-w" && length(ds) == 4 && content_hash_equal(left: k, right: c) + _ => false + }) + }) +} + +// One rank carries other bytes while the others are inactive with the declared bytes: every +// rank is drifted (declared bytes not running is drift, other bytes is drift) and the population +// names all four, the head and the other-bytes rank among them. A rank ACTIVE with the declared +// bytes is live and makes the occupancy not quiet -- the claim above -- never a drifted member. +test fn one_rank_with_other_bytes_is_drifted_naming_it() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match reconcile_occupancy(declared: d0w_t_declared(head: a, worker: b), ranks: d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: true, unit_digest: c }, w8: UnitInstalled { active: false, unit_digest: b })) { + DeclaredOccupantDrifted { drifted: ds } => length(ds) == 4 && any(ds, h => (h as String) == (operator_host_srv7 as String)) && any(ds, h => (h as String) == (operator_host_srv5 as String)) + _ => false + } + }) +} + +// RED -- a rank with no pair unit installed is UNREAD, not empty and not drifted: this +// transaction cannot tell it from a foreign occupant, and says so. It fences. (The head is quiet +// here so absence is corroborated and the worker's unread reading is what governs.) +test fn an_uninstalled_rank_is_unread_and_fences() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let ranks = d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: b }, w7: UnitNotInstalled, w8: UnitInstalled { active: true, unit_digest: b }) + (match reconcile_occupancy(declared: d0w_t_declared(head: a, worker: b), ranks: ranks) { + OccupancyUnread { hosts: hs, cause: _ } => length(hs) == 1 + _ => false + }) + && (match d0w_t_decide(declared: d0w_t_declared(head: a, worker: b), incumbent: d0w_silent(), ranks: ranks, h: c) { + D0DecidedFence { next: FencedRefusal { group: _, cause: _, disposition_authority: _ }, cause: D0OccupancyUnread { cause: _ } } => true + _ => false + }) + }) +} + +// RED -- a rank whose read refused is unread even when every other rank converged. +test fn one_unread_rank_makes_the_occupancy_unread() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match reconcile_occupancy(declared: d0w_t_declared(head: a, worker: b), ranks: d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitOccupancyUnread { refusal: IncarnationUnitNotActive { unit: spark_pair_worker_unit_name } }, w7: UnitInstalled { active: true, unit_digest: b }, w8: UnitInstalled { active: true, unit_digest: b })) { + OccupancyUnread { hosts: hs, cause: _ } => length(hs) == 1 && all(hs, h => (h as String) == (operator_host_srv6 as String)) + _ => false + } + }) +} + +// RED -- a declared rank with no reading at all is unread: a missing reading is not a reading. +test fn a_missing_rank_reading_is_unread() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match reconcile_occupancy(declared: d0w_t_declared(head: a, worker: b), ranks: [d0w_t_rank(host: operator_host_srv5, unit: spark_pair_head_unit_name, occupancy: UnitInstalled { active: false, unit_digest: a })]) { + OccupancyUnread { hosts: hs, cause: _ } => length(hs) == 3 + _ => false + } + }) +} + +// ── THE INCUMBENT QUESTION COMES FIRST ───────────────────────────────────────────────────────── + +// An incumbent answering with head bytes `a` and worker bytes `b`, judged against a declaration +// whose workers are `declared_worker`: equal to `b` for an established incumbent, other for drift. +fn d0w_t_answered(a: ContentHash, b: ContentHash, declared_worker: ContentHash, img: ContentHash) -> IncumbentReading { + let head = endpoint_incarnation_observation( + endpoint: vllm_endpoint_process_launch(group: FabricGroupA, endpoint: "http://10.0.0.5:8000" as NonEmptyStr, process_start_field: "1" as NonEmptyStr), + launch: vllm_serving_launch(group: FabricGroupA, head: operator_host_srv5, systemd_invocation: "inv" as NonEmptyStr, container_id: "c" as NonEmptyStr, executable_digest: img, unit_digest: a, started_at: 1), + container_env: [] as List, + receipt: "test.claim.spark.pair_serving_d0_witness fixture" as ObservationReceiptRef, + ) + let workers = [ + ObservedRankUnit { host: operator_host_srv6, unit_name: spark_pair_worker_unit_name, systemd_invocation: "inv" as NonEmptyStr, unit_digest: b }, + ObservedRankUnit { host: operator_host_srv7, unit_name: spark_pair_worker_unit_name, systemd_invocation: "inv" as NonEmptyStr, unit_digest: b }, + ObservedRankUnit { host: operator_host_srv8, unit_name: spark_pair_worker_unit_name, systemd_invocation: "inv" as NonEmptyStr, unit_digest: b }, + ] + IncumbentAnswered { + agreement: pair_realization_agreement( + declared: DeclaredPairRealizationKeyed { declared: d0w_t_declared(head: a, worker: declared_worker) }, + observed: ObservedPairRealizationRead { observed: observed_pair_realization_of(group: FabricGroupA, head: head, workers: workers, receipt: "test.claim.spark.pair_serving_d0_witness fixture" as ObservationReceiptRef) }, + ), + } +} + +// An incumbent answers and IS the declared realization on every rank: restoration is shown safe, +// the authority returns to EXACTLY the Active state it held -- unchanged, not upgraded to a keyed +// realization, because the running image is observed and uncompared -- and D0 is refused. +test fn a_live_declared_incumbent_restores_the_active_state_unchanged_and_refuses() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let d = d0w_t_declared(head: a, worker: b) + let decision = d0w_t_decide(declared: d, incumbent: d0w_t_answered(a: a, b: b, declared_worker: b, img: c), ranks: d0w_t_ranks(head: UnitInstalled { active: true, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: b }, w7: UnitInstalled { active: true, unit_digest: b }, w8: UnitInstalled { active: true, unit_digest: b }), h: c) + match decision { + D0DecidedRestore { next: PairServingActive { group: _, exact_realization: PairRealizationUnestablished { obligation: o } }, key: k2 } => + (o as String) == "witness" && content_hash_equal(left: k2, right: d.key) && !d0_decision_is_eligible(d: decision) + _ => false + } + }) +} + +// RED -- an incumbent answers but is NOT the declared realization (a worker's bytes differ): not +// restored, not suspended -- fenced for the operator, naming the standing. +test fn a_live_drifted_incumbent_is_fenced() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match d0w_t_decide(declared: d0w_t_declared(head: a, worker: c), incumbent: d0w_t_answered(a: a, b: b, declared_worker: c, img: c), ranks: d0w_t_ranks(head: UnitInstalled { active: true, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: c }, w7: UnitInstalled { active: true, unit_digest: c }, w8: UnitInstalled { active: true, unit_digest: c }), h: c) { + D0DecidedFence { next: FencedRefusal { group: _, cause: _, disposition_authority: _ }, cause: D0IncumbentLive { standing: _ } } => true + _ => false + } + }) +} + +// RED -- the route could not be read at all: the incumbent question is unanswered, and an +// unanswered first question fences before the ranks are consulted. +test fn an_unread_route_fences_before_reconciling() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match d0w_t_decide(declared: d0w_t_declared(head: a, worker: b), incumbent: IncumbentRouteUnread { cause: "no fleet ssh context" as NonEmptyStr }, ranks: d0w_t_ranks(head: UnitInstalled { active: true, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: b }, w7: UnitInstalled { active: true, unit_digest: b }, w8: UnitInstalled { active: true, unit_digest: b }), h: c) { + D0DecidedFence { next: _, cause: D0OccupancyUnread { cause: _ } } => true + _ => false + } + }) +} + +// RED -- the route ANSWERED but the document named no launch: an incumbent may be live and could +// not be identified. That is not silence, and it fences before the ranks are consulted -- +// whatever the ranks would have reconciled to. +test fn an_answering_but_unidentified_incumbent_fences() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match d0w_t_decide(declared: d0w_t_declared(head: a, worker: b), incumbent: IncumbentAnsweredUnread { cause: "the served metrics carried no process start-time series" as NonEmptyStr }, ranks: d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b }), h: c) { + D0DecidedFence { next: FencedRefusal { group: _, cause: _, disposition_authority: _ }, cause: D0IncumbentUnidentified { cause: _ } } => true + _ => false + } + }) +} + +// THE ENTRY IS NON-ADMITTING TODAY BY CONSTRUCTION: no authorization id resolves to a scoped +// authorization, so the wet entry refuses before its first write. This is the wall's executed +// red; the positive route is driven by the real-execution witness with a fixture grant. +test fn no_authorization_resolves_today_so_the_entry_cannot_admit() -> Bool { + match resolve_d0_authorization(group: FabricGroupA, authorization: "esc-anything") { + D0AuthorizationUnestablished { obligation: _ } => true + D0AuthorizationResolved { authorization: _ } => false + } +} + +fn d0w_port(n: Int) -> Port { + n +} + +fn d0w_with_four_hosts(scenario: fn(HostIdentity, HostIdentity, HostIdentity, HostIdentity) -> Bool) -> Bool { + let current = fabric_group_hosts(g: FabricGroupA) + if length(current) != 4 { false } else { + match get(xs: current, index: 0) { + Absent => false + Present { value: h1 } => + match get(xs: current, index: 1) { + Absent => false + Present { value: h2 } => + match get(xs: current, index: 2) { + Absent => false + Present { value: h3 } => + match get(xs: current, index: 3) { + Absent => false + Present { value: h4 } => scenario(h1, h2, h3, h4) + } + } + } + } + } +} + +fn d0w_t_subject_over(hs: List, successor: ContentHash, term: Nat) -> D0Subject { + D0Subject { group: FabricGroupA, hosts: hs, successor: successor, cleanup: QuiescentReservedBaseline, term: second(count: term) } +} + +// THE SUBJECT NAMES THE EXACT HOSTS. A grant whose host list is the group's current population, +// in any order, agrees; one with a host swapped, one dropped, or one duplicated differs -- by an +// identity join in both directions, so a count could not have passed the swap. +test fn the_grant_population_is_joined_to_the_current_group_by_identity() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + d0w_with_four_hosts(scenario: fn(h1, h2, h3, h4) { + (match d0_population(subject: d0w_t_subject_over(hs: [h4, h3, h2, h1], successor: c, term: 1000), current: [h1, h2, h3, h4]) { D0PopulationAgrees => true _ => false }) + && (match d0_population(subject: d0w_t_subject_over(hs: [h1, h2, h3], successor: c, term: 1000), current: [h1, h2, h3, h4]) { D0PopulationDiffers { granted: _, current: _ } => true _ => false }) + && (match d0_population(subject: d0w_t_subject_over(hs: [h1, h2, h3, h3], successor: c, term: 1000), current: [h1, h2, h3, h4]) { D0PopulationDiffers { granted: _, current: _ } => true _ => false }) + && (match d0_population(subject: d0w_t_subject_over(hs: [h1, h2, h3, operator_host_srv1], successor: c, term: 1000), current: [h1, h2, h3, h4]) { D0PopulationDiffers { granted: _, current: _ } => true _ => false }) + }) + }) +} + +// THE INTENT'S CANONICAL TEXT is one line per fact and the hosts are sorted: two subjects that +// list the same hosts in different orders encode identically, and any other fact moving -- +// successor, term, transaction -- changes the text. +test fn the_intent_text_is_canonical_over_the_operation() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + d0w_with_four_hosts(scenario: fn(h1, h2, h3, h4) { + let one = d0w_t_subject_over(hs: [h1, h2, h3, h4], successor: c, term: 1000) + let t = d0_intent_text(subject: one, group: FabricGroupA, transaction: d0w_t_tx) + t == d0_intent_text(subject: d0w_t_subject_over(hs: [h4, h2, h1, h3], successor: c, term: 1000), group: FabricGroupA, transaction: d0w_t_tx) + && t != d0_intent_text(subject: d0w_t_subject_over(hs: [h1, h2, h3, h4], successor: c, term: 1001), group: FabricGroupA, transaction: d0w_t_tx) + && t != d0_intent_text(subject: d0w_t_subject_over(hs: [h1, h2, h3, h4], successor: a, term: 1000), group: FabricGroupA, transaction: d0w_t_tx) + && t != d0_intent_text(subject: one, group: FabricGroupA, transaction: "tx-other" as NonEmptyStr) + && t.contains("suspend-pair-serving-authority") + }) + }) +} + +// ── RECOVERY: THE JOIN OVER THE CLAIM AND THE AUTHORITY HISTORY ──────────────────────────────── + +fn d0w_rec_of(id: String, tx: String, lifecycle: ContentHash?, next: PairServingGroupAuthority) -> AuthorityTransitionRecord { + AuthorityTransitionRecord { id: (id as NonEmptyStr) as EventId, transaction: tx as NonEmptyStr, lifecycle: lifecycle, next: next, placement: PlacementPreparationRef { id: (join(["p-", id], "") as NonEmptyStr) as EventId, group: FabricGroupA, previous_hosts: [] as List, next_hosts: committed_population(a: next), operation: tx as NonEmptyStr } } +} + +// A record written by THIS lifecycle (binding c, the same hash the claims pass as the lifecycle). +fn d0w_rec(id: String, tx: String, next: PairServingGroupAuthority) -> AuthorityTransitionRecord { + d0w_rec_of(id: id, tx: tx, lifecycle: none, next: next) +} + +fn d0w_mine(id: String, h: ContentHash, next: PairServingGroupAuthority) -> AuthorityTransitionRecord { + d0w_rec_of(id: id, tx: "tx-w", lifecycle: Present { value: h }, next: next) +} + +// A pending state as this witness's executor ("witness") wrote it: the lease owner is that +// executor's fingerprint, so a resume from the same executor is admitted and one from another +// executor is refused by the owner check. +fn d0w_pending(tx: String, h: ContentHash) -> PairServingGroupAuthority { + SuspensionPendingReconciliation { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), transaction: tx as NonEmptyStr, authorization_binding: h, lease: d0_held_lease(group: FabricGroupA, transaction: tx as NonEmptyStr, executor: "witness" as NonEmptyStr, generation: 1) } +} + +fn d0w_claimed(by: String) -> D0ClaimReading { + D0ClaimAt { state: ClaimedBy { attempt: (by as NonEmptyStr) as AttemptIdentity, claimed_at: "2026-09-18T00:00:00Z" }, generation: 1 } +} + +fn d0w_suspended(h: ContentHash) -> PairServingGroupAuthority { + SuspendedForAuthorizedSuccessor { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), authorization: "auth-w" as NonEmptyStr, exact_candidate_realization: PairRealizationKeyed { key: h }, lease: d0w_t_lease(h: h), cleanup: QuiescentReservedBaseline } +} + +// EVERY CRASH POINT HAS ONE DISPOSITION. No claim starts. A claim held by this transaction with no +// transition by it performs the first write when the authority is Active and aborts otherwise. A +// pending state under this transaction resumes. A settling transition by this transaction means +// only the claim's terminal is owed. A pending state written by this transaction that another +// writer moved is refused for an operator. +test fn the_recovery_join_dispatches_every_crash_point() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let active = d0w_t_active() + (match d0_recovery(claim: D0ClaimAbsent, transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: [] as List) { D0RecoverStart => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: active, transitions: [] as List) { D0RecoverFirstWrite { claim_generation: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "w" as NonEmptyStr, disposition_authority: "w" as NonEmptyStr }, transitions: [d0w_rec(id: "e1", tx: "tx-other", next: FencedRefusal { group: FabricGroupA, hosts: fabric_group_hosts(g: FabricGroupA), cause: "w" as NonEmptyStr, disposition_authority: "w" as NonEmptyStr })]) { D0RecoverAbort { claim_generation: _, cause: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: d0w_pending(tx: "tx-w", h: c), transitions: [d0w_mine(id: "e1", h: c, next: d0w_pending(tx: "tx-w", h: c))]) { D0RecoverResume { claim_generation: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: d0w_suspended(h: c), transitions: [d0w_mine(id: "e1", h: c, next: d0w_pending(tx: "tx-w", h: c)), d0w_mine(id: "e2", h: c, next: d0w_suspended(h: c))]) { D0RecoverComplete { claim_generation: _, settled: s } => (s.id as String) == "e2" _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: active, transitions: [d0w_mine(id: "e1", h: c, next: d0w_pending(tx: "tx-w", h: c)), d0w_mine(id: "e2", h: c, next: active), d0w_rec(id: "e3", tx: "tx-later", next: d0w_pending(tx: "tx-later", h: c)), d0w_rec(id: "e4", tx: "tx-later", next: active)]) { D0RecoverComplete { claim_generation: _, settled: s } => (s.id as String) == "e2" _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: active, transitions: [d0w_mine(id: "e1", h: c, next: d0w_pending(tx: "tx-w", h: c)), d0w_rec(id: "e2", tx: "tx-operator", next: active)]) { D0RecoveryRefused { cause: _ } => true _ => false }) + }) +} + +// RED: THE TRANSACTION WORD IS NOT THE LIFECYCLE. Escalation E1 under word tx-w settled (its +// transitions carry binding a); an operator restored the group; escalation E2 under the SAME word +// claimed (binding c) and died before its first write. E2's recovery must be FirstWrite -- never +// Complete from E1's terminal -- and a claim whose binding could not be digested identifies no +// history at all and is refused rather than joined by the word. AND THE LIFECYCLE IS NOT THE +// EXECUTOR: another executor facing THIS lifecycle's pending state is refused by the lease owner +// (no handoff), but that same executor completes a settled lifecycle -- suspended or restored to +// Active -- rather than aborting it or starting a second first write. +test fn a_reused_transaction_word_does_not_inherit_another_lifecycles_history() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let active = d0w_t_active() + let history = [ + d0w_rec_of(id: "e1", tx: "tx-w", lifecycle: Present { value: a }, next: d0w_pending(tx: "tx-w", h: a)), + d0w_rec_of(id: "e2", tx: "tx-w", lifecycle: Present { value: a }, next: d0w_suspended(h: a)), + d0w_rec(id: "e3", tx: "tx-operator-restore", next: active), + ] + (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: active, transitions: history) { D0RecoverFirstWrite { claim_generation: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: a }, executor: "witness" as NonEmptyStr, current: active, transitions: history) { D0RecoverComplete { claim_generation: _, settled: s } => (s.id as String) == "e2" _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: history) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "witness" as NonEmptyStr, current: d0w_pending(tx: "tx-w", h: a), transitions: concat(history, [d0w_rec_of(id: "e4", tx: "tx-w", lifecycle: Present { value: c }, next: d0w_pending(tx: "tx-w", h: c))])) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "another-executor" as NonEmptyStr, current: d0w_pending(tx: "tx-w", h: c), transitions: [d0w_mine(id: "e5", h: c, next: d0w_pending(tx: "tx-w", h: c))]) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "another-executor" as NonEmptyStr, current: d0w_suspended(h: c), transitions: [d0w_mine(id: "e5", h: c, next: d0w_pending(tx: "tx-w", h: c)), d0w_mine(id: "e6", h: c, next: d0w_suspended(h: c))]) { D0RecoverComplete { claim_generation: _, settled: st } => (st.id as String) == "e6" _ => false }) + && (match d0_recovery(claim: d0w_claimed(by: "tx-w"), transaction: d0w_t_tx, lifecycle: Present { value: c }, executor: "another-executor" as NonEmptyStr, current: active, transitions: [d0w_mine(id: "e5", h: c, next: d0w_pending(tx: "tx-w", h: c)), d0w_mine(id: "e6", h: c, next: active)]) { D0RecoverComplete { claim_generation: _, settled: st } => (st.id as String) == "e6" _ => false }) + }) +} + +// RED: consent that is not this run's is refused whatever the authority says -- held by another +// attempt, completed, aborted, or unreadable -- and an Active authority with a completed claim is +// NOT a start. +test fn a_claim_that_is_not_this_transactions_is_refused_before_any_write() -> Bool { + let active = d0w_t_active() + let none_yet = [] as List + (match d0_recovery(claim: d0w_claimed(by: "tx-other"), transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: none_yet) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: D0ClaimAt { state: CompletedBy { attempt: ("tx-w" as NonEmptyStr) as AttemptIdentity, completed_at: "t" }, generation: 2 }, transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: none_yet) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: D0ClaimAt { state: AbortedBy { attempt: ("tx-w" as NonEmptyStr) as AttemptIdentity, aborted_at: "t", cause: "w" as NonEmptyStr }, generation: 2 }, transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: none_yet) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: D0ClaimAt { state: Unclaimed, generation: 1 }, transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: none_yet) { D0RecoveryRefused { cause: _ } => true _ => false }) + && (match d0_recovery(claim: D0ClaimUnreadable { reason: "w" as NonEmptyStr }, transaction: d0w_t_tx, lifecycle: none, executor: "witness" as NonEmptyStr, current: active, transitions: none_yet) { D0RecoveryRefused { cause: _ } => true _ => false }) +} + +// A declaration that could not be keyed fences: there is nothing to reconcile against. +test fn an_unavailable_declaration_fences() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match d0w_t_grant(successor: c) { + Absent => false + Present { value: grant } => + match d0_decide( + declared: DeclaredPairRealizationUnavailable { group: FabricGroupA, obligation: "witness" as NonEmptyStr }, + observation: D0Observation { group: FabricGroupA, incumbent: d0w_silent(), ranks: [] as List }, + active: d0w_t_active(), lease: d0w_t_lease(h: c), grant: grant, + ) { + D0DecidedFence { next: _, cause: D0DeclarationUnavailable { obligation: _ } } => true + _ => false + } + } + }) +} + +// The suspended authority D0 writes is what the successor's launch gate reads: with the lease +// running it may launch, and the incumbent's apply is refused. +test fn the_suspension_d0_writes_admits_launch_and_refuses_apply() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + match d0w_t_decide(declared: d0w_t_declared(head: a, worker: b), incumbent: d0w_silent(), ranks: d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b }), h: c) { + D0DecidedSuspend { next: n, reconciliation: _ } => pair_serving_successor_may_launch(a: n) && !pair_serving_apply_admits(a: n) + _ => false + } + }) +} + +// ── ABSENCE IS CORROBORATED ON THE HEAD, AND THE FENCE KEEPS THE AUTHORIZED HOSTS ────────────── + +// The front door gave no status (a failed connect). Absence needs the head QUIET on both readings: +// no LISTEN on the enrolled port and no engine process (HeadEndpointQuiet), AND the declared unit +// inactive. A listener, an engine process, or an unread table fences as an unidentified incumbent +// (or unread); an active declared unit fences; an unread head fences; only a quiet head with an +// inactive unit proceeds to reconciliation. +test fn a_failed_connect_is_absence_only_when_the_head_unit_is_quiet() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + let declared = d0w_t_declared(head: a, worker: b) + let no_status = IncumbentDidNotAnswer { receipt: "the front door could not be connected (curl exit 7)" as NonEmptyStr, head: HeadEndpointQuiet } + let quiet_ranks = d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b }) + (match reconcile_occupancy(declared: declared, ranks: d0w_t_ranks(head: UnitInstalled { active: true, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b })) { OccupancyNotQuiet { live: lv } => length(lv) == 1 _ => false }) + && (match reconcile_occupancy(declared: declared, ranks: [] as List) { OccupancyUnread { hosts: _, cause: _ } => true _ => false }) + && (match reconcile_occupancy(declared: declared, ranks: d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: b }, w7: UnitInstalled { active: true, unit_digest: b }, w8: UnitInstalled { active: true, unit_digest: b })) { OccupancyNotQuiet { live: lv } => length(lv) == 3 && !any(lv, h => (h as String) == (operator_host_srv5 as String)) _ => false }) + && (match d0w_t_decide(declared: declared, incumbent: no_status, ranks: d0w_t_ranks(head: UnitInstalled { active: true, unit_digest: a }, w6: UnitInstalled { active: true, unit_digest: b }, w7: UnitInstalled { active: true, unit_digest: b }, w8: UnitInstalled { active: true, unit_digest: b }), h: c) { + D0DecidedFence { next: FencedRefusal { group: _, cause: _, disposition_authority: _ }, cause: D0IncumbentUnidentified { cause: _ } } => true + _ => false + }) + && (match d0w_t_decide(declared: declared, incumbent: no_status, ranks: d0w_t_ranks(head: UnitOccupancyUnread { refusal: IncarnationUnitNotActive { unit: spark_pair_head_unit_name } }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b }), h: c) { + D0DecidedFence { next: _, cause: D0OccupancyUnread { cause: _ } } => true + _ => false + }) + && (match d0w_t_decide(declared: declared, incumbent: no_status, ranks: quiet_ranks, h: c) { + D0DecidedSuspend { next: _, reconciliation: DeclaredOccupantDrifted { drifted: ds } } => length(ds) == 4 + _ => false + }) + && (match d0w_t_decide(declared: declared, incumbent: IncumbentDidNotAnswer { receipt: "curl 7" as NonEmptyStr, head: HeadEndpointListening { port: d0w_port(n: 8000), rows: [TcpSocketRow { local_address: "00000000:1F40", local_port: TcpPortBound { port: 8000 }, state: "0A" }] } }, ranks: quiet_ranks, h: c) { + D0DecidedFence { next: _, cause: D0IncumbentUnidentified { cause: _ } } => true + _ => false + }) + && (match d0w_t_decide(declared: declared, incumbent: IncumbentDidNotAnswer { receipt: "curl 7" as NonEmptyStr, head: HeadEndpointEngineProcess { listing: "4242 python -m vllm.entrypoints.openai.api_server" } }, ranks: quiet_ranks, h: c) { + D0DecidedFence { next: _, cause: D0IncumbentUnidentified { cause: _ } } => true + _ => false + }) + && (match d0w_t_decide(declared: declared, incumbent: IncumbentDidNotAnswer { receipt: "curl 7" as NonEmptyStr, head: HeadEndpointUnread { cause: "/proc/net/tcp unread" as NonEmptyStr } }, ranks: quiet_ranks, h: c) { + D0DecidedFence { next: _, cause: D0OccupancyUnread { cause: _ } } => true + _ => false + }) + }) +} + +fn d0w_same_order(hs: List, first: HostIdentity, last: HostIdentity) -> Bool { + length(hs) == 4 + && (match get(xs: hs, index: 0) { Present { value: x } => (x as String) == (first as String) Absent => false }) + && (match get(xs: hs, index: 3) { Present { value: x } => (x as String) == (last as String) Absent => false }) +} + +// EVERY STATE D0 WRITES CARRIES THE AUTHORIZED HOSTS, AND THE COMMITMENT READS THEM. The subject +// names [h4, h3, h2, h1] (the group's hosts, reordered); the suspension and the fence D0 decides +// commit exactly those, in the subject's order, whatever the roster says now -- and a resting +// Active commits the roster. +test fn the_states_d0_writes_commit_the_authorized_hosts_not_the_roster() -> Bool { + d0w_with_digests(scenario: fn(a, b, c) { + d0w_with_four_hosts(scenario: fn(h1, h2, h3, h4) { + let subject = d0w_t_subject_over(hs: [h4, h3, h2, h1], successor: c, term: 1000) + match d0w_t_grant(successor: c) { + Absent => false + Present { value: g0 } => { + let grant = OperatorGrant { escalation_id: g0.escalation_id, subject: subject, scopes: g0.scopes, purpose: g0.purpose, attempt: g0.attempt, intent_hash: g0.intent_hash, granted_by: g0.granted_by, granted_at: g0.granted_at, expires_at: g0.expires_at } + (match d0_decide(declared: DeclaredPairRealizationKeyed { declared: d0w_t_declared(head: a, worker: b) }, observation: D0Observation { group: FabricGroupA, incumbent: d0w_silent(), ranks: d0w_t_ranks(head: UnitInstalled { active: false, unit_digest: a }, w6: UnitInstalled { active: false, unit_digest: b }, w7: UnitInstalled { active: false, unit_digest: b }, w8: UnitInstalled { active: false, unit_digest: b }) }, active: d0w_t_active(), lease: d0w_t_lease(h: c), grant: grant) { + D0DecidedSuspend { next: n, reconciliation: _ } => d0w_same_order(hs: pair_serving_committed_hosts(a: n), first: h4, last: h1) + _ => false + }) + && (match d0_decide(declared: DeclaredPairRealizationKeyed { declared: d0w_t_declared(head: a, worker: b) }, observation: D0Observation { group: FabricGroupA, incumbent: IncumbentRouteUnread { cause: "w" as NonEmptyStr }, ranks: [] as List }, active: d0w_t_active(), lease: d0w_t_lease(h: c), grant: grant) { + D0DecidedFence { next: n, cause: _ } => d0w_same_order(hs: pair_serving_committed_hosts(a: n), first: h4, last: h1) + _ => false + }) + && length(pair_serving_committed_hosts(a: d0w_t_active())) == 4 + } + } + }) + }) +} + +// THE WET DOOR CONSUMES THE STORE'S WRITE-WALL DROP: while gunbc.rung_drop +// fabric_storage_append_principal_unrefused stands the door's first gate names it and refuses; when the +// row is retired the gate is absent. The gate is read from the row, so retiring the row is the +// only thing that opens it. +test fn the_wet_door_refuses_while_the_fabric_storage_write_wall_drop_stands() -> Bool { + match fabric_storage_append_principal_unrefused.standing { + Standing => (match d0_store_write_wall_standing() { Present { value: why } => why.contains("fabric_storage_append_principal_unrefused") && why.contains("does not run") Absent => false }) + Retired { trigger_fired: _ } => + match d0_store_operation_wall { + StoreOperationWallMissing { trigger: _ } => (match d0_store_write_wall_standing() { Present { value: why } => why.contains("d0_store_operation_wall") && why.contains("does not run") Absent => false }) + StoreOperationWallRestored { by: _ } => (match d0_store_write_wall_standing() { Absent => true Present { value: _ } => false }) + } + } +} diff --git a/dag/test/claim/spark/spark_host_commitment_witness_test.dag b/dag/test/claim/spark/spark_host_commitment_witness_test.dag index c4858f1a130..014831fd341 100644 --- a/dag/test/claim/spark/spark_host_commitment_witness_test.dag +++ b/dag/test/claim/spark/spark_host_commitment_witness_test.dag @@ -5,6 +5,7 @@ import std.algebra { FreeSemigroup, FreeMonoid, Empty, Cons } import product.placement_supply { HostIdentity, host_identity_eq } import std.decl_ref { DeclarationRef, decl_ref, declaration_ref_display_key } import v2.std.optional { Present, Absent } +import gunbc.spark.pair_serving_authority_log { HostEffectRecord } import gunbc.fleet_intent_network { operator_host_srv5, operator_host_srv6, operator_host_srv7, operator_host_srv8, operator_host_srv9, operator_host_srv10, operator_host_srv11, operator_host_srv12, @@ -33,6 +34,7 @@ import gunbc.spark.host_commitment { HoldsUndischargedRpcReservation, CellRoleUndecidable, OccupiedByServingArm, + OccupiedByHostEffect, SparkHostRulingCommitment, ReservedForServingSubject, ClaimedServingGroupMember, @@ -95,6 +97,7 @@ fn fixture_standings(held: List, arms: List) arm_claims: arms, held_reservation_hosts: held, serving_reservations: [], + host_effects: [] as List ) } @@ -112,6 +115,7 @@ fn fixture_standing(assignments: List, claimed: List, host: host, ) } @@ -128,6 +132,7 @@ fn is_cell_role(c: SparkHostCommitment) -> Bool { MemberOfClaimedServingGroup { group: _ } => false OccupiedByServingArm { claim: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false } RulingDerivedCommitment { cause: _ } => false @@ -142,6 +147,7 @@ fn is_group_member(c: SparkHostCommitment) -> Bool { CommittedToCellRole { role: _ } => false OccupiedByServingArm { claim: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false } RulingDerivedCommitment { cause: _ } => false @@ -156,6 +162,7 @@ fn is_held_reservation(c: SparkHostCommitment) -> Bool { MemberOfClaimedServingGroup { group: _ } => false CommittedToCellRole { role: _ } => false OccupiedByServingArm { claim: _ } => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false } RulingDerivedCommitment { cause: _ } => false @@ -243,6 +250,7 @@ test fn a_host_with_an_ambiguous_role_is_committed_rather_than_free() -> Bool { MemberOfClaimedServingGroup { group: _ } => false OccupiedByServingArm { claim: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false } RulingDerivedCommitment { cause: _ } => false } @@ -305,6 +313,7 @@ test fn a_multi_cause_host_renders_every_cause_with_both_separators() -> Bool { arm_claims: [], held_reservation_hosts: [operator_host_srv9], serving_reservations: [], + host_effects: [] as List, host: operator_host_srv9, ) { Absent => false @@ -384,6 +393,7 @@ fn placed_by_the_group_b_arm(host: HostIdentity) -> Bool { CommittedToCellRole { role: _ } => false MemberOfClaimedServingGroup { group: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false }) } @@ -446,6 +456,7 @@ test fn the_group_b_head_host_is_committed_and_its_cause_names_the_assignment() CommittedToCellRole { role: _ } => false MemberOfClaimedServingGroup { group: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false }) } @@ -468,7 +479,7 @@ test fn an_arm_claim_commits_only_the_host_it_names() -> Bool { // ── RED 5: THE ONE ADMISSION SEAM REFUSES A CLAIMED HOST, AND SAYS WHY ──────────────────────── // // The three effectful roots -- the source build, the image build entry and the published-image probe -// -- all reach a host through `admit_unplaced_host`. This is the control that the seam refuses a +// -- all reach a host through `admit_unplaced_host_live` (the resting projection here supplies the same standings shape). This is the control that the seam refuses a // claimed target and that its refusal names the actual cause rather than a generic sentence, which is // what the deleted hand-maintained producer list could never do. // @@ -658,6 +669,7 @@ test fn a_host_held_against_a_subject_names_the_unheld_causes() -> Bool { CommittedToCellRole { role: _ } => false MemberOfClaimedServingGroup { group: _ } => false HoldsUndischargedRpcReservation => false + OccupiedByHostEffect { purpose: _, executor: _ } => false CellRoleUndecidable { assignments: _ } => false } RulingDerivedCommitment { cause: _ } => false @@ -689,6 +701,7 @@ fn fixture_admissible_to_a(held: List) -> List { arm_claims: [ServingArmHostClaim { arm: fixture_arm, host: operator_host_srv9, basis: "fixture" as NonEmptyStr }], held_reservation_hosts: held, serving_reservations: [], + host_effects: [] as List ), subject: pair_unit_a, ) @@ -723,6 +736,7 @@ test fn a_declared_reservation_is_held_by_its_subject_and_against_every_other() hosts: [operator_host_srv8], basis: ServingRulingCitation { decider: "fixture" as NonEmptyStr, ruled_on: "fixture" as NonEmptyStr, ruled: "fixture" as NonEmptyStr }, }], + host_effects: [] as List, ) let to_arm = spark_admissible_hosts_to(standings: standings, subject: arm_subject) let to_a = spark_admissible_hosts_to(standings: standings, subject: pair_unit_a) @@ -749,6 +763,7 @@ test fn an_unclaimed_serving_role_and_any_training_role_are_held_by_nobody() -> arm_claims: [], held_reservation_hosts: [], serving_reservations: [], + host_effects: [] as List ) let to_a = spark_admissible_hosts_to(standings: standings, subject: pair_unit_a) !contains_host(hosts: to_a, host: operator_host_srv7) diff --git a/dag/test/claim/spark/spark_pair_serving_apply_wet_witness_test.dag b/dag/test/claim/spark/spark_pair_serving_apply_wet_witness_test.dag new file mode 100644 index 00000000000..871afd386c9 --- /dev/null +++ b/dag/test/claim/spark/spark_pair_serving_apply_wet_witness_test.dag @@ -0,0 +1,37 @@ +module test.claim.spark.spark_pair_serving_apply_wet_witness + +import std.logic { Bool } +import std.types { String, List } +import v2.std.optional { Present, Absent } +import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } +import gunbc.fabric_event_log_host { now_epoch_seconds } +import gunbc.spark.fabric_reach { ExecutorReachKnown, ExecutorReachUnknown, observe_executor_reach } +import gunbc.spark.pair_serving_apply { spark_pair_apply_plans, spark_pair_apply_plan_refusals } +import gunbc.spark.pair_serving_authority { spark_pair_serving_authorities } + +data live_tree_disposition: LiveTreeDisposition = SubstrateInputsOnly + +// THE APPLY SEAM'S LIVE BINDING, EXECUTED. test.claim.spark.spark_pair_serving_apply_witness drives +// the plan fold over a SUPPLIED roster; this is the other half of DESIGN §3's pairing obligation: +// spark_pair_apply_plans reads the executor's hostname, resolves its event log layout and reads +// every claimed group's authority from the log before planning anything. On the wet lane's +// runner that read does not reach a log, so every claimed group's plan is a refusal that NAMES +// THE READ -- "the serving authority could not be read" -- and nothing is planned from the +// resting row. Deleting the log read, or letting an unreadable log fall back to the row, turns +// these refusals into plans and this claim red. +test fn the_apply_seam_reads_the_authority_before_planning_and_refuses_off_fleet() -> Bool { + match observe_executor_reach() { + ExecutorReachUnknown { cause: _ } => false + ExecutorReachKnown { short_hostname: executor, path: _ } => + match now_epoch_seconds() { + Absent => false + Present { value: at } => { + let plans = spark_pair_apply_plans(executor: executor, at: at) + let refusals = spark_pair_apply_plan_refusals(plans: plans) + length(plans) == length(spark_pair_serving_authorities) + && length(refusals) == length(plans) + && all(refusals, r => string_contains(s: r, pattern: "the serving authority could not be read")) + } + } + } +} diff --git a/dag/test/claim/spark/spark_pair_serving_apply_witness_test.dag b/dag/test/claim/spark/spark_pair_serving_apply_witness_test.dag index 308bb323fae..6bdc6e92b90 100644 --- a/dag/test/claim/spark/spark_pair_serving_apply_witness_test.dag +++ b/dag/test/claim/spark/spark_pair_serving_apply_witness_test.dag @@ -6,13 +6,23 @@ import gunbc.fleet_intent_network { operator_host_srv5, operator_host_srv6, oper import gunbc.spark.pair_serving_realization { spark_pair_realizations } import gunbc.spark.pair_serving_apply { SparkPairApplyPlan, SparkPairApplyPlanned, SparkPairApplyRefused, SparkPairHostApplyPlan, - spark_pair_apply_plan, spark_pair_apply_plans, spark_pair_apply_plan_refusals, + spark_pair_apply_plan_current, spark_pair_apply_plans_current, spark_pair_apply_plan_refusals, +} +import gunbc.spark.pair_serving_authority { spark_pair_serving_authorities, pair_serving_authority_group } +import gunbc.spark.pair_serving_authority_log { CurrentAuthority, CurrentAuthorityRead, CurrentAuthorityUnread, AuthorityTransitionRecord } +import product.capacity.event_chain { HeadAbsent } + +// THE CURRENT AUTHORITY IS SUPPLIED: the resting row of every claimed group at generation 0, which +// is exactly what the log folds to when no transition has been recorded. The live read of the log +// is executed by test.claim.spark.spark_pair_serving_apply_wet_witness on the local-repo wet lane. +fn w_current() -> List { + map(spark_pair_serving_authorities, a => CurrentAuthorityRead { authority: a, head: HeadAbsent, generation: 0, grant: none, entry_state: none, previous: none, admitted_by: none, transitions: [] as List }) } fn w_plan() -> SparkPairApplyPlan { match first(spark_pair_realizations()) { Absent => SparkPairApplyRefused { cause: "no realization" } - Present { value: r } => spark_pair_apply_plan(r: r) + Present { value: r } => spark_pair_apply_plan_current(r: r, current: w_current()) } } @@ -56,21 +66,27 @@ test fn the_script_is_made_of_the_rendered_argv_and_carries_the_cache_over() -> }) } -// THE PAIR ROSTER IS GROUP A ALONE, BY DELIBERATE WITHDRAWAL. spark_pair_serving_groups has been -// [FabricGroupA] since 2026-09-08, when group B's hosts moved to the native GLM arm -- its plans -// are gunbc.spark.native_serving_apply spark_native_arm_steps, pinned by -// test.claim.spark.native_serving_roce_transport_witness_test. An earlier revision of this claim -// expected TWO plans with the second headed by srv9: it was left red from the withdrawal, -// asserting a roster the desired row no longer decides. What this claim pins now: the pair apply -// plans exactly the claimed roster, nothing refuses, and the plan is group A headed by srv6 with -// the other three cage-1 hosts as workers. -test fn every_promoted_group_plans_and_the_pair_plan_is_group_a_headed_by_srv6() -> Bool { - let plans = spark_pair_apply_plans() - length(plans) == 1 && length(spark_pair_apply_plan_refusals(plans: plans)) == 0 - && all(plans, p => match p { - SparkPairApplyRefused { cause: _ } => false - SparkPairApplyPlanned { workers: ws, head: h } => - length(ws) == 3 && host_identity_eq(a: h.host, b: operator_host_srv6) - && any(ws, w => host_identity_eq(a: w.host, b: operator_host_srv5)) - }) +// EVERY CLAIMED GROUP PLANS AND NONE REFUSES: one plan per authority row, so one apply carries +// the whole claimed roster. The roster is the authority population -- Group A alone since the +// membership became a projection of gunbc.spark.pair_serving_authority (#11501) -- and this claim +// reads its expected count from that population rather than from a literal, so an earlier +// spelling that named a second group headed by srv9 with srv12 was measuring a roster that no +// longer exists. The head-and-workers shape of the one plan is the claim above. +test fn every_claimed_group_plans_and_none_refuses() -> Bool { + let plans = spark_pair_apply_plans_current(current: w_current()) + length(plans) == length(spark_pair_serving_authorities) && length(plans) >= 1 + && length(spark_pair_apply_plan_refusals(plans: plans)) == 0 +} + +// RED -- AN UNREADABLE AUTHORITY IS NEVER PLANNED FROM THE RESTING ROW. The supplied roster says the +// log could not be read for the claimed group; the plan for that group is a refusal carrying the +// read's cause, and nothing renders. Folding the unread arm back onto the source row would plan +// the incumbent here and turn this red. +test fn an_unread_authority_refuses_the_plan_with_the_reads_cause() -> Bool { + let unread = map(spark_pair_serving_authorities, a => CurrentAuthorityUnread { group: pair_serving_authority_group(a: a), step: "fetch" as String, reason: "witness: the partition could not be fetched" as String }) + let plans = spark_pair_apply_plans_current(current: unread) + let refusals = spark_pair_apply_plan_refusals(plans: plans) + length(plans) == length(spark_pair_serving_authorities) && length(plans) >= 1 + && length(refusals) == length(plans) + && all(refusals, r => string_contains(s: r, pattern: "the serving authority could not be read") && string_contains(s: r, pattern: "witness: the partition could not be fetched")) } diff --git a/docs/design-rung-drops.md b/docs/design-rung-drops.md index 6b0118191db..1f081f364f4 100644 --- a/docs/design-rung-drops.md +++ b/docs/design-rung-drops.md @@ -330,7 +330,7 @@ two serving processes on one host can both open the single-writer approval store ### an append to the fabric event log by a principal that is not a fleet writer — declared 2026-09-19 -an append to the fabric event log by a principal that is not a fleet writer: RUNG DROP, mechanically preventable -> mitigatable (replacement staged: gunbc.fabric_storage_serve fabric_storage_serve_handle, reached only through tailscale serve on srv1 (tailnet membership is the only boundary)). Population: gunbc.fabric_storage_serve fabric_storage_serve_handle, gunbc.fabric_event_log event_log_append, gunbc.fabric_event_log fabric_seat_acquire. Restored when: a modeled roster of fabric writer principals, grounded in an observed reading of the identity each fleet writer presents through tailscale serve on srv1 -- SUFFICIENT FOR fabric_storage_serve_handle to refuse a put or advance from any identity outside the roster, including an absent identity, with a discriminating RED over the real handler. +an append to the fabric event log by a principal that is not a fleet writer: RUNG DROP, mechanically preventable -> mitigatable (replacement staged: gunbc.fabric_storage_serve fabric_storage_serve_handle, reached only through tailscale serve on srv1 (tailnet membership is the only boundary)). Population: gunbc.fabric_storage_serve fabric_storage_serve_handle, gunbc.fabric_event_log event_log_append, gunbc.fabric_event_log event_log_append_with, gunbc.fabric_event_log fabric_seat_acquire, gunbc.spark.pair_serving_authority_log (every writer to a pair-serving-authority partition and to host-placement: establishment, transition, entry state, cancellation, host-effect claim and release, placement prepare / claim / finalize / abort), gunbc.durable_cas_fabric_storage fabric_storage_compare_and_set (the D0 operator-consent claim slot). Restored when: a modeled roster of fabric writer principals, grounded in an observed reading of the identity each fleet writer presents through tailscale serve on srv1 -- SUFFICIENT FOR fabric_storage_serve_handle to refuse a put or advance from any identity outside the roster, including an absent identity, with a discriminating RED over the real handler. ### the required namespace-wave-admission phase: merge-path adjudication of closure, subject-membership and occurrence-binding deltas between merge base and head — declared 2026-09-19 diff --git a/src/v1/stage0/src/std_content_hash.rs b/src/v1/stage0/src/std_content_hash.rs index 6530c69c7dc..36426130a03 100644 --- a/src/v1/stage0/src/std_content_hash.rs +++ b/src/v1/stage0/src/std_content_hash.rs @@ -239,6 +239,14 @@ pub enum ContentHashComparison { ContentHashCrossFamilyIncomparable, } +pub fn content_hash_equal(left: Rc, right: Rc) -> bool { + match compare_content_hash(left.clone(), right.clone()) { + ContentHashComparison::ContentHashEqual => true, + ContentHashComparison::ContentHashDifferent => false, + ContentHashComparison::ContentHashCrossFamilyIncomparable => false, + } +} + pub fn compare_content_hash( left: Rc, right: Rc, @@ -304,6 +312,66 @@ pub fn serialize_content_hash(hash: Rc) -> String { } } +pub fn parse_content_hash_candidate(wire: String) -> Option> { + if v1_rt::starts_with(wire.clone(), "sha256:".to_string()) { + match Rc::new( + wire.clone() + .split(&"sha256:".to_string()) + .map(|s| s.to_string()) + .collect::>(), + ) + .get((1) as usize) + .cloned() + { + std::option::Option::None => std::option::Option::None, + Some(hex) => match sha256_hex_digest(hex.clone()) { + std::option::Option::None => std::option::Option::None, + Some(d) => Some(as_content_hash_cryptographic(d.clone())), + }, + } + } else { + if v1_rt::starts_with(wire.clone(), "sha512:".to_string()) { + match Rc::new( + wire.clone() + .split(&"sha512:".to_string()) + .map(|s| s.to_string()) + .collect::>(), + ) + .get((1) as usize) + .cloned() + { + std::option::Option::None => std::option::Option::None, + Some(hex) => match sha512_hex_digest(hex.clone()) { + std::option::Option::None => std::option::Option::None, + Some(d) => Some(as_content_hash_sha512(d.clone())), + }, + } + } else { + if content_hash_validate_lower_hex_length(wire.clone(), 40) { + match sha1_hex_digest(wire.clone()) { + std::option::Option::None => std::option::Option::None, + Some(d) => Some(as_content_hash_sha1(d.clone())), + } + } else { + content_hash_from_structural_digest(wire.clone()) + } + } + } +} + +pub fn parse_content_hash(wire: String) -> Option> { + match parse_content_hash_candidate(wire.clone()) { + std::option::Option::None => std::option::Option::None, + Some(parsed) => { + if (serialize_content_hash(parsed.clone()) == wire.clone()) { + Some(parsed.clone()) + } else { + std::option::Option::None + } + } + } +} + pub fn sha256_digest_wire_form(digest: Rc) -> String { Rc::new(vec!["sha256:".to_string(), digest.hex.clone()]).join(&"".to_string()) } diff --git a/src/v2/workflow/floor_route_gap.dag b/src/v2/workflow/floor_route_gap.dag index e0efde7c836..26a0c5d52c1 100644 --- a/src/v2/workflow/floor_route_gap.dag +++ b/src/v2/workflow/floor_route_gap.dag @@ -927,11 +927,101 @@ fn floor_route_gap_expectation_chunk_08() -> List { tail: Cons { head: FloorRouteGapExpectation { identity: "test.claim.devboot_text_blob_real_execution.a_lease_claim_is_stored_as_exactly_its_bytes_in_the_bare_store_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.a_transition_is_read_back_and_a_stale_one_names_the_current_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.an_unclaimed_group_is_refused_at_the_claim_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.the_production_route_settles_and_the_lease_governs_launch_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_rerun_resumes_its_own_pending_state_and_restores_the_exact_active_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_fence_and_a_restore_each_reach_their_terminal_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_seat_granted_after_the_suspension_landed_is_released_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.durable_cas_fabric_storage_real_execution.one_slot_is_one_head_and_two_bindings_see_one_generation_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.durable_cas_fabric_storage_real_execution.a_cycled_value_does_not_let_a_stale_writer_advance_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.the_same_grant_claimed_from_two_executors_is_held_once_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_crash_after_the_claim_and_before_the_first_write_is_recovered_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_crash_after_the_settling_write_and_before_completion_completes_the_claim_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_grant_over_another_host_population_is_refused_before_any_claim_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_front_door_real_execution.an_error_status_is_an_answer_and_no_listener_is_silence_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_front_door_real_execution.a_vacated_port_reads_as_no_response_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_front_door_real_execution.a_front_door_answering_500_fences_the_group_through_the_transaction_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_fresh_authorization_under_a_reused_word_is_not_completed_from_the_old_history_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_population_change_after_the_claim_aborts_before_a_write_and_fences_after_one_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_front_door_real_execution.a_listener_that_accepts_and_never_answers_fences_rather_than_reading_as_absent_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.a_second_executor_cannot_abort_a_pending_lifecycle_it_did_not_write_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_front_door_real_execution.a_failed_connect_fences_under_an_active_head_unit_and_suspends_only_under_a_quiet_one_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.any_executor_completes_a_settled_lifecycle_and_never_restarts_it_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_real_execution.the_production_admission_commits_the_frozen_hosts_of_a_fenced_lifecycle_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_d0_front_door_real_execution.a_listener_behind_a_failed_connect_is_read_on_the_host_and_fences_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.a_live_host_effect_claim_fences_the_reclaim_until_released_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.an_overdue_host_effect_claim_fences_until_released_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.an_ungrouped_host_is_claimed_on_the_placement_partition_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.a_claim_on_an_unestablished_groups_host_fences_its_establishment_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.a_release_needs_the_host_observed_quiet_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.a_stale_placement_read_cannot_place_or_commit_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.spark.pair_serving_authority_log_real_execution.a_saga_never_frees_a_host_early_and_a_consumed_preparation_cannot_be_aborted_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, + tail: Cons { head: FloorRouteGapExpectation { identity: "test.claim.devboot_text_blob_real_execution.a_staged_entry_is_stored_as_exactly_its_bytes_in_the_worktree_repository_by_real_execution", operation: "Dir", ground: NoMockResponse {} }, tail: Empty {} } } } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } } // The eight test.claim.materialization_store_local_wet_witness identities: the provider spine's diff --git a/src/v2/workflow/local_repo_wet_terminal.dag b/src/v2/workflow/local_repo_wet_terminal.dag index b6c96c76d70..8357ffbf07e 100644 --- a/src/v2/workflow/local_repo_wet_terminal.dag +++ b/src/v2/workflow/local_repo_wet_terminal.dag @@ -832,6 +832,198 @@ fn local_repo_wet_schedule() -> List { function: "and_with_true_left_removes_file", expectation: ExpectedToHold {} }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "a_transition_is_read_back_and_a_stale_one_names_the_current_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "a_transition_is_read_back_and_a_stale_one_names_the_current_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "an_unclaimed_group_is_refused_at_the_claim_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "an_unclaimed_group_is_refused_at_the_claim_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "the_production_route_settles_and_the_lease_governs_launch_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "the_production_route_settles_and_the_lease_governs_launch_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_rerun_resumes_its_own_pending_state_and_restores_the_exact_active_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_rerun_resumes_its_own_pending_state_and_restores_the_exact_active_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.fabric_capacity_standing_wet_witness", function: "the_live_entry_reads_the_authority_before_judging_and_refuses_off_fleet" }, + entry: "dag/test/claim/spark/fabric_capacity_standing_wet_witness_test.dag", + function: "the_live_entry_reads_the_authority_before_judging_and_refuses_off_fleet", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.spark_pair_serving_apply_wet_witness", function: "the_apply_seam_reads_the_authority_before_planning_and_refuses_off_fleet" }, + entry: "dag/test/claim/spark/spark_pair_serving_apply_wet_witness_test.dag", + function: "the_apply_seam_reads_the_authority_before_planning_and_refuses_off_fleet", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_fence_and_a_restore_each_reach_their_terminal_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_fence_and_a_restore_each_reach_their_terminal_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_seat_granted_after_the_suspension_landed_is_released_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_seat_granted_after_the_suspension_landed_is_released_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.durable_cas_fabric_storage_real_execution", function: "one_slot_is_one_head_and_two_bindings_see_one_generation_by_real_execution" }, + entry: "dag/test/claim/durable_cas_fabric_storage_real_execution_witness_test.dag", + function: "one_slot_is_one_head_and_two_bindings_see_one_generation_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.durable_cas_fabric_storage_real_execution", function: "a_cycled_value_does_not_let_a_stale_writer_advance_by_real_execution" }, + entry: "dag/test/claim/durable_cas_fabric_storage_real_execution_witness_test.dag", + function: "a_cycled_value_does_not_let_a_stale_writer_advance_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "the_same_grant_claimed_from_two_executors_is_held_once_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "the_same_grant_claimed_from_two_executors_is_held_once_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_crash_after_the_claim_and_before_the_first_write_is_recovered_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_crash_after_the_claim_and_before_the_first_write_is_recovered_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_crash_after_the_settling_write_and_before_completion_completes_the_claim_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_crash_after_the_settling_write_and_before_completion_completes_the_claim_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_grant_over_another_host_population_is_refused_before_any_claim_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_grant_over_another_host_population_is_refused_before_any_claim_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_front_door_real_execution", function: "an_error_status_is_an_answer_and_no_listener_is_silence_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag", + function: "an_error_status_is_an_answer_and_no_listener_is_silence_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_front_door_real_execution", function: "a_vacated_port_reads_as_no_response_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag", + function: "a_vacated_port_reads_as_no_response_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_front_door_real_execution", function: "a_front_door_answering_500_fences_the_group_through_the_transaction_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag", + function: "a_front_door_answering_500_fences_the_group_through_the_transaction_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_fresh_authorization_under_a_reused_word_is_not_completed_from_the_old_history_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_fresh_authorization_under_a_reused_word_is_not_completed_from_the_old_history_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_population_change_after_the_claim_aborts_before_a_write_and_fences_after_one_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_population_change_after_the_claim_aborts_before_a_write_and_fences_after_one_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_front_door_real_execution", function: "a_listener_that_accepts_and_never_answers_fences_rather_than_reading_as_absent_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag", + function: "a_listener_that_accepts_and_never_answers_fences_rather_than_reading_as_absent_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "a_second_executor_cannot_abort_a_pending_lifecycle_it_did_not_write_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "a_second_executor_cannot_abort_a_pending_lifecycle_it_did_not_write_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_front_door_real_execution", function: "a_failed_connect_fences_under_an_active_head_unit_and_suspends_only_under_a_quiet_one_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag", + function: "a_failed_connect_fences_under_an_active_head_unit_and_suspends_only_under_a_quiet_one_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "any_executor_completes_a_settled_lifecycle_and_never_restarts_it_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "any_executor_completes_a_settled_lifecycle_and_never_restarts_it_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_real_execution", function: "the_production_admission_commits_the_frozen_hosts_of_a_fenced_lifecycle_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_real_execution_witness_test.dag", + function: "the_production_admission_commits_the_frozen_hosts_of_a_fenced_lifecycle_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_d0_front_door_real_execution", function: "a_listener_behind_a_failed_connect_is_read_on_the_host_and_fences_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_d0_front_door_real_execution_witness_test.dag", + function: "a_listener_behind_a_failed_connect_is_read_on_the_host_and_fences_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "a_live_host_effect_claim_fences_the_reclaim_until_released_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "a_live_host_effect_claim_fences_the_reclaim_until_released_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "an_overdue_host_effect_claim_fences_until_released_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "an_overdue_host_effect_claim_fences_until_released_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "an_ungrouped_host_is_claimed_on_the_placement_partition_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "an_ungrouped_host_is_claimed_on_the_placement_partition_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "a_claim_on_an_unestablished_groups_host_fences_its_establishment_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "a_claim_on_an_unestablished_groups_host_fences_its_establishment_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "a_release_needs_the_host_observed_quiet_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "a_release_needs_the_host_observed_quiet_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "a_stale_placement_read_cannot_place_or_commit_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "a_stale_placement_read_cannot_place_or_commit_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.spark.pair_serving_authority_log_real_execution", function: "a_saga_never_frees_a_host_early_and_a_consumed_preparation_cannot_be_aborted_by_real_execution" }, + entry: "dag/test/claim/spark/pair_serving_authority_log_real_execution_witness_test.dag", + function: "a_saga_never_frees_a_host_early_and_a_consumed_preparation_cannot_be_aborted_by_real_execution", + expectation: ExpectedToHold {} + }, WetScheduledClaim { identity: WitnessIdentity { module_path: "test.claim.devboot_text_blob_real_execution", function: "a_lease_claim_is_stored_as_exactly_its_bytes_in_the_bare_store_by_real_execution" }, entry: "dag/test/claim/devboot_text_blob_real_execution_witness_test.dag",