diff --git a/DESIGN.md b/DESIGN.md index f758e22005f..fa8d0779a4d 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -99,7 +99,7 @@ The domain roster is a modeled fact, not this prose: `gunbc.design_argument` `co - **external facts** — each independently versioned upstream has its own module; this is the strongest instance and was, until this section, the only one with a reviewer. Home authorities: `extdeps.external_authority::ExternalModelScope`. - **materialization / realization** — materialization and realization are separate interfaces with admissible handlers: materialization presents and stores an admitted identity and result contract; realization orders, places and transports work. This row owns whether a handler, provider or placement faithfully realizes that admitted identity and result contract (completeness, readback, retention, transport, refusal, binding obligations) without substituting, truncating or bypassing the admitted key. Cache purity is one discriminator across that boundary, not the whole domain. Key RELATION -- what relation, what scope, what complete canonical preimage, equality and collision disposition, and whether declared inputs are complete -- is owned by conformance-identity. Home authorities: `std.realization::Materialization`, `std.materialization_provider::ArtifactRequest`, `std.artifact_store::ArtifactStore`, `std.cache_interface::StorageSurface`. -- **leasing / locking / grants / privileged access** — exclusive or bounded use and privileged access are one domain: leases with a declared deadline, compare-and-set expectations, scoped authorizations, grants naming who may perform an effect and how it is released, resource handles -- and, since 2026-09-19, which AUTHORIZATION PATTERN performs a privileged effect. That pattern is a §3d selection (gunbc.auth.authorization_pattern_selection consumes std.decision over the effect's frequency, reversibility, surface, workload-identity binding, minted reach and billing consequence): workload identity federation with a per-secret cell for recurring automated effects, one operator approval through the ntfy loop for one-off or witnessed effects, a named human-only step where no API exists, and a refusal for an operator token relayed by chat or between sessions. The census (gunbc.auth.privileged_effect_census) classifies every current site by executing that selection, so the reviewer's three-valued answer is derived for the sites that exist and asked of the site a change adds. Home authorities: `std.temporal_effect::HeldLease`, `std.durable_compare_and_set::CasExpectation`, `std.durable_exclusive_hold::DurableHoldState`, `std.scoped_authorization::ScopedAuthorization`, `std.effect_grant::Grant`, `std.resources::ResourceHandle`, `gunbc.auth.authorization_pattern_selection::select_authorization_pattern`, `gunbc.auth.github_gcp_federation::github_wif_provider`, `gunbc.auth.gcp_secret_access::SecretAccessGrant`, `gunbc.auth.access_request::AccessRequest`, `gunbc.auth.approval_capability::ApprovalCapability`, `gunbc.auth.approval_broker::redeem_capability`, `gunbc.auth.access_token_source::select_access_token_source`, `gunbc.auth.privileged_effect_census::privileged_effect_census`. +- **leasing / locking / grants / privileged access** — exclusive or bounded use and privileged access are one domain: leases with a declared deadline, compare-and-set expectations, scoped authorizations, grants naming who may perform an effect and how it is released, resource handles -- and, since 2026-09-19, which AUTHORIZATION PATTERN performs a privileged effect. That pattern is a §3d selection (gunbc.auth.authorization_pattern_selection consumes std.decision over the effect's frequency, reversibility, surface, workload-identity binding, minted reach and billing consequence): workload identity federation with a per-secret cell for recurring automated effects, one operator approval through the ntfy loop for one-off or witnessed effects, a named human-only step where no API exists, and a refusal for an operator token relayed by chat or between sessions. The census (gunbc.auth.privileged_effect_census) classifies every current site by executing that selection, so the reviewer's three-valued answer is derived for the sites that exist and asked of the site a change adds. A compare-and-set slot's generation line is kept whole by file_compare_and_set and observe_cas_slot_state (gunbc.durable_cas_file_store), because several slots read it as an audit trail; a slot whose only fact is its head -- the materialization store's occupancy index and family holds are the first -- uses the distinct windowed operations (file_compare_and_set_windowed, observe_cas_slot_state_windowed, and the file_hold_*_windowed family) with a CasGenerationWindow of k >= 2, so a bounded store's own bookkeeping does not grow with its commits. They are two contracts, not one operation with a mode: keep-all callers are untouched, and only a caller that names the windowed operation can delete generations. A window read takes the head from a listing of the slot root and verifies head+1 is absent, re-reading on a race and never guessing; a write reclaims below head-k only after its own commit, and a refused delete is reported, never fatal. Exclusion is unchanged -- losers re-read and retry -- so nothing waits on a hold. docs/plans/fabric-storage.md names the head-probe bound this also lifts and assigns its remedy to compaction -- a head that starts from a snapshot; a window is that head-starting-above-1 shape for a slot whose history needs no snapshot, and the fabric heads are not opted in. Home authorities: `std.temporal_effect::HeldLease`, `std.durable_compare_and_set::CasExpectation`, `gunbc.durable_cas_file_store::file_compare_and_set_windowed`, `std.durable_exclusive_hold::DurableHoldState`, `std.scoped_authorization::ScopedAuthorization`, `std.effect_grant::Grant`, `std.resources::ResourceHandle`, `gunbc.auth.authorization_pattern_selection::select_authorization_pattern`, `gunbc.auth.github_gcp_federation::github_wif_provider`, `gunbc.auth.gcp_secret_access::SecretAccessGrant`, `gunbc.auth.access_request::AccessRequest`, `gunbc.auth.approval_capability::ApprovalCapability`, `gunbc.auth.approval_broker::redeem_capability`, `gunbc.auth.access_token_source::select_access_token_source`, `gunbc.auth.privileged_effect_census::privileged_effect_census`. - **fabric / compute** — allocation, placement and negotiation of resources. The shared authority is product.capacity (lease, pool, seat events) and product.fabric (priced selection over a provider parameter); the subjects that consume it are compute cells (work contracts, compute layouts, cell reservations, capacity class admission) and inference serving (seat binding, co-tenancy admission). Serving is INSIDE this domain, not beside it -- but the sharing is partial and the boundary is the point: both subjects reach the same priced selection fold, and each carries its OWN occupancy producer and linearization (serving: a termed LeaseGrant settled by fabric_seat_acquire over an event chain; compute: a timeless LeaseIdentity settled by file_compare_and_set). They differ further in what the lease is over -- a compute cell is stateless between grants and settles money, a serving seat is over a replica with loaded state and is quality-conditioned instead. The argument is [the serving capacity home ruling](docs/plans/serving-capacity-home-ruling.md). Home authorities: `product.fabric.selection::select_supply`, `product.capacity.lease::LeaseGrant`, `product.capacity.pool_events::SeatRequest`, `product.fabric.work::ExecutionRequirements`, `gunbc.fabric_event_log::fabric_seat_acquire`, `gunbc.compute.work_request::WorkOperation`, `gunbc.compute.work_provider_local::ComputeLayout`, `gunbc.fabric_control_plane::CellReservation`, `gunbc.fabric_executor_class::CapacityClassAdmission`, `gunbc.harness.harness_seat::harness_bind_seat`, `gunbc.serving.turn_admission::serving_group_admission`. - **decision / selection** — a choice must remain attributable to its subject, candidate field, constraints, evidence and policy, and conformance exposes accidental forks of that authority; the hard selection laws -- no answer outrunning its evidence, no front standing as a winner -- stay with the §3d reviewer, and goal assessment (std.goal_assessment) is a boundary of this row, not a member of it. Home authorities: `std.decision::DecisionSubject`, `std.decision::RealizationSelectionResult`, `std.decision::SelectionReceipt`, `std.decision::select_realization`, `std.pareto::SelectionAxis`, `std.pareto::ParetoEntry`, `std.pareto::DominanceVerdict`. - **keys / hashing** — the row's scope today is the three keying relations that have a consumed home: structural node identity (v2.std.node content_hash over std.content_hash HashFamily), the request and artifact keys of std.materialization_provider (request_key, artifact_request_key), and occurrence identity within one source graph (std.occurrence_identity) -- plus the declared-equality admissibility of a key TYPE (std.algebra algebra_profile_equality_extensional), which governs whether distinct keys of a finite map are decidably distinct (consumed by the map-literal duplicate-key check, docs/plans/map-literal-introduction-design.md). That admissibility is instantiation-blind at this rung: an opaque brand or a type parameter admits unjudged, and its next-rung trigger is instantiation-grain admission (v1 infer equality_admission_wall_note). This row owns the key RELATION -- what relation, what scope, what complete canonical preimage derives it, what equality and collision disposition govern it -- and whether the declared inputs are complete. Faithful realization of an admitted identity and result contract (completeness, readback, retention, transport, refusal, binding; cache purity as one discriminator) is owned by conformance-realization. Two further relations -- demand identity (the question asked) and the dependency read set an answer consumed -- are a declared frontier, not members: they have no home authority yet, are modeled by docs/plans/demand-engine-program.md, and enter this row when M1 of that program lands their carriers; until then a reviewer asks nothing about them under this row (DESIGN §3b: a home that owns only part of a claimed scope is a scope mismatch, so the scope is stated at what the homes own). Each member is a named keying relation within a named scope, derived rather than authored; the relation model is docs/plans/keying-relation-design.md. Home authorities: `std.content_hash::HashFamily`, `std.materialization_provider::request_key`, `v2.std.node::content_hash`, `std.occurrence_identity::OccurrenceIdAllocator`, `extdeps.realization.cache_purity::CachePurityVerdict`, `std.computation_identity::ComputationIdentity`, `std.algebra::algebra_profile_equality_extensional`. diff --git a/dag/extdeps/cache.dag b/dag/extdeps/cache.dag index 01f45916b80..39e9a9e6a16 100644 --- a/dag/extdeps/cache.dag +++ b/dag/extdeps/cache.dag @@ -33,7 +33,7 @@ import extdeps.realization.reconcile_in_process { parse_cache_id, typed_module_cache_id, } -import extdeps.realization.materialization_store_local { materialization_store_local_facts } +import gunbc.materialization_store_budgets { materialization_store_local_facts } import std.cache_identity { CacheInterfaceId } import std.cache_interface { ArtifactIdentity, diff --git a/dag/extdeps/filesystem/filesystem_io.dag b/dag/extdeps/filesystem/filesystem_io.dag index 34bc35daeda..912ec9babdb 100644 --- a/dag/extdeps/filesystem/filesystem_io.dag +++ b/dag/extdeps/filesystem/filesystem_io.dag @@ -139,6 +139,41 @@ type FilesystemCreateNew | FilesystemCreateRefused { path: String, kind: FilesystemFailureKind, error: String } | FilesystemCreateKindUnrecognized { path: String, observed: String, error: String } +// A DELETE IS OBSERVED THE SAME WAY A CREATE IS: the host's error KIND, admitted through the one +// classifier, never its message. An absent target is its own arm because a caller confirming that a +// name is gone (a store retiring an evicted object) must tell "already gone" from "the host refused"; +// the second leaves bytes on disk and the first does not. +type FilesystemDelete + = FilesystemDeleted { path: String } + | FilesystemDeleteTargetAbsent { path: String } + | FilesystemDeleteRefused { path: String, kind: FilesystemFailureKind, error: String } + | FilesystemDeleteKindUnrecognized { path: String, observed: String, error: String } + +fn filesystem_delete( + path: String, + success: Bool, + error: String, + error_kind: String, +) -> FilesystemDelete { + if success { + FilesystemDeleted { path: path } + } else { + match admit_filesystem_failure_kind(observed: error_kind) { + FilesystemFailureKindUnrecognized { observed: o } => + FilesystemDeleteKindUnrecognized { path: path, observed: o, error: error } + FilesystemFailureKindAdmitted { kind: k } => + match k { + FilesystemNotFound => FilesystemDeleteTargetAbsent { path: path } + FilesystemAlreadyExists => FilesystemDeleteRefused { path: path, kind: k, error: error } + FilesystemPermissionDenied => FilesystemDeleteRefused { path: path, kind: k, error: error } + FilesystemNotDirectory => FilesystemDeleteRefused { path: path, kind: k, error: error } + FilesystemOtherFailure => FilesystemDeleteRefused { path: path, kind: k, error: error } + FilesystemCrossDevice => FilesystemDeleteRefused { path: path, kind: k, error: error } + } + } + } +} + fn filesystem_create_new( path: String, success: Bool, @@ -818,6 +853,7 @@ service Filesystem { output { success: Bool from "delete_success" error: String from "error" + error_kind: String from "error_kind" } transport file { path: "{path}", verb: "delete" } } diff --git a/dag/extdeps/realization/materialization_store_local.dag b/dag/extdeps/realization/materialization_store_local.dag index fcd958f4c96..3b4cd807aec 100644 --- a/dag/extdeps/realization/materialization_store_local.dag +++ b/dag/extdeps/realization/materialization_store_local.dag @@ -1,5 +1,6 @@ module extdeps.realization.materialization_store_local +import std.decl_ref { decl_ref } import extdeps.filesystem.filesystem_io { Filesystem, FilesystemFailureKind, FilesystemNotFound, FilesystemAlreadyExists, FilesystemPermissionDenied, @@ -9,16 +10,29 @@ import extdeps.filesystem.filesystem_io { FilesystemCreateNew, FilesystemCreated, FilesystemCreateTargetOccupied, FilesystemCreateRefused, FilesystemCreateKindUnrecognized, filesystem_exact_read, filesystem_create_new, filesystem_failure_kind_name, + filesystem_listing_observation, FilesystemDirectoryListed, FilesystemDirectoryListingRefused, + FilesystemDirectorySubjectRefused, filesystem_listing_entry_names, + FilesystemDelete, FilesystemDeleted, FilesystemDeleteTargetAbsent, FilesystemDeleteRefused, + FilesystemDeleteKindUnrecognized, filesystem_delete, FilesystemListingObservation, } +import std.decimal { decimal_digits_only } +import std.checked_arithmetic { int_inclusive_max } +import std.coercion { unicode_scalar_unfold, unicode_scalar_fold } +import v2.std.algebra { skip } +import v2.std.collection { empty_map, map_lookup, map_insert } import extdeps.external_authority { ExternalAuthority } import extdeps.uri { Uri, Https } import extdeps.cache.types { CacheInterfaceCatalogFacts, ContentAddressed, CatalogCasContentEncoding } import std.types { String, Bool, Int, List, NonEmptyStr } -import std.content_hash { ContentHash, content_hash_of_value } +import std.content_hash { ContentHash, content_hash_of_value, serialize_content_hash, parse_content_hash } import std.cache_identity { CacheInterfaceId } import std.cache_interface { ReleasedNever, - CapacityUnobserved, + CapacityBounded, + ByteCapacity, + ExactLimit, + ReplaceExisting, + LeastRecentlyUsed, ContentAddressLookup, WriteOnce, MissThenCreate, @@ -37,9 +51,51 @@ import v2.std.optional { Absent } import std.access { Permit, Deny } import std.effect_grant { Verb, Read, Write, NamespacePosition } import std.filesystem_path_grant { absolute_path_position } -import std.materialization_ladder { Frame } +import std.materialization_ladder { Frame, CacheProvider, provider_row, persistent_retention, CoversIdentities, ArtifactTier, ContentKeyed } +import std.artifact_store { + ArtifactStore, ArtifactRow, StoreConstruction, StoreReady, StoreRefusedRetention, + store_over_provider, store_put, StorePutOutcome, StorePutAdmitted, StorePutDidNotFit, + store_get, StoreHit, StoreMiss, store_bytes_used, +} +import std.cache_identity { ArtifactKindId } +import std.measure { ByteSize, byte_size, measure_count } +import std.bytes { utf8_encode_bytes, bytes_octets } +import std.durable_compare_and_set { + cas_attempt, CasOutcome, CasCommitted, CasPreconditionFailed, CasStoreRefused, + ExpectSlotAbsent, ExpectSlotGeneration, CasExpectation, + CasReadableAbsent, CasReadablePresent, CasObservedReadable, CasObservedUnreadable, + cas_unreadable_slot_detail, CasUnreadableSlot, CasStoreFailure, + CasSlotObservationRefused, CasGenerationPublicationRefused, CasGenerationSpaceExhausted, +} +import gunbc.durable_cas_file_store { + admit_cas_attempt, CasAttemptAdmitted, CasAttemptDigestMismatch, CasAttemptDigestIncomparable, CasAttemptKeyNotSlotAddressable, + file_compare_and_set_windowed, observe_cas_slot_state_windowed, DefaultAccessCreateOnly, + cas_generation_window, CasGenerationWindowAdmitted, CasGenerationWindowTooNarrow, + CasWindowReclamation, CasReclamationNotCommitted, CasAttemptAdmission, + cas_window_generations, CasWindowGenerations, CasWindowGenerationsListed, CasWindowGenerationsUnlisted, + cas_reclaim_below, CasGenerationWindow, +} +import std.durable_exclusive_hold { + DurableHoldOwnerRef, DurableHoldReleaseRef, DurableHoldHolderReport, DurableHoldReleaseRefusal, DurableHoldRecoveryRefusal, + HolderObservedLive, HolderObservedDead, HolderLivenessUnobservable, + DurableHoldObservation, HoldSlotAbsent, HoldSlotFree, HoldSlotHeld, HoldSlotUndecodable, HoldObservationUnavailable, + durable_hold_owner_equal, +} +import gunbc.durable_exclusive_hold_file_store { + file_hold_acquire_windowed, FileHoldAcquireOutcome, FileHoldAcquired, FileHoldOccupied, FileHoldAcquireLost, + FileHoldAcquireStoreUnavailable, FileHoldSlotUndecodable, FileHoldKeyNotSlotAddressable, + file_hold_release_assess_windowed, FileHoldReleasePlanned, FileHoldReleaseNotEligible, + file_hold_release_commit, FileHoldReleased, FileHoldReleaseLost, FileHoldReleaseStoreUnavailable, FileHoldReleaseKeyNotSlotAddressable, + file_hold_recovery_assess_windowed, FileHoldRecoveryPlanned, FileHoldRecoveryNotEligible, + observe_file_hold_windowed, +} +import gunbc.process_hold_identity { + process_hold_text, process_hold_marker, process_hold_decode, ProcessHoldHolder, + self_process_identity, SelfIdentified, SelfUnidentified, holder_process_liveness, +} +import std.durable_compare_and_set { CasGeneration, cas_generation_count } import std.materialization_provider { - ArtifactRequest, request_key, evaluation_store_address, + ResultContentIdentity, ArtifactRequest, request_key, request_kind, evaluation_store_address, Admitted, FamilyRefused } import std.materialization_store_grant { @@ -60,8 +116,10 @@ import std.materialization_object { StoreLookup, StoreLookupRefused, StoreLookupObjectUnavailable, StoreLookupStoreFamilyRefused, StoreCommit, StoreCommitRefused, StoreCommitPublishRefused, + StoreCommitRefusal, StoreCommitBudgetUndeclared, StoreCommitDidNotFit, StoreCommitOccupancyUnavailable, + StoreCommitCleanupOutstanding, StoreCommitPreparation, StoreCommitPrepared, StoreCommitNotPrepared, - store_object_name, store_lookup_from_address, store_commit_prepare, store_commit_settle, + store_object_name, store_object_format_version, store_lookup_from_address, store_commit_prepare, store_commit_settle, } data extdeps_external_authority_anchor: ExternalAuthority = ExternalAuthority { @@ -98,30 +156,161 @@ data materialization_store_local_id: CacheInterfaceId = "materialization_store_l data materialization_store_marker_name: String = "materialization-store.identity" -data materialization_store_marker_magic: String = "gunbc-materialization-store v1" - fn store_path(root_path: String, name: String) -> String { if ends_with(s: root_path, suffix: "/") { concat(root_path, name) } else { concat(root_path, "/", name) } } -fn store_marker_content(root: MaterializationStoreRoot, instance_label: NonEmptyStr) -> String { +// THE STORE'S FAMILY CATALOG -- every family AND its byte budget -- IS A FACT OF THE STORE, written once +// into its identity marker at initialization, and the budgets a writer enforces are DECODED from that +// marker, never taken from the caller. A reopen whose offered catalog differs from the decoded one in +// any family or any budget refuses before anything is swept, so a caller's partial roster cannot have +// the families it forgot treated as orphans, and a larger offered budget cannot be enforced. +// +// ONE CANONICAL ENCODING, AND ITS DECODER ACCEPTS EXACTLY THE ENCODER'S IMAGE. A family is written +// length-prefixed in Unicode scalars, ":=;", so no character of a family name is ever +// read as a delimiter; rows are sorted by family and a family appears once. The decoder is a total +// scan over the whole marker: exactly one catalog field, no leading zeros, no duplicate family, no +// unsorted or extra row, and exactly one instance line after it -- anything else is malformed and +// refuses, so one marker never authorizes two catalogs. +data materialization_store_marker_magic: String = "gunbc-materialization-store v3" + +fn store_marker_family_row(b: LocalStoreFamilyBudget) -> String { + let family = b.family as String + join([count(unicode_scalar_unfold(s: family)) as String, ":", family, "=", measure_count(m: b.budget) as String, ";"], "") +} + +fn store_marker_catalog_field(budgets: List) -> String { + join(budgets |> sort_by(b => b.family as String) |> map(b => store_marker_family_row(b: b)), "") +} + +fn store_marker_head(root: MaterializationStoreRoot) -> String { join([ materialization_store_marker_magic, concat("durability ", materialization_store_durability_name(d: materialization_store_durability(root: root))), - concat("instance ", instance_label as String), - "", + "families ", ], "\n") } -fn store_marker_recognized(root: MaterializationStoreRoot, content: String) -> Bool { - starts_with( - s: content, - prefix: join([ - materialization_store_marker_magic, - concat("durability ", materialization_store_durability_name(d: materialization_store_durability(root: root))), - "instance ", - ], "\n") - ) +fn store_marker_content(root: MaterializationStoreRoot, instance_label: NonEmptyStr, budgets: List) -> String { + join([store_marker_head(root: root), store_marker_catalog_field(budgets: budgets), "\ninstance ", instance_label as String, "\n"], "") +} + +// The offered catalog is encodable only when it names each family once and an instance label is one +// line; initialization refuses anything else before a marker is written. +fn store_marker_catalog_admissible(budgets: List) -> Bool { + count(budgets) > 0 && count(local_store_name_set(names: budgets |> map(b => b.family as String))) == count(budgets) +} + +fn store_marker_label_admissible(instance_label: NonEmptyStr) -> Bool { + !string_contains(s: instance_label as String, pattern: "\n") +} + +type StoreMarkerSpan { start: Int, length: Int, bytes: ByteSize } + +// The scan over the catalog field, one arm per phase, each carrying only what that phase needs. A +// refusal is its own arm, so no state can be both refused and holding rows, and no phase can be +// written that the grammar does not have. Positions are scalar offsets into the field. +type StoreMarkerScan + = StoreMarkerReadingCount { digits: StoreMarkerDigits, spans: List } + | StoreMarkerReadingName { start: Int, remaining: Int, spans: List } + | StoreMarkerExpectingEquals { start: Int, length: Int, spans: List } + | StoreMarkerReadingBytes { start: Int, length: Int, digits: StoreMarkerDigits, spans: List } + | StoreMarkerPastField { tail: Int, spans: List } + | StoreMarkerScanRefused + +// A decimal numeral being read: none read yet, or a canonical value (no leading zero, no overflow). +type StoreMarkerDigits + = StoreMarkerNoDigits + | StoreMarkerDigitsRead { value: Int } + +type StoreMarkerScanAt { position: Int, scan: StoreMarkerScan } + +fn store_marker_digit(d: StoreMarkerDigits, c: Int) -> StoreMarkerDigits? { + match d { + StoreMarkerNoDigits => Present { value: StoreMarkerDigitsRead { value: c - 48 } } + StoreMarkerDigitsRead { value: n } => + if n == 0 || n > (int_inclusive_max() - (c - 48)) / 10 { none } else { Present { value: StoreMarkerDigitsRead { value: n * 10 + (c - 48) } } } + } +} + +fn store_marker_scan_step(scan: StoreMarkerScan, position: Int, c: Int) -> StoreMarkerScan { + let is_digit = c >= 48 && c <= 57 + match scan { + StoreMarkerScanRefused => StoreMarkerScanRefused + StoreMarkerPastField { tail: t, spans: sp } => StoreMarkerPastField { tail: t, spans: sp } + StoreMarkerReadingCount { digits: d, spans: sp } => + if is_digit { + match store_marker_digit(d: d, c: c) { + Absent => StoreMarkerScanRefused + Present { value: d2 } => StoreMarkerReadingCount { digits: d2, spans: sp } + } + } else if c == 58 { + match d { + StoreMarkerDigitsRead { value: n } => if n > 0 { StoreMarkerReadingName { start: position + 1, remaining: n, spans: sp } } else { StoreMarkerScanRefused } + StoreMarkerNoDigits => StoreMarkerScanRefused + } + } else if c == 10 { + match d { + StoreMarkerNoDigits => if count(sp) > 0 { StoreMarkerPastField { tail: position + 1, spans: sp } } else { StoreMarkerScanRefused } + StoreMarkerDigitsRead { value: _ } => StoreMarkerScanRefused + } + } else { StoreMarkerScanRefused } + StoreMarkerReadingName { start: st, remaining: r, spans: sp } => + if r == 1 { StoreMarkerExpectingEquals { start: st, length: position + 1 - st, spans: sp } } else { StoreMarkerReadingName { start: st, remaining: r - 1, spans: sp } } + StoreMarkerExpectingEquals { start: st, length: l, spans: sp } => + if c == 61 { StoreMarkerReadingBytes { start: st, length: l, digits: StoreMarkerNoDigits, spans: sp } } else { StoreMarkerScanRefused } + StoreMarkerReadingBytes { start: st, length: l, digits: d, spans: sp } => + if is_digit { + match store_marker_digit(d: d, c: c) { + Absent => StoreMarkerScanRefused + Present { value: d2 } => StoreMarkerReadingBytes { start: st, length: l, digits: d2, spans: sp } + } + } else if c == 59 { + match d { + StoreMarkerDigitsRead { value: n } => + StoreMarkerReadingCount { digits: StoreMarkerNoDigits, spans: sp |> list_push(StoreMarkerSpan { start: st, length: l, bytes: byte_size(count: n) }) } + StoreMarkerNoDigits => StoreMarkerScanRefused + } + } else { StoreMarkerScanRefused } + } +} + +type StoreMarkerDecode + = StoreMarkerDecoded { catalog: List } + | StoreMarkerForeign + | StoreMarkerMalformed { detail: String } + +fn store_marker_decode(root: MaterializationStoreRoot, content: String) -> StoreMarkerDecode { + let head = store_marker_head(root: root) + if !starts_with(s: content, prefix: head) { StoreMarkerForeign } else { + let scalars = unicode_scalar_unfold(s: substring(s: content, start: string_length(s: head), end: string_length(s: content))) + let scan = fold(scalars, init: StoreMarkerScanAt { position: 0, scan: StoreMarkerReadingCount { digits: StoreMarkerNoDigits, spans: [] } }, f: (acc, c) => + StoreMarkerScanAt { position: acc.position + 1, scan: store_marker_scan_step(scan: acc.scan, position: acc.position, c: c) }) + match scan.scan { + StoreMarkerPastField { tail: tail_at, spans: spans } => { + let field = unicode_scalar_fold(xs: take(scalars, tail_at - 1)) + let tail = unicode_scalar_fold(xs: skip(scalars, tail_at)) + let catalog = spans |> map(sp => LocalStoreFamilyBudget { + family: unicode_scalar_fold(xs: take(skip(scalars, sp.start), sp.length)) as ArtifactKindId, + budget: sp.bytes + }) + if !starts_with(s: tail, prefix: "instance ") || string_length(s: tail) <= 10 || !ends_with(s: tail, suffix: "\n") || count(tail.split(delimiter: "\n")) != 2 { + StoreMarkerMalformed { detail: "the catalog field is not followed by exactly one instance line" } + } else if !store_marker_catalog_admissible(budgets: catalog) { + StoreMarkerMalformed { detail: "the family catalog names a family more than once" } + } else if store_marker_catalog_field(budgets: catalog) != field { + StoreMarkerMalformed { detail: "the family catalog is not in its canonical sorted encoding" } + } else { + StoreMarkerDecoded { catalog: catalog } + } + } + StoreMarkerReadingCount { digits: _, spans: _ } => StoreMarkerMalformed { detail: "the family catalog field is not a sequence of length-prefixed rows ending in one line feed" } + StoreMarkerReadingName { start: _, remaining: _, spans: _ } => StoreMarkerMalformed { detail: "the family catalog field is not a sequence of length-prefixed rows ending in one line feed" } + StoreMarkerExpectingEquals { start: _, length: _, spans: _ } => StoreMarkerMalformed { detail: "the family catalog field is not a sequence of length-prefixed rows ending in one line feed" } + StoreMarkerReadingBytes { start: _, length: _, digits: _, spans: _ } => StoreMarkerMalformed { detail: "the family catalog field is not a sequence of length-prefixed rows ending in one line feed" } + StoreMarkerScanRefused => StoreMarkerMalformed { detail: "the family catalog field is not a sequence of length-prefixed rows ending in one line feed" } + } + } } // --------------------------------------------------------------------------------------------- @@ -171,50 +360,146 @@ type LocalStoreUnavailableCause = LocalStoreGrantRefused { verb: Verb, target: NamespacePosition, frame: Frame } | LocalStoreNotInitialized { marker_path: String } | LocalStoreMarkerUnrecognized { marker_path: String } + | LocalStoreMarkerMalformed { marker_path: String, detail: String } + | LocalStoreCatalogMismatch { marker_path: String, offered: String } + | LocalStoreCatalogInadmissible { detail: String } | LocalStoreMarkerUnavailable { fault: StoreFault } + | LocalStoreSweepIncomplete { sweep: LocalStoreSweep } + +// THE VERIFIED BINDING: the root, the store instance its marker names, and the family catalog DECODED +// from that marker. It is minted only by local_store_open_decide after the marker decodes and matches +// the offered catalog, and it is what every hold carries, so a hold derives root, store and budgets +// from the marker it was verified against and from nothing a caller passed. +type LocalStoreBinding sole_constructor { + root: MaterializationStoreRoot + store: OpenedMaterializationStore + budgets: List +} + +// A VERIFIED BUT NOT YET WRITABLE STORE. The marker decision yields only this; no public operation takes +// it, so marker verification alone never confers the right to write. +type LocalStorePreOpen sole_constructor { + binding: LocalStoreBinding +} + +// THE PROOF THAT THE OPEN SWEEP ESTABLISHED THE BOUND: every family was held, the listing and every +// index were read, and every canonical name no index charges was confirmed gone. It is minted only +// from a sweep with no refused delete (local_store_open_swept), so no capability exists while an +// uncharged orphan the sweep saw is still on disk. +type LocalStoreCleanSweep sole_constructor { + removed: List + releases: List +} + +// THE OPENED STORE IS ONE CAPABILITY: a verified binding plus its clean-sweep proof. Every public +// operation consumes it and takes no separate root, store or budgets, so a store cannot be hand-built +// for an uninitialized root, a reopen cannot enforce a budget the store was not created with, one +// store's identity cannot be paired with another root, and a store whose open could not clear its +// orphans is never writable. +type LocalStoreCapability sole_constructor { + binding: LocalStoreBinding + sweep: LocalStoreCleanSweep +} type LocalStoreOpening - = LocalStoreOpened { root: MaterializationStoreRoot, store: OpenedMaterializationStore, durability: MaterializationStoreDurability } + = LocalStoreOpened { capability: LocalStoreCapability, durability: MaterializationStoreDurability } | LocalStoreUnavailable { root_path: String, cause: LocalStoreUnavailableCause } -// PURE: the open decision over the marker read. Hermetic controls call this directly. -fn local_store_open_decide(root: MaterializationStoreRoot, marker_read: FilesystemExactRead) -> LocalStoreOpening { +type LocalStorePreOpening + = LocalStorePreOpened { pre: LocalStorePreOpen, durability: MaterializationStoreDurability } + | LocalStorePreOpenRefused { root_path: String, cause: LocalStoreUnavailableCause } + +// PURE: the marker decision. It decodes the whole marker and binds the DECODED catalog, after checking +// that the offered catalog encodes to exactly the same canonical field (an identity comparison: the +// encoding is injective over catalogs that name each family once, and the offered one is refused +// otherwise). Hermetic controls call this directly; what they get is a pre-open, which writes nothing. +fn local_store_open_decide(root: MaterializationStoreRoot, marker_read: FilesystemExactRead, budgets: List) -> LocalStorePreOpening + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_open"), + decl_ref(module_path: "test.claim.materialization_store_witness", decl_name: "a_store_opens_only_on_its_own_marker"), + ] +{ let root_path = materialization_store_root_path(root: root) let marker_path = store_path(root_path: root_path, name: materialization_store_marker_name) match marker_read { FilesystemExactPathAbsent { path: _ } => - LocalStoreUnavailable { root_path: root_path, cause: LocalStoreNotInitialized { marker_path: marker_path } } + LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreNotInitialized { marker_path: marker_path } } FilesystemExactPathUnreadable { path: _, kind: k, error: e } => - LocalStoreUnavailable { root_path: root_path, cause: LocalStoreMarkerUnavailable { fault: filesystem_fault(kind: k, error: e) } } + LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreMarkerUnavailable { fault: filesystem_fault(kind: k, error: e) } } FilesystemExactPathKindUnrecognized { path: _, observed: o, error: e } => - LocalStoreUnavailable { root_path: root_path, cause: LocalStoreMarkerUnavailable { fault: StoreFault { class: StoreFaultTransportVocabulary, detail: concat(o, ": ", e) } } } + LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreMarkerUnavailable { fault: StoreFault { class: StoreFaultTransportVocabulary, detail: concat(o, ": ", e) } } } FilesystemExactPathRead { path: _, content: c } => - if store_marker_recognized(root: root, content: c) { - LocalStoreOpened { - root: root, - store: OpenedMaterializationStore { - provider: materialization_store_local_id, - instance: content_hash_of_value(value: c as NonEmptyStr) - }, - durability: materialization_store_durability(root: root) + if !store_marker_catalog_admissible(budgets: budgets) { + LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreCatalogInadmissible { detail: "the offered family catalog is empty or names a family more than once" } } + } else { + match store_marker_decode(root: root, content: c) { + StoreMarkerForeign => LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreMarkerUnrecognized { marker_path: marker_path } } + StoreMarkerMalformed { detail: d } => LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreMarkerMalformed { marker_path: marker_path, detail: d } } + StoreMarkerDecoded { catalog: decoded } => + if store_marker_catalog_field(budgets: decoded) != store_marker_catalog_field(budgets: budgets) { + LocalStorePreOpenRefused { root_path: root_path, cause: LocalStoreCatalogMismatch { marker_path: marker_path, offered: store_marker_catalog_field(budgets: budgets) } } + } else { + LocalStorePreOpened { + pre: LocalStorePreOpen { + binding: LocalStoreBinding { + root: root, + store: OpenedMaterializationStore { + provider: materialization_store_local_id, + instance: content_hash_of_value(value: c as NonEmptyStr) + }, + budgets: decoded + } + }, + durability: materialization_store_durability(root: root) + } + } } + } + } +} + +// The capability is minted HERE and nowhere else: only from a sweep that held every family, read every +// index and confirmed every orphan it saw gone. A sweep that could not run, or that left any delete +// unconfirmed, makes the store unavailable -- the orphan stays uncharged on disk, so writing beside it +// could exceed the budget. +fn local_store_open_swept(pre: LocalStorePreOpen, durability: MaterializationStoreDurability, sweep: LocalStoreSweep) -> LocalStoreOpening + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_open"), + ] +{ + let root_path = materialization_store_root_path(root: pre.binding.root) + match sweep { + LocalStoreSwept { removed: r, delete_refused: d, releases: rs } => + if count(d) == 0 { + LocalStoreOpened { capability: LocalStoreCapability { binding: pre.binding, sweep: LocalStoreCleanSweep { removed: r, releases: rs } }, durability: durability } } else { - LocalStoreUnavailable { root_path: root_path, cause: LocalStoreMarkerUnrecognized { marker_path: marker_path } } + LocalStoreUnavailable { root_path: root_path, cause: LocalStoreSweepIncomplete { sweep: sweep } } } + LocalStoreSweepUnavailable { refusal: _ } => LocalStoreUnavailable { root_path: root_path, cause: LocalStoreSweepIncomplete { sweep: sweep } } } } -fn local_store_read(path: String) -> FilesystemExactRead { +fn local_store_read(path: String) -> FilesystemExactRead + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_open"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_lookup_bound"), + ] +{ let r = Filesystem.Read(path: path) filesystem_exact_read(path: path, content: r.content, success: r.success, error: r.error, error_kind: r.error_kind) } -fn local_store_create_new(path: String, content: String) -> FilesystemCreateNew { +fn local_store_create_new(path: String, content: String) -> FilesystemCreateNew + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_initialize"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_reserved"), + ] +{ let w = Filesystem.WriteCreateNew(path: path, content: content) filesystem_create_new(path: path, success: w.success, error: w.error, error_kind: w.error_kind) } -fn local_store_open(root: MaterializationStoreRoot) -> LocalStoreOpening { +fn local_store_open(root: MaterializationStoreRoot, budgets: List) -> LocalStoreOpening { let root_path = materialization_store_root_path(root: root) match materialization_store_admissible(root: root, verb: Read) { Deny { refusal: _ } => @@ -223,10 +508,14 @@ fn local_store_open(root: MaterializationStoreRoot) -> LocalStoreOpening { cause: LocalStoreGrantRefused { verb: Read, target: absolute_path_position(path: root_path), frame: materialization_store_frame } } Permit => - local_store_open_decide( + match local_store_open_decide( root: root, - marker_read: local_store_read(path: store_path(root_path: root_path, name: materialization_store_marker_name)) - ) + marker_read: local_store_read(path: store_path(root_path: root_path, name: materialization_store_marker_name)), + budgets: budgets + ) { + LocalStorePreOpenRefused { root_path: p, cause: c } => LocalStoreUnavailable { root_path: p, cause: c } + LocalStorePreOpened { pre: pre, durability: d } => local_store_open_swept(pre: pre, durability: d, sweep: local_store_sweep(binding: pre.binding)) + } } } @@ -237,7 +526,7 @@ type LocalStoreInitialization // Initialization publishes the marker create-new and then OPENS the store it created, so the // returned opening is the same observation every later open makes -- a marker written by a // concurrent initializer with a different label opens as THAT store, never as ours. -fn local_store_initialize(root: MaterializationStoreRoot, instance_label: NonEmptyStr) -> LocalStoreInitialization { +fn local_store_initialize(root: MaterializationStoreRoot, instance_label: NonEmptyStr, budgets: List) -> LocalStoreInitialization { let root_path = materialization_store_root_path(root: root) match materialization_store_admissible(root: root, verb: Write) { Deny { refusal: _ } => @@ -246,22 +535,1104 @@ fn local_store_initialize(root: MaterializationStoreRoot, instance_label: NonEmp cause: LocalStoreGrantRefused { verb: Write, target: absolute_path_position(path: root_path), frame: materialization_store_frame } } Permit => + if !store_marker_catalog_admissible(budgets: budgets) || !store_marker_label_admissible(instance_label: instance_label) { + LocalStoreInitializationRefused { root_path: root_path, cause: LocalStoreCatalogInadmissible { detail: "the family catalog must name each family once and the instance label must be one line" } } + } else { match store_publish_observation(create: local_store_create_new( path: store_path(root_path: root_path, name: materialization_store_marker_name), - content: store_marker_content(root: root, instance_label: instance_label) + content: store_marker_content(root: root, instance_label: instance_label, budgets: budgets) )) { StoreObjectPublishRefused { fault: f } => LocalStoreInitializationRefused { root_path: root_path, cause: LocalStoreMarkerUnavailable { fault: f } } - StoreObjectPublished => LocalStoreInitialized { opening: local_store_open(root: root) } - StoreObjectOccupied => LocalStoreInitialized { opening: local_store_open(root: root) } + StoreObjectPublished => LocalStoreInitialized { opening: local_store_open(root: root, budgets: budgets) } + StoreObjectOccupied => LocalStoreInitialized { opening: local_store_open(root: root, budgets: budgets) } + } + } + } +} + +// --------------------------------------------------------------------------------------------- +// BOUNDED RETENTION: each request FAMILY has its own declared byte budget inside the one root, and the +// host ceiling is their declared sum (operator ruling via royal-moth-86, 2026-10-03, after an earlier +// on-disk cache with no ceiling or cleanup filled the CI runners' disks). The budget is not a number +// this module enforces by itself: it is a std.materialization_ladder CacheProvider row per family, and +// the store is CONSTRUCTED from that row by std.artifact_store store_over_provider, which refuses an +// unbounded or unobserved row and realizes ReplaceExisting { LeastRecentlyUsed } with a fit test +// before any eviction. So an unbounded family cannot be written here, an artifact larger than its +// family's budget refuses (StoreCommitDidNotFit) without evicting anything, and no family evicts +// another's entries -- each family's accounting is its own index. +// +// THE BUDGET VALUES ARE NOT THIS MODULE'S FACT (DESIGN section 3: interface, realization and policy +// are three facts). This transport owns the SHAPE of a family budget and how one is enforced; how many +// bytes a host grants a family is policy, owned by the store's consumer and passed in. The durable +// policy rows and the host ceiling are gunbc.materialization_store_budgets; a witness passes fixture +// rows of its own. Every operation that admits, evicts or sweeps takes the rows as a parameter, so +// revising a policy never edits this module. + +type LocalStoreFamilyBudget { + family: ArtifactKindId + budget: ByteSize +} + +fn local_store_budget_sum(budgets: List) -> ByteSize { + byte_size(count: fold(budgets, init: 0, f: (acc, b) => acc + measure_count(m: b.budget))) +} + +fn local_store_family_provider(family: ArtifactKindId, budget: ByteSize) -> CacheProvider { + provider_row( + id: concat("materialization_store_local:", family as String), + scope: [materialization_store_frame], + coverage: CoversIdentities { identities: [family as String] }, + tier: ArtifactTier { keying: ContentKeyed }, + retention: persistent_retention(capacity: CapacityBounded { + limit: ByteCapacity { limit: ExactLimit { value: budget } }, + at_capacity: ReplaceExisting { strategy: LeastRecentlyUsed } + }) + ) +} + +fn local_store_family_budget(budgets: List, family: ArtifactKindId) -> LocalStoreFamilyBudget? { + budgets |> filter(b => b.family == family) |> first +} + +// The object's size is its UTF-8 byte count, the same quantity std.materialization_object payload_part +// puts in a ByteSize -- never string_length, which counts code points. +fn local_store_object_bytes(object: String) -> ByteSize { + byte_size(count: bytes_octets(b: utf8_encode_bytes(s: object)) |> count()) +} + +// --------------------------------------------------------------------------------------------- +// THE OCCUPANCY INDEX: one CAS slot per family in the store root, retained as a generation WINDOW +// (gunbc.durable_cas_file_store KeepGenerationWindow, k = 2) because its history is not evidence -- +// its only fact is the head. Its canonical, versioned encoding is one header, the next ordinal, one +// row per live object sorted by name, then one `doomed` row per evicted object whose delete the host +// has not yet confirmed. A slot that does not decode REFUSES as corrupt, never reads as empty +// (docs/plans/demand-engine-program.md section 2.8, EvaluationIndexCorrupt). The recovery is the +// operator's: delete the family's occupancy slot files; the next open's sweep then removes every +// object the empty index no longer names, so a quarantine never leaves unaccounted bytes behind. +// +// DOOMED BYTES ARE CHARGED. An evicted row leaves the live set in the same CAS that admits the new +// row, and enters `doomed` with its size; it leaves `doomed` only in a later CAS made after the host +// CONFIRMED the object is gone (deleted, or absent). Live plus doomed bytes never exceed the family +// budget, so a host that refuses deletes fills the budget with doomed bytes and the store's writes +// refuse -- disk never grows past the ceiling by way of a delete that did not happen. + +data local_store_index_magic: String = "gunbc-store-occupancy v2" + +data local_store_index_window_k: Int = 2 + +fn local_store_index_key(family: ArtifactKindId) -> NonEmptyStr { + concat("occupancy-", family as String) as NonEmptyStr +} + +type LocalStoreIndexRow { + name: String + bytes: ByteSize + ordinal: Int +} + +type LocalStoreDoomedRow { + name: String + bytes: ByteSize +} + +type LocalStoreIndex { + next_ordinal: Int + rows: List + doomed: List +} + +type LocalStoreIndexDecode + = LocalStoreIndexDecoded { index: LocalStoreIndex } + | LocalStoreIndexCorrupt { detail: String } + +data local_store_empty_index: LocalStoreIndex = LocalStoreIndex { next_ordinal: 1, rows: [], doomed: [] } + +// THE ONE DELETION AUTHORITY. A name is this store's object exactly when it is what +// store_object_name produces: a serialized content hash that parses (std.content_hash +// parse_content_hash, the inverse at the same home) and re-serializes to itself, followed by the +// object format suffix. No other name -- a path with a separator, a traversal, a foreign file that +// merely contains the suffix -- is ever indexed, swept or deleted. +fn local_store_object_suffix() -> String { + concat(".materialization.", store_object_format_version) +} + +fn local_store_is_object_name(name: String) -> Bool { + let suffix = local_store_object_suffix() + let n = string_length(s: name) + let k = string_length(s: suffix) + if n <= k || !ends_with(s: name, suffix: suffix) { false } else { + let prefix = substring(s: name, start: 0, end: n - k) + match parse_content_hash(wire: prefix) { + Absent => false + Present { value: h } => serialize_content_hash(hash: h) as String == prefix + } + } +} + +fn local_store_index_row_line(r: LocalStoreIndexRow) -> String { + join([r.name, measure_count(m: r.bytes) as String, r.ordinal as String], " ") +} + +fn local_store_doomed_row_line(r: LocalStoreDoomedRow) -> String { + join(["doomed", r.name, measure_count(m: r.bytes) as String], " ") +} + +fn local_store_index_encode(index: LocalStoreIndex) -> String { + let rows = index.rows |> sort_by(r => r.name) |> map(r => local_store_index_row_line(r: r)) + let doomed = index.doomed |> sort_by(r => r.name) |> map(r => local_store_doomed_row_line(r: r)) + join(concat(concat([local_store_index_magic, concat("next ", index.next_ordinal as String)], rows), doomed), "\n") +} + +fn local_store_digits_int(s: String) -> Int? { + if s == "" || !decimal_digits_only(s: s) { none } else { + fold(s |> chars, init: Present { value: 0 }, f: (acc, c) => match acc { + Absent => none + Present { value: n } => if n > (int_inclusive_max() - (c - 48)) / 10 { none } else { Present { value: n * 10 + (c - 48) } } + }) + } +} + +type LocalStoreIndexLine + = LocalStoreLiveLine { row: LocalStoreIndexRow } + | LocalStoreDoomedLine { row: LocalStoreDoomedRow } + | LocalStoreUndecodableLine + +fn local_store_index_line_decode(line: String) -> LocalStoreIndexLine { + let parts: List = line.split(delimiter: " ") + if count(parts) != 3 { LocalStoreUndecodableLine } else { + let p0 = match first(parts) { Present { value: v } => v Absent => "" } + let p1 = match first(skip(parts, 1)) { Present { value: v } => v Absent => "" } + let p2 = match first(skip(parts, 2)) { Present { value: v } => v Absent => "" } + if p0 == "doomed" { + match local_store_digits_int(s: p2) { + Absent => LocalStoreUndecodableLine + Present { value: b } => + if local_store_is_object_name(name: p1) { LocalStoreDoomedLine { row: LocalStoreDoomedRow { name: p1, bytes: byte_size(count: b) } } } else { LocalStoreUndecodableLine } + } + } else { + match local_store_digits_int(s: p1) { + Absent => LocalStoreUndecodableLine + Present { value: b } => match local_store_digits_int(s: p2) { + Absent => LocalStoreUndecodableLine + Present { value: o } => + if local_store_is_object_name(name: p0) { LocalStoreLiveLine { row: LocalStoreIndexRow { name: p0, bytes: byte_size(count: b), ordinal: o } } } else { LocalStoreUndecodableLine } + } + } + } + } +} + +type LocalStoreIndexLines { + ok: Bool + rows: List + doomed: List +} + +fn local_store_index_decode(content: String) -> LocalStoreIndexDecode { + let lines: List = content.split(delimiter: "\n") + let magic = match first(lines) { Present { value: v } => v Absent => "" } + let next_line = match first(skip(lines, 1)) { Present { value: v } => v Absent => "" } + if magic != local_store_index_magic { + LocalStoreIndexCorrupt { detail: "the occupancy index does not start with its format header" } + } else if !starts_with(s: next_line, prefix: "next ") { + LocalStoreIndexCorrupt { detail: "the occupancy index has no next-ordinal line" } + } else { + match local_store_digits_int(s: substring(s: next_line, start: 5, end: string_length(s: next_line))) { + Absent => LocalStoreIndexCorrupt { detail: "the occupancy index's next ordinal is not a number" } + Present { value: next } => { + let decoded = fold(skip(lines, 2), init: LocalStoreIndexLines { ok: true, rows: [], doomed: [] }, f: (acc, l) => match local_store_index_line_decode(line: l) { + LocalStoreLiveLine { row: r } => LocalStoreIndexLines { ok: acc.ok, rows: acc.rows |> list_push(r), doomed: acc.doomed } + LocalStoreDoomedLine { row: d } => LocalStoreIndexLines { ok: acc.ok, rows: acc.rows, doomed: acc.doomed |> list_push(d) } + LocalStoreUndecodableLine => LocalStoreIndexLines { ok: false, rows: acc.rows, doomed: acc.doomed } + }) + if !decoded.ok { + LocalStoreIndexCorrupt { detail: "an occupancy index row does not decode as a canonical object name with its bytes" } + } else { + LocalStoreIndexDecoded { index: LocalStoreIndex { next_ordinal: next, rows: decoded.rows, doomed: decoded.doomed } } + } + } + } + } +} + +fn local_store_doomed_bytes(doomed: List) -> ByteSize { + byte_size(count: fold(doomed, init: 0, f: (acc, d) => acc + measure_count(m: d.bytes))) +} + +fn local_store_row_key(name: String) -> ContentHash { + content_hash_of_value(value: name as NonEmptyStr) +} + +// The index as the std.artifact_store fold sees it, with the budget from the family's provider row. +fn local_store_artifact_store(budget: ByteSize, index: LocalStoreIndex) -> ArtifactStore { + ArtifactStore { + budget: budget, + next_ordinal: index.next_ordinal, + rows: index.rows |> map(r => ArtifactRow { key: local_store_row_key(name: r.name), artifact_bytes: r.bytes, last_use_ordinal: r.ordinal }) + } +} + +// Back from the fold: each surviving row keeps its name, found through the key it was given. +// Each name is hashed ONCE into a key -> name map, and each surviving row finds its name by one +// lookup -- never by re-hashing every name per row (a quadratic fold, DESIGN section 6). +fn local_store_name_by_key(names: List) -> Map { + fold(names, init: empty_map(), f: (m, n) => map_insert(m, serialize_content_hash(hash: local_store_row_key(name: n)) as String, n)) +} + +fn local_store_index_of(store: ArtifactStore, names: List, doomed: List) -> LocalStoreIndex { + let by_key = local_store_name_by_key(names: names) + LocalStoreIndex { + next_ordinal: store.next_ordinal, + doomed: doomed, + rows: store.rows |> fold(init: [], f: (acc, row) => match map_lookup(by_key, serialize_content_hash(hash: row.key) as String) { + Absent => acc + Present { value: n } => acc |> list_push(LocalStoreIndexRow { name: n, bytes: row.artifact_bytes, ordinal: row.last_use_ordinal }) + }) + } +} + +fn local_store_name_set(names: List) -> Set { + fold(names, init: empty_set(), f: (acc, n) => set_insert(acc, n)) +} + +fn local_store_index_window() -> CasGenerationWindow? { + match cas_generation_window(k: local_store_index_window_k) { + CasGenerationWindowAdmitted { window: w } => Present { value: w } + CasGenerationWindowTooNarrow { k: _ } => none + } +} + +// A WINDOWED SLOT ADVANCES ONLY WHEN ITS WINDOW IS SETTLED. A windowed write reclaims generations at or +// below head - k after its own commit, but that reclamation is best effort -- a delete the host refuses +// leaves the generation behind and the write still commits. So before this store advances a windowed +// slot (its occupancy index, or its family hold) it lists the slot: no generation at or below head - k +// may remain. If one does, the CAS store's own cas_reclaim_below is retried once; if the stale +// generations still stand, the write REFUSES, typed, naming them. A slot whose cleanup keeps failing +// therefore stops advancing, and its generation files are bounded by the window plus the one write +// that found it unsettled -- they never grow with the writes that follow. +type LocalStoreSlotSettlement + = LocalStoreSlotSettled + | LocalStoreSlotGenerationsOutstanding { key: NonEmptyStr, stale: List } + | LocalStoreSlotUnlisted { key: NonEmptyStr, cause: String } + | LocalStoreSlotWindowTooNarrow { k: Int } + +fn local_store_stale_generations(gs: List, k: Int) -> List { + let head = fold(gs, init: 0, f: (acc, g) => if cas_generation_count(g: g) > acc { cas_generation_count(g: g) } else { acc }) + gs |> filter(g => cas_generation_count(g: g) <= head - k) +} + +fn local_store_settle_slot(root_path: String, key: NonEmptyStr) -> LocalStoreSlotSettlement + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_write_index_current"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_acquire_family_hold"), + ] +{ + match cas_generation_window(k: local_store_index_window_k) { + CasGenerationWindowTooNarrow { k: k } => LocalStoreSlotWindowTooNarrow { k: k } + CasGenerationWindowAdmitted { window: w } => + match cas_window_generations(root: root_path as NonEmptyStr, key: key) { + CasWindowGenerationsUnlisted { cause: c } => LocalStoreSlotUnlisted { key: key, cause: c } + CasWindowGenerationsListed { generations: gs } => { + let stale = local_store_stale_generations(gs: gs, k: local_store_index_window_k) + if count(stale) == 0 { LocalStoreSlotSettled } else { + let head = fold(gs, init: 0, f: (acc, g) => if cas_generation_count(g: g) > acc { cas_generation_count(g: g) } else { acc }) + let _reclaimed = cas_reclaim_below(root: root_path as NonEmptyStr, key: key, window: w, head: head) + match cas_window_generations(root: root_path as NonEmptyStr, key: key) { + CasWindowGenerationsUnlisted { cause: c } => LocalStoreSlotUnlisted { key: key, cause: c } + CasWindowGenerationsListed { generations: after } => { + let still = local_store_stale_generations(gs: after, k: local_store_index_window_k) + if count(still) == 0 { LocalStoreSlotSettled } else { LocalStoreSlotGenerationsOutstanding { key: key, stale: still } } + } + } + } + } + } + } +} + +fn local_store_settlement_text(s: LocalStoreSlotSettlement) -> String { + match s { + LocalStoreSlotSettled => "the slot's window is settled" + LocalStoreSlotGenerationsOutstanding { key: k, stale: st } => concat(concat("generations at or below the window could not be reclaimed in ", k as String), concat(": ", count(st) as String)) + LocalStoreSlotUnlisted { key: k, cause: c } => concat(concat("the slot ", k as String), concat(" could not be listed: ", c)) + LocalStoreSlotWindowTooNarrow { k: k } => concat("the slot window is narrower than its floor: k = ", k as String) + } +} + +// WHY THE INDEX COULD NOT BE USED, typed and located, never prose: the CAS layer's own causes are +// carried as they are, so a receipt names the slot observation, the attempt admission or the store +// failure that stopped the line. local_store_index_refusal_text is the ONE place they become text, +// and only because std.materialization_object's realization-agnostic refusal carries a String. +type LocalStoreIndexRefusal + = LocalStoreIndexWindowTooNarrow { k: Int } + | LocalStoreIndexSlotUnreadable { cause: CasUnreadableSlot } + | LocalStoreIndexUndecodable { detail: String } + | LocalStoreIndexAttemptNotAdmitted { admission: CasAttemptAdmission } + | LocalStoreIndexStoreRefused { cause: CasStoreFailure } + | LocalStoreIndexContended { attempts: Int } + | LocalStoreIndexUnsettled { settlement: LocalStoreSlotSettlement } + | LocalStoreIndexHoldStale { observed: DurableHoldObservation } + +fn local_store_index_refusal_text(refusal: LocalStoreIndexRefusal) -> String { + match refusal { + LocalStoreIndexWindowTooNarrow { k: k } => concat("the occupancy index window is narrower than its floor: k = ", k as String) + LocalStoreIndexSlotUnreadable { cause: c } => concat("the occupancy index could not be read: ", cas_unreadable_slot_detail(cause: c)) + LocalStoreIndexUndecodable { detail: d } => concat("the occupancy index is corrupt: ", d) + LocalStoreIndexAttemptNotAdmitted { admission: a } => match a { + CasAttemptDigestMismatch { claimed: _, actual: _ } => "the occupancy index write was not admitted: its digest did not match its bytes" + CasAttemptDigestIncomparable { claimed: _, actual: _ } => "the occupancy index write was not admitted: its digests are of incomparable families" + CasAttemptKeyNotSlotAddressable { key: k } => concat("the occupancy index write was not admitted: key is not slot-addressable: ", k as String) + CasAttemptAdmitted { verified: _ } => "the occupancy index write was admitted" + } + LocalStoreIndexStoreRefused { cause: c } => match c { + CasSlotObservationRefused { cause: u } => concat("the occupancy index slot could not be observed for the write: ", cas_unreadable_slot_detail(cause: u)) + CasGenerationPublicationRefused { detail: d } => concat("the occupancy index generation could not be published: ", d as String) + CasGenerationSpaceExhausted { head: _ } => "the occupancy index slot exhausted its generation space" + } + LocalStoreIndexContended { attempts: n } => concat("the occupancy index kept moving under concurrent writers past the attempt bound: ", n as String) + LocalStoreIndexUnsettled { settlement: st } => concat("the occupancy index cannot advance: ", local_store_settlement_text(s: st)) + LocalStoreIndexHoldStale { observed: _ } => "the occupancy index write was attempted under a hold that is no longer live" + } +} + +type LocalStoreIndexRead + = LocalStoreIndexAt { index: LocalStoreIndex, expected: CasExpectation } + | LocalStoreIndexUnavailable { refusal: LocalStoreIndexRefusal } + +fn local_store_read_index(root_path: String, family: ArtifactKindId) -> LocalStoreIndexRead + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_reserve"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_indexed_names"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_record_recency_current"), + decl_ref(module_path: "test.claim.materialization_store_local_wet_witness", decl_name: "only_canonical_object_names_are_indexed_or_swept_by_real_execution"), + decl_ref(module_path: "test.claim.materialization_store_local_wet_witness", decl_name: "index_row_count"), + ] +{ + match local_store_index_window() { + Absent => LocalStoreIndexUnavailable { refusal: LocalStoreIndexWindowTooNarrow { k: local_store_index_window_k } } + Present { value: window } => + match observe_cas_slot_state_windowed(root: root_path as NonEmptyStr, key: local_store_index_key(family: family), window: window) { + CasObservedUnreadable { cause: c } => LocalStoreIndexUnavailable { refusal: LocalStoreIndexSlotUnreadable { cause: c } } + CasObservedReadable { readable: r } => match r { + CasReadableAbsent => LocalStoreIndexAt { index: local_store_empty_index, expected: ExpectSlotAbsent } + CasReadablePresent { version: v } => match local_store_index_decode(content: v.value as String) { + LocalStoreIndexCorrupt { detail: d } => LocalStoreIndexUnavailable { refusal: LocalStoreIndexUndecodable { detail: d } } + LocalStoreIndexDecoded { index: i } => LocalStoreIndexAt { index: i, expected: ExpectSlotGeneration { generation: v.generation } } + } + } + } + } +} + +type LocalStoreIndexWrite + = LocalStoreIndexWritten { reclamation: CasWindowReclamation } + | LocalStoreIndexLostRace + | LocalStoreIndexWriteRefused { refusal: LocalStoreIndexRefusal } + +fn local_store_write_index(held: LocalStoreFamilyHeld, expected: CasExpectation, index: LocalStoreIndex) -> LocalStoreIndexWrite + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_reserve"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_record_recency_current"), + ] +{ + match local_store_hold_check(held: held) { + LocalStoreHoldNotCurrent { observed: o } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexHoldStale { observed: o } } + LocalStoreHoldCurrent => local_store_write_index_current(held: held, expected: expected, index: index) + } +} + +fn local_store_write_index_current(held: LocalStoreFamilyHeld, expected: CasExpectation, index: LocalStoreIndex) -> LocalStoreIndexWrite + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_write_index"), + ] +{ + match local_store_settle_slot(root_path: materialization_store_root_path(root: held.binding.root), key: local_store_index_key(family: held.family)) { + LocalStoreSlotSettled => local_store_write_index_settled(held: held, expected: expected, index: index) + LocalStoreSlotGenerationsOutstanding { key: k, stale: st } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexUnsettled { settlement: LocalStoreSlotGenerationsOutstanding { key: k, stale: st } } } + LocalStoreSlotUnlisted { key: k, cause: c } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexUnsettled { settlement: LocalStoreSlotUnlisted { key: k, cause: c } } } + LocalStoreSlotWindowTooNarrow { k: k } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexWindowTooNarrow { k: k } } + } +} + +fn local_store_write_index_settled(held: LocalStoreFamilyHeld, expected: CasExpectation, index: LocalStoreIndex) -> LocalStoreIndexWrite + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_write_index_current"), + ] +{ + let root_path = materialization_store_root_path(root: held.binding.root) + let family = held.family + let content = local_store_index_encode(index: index) as NonEmptyStr + match local_store_index_window() { + Absent => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexWindowTooNarrow { k: local_store_index_window_k } } + Present { value: window } => + match admit_cas_attempt(attempt: cas_attempt(key: local_store_index_key(family: family), expected: expected, proposed: content, proposed_content: content_hash_of_value(value: content))) { + CasAttemptDigestMismatch { claimed: c, actual: a } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexAttemptNotAdmitted { admission: CasAttemptDigestMismatch { claimed: c, actual: a } } } + CasAttemptDigestIncomparable { claimed: c, actual: a } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexAttemptNotAdmitted { admission: CasAttemptDigestIncomparable { claimed: c, actual: a } } } + CasAttemptKeyNotSlotAddressable { key: k } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexAttemptNotAdmitted { admission: CasAttemptKeyNotSlotAddressable { key: k } } } + CasAttemptAdmitted { verified: v } => { + let write = file_compare_and_set_windowed(root: root_path as NonEmptyStr, publication: DefaultAccessCreateOnly, window: window, verified: v) + match write.outcome { + CasCommitted { committed: _ } => LocalStoreIndexWritten { reclamation: write.reclamation } + CasPreconditionFailed { expected: _, observed: _ } => LocalStoreIndexLostRace + CasStoreRefused { cause: c } => LocalStoreIndexWriteRefused { refusal: LocalStoreIndexStoreRefused { cause: c } } + } + } } } } +// --------------------------------------------------------------------------------------------- +// THE FAMILY HOLD. Every write to a family -- reservation and its cleanup, deletion of what it evicted, +// publication, settlement, recency -- runs inside ONE std.durable_exclusive_hold per family, held by +// the writing process and released when the write settles; the open-time sweep holds every family. +// So no two writers ever interleave over one family's index and objects, and the interleavings that +// per-operation CAS had to handle one by one (a row evicted while its writer publishes, a stale cleaner +// deleting an object a later writer republished) have no schedule in which they can occur. Lookups +// take no hold: an object is published complete or not at all, and is read and verified by itself. +// +// A second writer REFUSES, typed -- it never waits, and a refused commit costs only a recomputation. +// The holder is this process (boot, pid, start time, pid namespace: gunbc.process_hold_identity). A +// holder that dies holding the family is freed by OBSERVATION, never by age: its process is looked up +// on this host, and only a holder observed dead is recovered through file_hold_recovery_assess. An +// unobservable holder stays held -- a family that cannot be shown free is not written. The hold slot +// is windowed like the occupancy index (k = 2), so its bookkeeping does not grow with commits. + +data local_store_hold_owner_prefix: String = "materialization-store" + +data local_store_hold_recovery_ref: DurableHoldReleaseRef = "materialization-store: holder observed dead" as DurableHoldReleaseRef + +data local_store_hold_release_ref: DurableHoldReleaseRef = "materialization-store: write settled" as DurableHoldReleaseRef + +fn local_store_hold_key(family: ArtifactKindId) -> NonEmptyStr { + concat("hold-", family as String) as NonEmptyStr +} + +type LocalStoreHoldRefusal + = LocalStoreFamilyBusy { holder: DurableHoldOwnerRef, evidence: NonEmptyStr } + | LocalStoreHolderUnobservable { holder: DurableHoldOwnerRef, cause: NonEmptyStr } + | LocalStoreHoldRecoveryRefused { refusal: DurableHoldRecoveryRefusal } + | LocalStoreHoldContended + | LocalStoreHoldStoreUnavailable { cause: CasStoreFailure } + | LocalStoreHoldSlotUndecodable { detail: NonEmptyStr } + | LocalStoreHoldKeyNotAddressable + | LocalStoreHoldSelfUnidentified { cause: NonEmptyStr } + | LocalStoreHoldWindowTooNarrow { k: Int } + | LocalStoreHoldUnsettled { settlement: LocalStoreSlotSettlement } + | LocalStoreHoldStale { observed: DurableHoldObservation } + | LocalStoreHoldWrongFamily { held: ArtifactKindId, requested: ArtifactKindId } + +fn local_store_hold_refusal_text(refusal: LocalStoreHoldRefusal) -> String { + match refusal { + LocalStoreFamilyBusy { holder: h, evidence: e } => concat(concat("the family is held by a live writer ", h as String), concat(": ", e as String)) + LocalStoreHolderUnobservable { holder: h, cause: c } => concat(concat("the family is held by a writer that cannot be observed ", h as String), concat(": ", c as String)) + LocalStoreHoldRecoveryRefused { refusal: _ } => "the family's dead holder could not be recovered" + LocalStoreHoldContended => "the family hold moved under a concurrent writer" + LocalStoreHoldStoreUnavailable { cause: _ } => "the family hold slot could not be written" + LocalStoreHoldSlotUndecodable { detail: d } => concat("the family hold slot is undecodable: ", d as String) + LocalStoreHoldKeyNotAddressable => "the family hold key is not slot-addressable" + LocalStoreHoldSelfUnidentified { cause: c } => concat("this process cannot name itself, so it cannot hold a family: ", c as String) + LocalStoreHoldWindowTooNarrow { k: k } => concat("the family hold window is narrower than its floor: k = ", k as String) + LocalStoreHoldUnsettled { settlement: st } => concat("the family hold slot cannot advance: ", local_store_settlement_text(s: st)) + LocalStoreHoldStale { observed: _ } => "the hold this write was given is no longer the family's live hold" + LocalStoreHoldWrongFamily { held: h, requested: r } => concat(concat("the hold is for family ", h as String), concat(", the write is for ", r as String)) + } +} + +type LocalStoreFamilyHeld sole_constructor { + binding: LocalStoreBinding + family: ArtifactKindId + owner: DurableHoldOwnerRef + generation: CasGeneration + window: CasGenerationWindow +} + +type LocalStoreHoldAcquisition + = LocalStoreHoldAcquired { held: LocalStoreFamilyHeld } + | LocalStoreHoldRefused { refusal: LocalStoreHoldRefusal } + +fn local_store_hold_owner_process(owner: DurableHoldOwnerRef) -> ProcessHoldHolder? { + let halves: List = (owner as String).split(delimiter: process_hold_marker) + if count(halves) != 2 || (match first(halves) { Present { value: v } => v Absent => "" }) != local_store_hold_owner_prefix { none } else { + process_hold_decode(text: concat("boot=", match first(skip(halves, 1)) { Present { value: v } => v Absent => "" })) + } +} + +fn local_store_acquire_family_hold(binding: LocalStoreBinding, family: ArtifactKindId, recovered: Bool) -> LocalStoreHoldAcquisition + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_with_family_of"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_acquire_all"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_hold_occupied"), + ] +{ + let root = binding.root + let root_path = materialization_store_root_path(root: root) + match local_store_index_window() { + Absent => LocalStoreHoldRefused { refusal: LocalStoreHoldWindowTooNarrow { k: local_store_index_window_k } } + Present { value: window } => + match self_process_identity() { + SelfUnidentified { cause: c } => LocalStoreHoldRefused { refusal: LocalStoreHoldSelfUnidentified { cause: c } } + SelfIdentified { process: me, pid_namespace: ns } => { + let owner = concat(local_store_hold_owner_prefix, process_hold_text(p: me, pid_namespace: ns)) as DurableHoldOwnerRef + let key = local_store_hold_key(family: family) + match local_store_settle_slot(root_path: root_path, key: key) { + LocalStoreSlotSettled => local_store_acquire_settled(binding: binding, family: family, key: key, owner: owner, window: window, recovered: recovered) + LocalStoreSlotGenerationsOutstanding { key: k, stale: st } => LocalStoreHoldRefused { refusal: LocalStoreHoldUnsettled { settlement: LocalStoreSlotGenerationsOutstanding { key: k, stale: st } } } + LocalStoreSlotUnlisted { key: k, cause: c } => LocalStoreHoldRefused { refusal: LocalStoreHoldUnsettled { settlement: LocalStoreSlotUnlisted { key: k, cause: c } } } + LocalStoreSlotWindowTooNarrow { k: k } => LocalStoreHoldRefused { refusal: LocalStoreHoldWindowTooNarrow { k: k } } + } + } + } + } +} + +fn local_store_acquire_settled(binding: LocalStoreBinding, family: ArtifactKindId, key: NonEmptyStr, owner: DurableHoldOwnerRef, window: CasGenerationWindow, recovered: Bool) -> LocalStoreHoldAcquisition + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_acquire_family_hold"), + ] +{ + let root = binding.root + let root_path = materialization_store_root_path(root: root) + match file_hold_acquire_windowed(root: root_path as NonEmptyStr, slot_key: key, requested_owner: owner, window: window) { + FileHoldAcquired { slot_key: _, owner: o, generation: g } => + LocalStoreHoldAcquired { held: LocalStoreFamilyHeld { binding: binding, family: family, owner: o, generation: g, window: window } } + FileHoldOccupied { slot_key: _, holder: h, generation: g } => + local_store_hold_occupied(binding: binding, family: family, key: key, holder: h, generation: g, window: window, recovered: recovered) + FileHoldAcquireLost { slot_key: _ } => LocalStoreHoldRefused { refusal: LocalStoreHoldContended } + FileHoldAcquireStoreUnavailable { slot_key: _, cause: c } => LocalStoreHoldRefused { refusal: LocalStoreHoldStoreUnavailable { cause: c } } + FileHoldSlotUndecodable { slot_key: _, generation: _, detail: d } => LocalStoreHoldRefused { refusal: LocalStoreHoldSlotUndecodable { detail: d } } + FileHoldKeyNotSlotAddressable { slot_key: _ } => LocalStoreHoldRefused { refusal: LocalStoreHoldKeyNotAddressable } + } +} + +// An occupied family is looked at ONCE: a live or unobservable holder refuses; a holder observed dead is +// recovered at the generation the refused acquire saw, and the acquire is tried once more. A family +// still held after that recovery is busy -- the loop is bounded by construction. +fn local_store_hold_occupied(binding: LocalStoreBinding, family: ArtifactKindId, key: NonEmptyStr, holder: DurableHoldOwnerRef, generation: CasGeneration, window: CasGenerationWindow, recovered: Bool) -> LocalStoreHoldAcquisition + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_acquire_settled"), + ] +{ + let root = binding.root + let root_path = materialization_store_root_path(root: root) + match local_store_hold_owner_process(owner: holder) { + Absent => LocalStoreHoldRefused { refusal: LocalStoreHolderUnobservable { holder: holder, cause: "the holder is not a materialization-store process" as NonEmptyStr } } + Present { value: hp } => { + let liveness = holder_process_liveness(p: hp.process, pid_namespace: hp.pid_namespace) + match liveness { + HolderObservedLive { evidence: e } => LocalStoreHoldRefused { refusal: LocalStoreFamilyBusy { holder: holder, evidence: e } } + HolderLivenessUnobservable { cause: c } => LocalStoreHoldRefused { refusal: LocalStoreHolderUnobservable { holder: holder, cause: c } } + HolderObservedDead { evidence: e } => + if recovered { + LocalStoreHoldRefused { refusal: LocalStoreHoldContended } + } else { + match file_hold_recovery_assess_windowed(root: root_path as NonEmptyStr, slot_key: key, report: DurableHoldHolderReport { holder: holder, generation: generation, liveness: liveness }, released_by: local_store_hold_recovery_ref, window: window) { + FileHoldRecoveryNotEligible { slot_key: _, refusal: r } => LocalStoreHoldRefused { refusal: LocalStoreHoldRecoveryRefused { refusal: r } } + FileHoldRecoveryPlanned { plan: pl, dead_holder: _, evidence: _ } => + match file_hold_release_commit(plan: pl) { + FileHoldReleased { slot_key: _, generation: _ } => local_store_acquire_family_hold(binding: binding, family: family, recovered: true) + FileHoldReleaseLost { slot_key: _ } => LocalStoreHoldRefused { refusal: LocalStoreHoldContended } + FileHoldReleaseStoreUnavailable { slot_key: _, cause: c } => LocalStoreHoldRefused { refusal: LocalStoreHoldStoreUnavailable { cause: c } } + FileHoldReleaseKeyNotSlotAddressable { slot_key: _ } => LocalStoreHoldRefused { refusal: LocalStoreHoldKeyNotAddressable } + } + } + } + } + } + } +} + +// A HELD VALUE IS ONLY AS GOOD AS THE SLOT SAYS. The bracket releases its hold when its body returns, +// and a body can still return the held value it was given, so possession of a LocalStoreFamilyHeld is +// not proof that the family is held NOW. Every mutation therefore re-observes the family's hold slot +// first: the slot must be held, at the very generation and by the very owner this value records, or +// the mutation refuses as a stale hold. Release-then-mutate is refused even with a leaked value. +type LocalStoreHoldCheck + = LocalStoreHoldCurrent + | LocalStoreHoldNotCurrent { observed: DurableHoldObservation } + +fn local_store_hold_check(held: LocalStoreFamilyHeld) -> LocalStoreHoldCheck + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_write_index"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep_held"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_record_recency_held"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_under"), + ] +{ + let observed = observe_file_hold_windowed(root: materialization_store_root_path(root: held.binding.root) as NonEmptyStr, key: local_store_hold_key(family: held.family), window: held.window) + match observed { + HoldSlotHeld { generation: g, owner: o } => + if cas_generation_count(g: g) == cas_generation_count(g: held.generation) && durable_hold_owner_equal(left: o, right: held.owner) { LocalStoreHoldCurrent } else { LocalStoreHoldNotCurrent { observed: observed } } + HoldSlotAbsent => LocalStoreHoldNotCurrent { observed: observed } + HoldSlotFree { generation: _, released_by: _ } => LocalStoreHoldNotCurrent { observed: observed } + HoldSlotUndecodable { generation: _, detail: _ } => LocalStoreHoldNotCurrent { observed: observed } + HoldObservationUnavailable { cause: _ } => LocalStoreHoldNotCurrent { observed: observed } + } +} + +// THE ONE BRACKET. A family is held only by running a body inside it: acquire, run the body with the +// held value, release. It is the only place a LocalStoreFamilyHeld is minted for a writer (the hold +// acquisition and its release are admitted to it and to the sweep alone), and the release happens +// here whatever the body returned. +type LocalStoreBracket + = LocalStoreBracketRan { value: T, hold_release: LocalStoreHoldRelease } + | LocalStoreBracketRefused { refusal: LocalStoreHoldRefusal } + +fn local_store_with_family(capability: LocalStoreCapability, of: ArtifactRequest, run: fn(LocalStoreFamilyHeld) -> T) -> LocalStoreBracket { + local_store_with_family_of(binding: capability.binding, family: request_kind(req: of), run: run) +} + +fn local_store_with_family_of(binding: LocalStoreBinding, family: ArtifactKindId, run: fn(LocalStoreFamilyHeld) -> T) -> LocalStoreBracket + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_with_family"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_record_recency"), + ] +{ + match local_store_acquire_family_hold(binding: binding, family: family, recovered: false) { + LocalStoreHoldRefused { refusal: r } => LocalStoreBracketRefused { refusal: r } + LocalStoreHoldAcquired { held: h } => { + let value = run(h) + LocalStoreBracketRan { value: value, hold_release: local_store_release_family_hold(held: h) } + } + } +} + +// The release is reported, never fatal to the write it closes: a hold that could not be freed stays +// held by this process, and once this process is gone the next writer recovers it by observation. +type LocalStoreHoldRelease + = LocalStoreHoldReleased + | LocalStoreHoldReleaseNotEligible { refusal: DurableHoldReleaseRefusal } + | LocalStoreHoldReleaseLost + | LocalStoreHoldReleaseStoreUnavailable { cause: CasStoreFailure } + | LocalStoreHoldReleaseKeyNotAddressable + +fn local_store_release_family_hold(held: LocalStoreFamilyHeld) -> LocalStoreHoldRelease + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_with_family_of"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_acquire_all"), + ] +{ + match file_hold_release_assess_windowed(root: materialization_store_root_path(root: held.binding.root) as NonEmptyStr, slot_key: local_store_hold_key(family: held.family), owner: held.owner, acquired_generation: held.generation, released_by: local_store_hold_release_ref, window: held.window) { + FileHoldReleaseNotEligible { slot_key: _, refusal: r } => LocalStoreHoldReleaseNotEligible { refusal: r } + FileHoldReleasePlanned { plan: pl } => + match file_hold_release_commit(plan: pl) { + FileHoldReleased { slot_key: _, generation: _ } => LocalStoreHoldReleased + FileHoldReleaseLost { slot_key: _ } => LocalStoreHoldReleaseLost + FileHoldReleaseStoreUnavailable { slot_key: _, cause: c } => LocalStoreHoldReleaseStoreUnavailable { cause: c } + FileHoldReleaseKeyNotSlotAddressable { slot_key: _ } => LocalStoreHoldReleaseKeyNotAddressable + } + } +} + +// --------------------------------------------------------------------------------------------- +// CONFIRMED DELETION. A name leaves `doomed` only when the host CONFIRMS the bytes are gone: the +// delete succeeded, or the target was already absent. A refused or unrecognized delete leaves the +// name doomed and its bytes charged. Every path below is store_path(root, name) for a name the +// recognizer admitted, so a delete never reaches outside the root. + +// One delete, observed once, carrying the host's typed answer itself: a receipt can say a delete was +// refused for permission, for another known kind, or for a kind the classifier does not recognize. +type LocalStoreDeleteAttempt { + name: String + outcome: FilesystemDelete +} + +// WHO MAY DELETE: a writer holding the family the names belong to, or the sweep holding every family. +// Both proofs are sealed records minted only by the hold acquisitions, so a delete without the hold +// cannot be written. +type LocalStoreWriteAuthority + = LocalStoreFamilyAuthority { held: LocalStoreFamilyHeld } + | LocalStoreSweepAuthority { all: LocalStoreAllFamiliesHeld } + +fn local_store_authority_root(authority: LocalStoreWriteAuthority) -> String { + match authority { + LocalStoreFamilyAuthority { held: h } => materialization_store_root_path(root: h.binding.root) + LocalStoreSweepAuthority { all: a } => materialization_store_root_path(root: a.binding.root) + } +} + +fn local_store_delete_attempts(authority: LocalStoreWriteAuthority, names: List) -> List + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_reserve"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_reserved"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep_current"), + ] +{ + let root_path = local_store_authority_root(authority: authority) + names |> map(n => { + let path = store_path(root_path: root_path, name: n) + let d = Filesystem.Delete(path: path) + LocalStoreDeleteAttempt { name: n, outcome: filesystem_delete(path: path, success: d.success, error: d.error, error_kind: d.error_kind) } + }) +} + +// The split a caller needs, made by matching the outcome once per attempt: the names now GONE (deleted +// or already absent), and the attempts that left bytes behind, with their typed outcomes intact. +type LocalStoreDeleteSplit { + gone: List + refused: List +} + +fn local_store_delete_split(attempts: List) -> LocalStoreDeleteSplit { + fold(attempts, init: LocalStoreDeleteSplit { gone: [], refused: [] }, f: (acc, t) => match t.outcome { + FilesystemDeleted { path: _ } => LocalStoreDeleteSplit { gone: acc.gone |> list_push(t.name), refused: acc.refused } + FilesystemDeleteTargetAbsent { path: _ } => LocalStoreDeleteSplit { gone: acc.gone |> list_push(t.name), refused: acc.refused } + FilesystemDeleteRefused { path: _, kind: _, error: _ } => LocalStoreDeleteSplit { gone: acc.gone, refused: acc.refused |> list_push(t) } + FilesystemDeleteKindUnrecognized { path: _, observed: _, error: _ } => LocalStoreDeleteSplit { gone: acc.gone, refused: acc.refused |> list_push(t) } + }) +} + +// --------------------------------------------------------------------------------------------- +// RESERVATION: the index row is committed BEFORE the object is published. One CAS does three things: +// it retires the doomed names whose delete the host has now confirmed, it admits the new row against +// the budget LESS the doomed bytes still charged, and it moves the rows that admission evicted into +// `doomed`. An artifact larger than the whole budget refuses as did-not-fit and evicts nothing; one +// that fits the budget but not what the undeleted bytes leave of it refuses as cleanup outstanding. +// A loser of the index CAS re-reads and retries, never overwrites; past the bound it refuses. An +// object already live is not re-reserved (put-if-absent is idempotent). Reserving is necessary but +// not sufficient for an object to stay accounted: the commit re-reads the index after publishing +// (local_store_commit), because another writer may evict this row while this one is publishing. + +data local_store_reservation_attempt_bound: Int = 8 + +// WHAT A RESERVATION ADMITTED, sealed: only local_store_reserve mints it, so the publish half cannot be +// handed an admission the index never made. +type LocalStoreAdmission sole_constructor { + budget: ByteSize + evicted: List + retired: List + reclamation: CasWindowReclamation +} + +type LocalStoreReservation + = LocalStoreReserved { admission: LocalStoreAdmission } + | LocalStoreReservationRefused { refusal: StoreCommitRefusal } + | LocalStoreReservationIndexRefused { refusal: LocalStoreIndexRefusal } + | LocalStoreReservationProviderRefused { construction: StoreConstruction } + +// A LIVE ROW IS NOT YET A PRESENT OBJECT. A row is written at reservation, before its object is +// published, so the object is asked for directly -- through the store's own lookup, which reads and +// VERIFIES it -- and the answer has three arms, not two. VERIFIED: the exact object is present, and a +// request whose row is live converges on it with no new bytes. ABSENT: established absent (the lookup is +// a miss), so a live row is an unfinished reservation that binds nothing, and the current request runs +// the ordinary admission path at ITS OWN size. UNAVAILABLE: the object could not be read or did not +// verify -- bytes may well be there -- so the commit refuses BEFORE any index write. +type LocalStoreObjectPresence + = LocalStoreObjectVerified + | LocalStoreObjectAbsent + | LocalStoreObjectUnavailable { lookup: StoreLookup } + +fn local_store_object_presence(lookup: StoreLookup) -> LocalStoreObjectPresence { + match lookup { + StoreLookupHit { artifact: _, payloads: _, receipt: _, evaluation_identity: _ } => LocalStoreObjectVerified + StoreLookupMiss { key: _ } => LocalStoreObjectAbsent + StoreLookupRefused { key: _, refusal: _ } => LocalStoreObjectUnavailable { lookup: lookup } + } +} + +fn local_store_presence_converges(presence: LocalStoreObjectPresence) -> Bool { + match presence { + LocalStoreObjectVerified => true + LocalStoreObjectAbsent => false + LocalStoreObjectUnavailable { lookup: _ } => false + } +} + +fn local_store_reserve(held: LocalStoreFamilyHeld, name: String, bytes: ByteSize, presence: LocalStoreObjectPresence, attempt: Int) -> LocalStoreReservation + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_held"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_reserve"), + ] +{ + let budgets = held.binding.budgets + let family = held.family + let root_path = materialization_store_root_path(root: held.binding.root) + if attempt > local_store_reservation_attempt_bound { + LocalStoreReservationIndexRefused { refusal: LocalStoreIndexContended { attempts: local_store_reservation_attempt_bound } } + } else { + match local_store_family_budget(budgets: budgets, family: family) { + Absent => LocalStoreReservationRefused { refusal: StoreCommitBudgetUndeclared { family: family } } + Present { value: b } => + match store_over_provider(provider: local_store_family_provider(family: family, budget: b.budget)) { + StoreRefusedRetention { provider_id: pid, declared: dcl, cause: why } => + LocalStoreReservationProviderRefused { construction: StoreRefusedRetention { provider_id: pid, declared: dcl, cause: why } } + StoreReady { store: ready } => + match local_store_read_index(root_path: root_path, family: family) { + LocalStoreIndexUnavailable { refusal: r } => LocalStoreReservationIndexRefused { refusal: r } + LocalStoreIndexAt { index: index, expected: expected } => { + let cleanup = local_store_delete_attempts(authority: LocalStoreFamilyAuthority { held: held }, names: index.doomed |> map(d => d.name)) + let cleanup_split = local_store_delete_split(attempts: cleanup) + let gone = local_store_name_set(names: cleanup_split.gone) + let still_doomed = index.doomed |> filter(d => !set_contains(gone, d.name)) + let undeleted = measure_count(m: local_store_doomed_bytes(doomed: still_doomed)) + let budget_total = measure_count(m: ready.budget) + let artifact = measure_count(m: bytes) + let live_names = local_store_name_set(names: index.rows |> map(r => r.name)) + if artifact > budget_total { + LocalStoreReservationRefused { refusal: StoreCommitDidNotFit { artifact_bytes: bytes, budget: ready.budget } } + } else if set_contains(live_names, name) && local_store_presence_converges(presence: presence) { + LocalStoreReserved { admission: LocalStoreAdmission { budget: ready.budget, evicted: [], retired: [], reclamation: CasReclamationNotCommitted } } + } else if artifact > budget_total - undeleted { + LocalStoreReservationRefused { refusal: StoreCommitCleanupOutstanding { artifact_bytes: bytes, budget: ready.budget, undeleted: byte_size(count: undeleted) } } + } else { + let cleaned = LocalStoreIndex { next_ordinal: index.next_ordinal, rows: index.rows, doomed: still_doomed } + match store_put(store: local_store_artifact_store(budget: byte_size(count: budget_total - undeleted), index: cleaned), key: local_store_row_key(name: name), artifact_bytes: bytes) { + StorePutDidNotFit { artifact_bytes: a, budget: _ } => + LocalStoreReservationRefused { refusal: StoreCommitCleanupOutstanding { artifact_bytes: a, budget: ready.budget, undeleted: byte_size(count: undeleted) } } + StorePutAdmitted { store: packed, evicted: _ } => { + let names = concat(index.rows |> map(r => r.name), [name]) + let survivors = local_store_index_of(store: packed, names: names, doomed: []).rows + let kept = local_store_name_set(names: survivors |> map(k => k.name)) + let evicted_rows = index.rows |> filter(r => !set_contains(kept, r.name)) + let next_index = LocalStoreIndex { + next_ordinal: packed.next_ordinal, + rows: survivors, + doomed: concat(still_doomed, evicted_rows |> map(r => LocalStoreDoomedRow { name: r.name, bytes: r.bytes })) + } + match local_store_write_index(held: held, expected: expected, index: next_index) { + LocalStoreIndexWritten { reclamation: rc } => + LocalStoreReserved { admission: LocalStoreAdmission { budget: ready.budget, evicted: evicted_rows |> map(r => LocalStoreDoomedRow { name: r.name, bytes: r.bytes }), retired: cleanup_split.gone, reclamation: rc } } + LocalStoreIndexLostRace => local_store_reserve(held: held, name: name, bytes: bytes, presence: presence, attempt: attempt + 1) + LocalStoreIndexWriteRefused { refusal: r } => LocalStoreReservationIndexRefused { refusal: r } + } + } + } + } + } + } + } + } + } +} + +// --------------------------------------------------------------------------------------------- +// SELF-CLEAN ON OPEN. The listing is taken FIRST and the family indexes are read AFTER it: an object +// in the listing was published after its reservation, so an index read after the listing names it +// unless it was since evicted -- in which case deleting it is right. Only this realization's object +// names (.materialization.) are candidates; the marker, the index slots and any +// other file are never touched, nothing is recursed into, and an index that cannot be read stops the +// sweep (nothing is deleted on a guess). + +// Why a sweep did not run, typed: the root's listing observation as the filesystem layer gave it, or +// the family whose index could not be read with that index's typed refusal. +type LocalStoreSweepRefusal + = LocalStoreSweepListingRefused { listing: FilesystemListingObservation } + | LocalStoreSweepIndexRefused { family: ArtifactKindId, refusal: LocalStoreIndexRefusal } + | LocalStoreSweepHoldRefused { family: ArtifactKindId, refusal: LocalStoreHoldRefusal, released: List } + +type LocalStoreSweep + = LocalStoreSwept { removed: List, delete_refused: List, releases: List } + | LocalStoreSweepUnavailable { refusal: LocalStoreSweepRefusal } + +type LocalStoreIndexedNames + = LocalStoreIndexedNamesRead { names: List } + | LocalStoreIndexedNamesRefused { family: ArtifactKindId, refusal: LocalStoreIndexRefusal } + +fn local_store_indexed_names(binding: LocalStoreBinding) -> LocalStoreIndexedNames + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep_current"), + ] +{ + let root = binding.root + let budgets = binding.budgets + let root_path = materialization_store_root_path(root: root) + fold(budgets, init: LocalStoreIndexedNamesRead { names: [] }, f: (acc, b) => match acc { + LocalStoreIndexedNamesRefused { family: f, refusal: r } => LocalStoreIndexedNamesRefused { family: f, refusal: r } + LocalStoreIndexedNamesRead { names: names } => match local_store_read_index(root_path: root_path, family: b.family) { + LocalStoreIndexUnavailable { refusal: r } => LocalStoreIndexedNamesRefused { family: b.family, refusal: r } + LocalStoreIndexAt { index: i, expected: _ } => LocalStoreIndexedNamesRead { names: concat(concat(names, i.rows |> map(r => r.name)), i.doomed |> map(d => d.name)) } + } + }) +} + +// THE SWEEP HOLDS EVERY FAMILY: an orphan's family is not known from its name, so no family may be +// writing while the listing and the indexes are compared. Holds are taken in budget order; a refusal +// releases the ones already taken and reports which family refused and why. +type LocalStoreAllHeld + = LocalStoreAllHeldAcquired { all: LocalStoreAllFamiliesHeld } + | LocalStoreAllHeldRefused { family: ArtifactKindId, refusal: LocalStoreHoldRefusal, released: List } + +type LocalStoreAllFamiliesHeld sole_constructor { + binding: LocalStoreBinding + held: List +} + +fn local_store_acquire_all(binding: LocalStoreBinding) -> LocalStoreAllHeld + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep"), + ] +{ + fold(binding.budgets, init: LocalStoreAllHeldAcquired { all: LocalStoreAllFamiliesHeld { binding: binding, held: [] } }, f: (acc, b) => match acc { + LocalStoreAllHeldRefused { family: f, refusal: r, released: rs } => LocalStoreAllHeldRefused { family: f, refusal: r, released: rs } + LocalStoreAllHeldAcquired { all: a } => match local_store_acquire_family_hold(binding: binding, family: b.family, recovered: false) { + LocalStoreHoldAcquired { held: h } => LocalStoreAllHeldAcquired { all: LocalStoreAllFamiliesHeld { binding: a.binding, held: a.held |> list_push(h) } } + LocalStoreHoldRefused { refusal: r } => LocalStoreAllHeldRefused { family: b.family, refusal: r, released: a.held |> map(x => local_store_release_family_hold(held: x)) } + } + }) +} + +fn local_store_sweep(binding: LocalStoreBinding) -> LocalStoreSweep + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_open"), + ] +{ + let root = binding.root + let budgets = binding.budgets + let root_path = materialization_store_root_path(root: root) + match local_store_acquire_all(binding: binding) { + LocalStoreAllHeldRefused { family: f, refusal: r, released: rs } => LocalStoreSweepUnavailable { refusal: LocalStoreSweepHoldRefused { family: f, refusal: r, released: rs } } + LocalStoreAllHeldAcquired { all: a } => { + let swept = local_store_sweep_held(all: a) + let releases = a.held |> map(x => local_store_release_family_hold(held: x)) + match swept { + LocalStoreSwept { removed: r, delete_refused: d, releases: _ } => LocalStoreSwept { removed: r, delete_refused: d, releases: releases } + LocalStoreSweepUnavailable { refusal: x } => LocalStoreSweepUnavailable { refusal: x } + } + } + } +} + +fn local_store_sweep_held(all: LocalStoreAllFamiliesHeld) -> LocalStoreSweep + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep"), + ] +{ + let root = all.binding.root + let root_path = materialization_store_root_path(root: root) + let stale = fold(all.held, init: none, f: (acc, h) => match acc { + Present { value: _ } => acc + Absent => match local_store_hold_check(held: h) { + LocalStoreHoldCurrent => none + LocalStoreHoldNotCurrent { observed: o } => Present { value: LocalStoreSweepHoldRefused { family: h.family, refusal: LocalStoreHoldStale { observed: o }, released: [] } } + } + }) + match stale { + Present { value: r } => LocalStoreSweepUnavailable { refusal: r } + Absent => local_store_sweep_current(all: all) + } +} + +fn local_store_sweep_current(all: LocalStoreAllFamiliesHeld) -> LocalStoreSweep + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_sweep_held"), + ] +{ + let root = all.binding.root + let root_path = materialization_store_root_path(root: root) + let listing = Filesystem.List(path: root_path) + let observed = filesystem_listing_observation(directory: root_path, success: listing.success, entries: listing.entries, error: listing.error) + match observed { + FilesystemDirectorySubjectRefused { directory: _, cause: _ } => LocalStoreSweepUnavailable { refusal: LocalStoreSweepListingRefused { listing: observed } } + FilesystemDirectoryListingRefused { directory: _, error: _ } => LocalStoreSweepUnavailable { refusal: LocalStoreSweepListingRefused { listing: observed } } + FilesystemDirectoryListed(listed) => + match local_store_indexed_names(binding: all.binding) { + LocalStoreIndexedNamesRefused { family: f, refusal: r } => LocalStoreSweepUnavailable { refusal: LocalStoreSweepIndexRefused { family: f, refusal: r } } + LocalStoreIndexedNamesRead { names: indexed } => { + let indexed_set = local_store_name_set(names: indexed) + let orphans = filesystem_listing_entry_names(listing: listed) + |> filter(n => local_store_is_object_name(name: n) && !set_contains(indexed_set, n)) + let split = local_store_delete_split(attempts: local_store_delete_attempts(authority: LocalStoreSweepAuthority { all: all }, names: orphans)) + LocalStoreSwept { removed: split.gone, delete_refused: split.refused, releases: [] } + } + } + } +} + +// --------------------------------------------------------------------------------------------- +// RECENCY IS ADVISORY. A lookup never writes. A process records the names it served ONCE, at the end +// of its run, as one CAS per family index; losing that CAS, or failing to read the index, only makes +// the next eviction less precise, so this function cannot fail a lookup or a commit and has no +// refusal arm -- its outcome is reported and nothing branches on it for correctness. + +type LocalStoreRecency + = LocalStoreRecencyRecorded { family: ArtifactKindId, bumped: Int } + | LocalStoreRecencyNothingToRecord { family: ArtifactKindId } + | LocalStoreRecencyLost { family: ArtifactKindId, refusal: LocalStoreIndexRefusal } + | LocalStoreRecencyRaced { family: ArtifactKindId } + | LocalStoreRecencyHoldRefused { family: ArtifactKindId, refusal: LocalStoreHoldRefusal } + +type LocalStoreRecencyReceipt { + family: ArtifactKindId + recency: LocalStoreRecency + hold_release: LocalStoreHoldRelease? +} + +// Recency is recorded for the requests a process served, once per family that has any: the family +// is the request's own kind and the name its own object name, so nothing beside the capability says +// which family or which file. +fn local_store_record_recency(capability: LocalStoreCapability, served: List) -> List { + capability.binding.budgets |> flat_map(b => { + let names = served |> filter(q => request_kind(req: q) == b.family) |> flat_map(q => match evaluation_store_address(req: q) { + Admitted { address: addr } => [store_object_name(address: addr)] + FamilyRefused { family: _, scope: _ } => [] + }) + if count(names) == 0 { [] } else { + [match local_store_with_family_of(binding: capability.binding, family: b.family, run: h => local_store_record_recency_held(held: h, served: names)) { + LocalStoreBracketRefused { refusal: r } => LocalStoreRecencyReceipt { family: b.family, recency: LocalStoreRecencyHoldRefused { family: b.family, refusal: r }, hold_release: none } + LocalStoreBracketRan { value: recency, hold_release: rel } => LocalStoreRecencyReceipt { family: b.family, recency: recency, hold_release: Present { value: rel } } + }] + } + }) +} + +fn local_store_record_recency_held(held: LocalStoreFamilyHeld, served: List) -> LocalStoreRecency + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_record_recency"), + ] +{ + let family = held.family + let root_path = materialization_store_root_path(root: held.binding.root) + match local_store_hold_check(held: held) { + LocalStoreHoldNotCurrent { observed: o } => LocalStoreRecencyHoldRefused { family: family, refusal: LocalStoreHoldStale { observed: o } } + LocalStoreHoldCurrent => local_store_record_recency_current(held: held, served: served) + } +} + +fn local_store_record_recency_current(held: LocalStoreFamilyHeld, served: List) -> LocalStoreRecency + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_record_recency_held"), + ] +{ + let family = held.family + let root_path = materialization_store_root_path(root: held.binding.root) + match local_store_read_index(root_path: root_path, family: family) { + LocalStoreIndexUnavailable { refusal: r } => LocalStoreRecencyLost { family: family, refusal: r } + LocalStoreIndexAt { index: index, expected: expected } => { + let names = index.rows |> map(r => r.name) + let live_names = local_store_name_set(names: names) + let hits = served |> filter(n => set_contains(live_names, n)) + if count(hits) == 0 { LocalStoreRecencyNothingToRecord { family: family } } else { + let bumped = fold(hits, init: local_store_artifact_store(budget: byte_size(count: 0), index: index), f: (st, n) => match store_get(store: st, key: local_store_row_key(name: n)) { + StoreHit { store: s2, row: _ } => s2 + StoreMiss { store: s2 } => s2 + }) + match local_store_write_index(held: held, expected: expected, index: local_store_index_of(store: bumped, names: names, doomed: index.doomed)) { + LocalStoreIndexWritten { reclamation: _ } => LocalStoreRecencyRecorded { family: family, bumped: count(hits) } + LocalStoreIndexLostRace => LocalStoreRecencyRaced { family: family } + LocalStoreIndexWriteRefused { refusal: r } => LocalStoreRecencyLost { family: family, refusal: r } + } + } + } + } +} + // --------------------------------------------------------------------------------------------- // Lookup and commit over an opened store. -fn local_store_lookup(root: MaterializationStoreRoot, store: OpenedMaterializationStore, req: ArtifactRequest) -> StoreLookup { +fn local_store_lookup(capability: LocalStoreCapability, req: ArtifactRequest) -> StoreLookup { + local_store_lookup_bound(binding: capability.binding, req: req) +} + +fn local_store_lookup_bound(binding: LocalStoreBinding, req: ArtifactRequest) -> StoreLookup + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_lookup"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_held"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_reserved"), + ] +{ + let root = binding.root + let store = binding.store let root_path = materialization_store_root_path(root: root) match materialization_store_admissible(root: root, verb: Read) { Deny { refusal: _ } => @@ -289,34 +1660,170 @@ fn local_store_lookup(root: MaterializationStoreRoot, store: OpenedMaterializati } } -fn local_store_commit( - root: MaterializationStoreRoot, - store: OpenedMaterializationStore, +// THE COMMIT RECEIPT carries what the bound did beside the store-level commit: the names this commit +// evicted, the eviction deletes the host did not confirm WITH their typed outcomes (those names stay +// doomed and CHARGED until a later reservation confirms them gone), the doomed names this commit's +// reservation retired, the index slot's own generation reclamation, the typed refusal when the +// occupancy index or the family hold stopped the commit (the store-level StoreCommitOccupancyUnavailable +// carries only its rendering, because std.materialization_object is realization-agnostic), and how the +// family hold was released. +type LocalStoreCommit { + commit: StoreCommit + evicted: List + eviction_delete_refused: List + retired: List + index_reclamation: CasWindowReclamation + index_refusal: LocalStoreIndexRefusal? + hold_refusal: LocalStoreHoldRefusal? + hold_release: LocalStoreHoldRelease? +} + +fn local_store_commit_refused(commit: StoreCommit) -> LocalStoreCommit { + LocalStoreCommit { commit: commit, evicted: [], eviction_delete_refused: [], retired: [], index_reclamation: CasReclamationNotCommitted, index_refusal: none, hold_refusal: none, hold_release: none } +} + +fn local_store_commit_index_refused(key: ContentHash, refusal: LocalStoreIndexRefusal) -> LocalStoreCommit { + LocalStoreCommit { + commit: StoreCommitRefused { key: key, refusal: StoreCommitOccupancyUnavailable { cause: local_store_index_refusal_text(refusal: refusal) } }, + evicted: [], eviction_delete_refused: [], retired: [], index_reclamation: CasReclamationNotCommitted, + index_refusal: Present { value: refusal }, hold_refusal: none, hold_release: none + } +} + +fn local_store_commit_hold_refused(key: ContentHash, refusal: LocalStoreHoldRefusal) -> LocalStoreCommit { + LocalStoreCommit { + commit: StoreCommitRefused { key: key, refusal: StoreCommitOccupancyUnavailable { cause: local_store_hold_refusal_text(refusal: refusal) } }, + evicted: [], eviction_delete_refused: [], retired: [], index_reclamation: CasReclamationNotCommitted, + index_refusal: none, hold_refusal: Present { value: refusal }, hold_release: none + } +} + +fn local_store_commit_released(c: LocalStoreCommit, settled_release: LocalStoreHoldRelease) -> LocalStoreCommit { + LocalStoreCommit { + commit: c.commit, evicted: c.evicted, eviction_delete_refused: c.eviction_delete_refused, retired: c.retired, + index_reclamation: c.index_reclamation, index_refusal: c.index_refusal, hold_refusal: c.hold_refusal, hold_release: Present { value: settled_release } + } +} + +// THE WRITE, INSIDE THE FAMILY HOLD: ask whether the exact object is already present (verified, absent, +// or unavailable -- the last refuses before any index write), reserve, delete what the reservation +// evicted, and ONLY IF THE HOST CONFIRMED EVERY ONE OF THOSE DELETES publish -- otherwise refuse as +// cleanup outstanding, leaving the evicted names doomed and charged, so on-disk bytes never exceed the +// budget because a delete did not happen. No other writer runs while this one holds the family, so +// what the reservation committed is still the index when the object appears. +fn local_store_commit_held( + held: LocalStoreFamilyHeld, req: ArtifactRequest, - payloads: List -) -> StoreCommit { + key: ContentHash, + name: String, + object: String, + offered: ContentHash, + offered_result: ResultContentIdentity +) -> LocalStoreCommit + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_under"), + ] +{ + let store = held.binding.store + let root = held.binding.root let root_path = materialization_store_root_path(root: root) - match store_commit_prepare(req: req, payloads: payloads) { - StoreCommitNotPrepared { commit: c } => c - StoreCommitPrepared { key: k, object_name: name, object: object, offered_digest: offered, offered_result: offered_result } => - match materialization_store_admissible(root: root, verb: Write) { - Deny { refusal: _ } => - StoreCommitRefused { - key: k, - refusal: StoreCommitPublishRefused { fault: StoreFault { class: StoreFaultUnauthorized, detail: concat("write grant refused for ", root_path) } } - } - Permit => { - let publish = store_publish_observation(create: local_store_create_new(path: store_path(root_path: root_path, name: name), content: object)) - store_commit_settle( + let presence = local_store_object_presence(lookup: local_store_lookup_bound(binding: held.binding, req: req)) + match presence { + LocalStoreObjectUnavailable { lookup: _ } => + local_store_commit_refused(commit: StoreCommitRefused { key: key, refusal: StoreCommitOccupancyUnavailable { cause: "the object's presence could not be established, so its row is neither converged on nor replaced" } }) + LocalStoreObjectVerified => local_store_commit_reserved(held: held, req: req, key: key, name: name, object: object, offered: offered, offered_result: offered_result, + reservation: local_store_reserve(held: held, name: name, bytes: local_store_object_bytes(object: object), presence: presence, attempt: 1)) + LocalStoreObjectAbsent => local_store_commit_reserved(held: held, req: req, key: key, name: name, object: object, offered: offered, offered_result: offered_result, + reservation: local_store_reserve(held: held, name: name, bytes: local_store_object_bytes(object: object), presence: presence, attempt: 1)) + } +} + +fn local_store_commit_reserved( + held: LocalStoreFamilyHeld, + req: ArtifactRequest, + key: ContentHash, + name: String, + object: String, + offered: ContentHash, + offered_result: ResultContentIdentity, + reservation: LocalStoreReservation +) -> LocalStoreCommit + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_commit_held"), + ] +{ + let store = held.binding.store + let root = held.binding.root + let root_path = materialization_store_root_path(root: root) + match reservation { + LocalStoreReservationRefused { refusal: r } => local_store_commit_refused(commit: StoreCommitRefused { key: key, refusal: r }) + LocalStoreReservationIndexRefused { refusal: r } => local_store_commit_index_refused(key: key, refusal: r) + LocalStoreReservationProviderRefused { construction: _ } => + local_store_commit_refused(commit: StoreCommitRefused { key: key, refusal: StoreCommitOccupancyUnavailable { cause: concat("the family's provider row does not construct a bounded store: ", request_kind(req: req) as String) } }) + LocalStoreReserved { admission: adm } => { + let budget = adm.budget + let evicted = adm.evicted + let retired = adm.retired + let rc = adm.reclamation + let split = local_store_delete_split(attempts: local_store_delete_attempts(authority: LocalStoreFamilyAuthority { held: held }, names: evicted |> map(e => e.name))) + let unconfirmed_set = local_store_name_set(names: split.refused |> map(t => t.name)) + if count(split.refused) > 0 { + LocalStoreCommit { + commit: StoreCommitRefused { key: key, refusal: StoreCommitCleanupOutstanding { artifact_bytes: local_store_object_bytes(object: object), budget: budget, undeleted: local_store_doomed_bytes(doomed: evicted |> filter(e => set_contains(unconfirmed_set, e.name))) } }, + evicted: evicted |> map(e => e.name), eviction_delete_refused: split.refused, retired: retired, index_reclamation: rc, + index_refusal: none, hold_refusal: none, hold_release: none + } + } else { + LocalStoreCommit { + commit: store_commit_settle( store: store, - prepared_key: k, + prepared_key: key, offered_digest: offered, offered_result: offered_result, - publish: publish, - read_back: local_store_lookup(root: root, store: store, req: req) - ) + publish: store_publish_observation(create: local_store_create_new(path: store_path(root_path: root_path, name: name), content: object)), + read_back: local_store_lookup_bound(binding: held.binding, req: req) + ), + evicted: evicted |> map(e => e.name), eviction_delete_refused: [], retired: retired, index_reclamation: rc, + index_refusal: none, hold_refusal: none, hold_release: none } } + } + } +} + +fn local_store_commit(capability: LocalStoreCapability, req: ArtifactRequest, payloads: List) -> LocalStoreCommit { + match local_store_with_family(capability: capability, of: req, run: h => local_store_commit_under(held: h, req: req, payloads: payloads)) { + LocalStoreBracketRefused { refusal: r } => local_store_commit_hold_refused(key: request_key(req: req), refusal: r) + LocalStoreBracketRan { value: written, hold_release: rel } => local_store_commit_released(c: written, settled_release: rel) + } +} + +// A COMMIT UNDER A GIVEN HOLD. The hold carries the store root and the family; nothing here takes either +// separately, so authority for one root or family cannot write another. The hold is re-observed first, +// so a value smuggled out of a finished bracket is refused as stale rather than used. +fn local_store_commit_under( + held: LocalStoreFamilyHeld, + req: ArtifactRequest, + payloads: List +) -> LocalStoreCommit { + let root = held.binding.root + let root_path = materialization_store_root_path(root: root) + if request_kind(req: req) != held.family { + local_store_commit_hold_refused(key: request_key(req: req), refusal: LocalStoreHoldWrongFamily { held: held.family, requested: request_kind(req: req) }) + } else { + match local_store_hold_check(held: held) { + LocalStoreHoldNotCurrent { observed: o } => local_store_commit_hold_refused(key: request_key(req: req), refusal: LocalStoreHoldStale { observed: o }) + LocalStoreHoldCurrent => + match store_commit_prepare(req: req, payloads: payloads) { + StoreCommitNotPrepared { commit: c } => local_store_commit_refused(commit: c) + StoreCommitPrepared { key: k, object_name: name, object: object, offered_digest: offered, offered_result: offered_result } => + match materialization_store_admissible(root: root, verb: Write) { + Deny { refusal: _ } => + local_store_commit_refused(commit: StoreCommitRefused { key: k, refusal: StoreCommitPublishRefused { fault: StoreFault { class: StoreFaultUnauthorized, detail: concat("write grant refused for ", root_path) } } }) + Permit => local_store_commit_held(held: held, req: req, key: k, name: name, object: object, offered: offered, offered_result: offered_result) + } + } + } } } @@ -325,20 +1832,18 @@ fn local_store_commit( // miss, and none is answered from anywhere else. type LocalStoreBatchLookup = LocalStoreBatchLooked { store: OpenedMaterializationStore, lookups: List } - | LocalStoreBatchUnavailable { root_path: String, cause: LocalStoreUnavailableCause } -fn local_store_lookup_batch(root: MaterializationStoreRoot, reqs: List) -> LocalStoreBatchLookup { - match local_store_open(root: root) { - LocalStoreUnavailable { root_path: p, cause: c } => LocalStoreBatchUnavailable { root_path: p, cause: c } - LocalStoreOpened { root: r, store: s, durability: _ } => - LocalStoreBatchLooked { store: s, lookups: reqs |> map(req => local_store_lookup(root: r, store: s, req: req)) } - } +fn local_store_lookup_batch(capability: LocalStoreCapability, reqs: List) -> LocalStoreBatchLookup { + LocalStoreBatchLooked { store: capability.binding.store, lookups: reqs |> map(req => local_store_lookup(capability: capability, req: req)) } } // --------------------------------------------------------------------------------------------- -// The cache-interface row this realization's receipts name. +// The cache-interface row this realization's receipts name, as a function of the ceiling it is deployed +// at: everything below is a fact of this realization except that one figure, which is policy and is +// applied by the deploying layer (gunbc.materialization_store_budgets materialization_store_local_facts). -data materialization_store_local_facts: CacheInterfaceCatalogFacts = { +fn materialization_store_local_facts_at(ceiling: ByteSize) -> CacheInterfaceCatalogFacts { + { identity: materialization_store_local_id backing_surface: "gunbc-materialization-store-persistent-host-volume" transport_encoding: CatalogCasContentEncoding @@ -358,7 +1863,7 @@ data materialization_store_local_facts: CacheInterfaceCatalogFacts = { placement: { value_shape: StructuredArtifact locality: PerHostFilesystem - retention: { release_policy: ReleasedNever, capacity: CapacityUnobserved } + retention: { release_policy: ReleasedNever, capacity: CapacityBounded { limit: ByteCapacity { limit: ExactLimit { value: ceiling } }, at_capacity: ReplaceExisting { strategy: LeastRecentlyUsed } } } atomicity: WriteThenCommit auth: FilesystemPerms read_latency: LocalDiskUs @@ -375,9 +1880,10 @@ data materialization_store_local_facts: CacheInterfaceCatalogFacts = { { field: "retention" evidence: OperatorObserved { - note: "no capacity ceiling or eviction is applied by this realization yet: retention and capacity are owned by std.artifact_store under the provider, and this row stays CapacityUnobserved until that accounting binds to the store" + note: "the host-wide ceiling is the deploying layer's figure (gunbc.materialization_store_budgets materialization_store_host_ceiling_bytes, the declared sum of materialization_store_durable_family_budgets), passed in as this function's ceiling; each family's store is constructed by std.artifact_store store_over_provider from its own CapacityBounded/ExactLimit/LeastRecentlyUsed provider row, so one family never evicts another's entries and an artifact over its family's budget refuses before any eviction. Occupancy is the family's windowed CAS index slot; recency is advisory and recorded once per process, and losing that write only makes eviction less precise. An open sweeps objects no index names." observed_at: Absent } }, ] + } } diff --git a/dag/gunbc/design_argument.dag b/dag/gunbc/design_argument.dag index 9213329cec1..626165a13f7 100644 --- a/dag/gunbc/design_argument.dag +++ b/dag/gunbc/design_argument.dag @@ -115,6 +115,7 @@ data conformance_domains: List = [ first: decl_ref(module_path: "std.temporal_effect", decl_name: "HeldLease"), rest: [ decl_ref(module_path: "std.durable_compare_and_set", decl_name: "CasExpectation"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "file_compare_and_set_windowed"), decl_ref(module_path: "std.durable_exclusive_hold", decl_name: "DurableHoldState"), decl_ref(module_path: "std.scoped_authorization", decl_name: "ScopedAuthorization"), decl_ref(module_path: "std.effect_grant", decl_name: "Grant"), @@ -132,13 +133,14 @@ data conformance_domains: List = [ tells: [ "a flag file, a sleep, a retry loop or a bare Bool standing where a lease with a declared deadline was available", "a write to a shared head that does not carry the expectation it was computed against", + "a compare-and-set slot that deletes its own generations other than through the windowed operations, or a slot whose generation history is its audit trail written through file_compare_and_set_windowed", "an effect performed without the grant that names who may perform it and how it is released", "a privileged effect whose token source is bound at the call site (GcloudPrintToken or OperatorSuppliedToken spelled in the entry) rather than resolved by gunbc.auth.access_token_source, with no row in gunbc.auth.privileged_effect_census stating why", "a PR body, message or brief that asks the operator to paste, forward or relay an access token; a GUNBC_GCP_ACCESS_TOKEN_FILE filled from chat; a token value crossing a session boundary by any channel other than the one its issuer issued it on", "a one-off or irreversible effect performed by a dispatched run under the federated identity with no approval request, or a recurring effect gated on a per-run phone tap", "a dashboard-only or checkout-only action performed by prose instruction with no std.human_intervention row naming the step" ], - narrative: "exclusive or bounded use and privileged access are one domain: leases with a declared deadline, compare-and-set expectations, scoped authorizations, grants naming who may perform an effect and how it is released, resource handles -- and, since 2026-09-19, which AUTHORIZATION PATTERN performs a privileged effect. That pattern is a §3d selection (gunbc.auth.authorization_pattern_selection consumes std.decision over the effect's frequency, reversibility, surface, workload-identity binding, minted reach and billing consequence): workload identity federation with a per-secret cell for recurring automated effects, one operator approval through the ntfy loop for one-off or witnessed effects, a named human-only step where no API exists, and a refusal for an operator token relayed by chat or between sessions. The census (gunbc.auth.privileged_effect_census) classifies every current site by executing that selection, so the reviewer's three-valued answer is derived for the sites that exist and asked of the site a change adds", + narrative: "exclusive or bounded use and privileged access are one domain: leases with a declared deadline, compare-and-set expectations, scoped authorizations, grants naming who may perform an effect and how it is released, resource handles -- and, since 2026-09-19, which AUTHORIZATION PATTERN performs a privileged effect. That pattern is a §3d selection (gunbc.auth.authorization_pattern_selection consumes std.decision over the effect's frequency, reversibility, surface, workload-identity binding, minted reach and billing consequence): workload identity federation with a per-secret cell for recurring automated effects, one operator approval through the ntfy loop for one-off or witnessed effects, a named human-only step where no API exists, and a refusal for an operator token relayed by chat or between sessions. The census (gunbc.auth.privileged_effect_census) classifies every current site by executing that selection, so the reviewer's three-valued answer is derived for the sites that exist and asked of the site a change adds. A compare-and-set slot's generation line is kept whole by file_compare_and_set and observe_cas_slot_state (gunbc.durable_cas_file_store), because several slots read it as an audit trail; a slot whose only fact is its head -- the materialization store's occupancy index and family holds are the first -- uses the distinct windowed operations (file_compare_and_set_windowed, observe_cas_slot_state_windowed, and the file_hold_*_windowed family) with a CasGenerationWindow of k >= 2, so a bounded store's own bookkeeping does not grow with its commits. They are two contracts, not one operation with a mode: keep-all callers are untouched, and only a caller that names the windowed operation can delete generations. A window read takes the head from a listing of the slot root and verifies head+1 is absent, re-reading on a race and never guessing; a write reclaims below head-k only after its own commit, and a refused delete is reported, never fatal. Exclusion is unchanged -- losers re-read and retry -- so nothing waits on a hold. docs/plans/fabric-storage.md names the head-probe bound this also lifts and assigns its remedy to compaction -- a head that starts from a snapshot; a window is that head-starting-above-1 shape for a slot whose history needs no snapshot, and the fabric heads are not opted in", }, ConformanceDomain { key: "conformance-compute", diff --git a/dag/gunbc/durable_cas_file_store.dag b/dag/gunbc/durable_cas_file_store.dag index a00fa7889ce..29889384b6d 100644 --- a/dag/gunbc/durable_cas_file_store.dag +++ b/dag/gunbc/durable_cas_file_store.dag @@ -306,6 +306,7 @@ type CasSlotProbe | ProbedHead { generation: CasGeneration, value: NonEmptyStr } | ProbedUnreadable { generation: CasGeneration, kind: FilesystemFailureKind } | ProbedReadKindUnrecognized { generation: CasGeneration, observed: String } + | ProbedWindowHeadUnsettled { attempts: Int } // One generation read, classified: present with its content, absent, or a typed refusal. type CasGenerationRead @@ -412,6 +413,163 @@ fn cas_observe_slot(root: NonEmptyStr, key: NonEmptyStr) -> CasSlotProbe { } } +// SLOT RETENTION IS DECLARED PER SLOT, AND THE DEFAULT IS TODAY'S BEHAVIOUR. +// +// Every slot above keeps its whole generation line, and several callers read that line as their +// audit trail (authorization claims, approval receipts and device redemption, the SCM repository +// slot, compute attempt lifecycles, secret-access admission, the fabric storage heads). Those keep +// KeepAllGenerations, which each declares at its own call site rather than inheriting it -- a +// default would let a future slot opt into deletion by omission. +// +// KeepGenerationWindow exists for a slot whose history is NOT evidence: a mutable index whose only +// fact is its head (extdeps.realization.materialization_store_local's occupancy index is the first). +// Its live generations are a contiguous WINDOW ending at the head; a writer, after its OWN commit, +// deletes the generations that fell out of it, so the slot's file count stays bounded by the window +// rather than by the number of commits. Because the window starts above 1, the gallop above cannot +// find its head; a window read takes the head from a listing of the slot root and then VERIFIES it +// -- the head must read present and head+1 absent -- and a reading that moved under it re-reads, up +// to a bound, and otherwise refuses. It never guesses a head. Exclusion is unchanged: commits are +// still create-new of head+1, so a losing writer still re-reads and retries and nothing waits on a +// hold (which is why this, and not gunbc.durable_exclusive_hold_file_store, bounds the index). +// +// docs/plans/fabric-storage.md names the head-probe bound this shape also lifts, and assigns its +// remedy to compaction -- a head that starts from a snapshot object. A window is that same +// head-starting-above-1 shape for a slot whose history needs no snapshot. The fabric storage heads +// are deliberately NOT opted in here: their history is their record, and that is their lane's call. +// +// k >= 2 is construction, not a check at use: CasGenerationWindow has one constructor, reached +// only through cas_generation_window, so a narrower window cannot be written. Two is the floor +// because a reader that listed head N while a writer committed N+1 must still find N present. +type CasGenerationWindow sole_constructor { + k: Int +} + +type CasGenerationWindowAdmission + = CasGenerationWindowAdmitted { window: CasGenerationWindow } + | CasGenerationWindowTooNarrow { k: Int } + +fn cas_generation_window(k: Int) -> CasGenerationWindowAdmission { + if k >= 2 { CasGenerationWindowAdmitted { window: CasGenerationWindow { k: k } } } else { CasGenerationWindowTooNarrow { k: k } } +} + +type CasSlotRetention + = KeepAllGenerations + | KeepGenerationWindow { window: CasGenerationWindow } + +// A window read that keeps observing a moving head stops here and refuses with the attempt count. +data cas_window_observation_attempt_bound: Int = 8 + +// "" -> the generation it names; none for anything else, for zero, and for a value past the +// Int maximum. The multiplication is pre-checked (std.checked_arithmetic: check before, never after). +fn cas_generation_of_digits(s: String) -> CasGeneration? { + if !cas_name_is_digits(s: s) { none } else { + let parsed = fold(s |> chars, init: Present { value: 0 }, f: (acc, c) => match acc { + Absent => none + Present { value: n } => + if n > (int_inclusive_max() - (c - 48)) / 10 { none } else { Present { value: n * 10 + (c - 48) } } + }) + match parsed { + Absent => none + Present { value: n } => if n >= 1 { Present { value: n } } else { none } + } + } +} + +// THIS key's generations named by a listing of the slot root. The listing is the same filesystem +// authority cas_first_generation_absent consults, and a root that cannot be listed carries its cause. +type CasWindowGenerations + = CasWindowGenerationsListed { generations: List } + | CasWindowGenerationsUnlisted { cause: String } + +fn cas_key_generation_of_name(key: NonEmptyStr, name: String) -> CasGeneration? { + let prefix = concat(key as String, ".") + if !starts_with(s: name, prefix: prefix) { none } else { + cas_generation_of_digits(s: substring(s: name, start: string_length(s: prefix), end: string_length(s: name))) + } +} + +fn cas_window_generations(root: NonEmptyStr, key: NonEmptyStr) -> CasWindowGenerations + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_settle_slot"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_observe_window"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_reclaim_below"), + ] +{ + let listing = Filesystem.List(path: root as String) + match filesystem_listing_observation(directory: root as String, success: listing.success, entries: listing.entries, error: listing.error) { + FilesystemDirectorySubjectRefused { directory: _, cause: c } => CasWindowGenerationsUnlisted { cause: c } + FilesystemDirectoryListingRefused { directory: _, error: e } => CasWindowGenerationsUnlisted { cause: e } + FilesystemDirectoryListed(listed) => + CasWindowGenerationsListed { + generations: fold(filesystem_listing_entry_names(listing: listed), init: [], f: (acc, n) => match cas_key_generation_of_name(key: key, name: trim(s: n)) { + Absent => acc + Present { value: g } => acc |> list_push(g) + }) + } + } +} + +fn cas_window_max_generation(generations: List) -> CasGeneration? { + fold(generations, init: none, f: (best, g) => match best { + Absent => Present { value: g } + Present { value: b } => if cas_generation_count(g: g) > cas_generation_count(g: b) { Present { value: g } } else { best } + }) +} + +fn cas_observe_window(root: NonEmptyStr, key: NonEmptyStr, attempt: Int) -> CasSlotProbe + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_observe_retained_slot"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_observe_window"), + ] +{ + if attempt > cas_window_observation_attempt_bound { + ProbedWindowHeadUnsettled { attempts: cas_window_observation_attempt_bound } + } else { + match cas_window_generations(root: root, key: key) { + CasWindowGenerationsUnlisted { cause: c } => ProbedAbsenceUnestablished { cause: c } + CasWindowGenerationsListed { generations: gs } => + match cas_window_max_generation(generations: gs) { + Absent => ProbedAbsent + Present { value: head } => + match cas_read_generation(root: root, key: key, generation: head) { + GenerationRefused { probe: p } => p + GenerationAbsent => cas_observe_window(root: root, key: key, attempt: attempt + 1) + GenerationPresent { content: c } => + match cas_generation_successor(g: head) { + CasSuccessorExhausted { head: _ } => ProbedHead { generation: head, value: c } + CasSuccessorGeneration { generation: next } => + match cas_read_generation(root: root, key: key, generation: next) { + GenerationRefused { probe: p } => p + GenerationAbsent => ProbedHead { generation: head, value: c } + GenerationPresent { content: _ } => cas_observe_window(root: root, key: key, attempt: attempt + 1) + } + } + } + } + } + } +} + +// THE ONE PLACE A SLOT'S RETENTION SELECTS ITS OBSERVATION. +fn cas_observe_retained_slot(root: NonEmptyStr, key: NonEmptyStr, retention: CasSlotRetention) -> CasSlotProbe + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "observe_cas_slot_state_retained"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_owner_only_failed_publication"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_compare_and_set_outcome"), + ] +{ + match retention { + KeepAllGenerations => cas_observe_slot(root: root, key: key) + KeepGenerationWindow { window: _ } => cas_observe_window(root: root, key: key, attempt: 1) + } +} + +fn cas_window_head_unsettled(attempts: Int) -> CasUnreadableSlot { + CasUnreadableReadRefused { + detail: concat(concat("the window head moved during ", attempts as String), " observations and was not settled; a head is never guessed") as NonEmptyStr + } +} + // THE PROJECTION TAKES A HEAD, NOT A PROBE, AND THE DELETED ARM IS THE REPAIR. // // An earlier revision was `cas_probe_as_readable(probe: CasSlotProbe)`, total over the probe, mapping @@ -469,6 +627,7 @@ fn cas_probe_as_observation(probe: CasSlotProbe) -> CasSlotObservation CasObservedReadable { readable: CasReadableAbsent } ProbedAbsenceUnestablished { cause: c } => CasObservedUnreadable { cause: cas_absence_unestablished(cause: c) } + ProbedWindowHeadUnsettled { attempts: a } => CasObservedUnreadable { cause: cas_window_head_unsettled(attempts: a) } ProbedHead { generation: h, value: v } => CasObservedReadable { readable: cas_head_as_readable(generation: h, value: v) } } @@ -495,14 +654,34 @@ fn cas_probe_as_observation(probe: CasSlotProbe) -> CasSlotObservation CasSlotObservation { +fn observe_cas_slot_state_retained(root: NonEmptyStr, key: NonEmptyStr, retention: CasSlotRetention) -> CasSlotObservation + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "observe_cas_slot_state"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "observe_cas_slot_state_windowed"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_commit_at"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "observe_file_hold_retained"), + ] +{ if !cas_key_is_slot_addressable(key: key) { CasObservedUnreadable { cause: CasUnreadableReadRefused { detail: cas_key_not_slot_addressable_detail } } } else { - cas_probe_as_observation(probe: cas_observe_slot(root: root, key: key)) + cas_probe_as_observation(probe: cas_observe_retained_slot(root: root, key: key, retention: retention)) } } +fn observe_cas_slot_state(root: NonEmptyStr, key: NonEmptyStr) -> CasSlotObservation { + observe_cas_slot_state_retained(root: root, key: key, retention: KeepAllGenerations) +} + +fn observe_cas_slot_state_windowed(root: NonEmptyStr, key: NonEmptyStr, window: CasGenerationWindow) -> CasSlotObservation + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_read_index"), + decl_ref(module_path: "test.claim.durable_cas_file_store_wet_witness", decl_name: "a_window_slot_holds_at_most_k_plus_one_generations_by_real_execution"), + ] +{ + observe_cas_slot_state_retained(root: root, key: key, retention: KeepGenerationWindow { window: window }) +} + data cas_key_not_slot_addressable_detail: NonEmptyStr = "key is not slot-addressable: it carries a path separator" @@ -597,6 +776,7 @@ type CasGenerationPublication type AdmittedCasPublication sole_constructor { root: NonEmptyStr publication: CasGenerationPublication + retention: CasSlotRetention attempt: CasAttempt target: CasGeneration } @@ -621,7 +801,7 @@ fn cas_commit_at(admitted: AdmittedCasPublication) -> CasOutcome { attempt: attempt, target: target, created: filesystem_create_new(path: write.path, success: write.success, error: write.error, error_kind: write.error_kind), - post: observe_cas_slot_state(root: root, key: attempt.key), + post: observe_cas_slot_state_retained(root: root, key: attempt.key, retention: admitted.retention), ) } DeclaredModeCreateOnly { mode } => { @@ -630,7 +810,7 @@ fn cas_commit_at(admitted: AdmittedCasPublication) -> CasOutcome { attempt: attempt, target: target, created: filesystem_create_new(path: write.path, success: write.success, error: write.error, error_kind: write.error_kind), - post: observe_cas_slot_state(root: root, key: attempt.key), + post: observe_cas_slot_state_retained(root: root, key: attempt.key, retention: admitted.retention), ) } OwnerOnlyCreateOnly => { @@ -640,7 +820,7 @@ fn cas_commit_at(admitted: AdmittedCasPublication) -> CasOutcome { committed: CasSlotVersion { generation: target, content: attempt.proposed_content, value: attempt.proposed } } } else { - cas_owner_only_failed_publication(root: root, attempt: attempt, target: target, detail: write.error) + cas_owner_only_failed_publication(root: root, retention: admitted.retention, attempt: attempt, target: target, detail: write.error) } } } @@ -718,11 +898,16 @@ fn cas_occupied_publication_outcome( // distinction, so this says what happened. fn cas_owner_only_failed_publication( root: NonEmptyStr, + retention: CasSlotRetention, attempt: CasAttempt, target: CasGeneration, detail: String, -) -> CasOutcome { - let probe = cas_observe_slot(root: root, key: attempt.key) +) -> CasOutcome + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "cas_commit_at"), + ] +{ + let probe = cas_observe_retained_slot(root: root, key: attempt.key, retention: retention) match probe { ProbedHead { generation: h, value: v } => if cas_generation_count(g: h) >= cas_generation_count(g: target) { @@ -741,6 +926,8 @@ fn cas_owner_only_failed_publication( CasStoreRefused { cause: CasSlotObservationRefused { cause: cas_generation_read_kind_unrecognized(generation: g, observed: o) } } + ProbedWindowHeadUnsettled { attempts: a } => + CasStoreRefused { cause: CasSlotObservationRefused { cause: cas_window_head_unsettled(attempts: a) } } } } @@ -757,13 +944,18 @@ fn cas_publication_refused(detail: String) -> CasOutcome { // EXPECTATION. That is the whole correction: an expectation is a claim about the // store, and deriving the write target from the claim rather than from the store // is what let an attempt expecting generation 7 commit against an empty slot. -fn file_compare_and_set( +fn cas_compare_and_set_outcome( root: NonEmptyStr, publication: CasGenerationPublication, + retention: CasSlotRetention, verified: VerifiedCasAttempt, -) -> CasOutcome { +) -> CasOutcome + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "file_compare_and_set_retained"), + ] +{ let attempt = verified.attempt - let probe = cas_observe_slot(root: root, key: attempt.key) + let probe = cas_observe_retained_slot(root: root, key: attempt.key, retention: retention) match probe { ProbedUnreadable { generation: g, kind: k } => CasStoreRefused { cause: CasSlotObservationRefused { cause: cas_generation_unreadable(generation: g, kind: k) } } @@ -773,12 +965,14 @@ fn file_compare_and_set( } ProbedAbsenceUnestablished { cause: c } => CasStoreRefused { cause: CasSlotObservationRefused { cause: cas_absence_unestablished(cause: c) } } + ProbedWindowHeadUnsettled { attempts: a } => + CasStoreRefused { cause: CasSlotObservationRefused { cause: cas_window_head_unsettled(attempts: a) } } ProbedAbsent => match attempt.expected { ExpectSlotAbsent => cas_commit_at( admitted: AdmittedCasPublication { - root: root, publication: publication, attempt: attempt, + root: root, publication: publication, retention: retention, attempt: attempt, target: cas_generation_first(), } ) @@ -796,7 +990,7 @@ fn file_compare_and_set( CasSuccessorGeneration { generation: next } => cas_commit_at( admitted: AdmittedCasPublication { - root: root, publication: publication, attempt: attempt, + root: root, publication: publication, retention: retention, attempt: attempt, target: next, } ) @@ -808,6 +1002,102 @@ fn file_compare_and_set( } } +// RECLAMATION FOLLOWS THE WRITER'S OWN COMMIT, AND IT CANNOT FAIL THE COMMIT. +// +// Only a KeepGenerationWindow slot reclaims, and only after this writer's CasCommitted: it deletes +// THIS key's generations at or below head-k, so head-k+1 .. head stay readable for a racing reader. +// A delete the host refuses is REPORTED in the write's receipt and counted there -- the commit has +// already happened and is the answer -- and the next committed write retries the same names, so a +// transient refusal does not grow the slot past the next successful reclamation. +// The host's own refusal text, carried verbatim: Filesystem.Delete answers success and error but no +// error kind, so the text is the most this receipt can say and it is never replaced by a summary. +type CasGenerationReclaimFailure { + generation: CasGeneration + host_error: String +} + +// One delete, observed once: the receipt is split from these, never re-derived by searching. +type CasGenerationDeleteAttempt { + generation: CasGeneration + success: Bool + error: String +} + +type CasWindowReclamation + = CasReclamationNotDeclared + | CasReclamationNotCommitted + | CasReclaimed { removed: List } + | CasReclamationIncomplete { removed: List, failed: List } + | CasReclamationUnlisted { cause: String } + +type CasFileWrite { + outcome: CasOutcome + reclamation: CasWindowReclamation +} + +fn cas_reclaim_below(root: NonEmptyStr, key: NonEmptyStr, window: CasGenerationWindow, head: CasGeneration) -> CasWindowReclamation + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "file_compare_and_set_retained"), + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_settle_slot"), + ] +{ + match cas_window_generations(root: root, key: key) { + CasWindowGenerationsUnlisted { cause: c } => CasReclamationUnlisted { cause: c } + CasWindowGenerationsListed { generations: gs } => { + let floor = cas_generation_count(g: head) - window.k + let attempts = gs |> filter(g => cas_generation_count(g: g) <= floor) |> map(g => { + let d = Filesystem.Delete(path: cas_file_slot_path(root: root, key: key, generation: g)) + CasGenerationDeleteAttempt { generation: g, success: d.success, error: d.error } + }) + let removed = attempts |> filter(t => t.success) |> map(t => t.generation) + let failed = attempts |> filter(t => !t.success) |> map(t => CasGenerationReclaimFailure { generation: t.generation, host_error: t.error }) + if count(failed) == 0 { CasReclaimed { removed: removed } } else { CasReclamationIncomplete { removed: removed, failed: failed } } + } + } +} + +// THE STORE-OWNED COMPARE-AND-SET, WITH THE SLOT'S DECLARED RETENTION. Every caller names its slot's +// retention here; KeepAllGenerations callers read .outcome and lose nothing, because their +// reclamation is CasReclamationNotDeclared by construction. +fn file_compare_and_set_retained( + root: NonEmptyStr, + publication: CasGenerationPublication, + retention: CasSlotRetention, + verified: VerifiedCasAttempt, +) -> CasFileWrite + admit_callers: [ + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "file_compare_and_set"), + decl_ref(module_path: "gunbc.durable_cas_file_store", decl_name: "file_compare_and_set_windowed"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_acquire_commit"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_release_commit"), + ] +{ + let outcome = cas_compare_and_set_outcome(root: root, publication: publication, retention: retention, verified: verified) + match retention { + KeepAllGenerations => CasFileWrite { outcome: outcome, reclamation: CasReclamationNotDeclared } + KeepGenerationWindow { window: w } => + match outcome { + CasCommitted { committed: v } => + CasFileWrite { outcome: outcome, reclamation: cas_reclaim_below(root: root, key: verified.attempt.key, window: w, head: v.generation) } + CasPreconditionFailed { expected: _, observed: _ } => CasFileWrite { outcome: outcome, reclamation: CasReclamationNotCommitted } + CasStoreRefused { cause: _ } => CasFileWrite { outcome: outcome, reclamation: CasReclamationNotCommitted } + } + } +} + +fn file_compare_and_set(root: NonEmptyStr, publication: CasGenerationPublication, verified: VerifiedCasAttempt) -> CasOutcome { + file_compare_and_set_retained(root: root, publication: publication, retention: KeepAllGenerations, verified: verified).outcome +} + +fn file_compare_and_set_windowed(root: NonEmptyStr, publication: CasGenerationPublication, verified: VerifiedCasAttempt, window: CasGenerationWindow) -> CasFileWrite + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_write_index_settled"), + decl_ref(module_path: "test.claim.durable_cas_file_store_wet_witness", decl_name: "run_cas_in"), + ] +{ + file_compare_and_set_retained(root: root, publication: publication, retention: KeepGenerationWindow { window: window }, verified: verified) +} + // ── Which slots exist ──────────────────────────────────────────────────────────────────────── // THE STORE OWNS BOTH DIRECTIONS OF ITS LAYOUT. cas_file_slot_path renders "/.", so // the inverse over a directory listing strips exactly one trailing "." -- the generation diff --git a/dag/gunbc/durable_exclusive_hold_file_store.dag b/dag/gunbc/durable_exclusive_hold_file_store.dag index 706722dc49f..fa5efa93c9d 100644 --- a/dag/gunbc/durable_exclusive_hold_file_store.dag +++ b/dag/gunbc/durable_exclusive_hold_file_store.dag @@ -42,7 +42,7 @@ import std.durable_exclusive_hold { DurableHoldHolderReport, DurableHoldRecoveryRefusal, HoldRecoveryEligible, HoldRecoveryRefused, durable_hold_recovery_assess, } -import gunbc.durable_cas_file_store { +import gunbc.durable_cas_file_store { KeepAllGenerations, CasSlotRetention, DefaultAccessCreateOnly, DerivedPayloadAdmitted, DerivedPayloadKeyNotSlotAddressable, admit_cas_attempt_for_derived_payload, file_compare_and_set, observe_cas_slot_state, } @@ -135,8 +135,32 @@ fn hold_observation_of_cas(observed: CasSlotObservation) -> Durable } } +// RETENTION IS THE SLOT'S DECLARATION, passed by its consumer (gunbc.durable_cas_file_store +// CasSlotRetention): a hold whose history is an audit trail keeps every generation; a hold taken and +// released on every commit of a bounded store keeps a window, or its own bookkeeping would grow +// without bound. Read, plan and commit carry the same declaration. +fn observe_file_hold_retained(root: NonEmptyStr, key: NonEmptyStr, retention: CasSlotRetention) -> DurableHoldObservation + admit_callers: [ + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "observe_file_hold"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "observe_file_hold_windowed"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_acquire_prepare_retained"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_release_assess_retained"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_recovery_assess_retained"), + ] +{ + hold_observation_of_cas(observed: observe_cas_slot_state_retained(root: root, key: key, retention: retention)) +} + fn observe_file_hold(root: NonEmptyStr, key: NonEmptyStr) -> DurableHoldObservation { - hold_observation_of_cas(observed: observe_cas_slot_state(root: root, key: key)) + observe_file_hold_retained(root: root, key: key, retention: KeepAllGenerations) +} + +fn observe_file_hold_windowed(root: NonEmptyStr, key: NonEmptyStr, window: CasGenerationWindow) -> DurableHoldObservation + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_hold_check"), + ] +{ + observe_file_hold_retained(root: root, key: key, retention: KeepGenerationWindow { window: window }) } // EVERY EFFECTFUL HOLD WRITE IS LIVE READ -> SEALED PLAN -> CAS COMMIT, AND NO @@ -165,6 +189,7 @@ type FileHoldAcquirePlan sole_constructor { owner: DurableHoldOwnerRef expected: CasExpectation held_payload: NonEmptyStr + retention: CasSlotRetention } type FileHoldAcquirePreparation @@ -173,12 +198,18 @@ type FileHoldAcquirePreparation // The live read and the protocol's judgement of it. Only an absent or a free // slot yields a plan; the expectation on the plan is derived from what was read. -fn file_hold_acquire_prepare( +fn file_hold_acquire_prepare_retained( root: NonEmptyStr, slot_key: NonEmptyStr, requested_owner: DurableHoldOwnerRef, -) -> FileHoldAcquirePreparation { - match durable_hold_acquire_assess(observed: observe_file_hold(root: root, key: slot_key)) { + retention: CasSlotRetention, +) -> FileHoldAcquirePreparation + admit_callers: [ + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_acquire_prepare"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_acquire_retained"), + ] +{ + match durable_hold_acquire_assess(observed: observe_file_hold_retained(root: root, key: slot_key, retention: retention)) { HoldAcquireRefused { refusal: r } => FileHoldAcquireNotEligible { slot_key: slot_key, refusal: r } HoldAcquireEligible { expected: e } => FileHoldAcquirePlanned { @@ -188,11 +219,16 @@ fn file_hold_acquire_prepare( owner: requested_owner, expected: e, held_payload: hold_slot_payload(state: DurableHoldHeld { owner: requested_owner }), + retention: retention, }, } } } +fn file_hold_acquire_prepare(root: NonEmptyStr, slot_key: NonEmptyStr, requested_owner: DurableHoldOwnerRef) -> FileHoldAcquirePreparation { + file_hold_acquire_prepare_retained(root: root, slot_key: slot_key, requested_owner: requested_owner, retention: KeepAllGenerations) +} + // EVERY WAY AN ACQUIRE ENDS, TYPED AND SEPARATE, because the remedies differ. // Occupied is the protocol's answer and retrying will not change it until the // holder releases; Lost is a race after this caller's own read and the remedy @@ -217,7 +253,7 @@ fn file_hold_acquire_commit(plan: FileHoldAcquirePlan) -> FileHoldAcquireOutcome match admit_cas_attempt_for_derived_payload(key: plan.slot_key, expected: plan.expected, proposed: plan.held_payload) { DerivedPayloadKeyNotSlotAddressable { key: _ } => FileHoldKeyNotSlotAddressable { slot_key: plan.slot_key } DerivedPayloadAdmitted { verified: v } => - match file_compare_and_set(root: plan.root, publication: DefaultAccessCreateOnly, verified: v) { + match file_compare_and_set_retained(root: plan.root, publication: DefaultAccessCreateOnly, retention: plan.retention, verified: v).outcome { CasCommitted { committed: c } => FileHoldAcquired { slot_key: plan.slot_key, owner: plan.owner, generation: c.generation } CasPreconditionFailed { expected: _, observed: _ } => FileHoldAcquireLost { slot_key: plan.slot_key } @@ -231,12 +267,18 @@ fn file_hold_acquire_commit(plan: FileHoldAcquirePlan) -> FileHoldAcquireOutcome // free, derives the expectation from that read, and writes. A caller cannot // present a generation, so it cannot overwrite a hold whose generation it // happens to know. -fn file_hold_acquire( +fn file_hold_acquire_retained( root: NonEmptyStr, slot_key: NonEmptyStr, requested_owner: DurableHoldOwnerRef, -) -> FileHoldAcquireOutcome { - match file_hold_acquire_prepare(root: root, slot_key: slot_key, requested_owner: requested_owner) { + retention: CasSlotRetention, +) -> FileHoldAcquireOutcome + admit_callers: [ + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_acquire"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_acquire_windowed"), + ] +{ + match file_hold_acquire_prepare_retained(root: root, slot_key: slot_key, requested_owner: requested_owner, retention: retention) { FileHoldAcquirePlanned { plan: p } => file_hold_acquire_commit(plan: p) FileHoldAcquireNotEligible { slot_key: _, refusal: r } => match r { @@ -250,6 +292,19 @@ fn file_hold_acquire( } } +fn file_hold_acquire(root: NonEmptyStr, slot_key: NonEmptyStr, requested_owner: DurableHoldOwnerRef) -> FileHoldAcquireOutcome { + file_hold_acquire_retained(root: root, slot_key: slot_key, requested_owner: requested_owner, retention: KeepAllGenerations) +} + +fn file_hold_acquire_windowed(root: NonEmptyStr, slot_key: NonEmptyStr, requested_owner: DurableHoldOwnerRef, window: CasGenerationWindow) -> FileHoldAcquireOutcome + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_acquire_settled"), + decl_ref(module_path: "test.claim.materialization_store_local_wet_witness", decl_name: "a_hold_left_by_a_dead_writer_is_recovered_by_observation_by_real_execution"), + ] +{ + file_hold_acquire_retained(root: root, slot_key: slot_key, requested_owner: requested_owner, retention: KeepGenerationWindow { window: window }) +} + // RELEASE IS ASSESS-THEN-COMMIT, NOT ONE INDIVISIBLE FREE, because a consumer // has settlement to do between the two: the fabric releases or settles money // and frees the cell only if the ledger advanced. So the assessment produces a @@ -263,6 +318,7 @@ type FileHoldReleasePlan sole_constructor { owner: DurableHoldOwnerRef generation: CasGeneration free_payload: NonEmptyStr + retention: CasSlotRetention } type FileHoldReleaseAssessment @@ -274,15 +330,21 @@ type FileHoldReleaseAssessment // such a mint would produce a plan that frees a real hold of B's at N, and the // plan being sole_constructor would then protect nothing. Every field on the // plan is copied from this live read. -fn file_hold_release_assess( +fn file_hold_release_assess_retained( root: NonEmptyStr, slot_key: NonEmptyStr, owner: DurableHoldOwnerRef, acquired_generation: CasGeneration, released_by: DurableHoldReleaseRef, -) -> FileHoldReleaseAssessment { + retention: CasSlotRetention, +) -> FileHoldReleaseAssessment + admit_callers: [ + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_release_assess"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_release_assess_windowed"), + ] +{ match durable_hold_release_assess( - observed: observe_file_hold(root: root, key: slot_key), + observed: observe_file_hold_retained(root: root, key: slot_key, retention: retention), owner: owner, acquired_generation: acquired_generation, ) { @@ -295,11 +357,24 @@ fn file_hold_release_assess( owner: owner, generation: g, free_payload: hold_slot_payload(state: DurableHoldFree { released_by: released_by }), + retention: retention, }, } } } +fn file_hold_release_assess(root: NonEmptyStr, slot_key: NonEmptyStr, owner: DurableHoldOwnerRef, acquired_generation: CasGeneration, released_by: DurableHoldReleaseRef) -> FileHoldReleaseAssessment { + file_hold_release_assess_retained(root: root, slot_key: slot_key, owner: owner, acquired_generation: acquired_generation, released_by: released_by, retention: KeepAllGenerations) +} + +fn file_hold_release_assess_windowed(root: NonEmptyStr, slot_key: NonEmptyStr, owner: DurableHoldOwnerRef, acquired_generation: CasGeneration, released_by: DurableHoldReleaseRef, window: CasGenerationWindow) -> FileHoldReleaseAssessment + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_release_family_hold"), + ] +{ + file_hold_release_assess_retained(root: root, slot_key: slot_key, owner: owner, acquired_generation: acquired_generation, released_by: released_by, retention: KeepGenerationWindow { window: window }) +} + // The commit still tells a lost race from an unavailable store. A release // assessed at generation N and committed after the slot moved is // FileHoldReleaseLost -- the hold this plan described no longer exists -- and @@ -319,7 +394,7 @@ fn file_hold_release_commit(plan: FileHoldReleasePlan) -> FileHoldReleaseOutcome ) { DerivedPayloadKeyNotSlotAddressable { key: _ } => FileHoldReleaseKeyNotSlotAddressable { slot_key: plan.slot_key } DerivedPayloadAdmitted { verified: v } => - match file_compare_and_set(root: plan.root, publication: DefaultAccessCreateOnly, verified: v) { + match file_compare_and_set_retained(root: plan.root, publication: DefaultAccessCreateOnly, retention: plan.retention, verified: v).outcome { CasCommitted { committed: c } => FileHoldReleased { slot_key: plan.slot_key, generation: c.generation } CasPreconditionFailed { expected: _, observed: _ } => FileHoldReleaseLost { slot_key: plan.slot_key } CasStoreRefused { cause: c } => FileHoldReleaseStoreUnavailable { slot_key: plan.slot_key, cause: c } @@ -341,17 +416,19 @@ type FileHoldRecoveryAssessment = FileHoldRecoveryPlanned { plan: FileHoldReleasePlan, dead_holder: DurableHoldOwnerRef, evidence: NonEmptyStr } | FileHoldRecoveryNotEligible { slot_key: NonEmptyStr, refusal: DurableHoldRecoveryRefusal } -fn file_hold_recovery_assess( +fn file_hold_recovery_assess_retained( root: NonEmptyStr, slot_key: NonEmptyStr, report: DurableHoldHolderReport, released_by: DurableHoldReleaseRef, + retention: CasSlotRetention, ) -> FileHoldRecoveryAssessment admit_callers: [ - decl_ref(module_path: "gunbc.managed_host_unit_hold", decl_name: "unit_hold_recover_dead_holder"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_recovery_assess"), + decl_ref(module_path: "gunbc.durable_exclusive_hold_file_store", decl_name: "file_hold_recovery_assess_windowed"), ] { - match durable_hold_recovery_assess(observed: observe_file_hold(root: root, key: slot_key), report: report) { + match durable_hold_recovery_assess(observed: observe_file_hold_retained(root: root, key: slot_key, retention: retention), report: report) { HoldRecoveryRefused { refusal: r } => FileHoldRecoveryNotEligible { slot_key: slot_key, refusal: r } HoldRecoveryEligible { generation: g, dead_holder: h, evidence: e } => FileHoldRecoveryPlanned { @@ -361,9 +438,26 @@ fn file_hold_recovery_assess( owner: h, generation: g, free_payload: hold_slot_payload(state: DurableHoldFree { released_by: released_by }), + retention: retention, }, dead_holder: h, evidence: e, } } } + +fn file_hold_recovery_assess(root: NonEmptyStr, slot_key: NonEmptyStr, report: DurableHoldHolderReport, released_by: DurableHoldReleaseRef) -> FileHoldRecoveryAssessment + admit_callers: [ + decl_ref(module_path: "gunbc.managed_host_unit_hold", decl_name: "unit_hold_recover_dead_holder"), + ] +{ + file_hold_recovery_assess_retained(root: root, slot_key: slot_key, report: report, released_by: released_by, retention: KeepAllGenerations) +} + +fn file_hold_recovery_assess_windowed(root: NonEmptyStr, slot_key: NonEmptyStr, report: DurableHoldHolderReport, released_by: DurableHoldReleaseRef, window: CasGenerationWindow) -> FileHoldRecoveryAssessment + admit_callers: [ + decl_ref(module_path: "extdeps.realization.materialization_store_local", decl_name: "local_store_hold_occupied"), + ] +{ + file_hold_recovery_assess_retained(root: root, slot_key: slot_key, report: report, released_by: released_by, retention: KeepGenerationWindow { window: window }) +} diff --git a/dag/gunbc/host/managed_host_unit_hold.dag b/dag/gunbc/host/managed_host_unit_hold.dag index 8ea7e6e1d95..72476f678fd 100644 --- a/dag/gunbc/host/managed_host_unit_hold.dag +++ b/dag/gunbc/host/managed_host_unit_hold.dag @@ -1,5 +1,8 @@ module gunbc.managed_host_unit_hold +import gunbc.process_hold_identity { + process_hold_text, process_hold_marker, process_hold_decode, text_at, self_process_identity, SelfIdentified, SelfUnidentified, holder_process_liveness, +} import std.types { Bool, Int, List, NonEmptyStr, String } import std.process { ProcessExit, ExitSuccess, ExitFailure, exit_failure } import std.durable_compare_and_set { CasGeneration } @@ -271,60 +274,13 @@ fn unit_hold_owner_tag(o: UnitHoldOwner) -> String { } } -// THE PROCESS PART OF A BOOT OWNER, rendered after the run id and decoded back here and nowhere else: -// `@boot=,pid=,start=,pidns=`. -data unit_hold_process_marker: String = "@boot=" - -fn unit_hold_process_text(p: ProcessIdentity, pid_namespace: NonEmptyStr) -> String { - join([unit_hold_process_marker, p.boot_id as String, ",pid=", to_string(p.pid), ",start=", p.start_time as String, ",pidns=", pid_namespace as String], "") -} - -type UnitHoldHolderProcess { - process: ProcessIdentity - pid_namespace: NonEmptyStr -} - -fn text_at(xs: List, i: Int) -> String { - match xs.skip(n: i).first() { - Present { value: v } => v - Absent => "" - } -} - -fn unit_hold_process_decode(text: String) -> UnitHoldHolderProcess? { - let parts = split(s: text, delimiter: ",") - if count(parts) != 4 { - none - } else { - let b = text_at(xs: parts, i: 0) - let pid_text = text_at(xs: parts, i: 1) - let start_text = text_at(xs: parts, i: 2) - let ns_text = text_at(xs: parts, i: 3) - if !starts_with(s: b, prefix: "boot=") || !starts_with(s: pid_text, prefix: "pid=") || !starts_with(s: start_text, prefix: "start=") || !starts_with(s: ns_text, prefix: "pidns=") { - none - } else { - let boot = substring(s: b, start: 5, end: string_length(b)) - let start = substring(s: start_text, start: 6, end: string_length(start_text)) - let ns = substring(s: ns_text, start: 6, end: string_length(ns_text)) - match parse_int(s: substring(s: pid_text, start: 4, end: string_length(pid_text))) { - Absent => none - Present { value: n } => - match checked_int_to_nat(n: n) { - Absent => none - Present { value: pid } => - if boot == "" || start == "" || ns == "" { none } else { Present { value: UnitHoldHolderProcess { process: ProcessIdentity { boot_id: boot as NonEmptyStr, pid: pid, start_time: start as NonEmptyStr }, pid_namespace: ns as NonEmptyStr } } } - } - } - } - } -} fn unit_hold_owner_detail(o: UnitHoldOwner) -> String { match o { OperatorMaintenance { reason: r } => r as String - BootRun { run_id: r, process: p, pid_namespace: n } => concat(r as String, unit_hold_process_text(p: p, pid_namespace: n)) + BootRun { run_id: r, process: p, pid_namespace: n } => concat(r as String, process_hold_text(p: p, pid_namespace: n)) HostResetReturn { attempt: a } => a as String - KvmObserverProbe { run_id: r, process: p, pid_namespace: n } => concat(r as String, unit_hold_process_text(p: p, pid_namespace: n)) + KvmObserverProbe { run_id: r, process: p, pid_namespace: n } => concat(r as String, process_hold_text(p: p, pid_namespace: n)) } } @@ -364,11 +320,11 @@ fn decode_unit_hold_owner(owner: DurableHoldOwnerRef) -> UnitHoldOwnerDecode { match unit_hold_owner_detail_after(raw: raw, tag: boot) { Absent => UnitHoldOwnerUnrecognized { raw: raw } Present { value: d } => { - let halves = split(s: d as String, delimiter: unit_hold_process_marker) + let halves = split(s: d as String, delimiter: process_hold_marker) if count(halves) != 2 || text_at(xs: halves, i: 0) == "" { UnitHoldOwnerUnrecognized { raw: raw } } else { - match unit_hold_process_decode(text: concat("boot=", text_at(xs: halves, i: 1))) { + match process_hold_decode(text: concat("boot=", text_at(xs: halves, i: 1))) { Present { value: h } => UnitHoldOwnerDecoded { owner: BootRun { run_id: text_at(xs: halves, i: 0) as NonEmptyStr, process: h.process, pid_namespace: h.pid_namespace } } Absent => UnitHoldOwnerUnrecognized { raw: raw } } @@ -379,11 +335,11 @@ fn decode_unit_hold_owner(owner: DurableHoldOwnerRef) -> UnitHoldOwnerDecode { match unit_hold_owner_detail_after(raw: raw, tag: probe) { Absent => UnitHoldOwnerUnrecognized { raw: raw } Present { value: d } => { - let halves = split(s: d as String, delimiter: unit_hold_process_marker) + let halves = split(s: d as String, delimiter: process_hold_marker) if count(halves) != 2 || text_at(xs: halves, i: 0) == "" { UnitHoldOwnerUnrecognized { raw: raw } } else { - match unit_hold_process_decode(text: concat("boot=", text_at(xs: halves, i: 1))) { + match process_hold_decode(text: concat("boot=", text_at(xs: halves, i: 1))) { Present { value: h } => UnitHoldOwnerDecoded { owner: KvmObserverProbe { run_id: text_at(xs: halves, i: 0) as NonEmptyStr, process: h.process, pid_namespace: h.pid_namespace } } Absent => UnitHoldOwnerUnrecognized { raw: raw } } @@ -449,152 +405,6 @@ fn unit_hold_refusal(outcome: FileHoldAcquireOutcome) -> NonEmptyStr? { } } -// ── WHETHER A HOLDING PROCESS STILL RUNS, OBSERVED ───────────────────────────────────────────── -// The store and every holder that can be recovered run on one host (the admitted subject's -// store_host; each hold-owning root refuses any other), so the holder's process is observable through this host's -// procfs. Three reads decide it, each typed by the host's own error kind: -// - /proc/sys/kernel/random/boot_id differs from the holder's boot -> the host rebooted since, so -// every process of that boot is gone: OBSERVED DEAD; -// - same boot, /proc//stat is ABSENT (the host answered NotFound) -> no such process: DEAD; -// - same boot, the stat reads and its starttime differs -> the pid was reused: DEAD; -// - same boot, same starttime -> the holder runs: LIVE. -// Any read the host refused for another reason, and any content that does not parse, is -// UNOBSERVABLE: the holder may well be alive, and nothing here concludes otherwise from a failed look. -data proc_boot_id_path: String = "/proc/sys/kernel/random/boot_id" - -data proc_self_stat_path: String = "/proc/self/stat" - -// THE READER'S PID NAMESPACE (namespaces(7), pid_namespaces(7)): /proc/self/ns/pid is a symbolic link -// whose target, `pid:[]`, identifies the namespace; two processes share a pid namespace exactly -// when the targets are equal. It is read with readlink(1), because the link is not a readable file. -data proc_self_pid_namespace_path: String = "/proc/self/ns/pid" - -type PidNamespaceReading - = PidNamespaceRead { namespace: NonEmptyStr } - | PidNamespaceUnread { cause: NonEmptyStr } - -fn own_pid_namespace() -> PidNamespaceReading { - match run_shell_command_capture(command: readlink_command(path: proc_self_pid_namespace_path), transport: LocalExec) { - ProcessOutputPresent { text: t } => - if trim(s: t) == "" { PidNamespaceUnread { cause: "readlink /proc/self/ns/pid printed nothing" as NonEmptyStr } } - else { PidNamespaceRead { namespace: trim(s: t) as NonEmptyStr } } - ProcessOutputAbsent => PidNamespaceUnread { cause: "readlink /proc/self/ns/pid printed nothing" as NonEmptyStr } - ProcessRefused { exit_code: c, stderr: e } => PidNamespaceUnread { cause: join(["readlink /proc/self/ns/pid exited ", to_string(c), ": ", e], "") as NonEmptyStr } - } -} - -fn exact_read_of(path: String) -> FilesystemExactRead { - let read = Filesystem.Read(path: path) - filesystem_exact_read(path: path, content: read.content, success: read.success, error: read.error, error_kind: read.error_kind) -} - -type ExactText - = ExactTextRead { text: String } - | ExactTextAbsent - | ExactTextUnread { cause: NonEmptyStr } - -fn exact_text(path: String) -> ExactText { - match exact_read_of(path: path) { - FilesystemExactPathRead { path: _, content: c } => ExactTextRead { text: c } - FilesystemExactPathAbsent { path: _ } => ExactTextAbsent - FilesystemExactPathUnreadable { path: _, kind: _, error: e } => ExactTextUnread { cause: join([path, " could not be read: ", e], "") as NonEmptyStr } - FilesystemExactPathKindUnrecognized { path: _, observed: o, error: e } => ExactTextUnread { cause: join([path, " failed with an unrecognized error kind ", o, ": ", e], "") as NonEmptyStr } - } -} - -fn start_time_text(stat: String) -> String? { - match proc_pid_stat_start_time(content: stat) { - StartTimeObserved { ticks_since_boot: t } => Present { value: to_string(t) } - _ => none - } -} - -// THE NAMESPACE GATE COMES FIRST: only a reader in the holder's own pid namespace can conclude -// anything from a pid, so a reader elsewhere -- or one that cannot name its namespace -- cannot look. -fn holder_process_liveness(p: ProcessIdentity, pid_namespace: NonEmptyStr) -> DurableHoldHolderLiveness { - match own_pid_namespace() { - PidNamespaceUnread { cause: c } => HolderLivenessUnobservable { cause: c } - PidNamespaceRead { namespace: mine } => - if (mine as String) != (pid_namespace as String) { - HolderLivenessUnobservable { cause: join(["the holder's pid namespace is ", pid_namespace as String, " and this reader's is ", mine as String, "; its pid cannot be looked up from here"], "") as NonEmptyStr } - } else { - holder_process_liveness_in_namespace(p: p) - } - } -} - -fn holder_process_liveness_in_namespace(p: ProcessIdentity) -> DurableHoldHolderLiveness { - match exact_text(path: proc_boot_id_path) { - ExactTextAbsent => HolderLivenessUnobservable { cause: concat(proc_boot_id_path, " is absent on this host") as NonEmptyStr } - ExactTextUnread { cause: c } => HolderLivenessUnobservable { cause: c } - ExactTextRead { text: t } => { - let now = trim(s: t) - if now == "" { - HolderLivenessUnobservable { cause: concat(proc_boot_id_path, " was empty") as NonEmptyStr } - } else if now != (p.boot_id as String) { - HolderObservedDead { evidence: join(["the host has booted since the holder started (boot ", p.boot_id as String, ", now ", now, ")"], "") as NonEmptyStr } - } else { - let stat_path = join(["/proc/", to_string(p.pid), "/stat"], "") - match exact_text(path: stat_path) { - ExactTextAbsent => HolderObservedDead { evidence: join(["no process ", to_string(p.pid), " on the holder's boot (", stat_path, " not found)"], "") as NonEmptyStr } - ExactTextUnread { cause: c } => HolderLivenessUnobservable { cause: c } - ExactTextRead { text: stat } => - match start_time_text(stat: stat) { - Absent => HolderLivenessUnobservable { cause: concat(stat_path, " carried no readable starttime") as NonEmptyStr } - Present { value: st } => - if st == (p.start_time as String) { - HolderObservedLive { evidence: join(["process ", to_string(p.pid), " started at tick ", st, " still runs"], "") as NonEmptyStr } - } else { - HolderObservedDead { evidence: join(["pid ", to_string(p.pid), " now names another process (start tick ", st, ", holder's ", p.start_time as String, ")"], "") as NonEmptyStr } - } - } - } - } - } - } -} - -// THIS PROCESS'S OWN IDENTITY, read from the same procfs a successor will read it back through: the -// boot id, and pid and starttime from ONE read of /proc/self/stat. A boot that cannot name itself does -// not take the unit -- a hold it could never be shown dead in is the lockout this replaces. -type SelfProcessIdentity - = SelfIdentified { process: ProcessIdentity, pid_namespace: NonEmptyStr } - | SelfUnidentified { cause: NonEmptyStr } - -fn self_process_identity() -> SelfProcessIdentity { - match exact_text(path: proc_boot_id_path) { - ExactTextAbsent => SelfUnidentified { cause: concat(proc_boot_id_path, " is absent on this host") as NonEmptyStr } - ExactTextUnread { cause: c } => SelfUnidentified { cause: c } - ExactTextRead { text: b } => { - let boot = trim(s: b) - match exact_text(path: proc_self_stat_path) { - ExactTextAbsent => SelfUnidentified { cause: concat(proc_self_stat_path, " is absent on this host") as NonEmptyStr } - ExactTextUnread { cause: c } => SelfUnidentified { cause: c } - ExactTextRead { text: stat } => - match proc_pid_stat_pid(content: stat) { - PidUnparseable { field: f } => SelfUnidentified { cause: concat("/proc/self/stat carried no pid: ", f) as NonEmptyStr } - PidObserved { pid: n } => - match start_time_text(stat: stat) { - Absent => SelfUnidentified { cause: "/proc/self/stat carried no readable starttime" as NonEmptyStr } - Present { value: st } => - match checked_int_to_nat(n: n) { - Absent => SelfUnidentified { cause: "/proc/self/stat pid is not a natural number" as NonEmptyStr } - Present { value: pid } => - if boot == "" { SelfUnidentified { cause: concat(proc_boot_id_path, " was empty") as NonEmptyStr } } - else { - match own_pid_namespace() { - PidNamespaceUnread { cause: c } => SelfUnidentified { cause: c } - PidNamespaceRead { namespace: n } => SelfIdentified { process: ProcessIdentity { boot_id: boot as NonEmptyStr, pid: pid, start_time: st as NonEmptyStr }, pid_namespace: n } - } - } - } - } - } - } - } - } -} - // ── RECOVERING A HOLD WHOSE HOLDER WAS OBSERVED DEAD ─────────────────────────────────────────── // Only a holder whose owner names a process can be observed, so only a decoded BootRun or // KvmObserverProbe is looked at; diff --git a/dag/gunbc/materialization_store_budgets.dag b/dag/gunbc/materialization_store_budgets.dag new file mode 100644 index 00000000000..4972a14f844 --- /dev/null +++ b/dag/gunbc/materialization_store_budgets.dag @@ -0,0 +1,36 @@ +module gunbc.materialization_store_budgets + +import std.types { List, NonEmptyStr } +import std.dissolution { DissolutionCondition, unbound_dissolution } +import std.measure { ByteSize, byte_size } +import std.cache_identity { typed_module_artifact_kind } +import extdeps.realization.materialization_store_local { LocalStoreFamilyBudget, local_store_budget_sum, materialization_store_local_facts_at } +import extdeps.cache.types { CacheInterfaceCatalogFacts } + +// THE DURABLE STORE'S BUDGETS ARE POLICY, AND POLICY IS THIS LAYER'S FACT (DESIGN section 3). The +// transport (extdeps.realization.materialization_store_local) owns the budget's shape and its +// enforcement and receives these rows as a parameter; the per-host figures live here, beside the +// consumer that opens the durable root, so revising one never edits the transport. One root per host +// is structural (std.materialization_store_grant DurableHostVolumeRoot names one path), so the host +// ceiling is the declared sum of these rows and nothing else. + +// POLICY, NAMED AS POLICY: 4 GiB for typed-module results per host. It is not derived from a +// measurement -- the only measured predecessor stored closure snapshots of ~3.65 GB each, which is not +// the per-module grain the store holds -- and its REVISION TRIGGER is the TypecheckModuleRequest +// consumer (PR C2) measuring real per-module entries on the durable root. +data materialization_store_typed_module_budget_policy: ByteSize = byte_size(count: 4294967296) + +data materialization_store_durable_family_budgets: List = [ + LocalStoreFamilyBudget { family: typed_module_artifact_kind, budget: materialization_store_typed_module_budget_policy }, +] + +// The one host-wide figure: the sum of the durable root's declared family budgets. +data materialization_store_host_ceiling_bytes: ByteSize = local_store_budget_sum(budgets: materialization_store_durable_family_budgets) + +// The catalog row for the store as DEPLOYED: the realization's facts at this layer's host ceiling. +data materialization_store_local_facts: CacheInterfaceCatalogFacts = materialization_store_local_facts_at(ceiling: materialization_store_host_ceiling_bytes) + +// THE PRODUCTION CONSUMER IS A DECLARED FRONTIER (DESIGN section 3c): no process opens the durable +// root in this change; the witnesses pass fixture rows of their own, and the host-ceiling control +// reads the figure above. +data materialization_store_durable_budgets_consumer_frontier: DissolutionCondition = unbound_dissolution(description: "TRIGGER: the TypecheckModuleRequest consumer (PR C2) opens DurableHostVolumeRoot through extdeps.realization.materialization_store_local passing materialization_store_durable_family_budgets. SUFFICIENT FOR: these rows bound a production store on every run that consumes it; until then they are read only by the host-ceiling control, and this row is retired by that landing and nothing else." as NonEmptyStr) diff --git a/dag/gunbc/non_fold_residue.dag b/dag/gunbc/non_fold_residue.dag index 054b809d145..87eb7688fe0 100644 --- a/dag/gunbc/non_fold_residue.dag +++ b/dag/gunbc/non_fold_residue.dag @@ -808,7 +808,6 @@ data non_fold_residue_frontier: List = [ FrontierRow { subject: PathSubject { path: "dag/gunbc/harness/harness_wire.dag::harness_member_int" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, FrontierRow { subject: PathSubject { path: "dag/gunbc/harness/harness_wire.dag::harness_member_string" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, FrontierRow { subject: PathSubject { path: "dag/gunbc/harness/harness_wire.dag::harness_usage_read" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, - FrontierRow { subject: PathSubject { path: "dag/gunbc/host/managed_host_unit_hold.dag::start_time_text" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, FrontierRow { subject: PathSubject { path: "dag/gunbc/host/verified_archive_install.dag::stage_archive_wet" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, FrontierRow { subject: PathSubject { path: "dag/gunbc/host/verified_archive_install.dag::verify_archive_digest" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, FrontierRow { subject: PathSubject { path: "dag/gunbc/instruments/approval_store_live_probe.dag::approval_submission_live_probe" }, reason: nfr_reason_typed_census_ca5ed1724b, dissolution: nfr_dissolve_owning_fold }, diff --git a/dag/gunbc/process_hold_identity.dag b/dag/gunbc/process_hold_identity.dag new file mode 100644 index 00000000000..5aed7587314 --- /dev/null +++ b/dag/gunbc/process_hold_identity.dag @@ -0,0 +1,221 @@ +module gunbc.process_hold_identity + +import std.types { Bool, Int, List, NonEmptyStr, String } +import v2.std.optional { Present, Absent } +import std.checked_arithmetic { checked_int_to_nat } +import std.algebra { trim } +import std.durable_exclusive_hold { DurableHoldHolderLiveness, HolderObservedLive, HolderObservedDead, HolderLivenessUnobservable } +import gunbc.build_cache_instance { ProcessIdentity } +import gunbc.command_runner { run_shell_command_capture } +import extdeps.exec.command { LocalExec } +import extdeps.shell.exec { ProcessOutcome, ProcessOutputPresent, ProcessOutputAbsent, ProcessRefused } +import extdeps.tools.gnu_coreutils { readlink_command } +import extdeps.linux.proc_pid_stat { + proc_pid_stat_start_time, StartTimeObserved, StartTimeCommUnterminated, StartTimeFieldAbsent, StartTimeUnparseable, + proc_pid_stat_pid, PidObserved, PidUnparseable, +} +import extdeps.filesystem.filesystem_io { + Filesystem, filesystem_exact_read, FilesystemExactRead, FilesystemExactPathRead, FilesystemExactPathAbsent, + FilesystemExactPathUnreadable, FilesystemExactPathKindUnrecognized, +} + +// A HOLDER THAT IS A PROCESS ON THIS HOST: its identity as this process reads it, its rendering into a +// hold owner, and whether it still runs -- observed, never aged. Moved MECHANICALLY out of +// gunbc.managed_host_unit_hold when the materialization store became a second holder of the same +// kind; only the codec's unit_hold_ prefix became process_hold_, and the bodies are unchanged. + +// THE PROCESS PART OF A BOOT OWNER, rendered after the run id and decoded back here and nowhere else: +// `@boot=,pid=,start=,pidns=`. +data process_hold_marker: String = "@boot=" + +fn process_hold_text(p: ProcessIdentity, pid_namespace: NonEmptyStr) -> String { + join([process_hold_marker, p.boot_id as String, ",pid=", to_string(p.pid), ",start=", p.start_time as String, ",pidns=", pid_namespace as String], "") +} + +type ProcessHoldHolder { + process: ProcessIdentity + pid_namespace: NonEmptyStr +} + +fn text_at(xs: List, i: Int) -> String { + match xs.skip(n: i).first() { + Present { value: v } => v + Absent => "" + } +} + +fn process_hold_decode(text: String) -> ProcessHoldHolder? { + let parts = split(s: text, delimiter: ",") + if count(parts) != 4 { + none + } else { + let b = text_at(xs: parts, i: 0) + let pid_text = text_at(xs: parts, i: 1) + let start_text = text_at(xs: parts, i: 2) + let ns_text = text_at(xs: parts, i: 3) + if !starts_with(s: b, prefix: "boot=") || !starts_with(s: pid_text, prefix: "pid=") || !starts_with(s: start_text, prefix: "start=") || !starts_with(s: ns_text, prefix: "pidns=") { + none + } else { + let boot = substring(s: b, start: 5, end: string_length(b)) + let start = substring(s: start_text, start: 6, end: string_length(start_text)) + let ns = substring(s: ns_text, start: 6, end: string_length(ns_text)) + match parse_int(s: substring(s: pid_text, start: 4, end: string_length(pid_text))) { + Absent => none + Present { value: n } => + match checked_int_to_nat(n: n) { + Absent => none + Present { value: pid } => + if boot == "" || start == "" || ns == "" { none } else { Present { value: ProcessHoldHolder { process: ProcessIdentity { boot_id: boot as NonEmptyStr, pid: pid, start_time: start as NonEmptyStr }, pid_namespace: ns as NonEmptyStr } } } + } + } + } + } +} + +// ── WHETHER A HOLDING PROCESS STILL RUNS, OBSERVED ───────────────────────────────────────────── +// The store and every holder that can be recovered run on one host (the admitted subject's +// store_host; each hold-owning root refuses any other), so the holder's process is observable through this host's +// procfs. Three reads decide it, each typed by the host's own error kind: +// - /proc/sys/kernel/random/boot_id differs from the holder's boot -> the host rebooted since, so +// every process of that boot is gone: OBSERVED DEAD; +// - same boot, /proc//stat is ABSENT (the host answered NotFound) -> no such process: DEAD; +// - same boot, the stat reads and its starttime differs -> the pid was reused: DEAD; +// - same boot, same starttime -> the holder runs: LIVE. +// Any read the host refused for another reason, and any content that does not parse, is +// UNOBSERVABLE: the holder may well be alive, and nothing here concludes otherwise from a failed look. +data proc_boot_id_path: String = "/proc/sys/kernel/random/boot_id" + +data proc_self_stat_path: String = "/proc/self/stat" + +// THE READER'S PID NAMESPACE (namespaces(7), pid_namespaces(7)): /proc/self/ns/pid is a symbolic link +// whose target, `pid:[]`, identifies the namespace; two processes share a pid namespace exactly +// when the targets are equal. It is read with readlink(1), because the link is not a readable file. +data proc_self_pid_namespace_path: String = "/proc/self/ns/pid" + +type PidNamespaceReading + = PidNamespaceRead { namespace: NonEmptyStr } + | PidNamespaceUnread { cause: NonEmptyStr } + +fn own_pid_namespace() -> PidNamespaceReading { + match run_shell_command_capture(command: readlink_command(path: proc_self_pid_namespace_path), transport: LocalExec) { + ProcessOutputPresent { text: t } => + if trim(s: t) == "" { PidNamespaceUnread { cause: "readlink /proc/self/ns/pid printed nothing" as NonEmptyStr } } + else { PidNamespaceRead { namespace: trim(s: t) as NonEmptyStr } } + ProcessOutputAbsent => PidNamespaceUnread { cause: "readlink /proc/self/ns/pid printed nothing" as NonEmptyStr } + ProcessRefused { exit_code: c, stderr: e } => PidNamespaceUnread { cause: join(["readlink /proc/self/ns/pid exited ", to_string(c), ": ", e], "") as NonEmptyStr } + } +} + +fn exact_read_of(path: String) -> FilesystemExactRead { + let read = Filesystem.Read(path: path) + filesystem_exact_read(path: path, content: read.content, success: read.success, error: read.error, error_kind: read.error_kind) +} + +type ExactText + = ExactTextRead { text: String } + | ExactTextAbsent + | ExactTextUnread { cause: NonEmptyStr } + +fn exact_text(path: String) -> ExactText { + match exact_read_of(path: path) { + FilesystemExactPathRead { path: _, content: c } => ExactTextRead { text: c } + FilesystemExactPathAbsent { path: _ } => ExactTextAbsent + FilesystemExactPathUnreadable { path: _, kind: _, error: e } => ExactTextUnread { cause: join([path, " could not be read: ", e], "") as NonEmptyStr } + FilesystemExactPathKindUnrecognized { path: _, observed: o, error: e } => ExactTextUnread { cause: join([path, " failed with an unrecognized error kind ", o, ": ", e], "") as NonEmptyStr } + } +} + +fn start_time_text(stat: String) -> String? { + match proc_pid_stat_start_time(content: stat) { + StartTimeObserved { ticks_since_boot: t } => Present { value: to_string(t) } + StartTimeCommUnterminated => none + StartTimeFieldAbsent { fields_after_comm: _ } => none + StartTimeUnparseable { field: _ } => none + } +} + +// THE NAMESPACE GATE COMES FIRST: only a reader in the holder's own pid namespace can conclude +// anything from a pid, so a reader elsewhere -- or one that cannot name its namespace -- cannot look. +fn holder_process_liveness(p: ProcessIdentity, pid_namespace: NonEmptyStr) -> DurableHoldHolderLiveness { + match own_pid_namespace() { + PidNamespaceUnread { cause: c } => HolderLivenessUnobservable { cause: c } + PidNamespaceRead { namespace: mine } => + if (mine as String) != (pid_namespace as String) { + HolderLivenessUnobservable { cause: join(["the holder's pid namespace is ", pid_namespace as String, " and this reader's is ", mine as String, "; its pid cannot be looked up from here"], "") as NonEmptyStr } + } else { + holder_process_liveness_in_namespace(p: p) + } + } +} + +fn holder_process_liveness_in_namespace(p: ProcessIdentity) -> DurableHoldHolderLiveness { + match exact_text(path: proc_boot_id_path) { + ExactTextAbsent => HolderLivenessUnobservable { cause: concat(proc_boot_id_path, " is absent on this host") as NonEmptyStr } + ExactTextUnread { cause: c } => HolderLivenessUnobservable { cause: c } + ExactTextRead { text: t } => { + let now = trim(s: t) + if now == "" { + HolderLivenessUnobservable { cause: concat(proc_boot_id_path, " was empty") as NonEmptyStr } + } else if now != (p.boot_id as String) { + HolderObservedDead { evidence: join(["the host has booted since the holder started (boot ", p.boot_id as String, ", now ", now, ")"], "") as NonEmptyStr } + } else { + let stat_path = join(["/proc/", to_string(p.pid), "/stat"], "") + match exact_text(path: stat_path) { + ExactTextAbsent => HolderObservedDead { evidence: join(["no process ", to_string(p.pid), " on the holder's boot (", stat_path, " not found)"], "") as NonEmptyStr } + ExactTextUnread { cause: c } => HolderLivenessUnobservable { cause: c } + ExactTextRead { text: stat } => + match start_time_text(stat: stat) { + Absent => HolderLivenessUnobservable { cause: concat(stat_path, " carried no readable starttime") as NonEmptyStr } + Present { value: st } => + if st == (p.start_time as String) { + HolderObservedLive { evidence: join(["process ", to_string(p.pid), " started at tick ", st, " still runs"], "") as NonEmptyStr } + } else { + HolderObservedDead { evidence: join(["pid ", to_string(p.pid), " now names another process (start tick ", st, ", holder's ", p.start_time as String, ")"], "") as NonEmptyStr } + } + } + } + } + } + } +} + +// THIS PROCESS'S OWN IDENTITY, read from the same procfs a successor will read it back through: the +// boot id, and pid and starttime from ONE read of /proc/self/stat. A boot that cannot name itself does +// not take the unit -- a hold it could never be shown dead in is the lockout this replaces. +type SelfProcessIdentity + = SelfIdentified { process: ProcessIdentity, pid_namespace: NonEmptyStr } + | SelfUnidentified { cause: NonEmptyStr } + +fn self_process_identity() -> SelfProcessIdentity { + match exact_text(path: proc_boot_id_path) { + ExactTextAbsent => SelfUnidentified { cause: concat(proc_boot_id_path, " is absent on this host") as NonEmptyStr } + ExactTextUnread { cause: c } => SelfUnidentified { cause: c } + ExactTextRead { text: b } => { + let boot = trim(s: b) + match exact_text(path: proc_self_stat_path) { + ExactTextAbsent => SelfUnidentified { cause: concat(proc_self_stat_path, " is absent on this host") as NonEmptyStr } + ExactTextUnread { cause: c } => SelfUnidentified { cause: c } + ExactTextRead { text: stat } => + match proc_pid_stat_pid(content: stat) { + PidUnparseable { field: f } => SelfUnidentified { cause: concat("/proc/self/stat carried no pid: ", f) as NonEmptyStr } + PidObserved { pid: n } => + match start_time_text(stat: stat) { + Absent => SelfUnidentified { cause: "/proc/self/stat carried no readable starttime" as NonEmptyStr } + Present { value: st } => + match checked_int_to_nat(n: n) { + Absent => SelfUnidentified { cause: "/proc/self/stat pid is not a natural number" as NonEmptyStr } + Present { value: pid } => + if boot == "" { SelfUnidentified { cause: concat(proc_boot_id_path, " was empty") as NonEmptyStr } } + else { + match own_pid_namespace() { + PidNamespaceUnread { cause: c } => SelfUnidentified { cause: c } + PidNamespaceRead { namespace: n } => SelfIdentified { process: ProcessIdentity { boot_id: boot as NonEmptyStr, pid: pid, start_time: st as NonEmptyStr }, pid_namespace: n } + } + } + } + } + } + } + } + } +} diff --git a/dag/std/materialization_object.dag b/dag/std/materialization_object.dag index d28e969d502..0c0fba24562 100644 --- a/dag/std/materialization_object.dag +++ b/dag/std/materialization_object.dag @@ -638,6 +638,14 @@ fn store_lookup_decide_present( // --------------------------------------------------------------------------------------------- // Commit: prepare (pure), publish (the realization's one effect), read back (a lookup), settle. +// The last three arms are CAPACITY REFUSALS, owned by the store realization's declared budgets +// (std.artifact_store over the family's CacheProvider). A family with no declared budget refuses -- +// an unbudgeted family never borrows another's -- and an artifact larger than its family's whole +// budget refuses before any eviction (store_put's fit test), so it can never empty the store to make +// room for itself. An occupancy record that cannot be read refuses rather than reading as empty. +// Bytes an eviction could not yet delete stay CHARGED against the family's budget: an artifact that +// fits the budget but not the budget less those bytes refuses as cleanup outstanding, so a host that +// keeps refusing deletes stops the store's writes instead of letting disk grow past the ceiling. type StoreCommitRefusal = StoreCommitManifestUninterpretable { decode: ManifestDecode } | StoreCommitContractRefused { binding: RequestArtifactBinding } @@ -647,6 +655,10 @@ type StoreCommitRefusal | StoreCommitResultContentConflict { binding: EvaluationResultBinding, observed: ResultContentIdentity } | StoreCommitResultHashFamilyIncomparable { left: HashFamily, right: HashFamily } | StoreCommitResultIdentityUnverifiable { cause: String } + | StoreCommitBudgetUndeclared { family: ArtifactKindId } + | StoreCommitDidNotFit { artifact_bytes: ByteSize, budget: ByteSize } + | StoreCommitCleanupOutstanding { artifact_bytes: ByteSize, budget: ByteSize, undeleted: ByteSize } + | StoreCommitOccupancyUnavailable { cause: String } type StoreCommit = StoreCommitSettled { receipt: RealizationReceipt } @@ -879,6 +891,10 @@ fn store_commit_refusal_tag(c: StoreCommit) -> String { StoreCommitResultContentConflict { binding: _, observed: _ } => "result_content_conflict" StoreCommitResultHashFamilyIncomparable { left: _, right: _ } => "result_hash_family_incomparable" StoreCommitResultIdentityUnverifiable { cause: _ } => "result_identity_unverifiable" + StoreCommitBudgetUndeclared { family: _ } => "budget_undeclared" + StoreCommitDidNotFit { artifact_bytes: _, budget: _ } => "did_not_fit" + StoreCommitCleanupOutstanding { artifact_bytes: _, budget: _, undeleted: _ } => "cleanup_outstanding" + StoreCommitOccupancyUnavailable { cause: _ } => "occupancy_unavailable" } ) } diff --git a/dag/test/claim/durable_cas_file_store_wet_witness_test.dag b/dag/test/claim/durable_cas_file_store_wet_witness_test.dag index dc2be56e479..b0e73e9924d 100644 --- a/dag/test/claim/durable_cas_file_store_wet_witness_test.dag +++ b/dag/test/claim/durable_cas_file_store_wet_witness_test.dag @@ -15,6 +15,8 @@ import std.checked_arithmetic { int_inclusive_max } import gunbc.durable_cas_file_store { CasAttemptAdmission, CasAttemptAdmitted, CasAttemptDigestMismatch, CasAttemptDigestIncomparable, CasAttemptKeyNotSlotAddressable, admit_cas_attempt, file_compare_and_set, DefaultAccessCreateOnly, observe_cas_slot_state, cas_file_slot_path, + file_compare_and_set_windowed, observe_cas_slot_state_windowed, CasGenerationWindow, + cas_generation_window, CasGenerationWindowAdmitted, CasGenerationWindowTooNarrow, CasSlotProbe, ProbedHead, cas_probe_gallop, } @@ -198,3 +200,101 @@ test fn an_established_empty_root_reads_the_slot_absent_by_real_execution() -> B let removed = shell.Remove.RecursiveForce(path: root_path) absent && removed.success } + +// WINDOWED SLOTS, BY REAL EXECUTION. file_compare_and_set_windowed keeps at most k+1 generation files +// however many commits it takes, while file_compare_and_set -- unchanged -- keeps every one; and a +// second writer under a window that loses the race re-reads and lands its commit, so no commit is lost. + +fn window_of(k: Int) -> CasGenerationWindow? { + match cas_generation_window(k: k) { + CasGenerationWindowAdmitted { window: w } => Present { value: w } + CasGenerationWindowTooNarrow { k: _ } => none + } +} + +// A window slot's commit, or the unchanged keep-all commit when no window is given. +fn run_cas_in(root: NonEmptyStr, window: CasGenerationWindow?, attempt: CasAttempt) -> CasOutcome { + match window { + Absent => run_cas(root: root, attempt: attempt) + Present { value: w } => + match admit_cas_attempt(attempt: attempt) { + CasAttemptAdmitted { verified: v } => file_compare_and_set_windowed(root: root, publication: DefaultAccessCreateOnly, verified: v, window: w).outcome + CasAttemptDigestMismatch { claimed: _, actual: _ } => + CasStoreRefused { cause: CasGenerationPublicationRefused { detail: "mint digest mismatch" as NonEmptyStr } } + CasAttemptDigestIncomparable { claimed: _, actual: _ } => + CasStoreRefused { cause: CasGenerationPublicationRefused { detail: "mint digest incomparable" as NonEmptyStr } } + CasAttemptKeyNotSlotAddressable { key: _ } => + CasStoreRefused { cause: CasGenerationPublicationRefused { detail: "key not slot-addressable" as NonEmptyStr } } + } + } +} + +fn slot_file_count(root_path: String, key: String) -> Int { + let listed = Filesystem.List(path: root_path) + if !listed.success { 0 - 1 } else { + fold(listed.entries.split(delimiter: "\n"), init: 0, f: (acc, n) => if starts_with(s: n, prefix: concat(key, ".")) { acc + 1 } else { acc }) + } +} + +// Commit n successive generations through the slot's own observation of its head. +fn commit_successively(root: NonEmptyStr, window: CasGenerationWindow?, key: NonEmptyStr, n: Int) -> Bool { + [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] |> filter(i => i <= n) |> all(i => { + let payload = concat("v", i as String) as NonEmptyStr + let expected = if i == 1 { ExpectSlotAbsent } else { ExpectSlotGeneration { generation: i - 1 } } + is_committed(o: run_cas_in(root: root, window: window, attempt: honest(key: key, payload: payload, expected: expected)), payload: payload, generation: i) + }) +} + +test fn a_window_slot_holds_at_most_k_plus_one_generations_by_real_execution() -> Bool { + let root_path = fresh_root() + let root = root_path as NonEmptyStr + let w = window_of(k: 2) + let committed = commit_successively(root: root, window: w, key: "index", n: 10) + let files = slot_file_count(root_path: root_path, key: "index") + let observed_head = match w { + Absent => false + Present { value: win } => match observe_cas_slot_state_windowed(root: root, key: "index", window: win) { + CasObservedReadable { readable: r } => match r { + CasReadablePresent { version: v } => v.value == "v10" && cas_generation_count(g: v.generation) == 10 + CasReadableAbsent => false + } + CasObservedUnreadable { cause: _ } => false + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + committed && files >= 1 && files <= 3 && observed_head && removed.success +} + +test fn the_unwindowed_compare_and_set_keeps_every_generation_by_real_execution() -> Bool { + let root_path = fresh_root() + let root = root_path as NonEmptyStr + let committed = commit_successively(root: root, window: none, key: "audit", n: 10) + let files = slot_file_count(root_path: root_path, key: "audit") + let removed = shell.Remove.RecursiveForce(path: root_path) + committed && files == 10 && removed.success +} + +// Two writers read the same head; the first commits, the second loses with the winner observed, +// re-reads the window's head and commits on top of it. Both commits stand. +test fn a_racing_writer_under_a_window_retries_and_loses_no_commit_by_real_execution() -> Bool { + let root_path = fresh_root() + let root = root_path as NonEmptyStr + let w = window_of(k: 2) + let setup = commit_successively(root: root, window: w, key: "index", n: 3) + let first = run_cas_in(root: root, window: w, attempt: honest(key: "index", payload: "writer-a", expected: ExpectSlotGeneration { generation: 3 })) + let lost = run_cas_in(root: root, window: w, attempt: honest(key: "index", payload: "writer-b", expected: ExpectSlotGeneration { generation: 3 })) + let retried = run_cas_in(root: root, window: w, attempt: honest(key: "index", payload: "writer-b", expected: ExpectSlotGeneration { generation: 4 })) + let removed = shell.Remove.RecursiveForce(path: root_path) + setup && is_committed(o: first, payload: "writer-a", generation: 4) + && is_precondition_present(o: lost, payload: "writer-a") + && is_committed(o: retried, payload: "writer-b", generation: 5) + && removed.success +} + +// The window's floor is construction: k < 2 has no admitted window. +test fn a_window_narrower_than_two_is_not_admitted() -> Bool { + match cas_generation_window(k: 1) { + CasGenerationWindowTooNarrow { k: k } => k == 1 + CasGenerationWindowAdmitted { window: _ } => false + } +} diff --git a/dag/test/claim/materialization_store_local_seal_witness_test.dag b/dag/test/claim/materialization_store_local_seal_witness_test.dag new file mode 100644 index 00000000000..e063e2843c2 --- /dev/null +++ b/dag/test/claim/materialization_store_local_seal_witness_test.dag @@ -0,0 +1,130 @@ +module test.claim.materialization_store_local_seal_witness + +import std.types { Bool, Int, String } +import gunbc.compile_census_probe { blocking_subject_count_for } + +// A FAMILY MUTATION WITHOUT THE FAMILY HOLD IS UNWRITABLE. Every mutator of +// extdeps.realization.materialization_store_local takes a LocalStoreFamilyHeld (or, for the sweep, a +// LocalStoreAllFamiliesHeld), and both are sole_constructor records minted only by the hold +// acquisitions. So a caller that wants to write without holding the family has to spell one of them, +// and that spelling is a located compile refusal. Each probe asserts EXACTLY ONE refusal on its subject. + +fn seal_violations(source: String, subject: String) -> Int { + blocking_subject_count_for(source: source, wanted_class: "SoleConstructorViolation", wanted_subject: subject) +} + +fn admission_refusals(source: String, callee: String) -> Int { + blocking_subject_count_for(source: source, wanted_class: "ConstructorCallAdmissionRefused", wanted_subject: callee) +} + +data forged_family_hold_source: String = "module probe_store_forged_family_hold\nimport extdeps.realization.materialization_store_local { LocalStoreFamilyHeld }\nfn forged(h: LocalStoreFamilyHeld) -> LocalStoreFamilyHeld {\n LocalStoreFamilyHeld { root_path: h.root_path, family: h.family, owner: h.owner, generation: h.generation, retention: h.retention }\n}\n" + +data forged_all_families_source: String = "module probe_store_forged_all_families\nimport extdeps.realization.materialization_store_local { LocalStoreAllFamiliesHeld }\nfn forged(a: LocalStoreAllFamiliesHeld) -> LocalStoreAllFamiliesHeld {\n LocalStoreAllFamiliesHeld { root_path: a.root_path, held: a.held }\n}\n" + +test fn a_forged_family_hold_is_refused_at_compile() -> Bool { + seal_violations(source: forged_family_hold_source, subject: "LocalStoreFamilyHeld") == 1 +} + +test fn a_forged_all_families_hold_is_refused_at_compile() -> Bool { + seal_violations(source: forged_all_families_source, subject: "LocalStoreAllFamiliesHeld") == 1 +} + + +// FORGED ADMISSION: the publish half takes what a reservation admitted, and only local_store_reserve +// mints it, so skipping admission means spelling one. +data forged_admission_source: String = "module probe_store_forged_admission\nimport extdeps.realization.materialization_store_local { LocalStoreAdmission }\nfn forged(a: LocalStoreAdmission) -> LocalStoreAdmission {\n LocalStoreAdmission { budget: a.budget, evicted: a.evicted, retired: a.retired, reclamation: a.reclamation }\n}\n" + +test fn a_forged_reservation_admission_is_refused_at_compile() -> Bool { + seal_violations(source: forged_admission_source, subject: "LocalStoreAdmission") == 1 +} + +// THE RAW HELPERS ARE SEALED TO THE BRACKET'S OWN PATH: a caller outside it is an admission refusal, +// even holding a real held value. +data direct_write_index_source: String = "module probe_store_direct_write_index\nimport extdeps.realization.materialization_store_local { LocalStoreFamilyHeld, LocalStoreIndex, LocalStoreIndexWrite, local_store_write_index }\nimport std.durable_compare_and_set { CasExpectation }\nfn direct(h: LocalStoreFamilyHeld, e: CasExpectation, i: LocalStoreIndex) -> LocalStoreIndexWrite {\n local_store_write_index(held: h, expected: e, index: i)\n}\n" + +data direct_acquire_settled_source: String = "module probe_store_direct_acquire_settled\nimport std.types { Bool, NonEmptyStr }\nimport std.cache_identity { ArtifactKindId }\nimport std.durable_exclusive_hold { DurableHoldOwnerRef }\nimport gunbc.durable_cas_file_store { CasGenerationWindow }\nimport extdeps.realization.materialization_store_local { LocalStoreBinding, LocalStoreHoldAcquisition, local_store_acquire_settled }\nfn direct(b: LocalStoreBinding, f: ArtifactKindId, k: NonEmptyStr, o: DurableHoldOwnerRef, w: CasGenerationWindow) -> LocalStoreHoldAcquisition {\n local_store_acquire_settled(binding: b, family: f, key: k, owner: o, window: w, recovered: false)\n}\n" + +test fn a_direct_index_write_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_write_index_source, callee: "local_store_write_index") == 1 +} + +test fn a_direct_settled_acquire_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_acquire_settled_source, callee: "local_store_acquire_settled") == 1 +} + +// THE REMAINING EFFECTFUL HELPERS ARE SEALED: creating a store file, and reclaiming a slot's +// generations, are reachable only from the store's own paths -- a direct call could write an +// unindexed file, or apply this store's window to a slot that keeps every generation. +data direct_create_new_source: String = "module probe_store_direct_create_new\nimport std.types { String }\nimport extdeps.filesystem.filesystem_io { FilesystemCreateNew }\nimport extdeps.realization.materialization_store_local { local_store_create_new }\nfn direct(p: String, c: String) -> FilesystemCreateNew {\n local_store_create_new(path: p, content: c)\n}\n" + +data direct_settle_slot_source: String = "module probe_store_direct_settle_slot\nimport std.types { String, NonEmptyStr }\nimport extdeps.realization.materialization_store_local { LocalStoreSlotSettlement, local_store_settle_slot }\nfn direct(r: String, k: NonEmptyStr) -> LocalStoreSlotSettlement {\n local_store_settle_slot(root_path: r, key: k)\n}\n" + +test fn a_direct_store_file_create_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_create_new_source, callee: "local_store_create_new") == 1 +} + +test fn a_direct_slot_settle_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_settle_slot_source, callee: "local_store_settle_slot") == 1 +} + +// THE OPENED STORE IS UNFORGEABLE. A capability is a verified binding plus its clean-sweep proof, minted +// only by the open after its sweep cleared every orphan. A pre-open (what marker verification alone +// yields) cannot be promoted by spelling a capability around its binding, a binding cannot be +// spelled with different budgets, and a clean-sweep proof cannot be spelled for a sweep that never ran. +data forged_capability_source: String = "module probe_store_forged_capability\nimport extdeps.realization.materialization_store_local { LocalStoreCapability, LocalStorePreOpen }\nfn forged(p: LocalStorePreOpen, c: LocalStoreCapability) -> LocalStoreCapability {\n LocalStoreCapability { binding: p.binding, sweep: c.sweep }\n}\n" + +data forged_binding_source: String = "module probe_store_forged_binding\nimport extdeps.realization.materialization_store_local { LocalStoreBinding, LocalStoreCapability }\nfn forged(c: LocalStoreCapability) -> LocalStoreBinding {\n LocalStoreBinding { root: c.binding.root, store: c.binding.store, budgets: [] }\n}\n" + +data forged_clean_sweep_source: String = "module probe_store_forged_clean_sweep\nimport extdeps.realization.materialization_store_local { LocalStoreCleanSweep }\nfn forged() -> LocalStoreCleanSweep {\n LocalStoreCleanSweep { removed: [], releases: [] }\n}\n" + +test fn a_forged_store_capability_is_refused_at_compile() -> Bool { + seal_violations(source: forged_capability_source, subject: "LocalStoreCapability") == 1 +} + +test fn a_forged_store_binding_is_refused_at_compile() -> Bool { + seal_violations(source: forged_binding_source, subject: "LocalStoreBinding") == 1 +} + +test fn a_forged_clean_sweep_proof_is_refused_at_compile() -> Bool { + seal_violations(source: forged_clean_sweep_source, subject: "LocalStoreCleanSweep") == 1 +} + +// THE WINDOWED CONTRACT IS THE STORE'S ALONE. Keep-all callers use the unchanged file_compare_and_set +// family; the windowed operations, and the raw helpers beneath both contracts, admit only their named +// callers. Each probe below is a module outside that list calling one of them, and each is exactly one +// ConstructorCallAdmissionRefused naming the callee. + +// a keep-all caller committing through the windowed operation +data direct_windowed_compare_and_set_source: String = "module probe_direct_windowed_compare_and_set\nimport std.types { NonEmptyStr }\nimport gunbc.durable_cas_file_store { CasGenerationPublication, VerifiedCasAttempt, CasGenerationWindow, CasFileWrite, file_compare_and_set_windowed }\nfn direct(r: NonEmptyStr, p: CasGenerationPublication, v: VerifiedCasAttempt, w: CasGenerationWindow) -> CasFileWrite {\n file_compare_and_set_windowed(root: r, publication: p, verified: v, window: w)\n}\n" + +// a hold taken on a windowed slot outside the store +data direct_windowed_hold_source: String = "module probe_direct_windowed_hold\nimport std.types { NonEmptyStr }\nimport std.durable_exclusive_hold { DurableHoldOwnerRef }\nimport gunbc.durable_cas_file_store { CasGenerationWindow }\nimport gunbc.durable_exclusive_hold_file_store { FileHoldAcquireOutcome, file_hold_acquire_windowed }\nfn direct(r: NonEmptyStr, k: NonEmptyStr, o: DurableHoldOwnerRef, w: CasGenerationWindow) -> FileHoldAcquireOutcome {\n file_hold_acquire_windowed(root: r, slot_key: k, requested_owner: o, window: w)\n}\n" + +// the raw commit with a caller-chosen retention +data direct_raw_outcome_source: String = "module probe_direct_raw_outcome\nimport std.types { NonEmptyStr }\nimport gunbc.durable_cas_file_store { CasGenerationPublication, VerifiedCasAttempt, CasSlotRetention, cas_compare_and_set_outcome }\nimport std.durable_compare_and_set { CasOutcome }\nfn direct(r: NonEmptyStr, p: CasGenerationPublication, t: CasSlotRetention, v: VerifiedCasAttempt) -> CasOutcome {\n cas_compare_and_set_outcome(root: r, publication: p, retention: t, verified: v)\n}\n" + +// the raw observation with a caller-chosen retention +data direct_raw_observe_source: String = "module probe_direct_raw_observe\nimport std.types { NonEmptyStr }\nimport gunbc.durable_cas_file_store { CasSlotRetention, CasSlotProbe, cas_observe_retained_slot }\nfn direct(r: NonEmptyStr, k: NonEmptyStr, t: CasSlotRetention) -> CasSlotProbe {\n cas_observe_retained_slot(root: r, key: k, retention: t)\n}\n" + +// deleting an audit slot's generations below a caller-chosen head +data direct_raw_reclaim_source: String = "module probe_direct_raw_reclaim\nimport std.types { NonEmptyStr }\nimport std.durable_compare_and_set { CasGeneration }\nimport gunbc.durable_cas_file_store { CasGenerationWindow, CasWindowReclamation, cas_reclaim_below }\nfn direct(r: NonEmptyStr, k: NonEmptyStr, w: CasGenerationWindow, h: CasGeneration) -> CasWindowReclamation {\n cas_reclaim_below(root: r, key: k, window: w, head: h)\n}\n" + +test fn a_direct_windowed_compare_and_set_call_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_windowed_compare_and_set_source, callee: "file_compare_and_set_windowed") == 1 +} + +test fn a_direct_windowed_hold_call_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_windowed_hold_source, callee: "file_hold_acquire_windowed") == 1 +} + +test fn a_direct_raw_outcome_call_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_raw_outcome_source, callee: "cas_compare_and_set_outcome") == 1 +} + +test fn a_direct_raw_observe_call_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_raw_observe_source, callee: "cas_observe_retained_slot") == 1 +} + +test fn a_direct_raw_reclaim_call_is_an_admission_refusal() -> Bool { + admission_refusals(source: direct_raw_reclaim_source, callee: "cas_reclaim_below") == 1 +} diff --git a/dag/test/claim/materialization_store_local_wet_witness_test.dag b/dag/test/claim/materialization_store_local_wet_witness_test.dag index 3e46f082d6e..1d95a21a099 100644 --- a/dag/test/claim/materialization_store_local_wet_witness_test.dag +++ b/dag/test/claim/materialization_store_local_wet_witness_test.dag @@ -25,13 +25,40 @@ import std.materialization_object { store_commit_refusal_tag, store_commit_fold, } import extdeps.realization.materialization_store_local { + LocalStoreCapability, LocalStoreOpening, LocalStoreOpened, LocalStoreUnavailable, LocalStoreInitialization, LocalStoreInitialized, LocalStoreInitializationRefused, LocalStoreNotInitialized, LocalStoreGrantRefused, - LocalStoreBatchLookup, LocalStoreBatchLooked, LocalStoreBatchUnavailable, + LocalStoreBatchLookup, LocalStoreBatchLooked, local_store_initialize, local_store_open, local_store_lookup, local_store_commit, local_store_lookup_batch, store_path, materialization_store_marker_name, + LocalStoreCommit, LocalStoreSweep, LocalStoreSwept, LocalStoreSweepUnavailable, + LocalStoreIndexRead, LocalStoreIndexAt, LocalStoreIndexUnavailable, local_store_read_index, local_store_index_key, + LocalStoreRecency, LocalStoreRecencyRecorded, LocalStoreRecencyNothingToRecord, LocalStoreRecencyLost, LocalStoreRecencyRaced, + LocalStoreDeleteAttempt, + local_store_record_recency, local_store_is_object_name, local_store_object_bytes, local_store_family_budget, + LocalStoreFamilyBudget, local_store_budget_sum, + LocalStoreRecencyHoldRefused, LocalStoreRecencyReceipt, + local_store_with_family, LocalStoreBracket, LocalStoreBracketRan, LocalStoreBracketRefused, local_store_commit_under, LocalStoreHoldStale, LocalStoreHoldWrongFamily, + local_store_hold_key, local_store_hold_owner_prefix, + LocalStoreHoldAcquisition, LocalStoreHoldAcquired, LocalStoreHoldRefused, LocalStoreHoldRefusal, LocalStoreFamilyBusy, + LocalStoreHolderUnobservable, LocalStoreHoldRecoveryRefused, LocalStoreHoldContended, LocalStoreHoldStoreUnavailable, + LocalStoreHoldSlotUndecodable, LocalStoreHoldKeyNotAddressable, LocalStoreHoldSelfUnidentified, LocalStoreHoldWindowTooNarrow, LocalStoreHoldUnsettled, + LocalStoreCatalogMismatch, LocalStoreCatalogInadmissible, LocalStoreSweepIncomplete, LocalStoreUnavailableCause, + local_store_index_window, } +import gunbc.process_hold_identity { self_process_identity, SelfIdentified, SelfUnidentified, process_hold_text } +import gunbc.build_cache_instance { ProcessIdentity } +import gunbc.durable_exclusive_hold_file_store { file_hold_acquire_windowed, FileHoldAcquired, FileHoldOccupied, FileHoldAcquireLost, FileHoldAcquireStoreUnavailable, FileHoldSlotUndecodable, FileHoldKeyNotSlotAddressable } +import std.durable_exclusive_hold { DurableHoldOwnerRef } +import std.cache_identity { native_module_verdict_bundle_artifact_kind, typed_module_artifact_kind } +import gunbc.materialization_store_budgets { + materialization_store_host_ceiling_bytes, materialization_store_typed_module_budget_policy, materialization_store_durable_family_budgets, +} +import std.measure { byte_size } +import std.measure { measure_count } +import std.render_repeat_string_bootstrap { repeat_string } +import extdeps.filesystem.filesystem_io { FilesystemDeleted, FilesystemDeleteTargetAbsent, FilesystemDeleteRefused, FilesystemDeleteKindUnrecognized, filesystem_listing_observation, FilesystemDirectoryListed, FilesystemDirectoryListingRefused, FilesystemDirectorySubjectRefused, filesystem_listing_entry_names } // WET CONTROLS FOR THE LOCAL PERSISTENT REALIZATION: every control of the provider-spine brief, // executed against a REAL filesystem through extdeps.realization.materialization_store_local. Each @@ -109,6 +136,39 @@ fn object_path(root_path: String, req: ArtifactRequest) -> String { } } +fn object_name(req: ArtifactRequest) -> String { + match evaluation_store_admitted_address(admission: evaluation_store_address(req: req)) { + Absent => "" + Present { value: addr } => store_object_name(address: addr) + } +} + +// The receipt carries the host's typed answer, not a flag: a directory where a file should be is a +// delete the host REFUSED, distinct from an absent target or an unrecognized kind. +fn delete_was_refused_by_the_host(t: LocalStoreDeleteAttempt) -> Bool { + match t.outcome { + FilesystemDeleteRefused { path: _, kind: _, error: _ } => true + FilesystemDeleted { path: _ } => false + FilesystemDeleteTargetAbsent { path: _ } => false + FilesystemDeleteKindUnrecognized { path: _, observed: _, error: _ } => false + } +} + +// FIXTURE BUDGETS, this file's own: small enough that a wet witness overruns them in a few commits, so +// eviction and the fit refusal are reached by execution. The store receives them as its parameter, +// exactly as a production consumer passes gunbc.materialization_store_budgets. +data witness_family_budgets: List = [ + LocalStoreFamilyBudget { family: native_module_verdict_bundle_artifact_kind, budget: byte_size(count: 4096) }, + LocalStoreFamilyBudget { family: typed_module_artifact_kind, budget: byte_size(count: 4096) }, +] + +fn first_recency(receipts: List) -> LocalStoreRecency { + match first(receipts) { + Present { value: r } => r.recency + Absent => LocalStoreRecencyRaced { family: native_module_verdict_bundle_artifact_kind } + } +} + fn commit_tag(c: StoreCommit) -> String { store_commit_refusal_tag(c: c) } @@ -117,10 +177,10 @@ fn fresh_root_path() -> String { shell.Mktemp.DirWithTemplate(template: "/tmp/gunbc_matstore.XXXXXX").path } -fn opened(root: MaterializationStoreRoot) -> OpenedMaterializationStore? { - match local_store_initialize(root: root, instance_label: "wet-witness") { +fn opened(root: MaterializationStoreRoot) -> LocalStoreCapability? { + match local_store_initialize(root: root, instance_label: "wet-witness", budgets: witness_family_budgets) { LocalStoreInitialized { opening: o } => match o { - LocalStoreOpened { root: _, store: s, durability: _ } => Present { value: s } + LocalStoreOpened { capability: c, durability: _ } => Present { value: c } LocalStoreUnavailable { root_path: _, cause: _ } => none } LocalStoreInitializationRefused { root_path: _, cause: _ } => none @@ -134,8 +194,8 @@ test fn commit_then_read_back_by_real_execution() -> Bool { let verdict = match opened(root: root) { Absent => false Present { value: store } => { - let committed = commit_tag(c: local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass\na.t2 fail"))) - let read = local_store_lookup(root: root, store: store, req: bundle_req(module: "a")) + let committed = commit_tag(c: local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass\na.t2 fail")).commit) + let read = local_store_lookup(capability: store, req: bundle_req(module: "a")) let bytes_ok = match store_lookup_payload(l: read, id: output_module_test_verdicts) { Present { value: v } => v == "a.t1 pass\na.t2 fail" Absent => false @@ -154,10 +214,10 @@ test fn corruption_is_an_integrity_refusal_by_real_execution() -> Bool { let verdict = match opened(root: root) { Absent => false Present { value: store } => { - let committed = commit_tag(c: local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass"))) + let committed = commit_tag(c: local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")).commit) let good = Filesystem.Read(path: object_path(root_path: root_path, req: bundle_req(module: "a"))) let corrupt = Filesystem.Write(path: object_path(root_path: root_path, req: bundle_req(module: "a")), content: replace(good.content, "a.t1%20pass", "a.t1%20PASS")) - let read = local_store_lookup(root: root, store: store, req: bundle_req(module: "a")) + let read = local_store_lookup(capability: store, req: bundle_req(module: "a")) committed == "committed" && good.success && corrupt.success && store_lookup_refusal_tag(l: read) == "contract_wrong_content" && store_lookup_is_miss(l: read) == false @@ -180,7 +240,7 @@ test fn wrong_kind_refuses_by_real_execution() -> Bool { Absent => false Present { value: store } => { let planted = Filesystem.WriteCreateNew(path: object_path(root_path: root_path, req: lookup), content: forged) - let read = local_store_lookup(root: root, store: store, req: lookup) + let read = local_store_lookup(capability: store, req: lookup) planted.success && (forged == universe) == false && store_lookup_refusal_tag(l: read) == "contract_kind_mismatch" } @@ -217,7 +277,7 @@ test fn incomplete_refuses_by_real_execution() -> Bool { ) ) let planted = Filesystem.WriteCreateNew(path: object_path(root_path: root_path, req: req), content: incomplete) - let read = local_store_lookup(root: root, store: store, req: req) + let read = local_store_lookup(capability: store, req: req) planted.success && store_lookup_refusal_tag(l: read) == "contract_incomplete" } } @@ -233,8 +293,8 @@ test fn a_second_writer_converges_on_the_committed_artifact_by_real_execution() let verdict = match opened(root: root) { Absent => false Present { value: store } => { - let first = local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")) - let second = local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")) + let first = local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")).commit + let second = local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")).commit let same_content = match first { StoreCommitSettled { receipt: r1 } => match second { StoreCommitSettled { receipt: r2 } => r1.content_digest == r2.content_digest && r1.request_key == r2.request_key @@ -243,7 +303,7 @@ test fn a_second_writer_converges_on_the_committed_artifact_by_real_execution() StoreCommitRefused { key: _, refusal: _ } => false } commit_tag(c: first) == "committed" && commit_tag(c: second) == "converged_on_prior_commit" && same_content - && store_lookup_is_hit(l: local_store_lookup(root: root, store: store, req: bundle_req(module: "a"))) + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: bundle_req(module: "a"))) } } let removed = shell.Remove.RecursiveForce(path: root_path) @@ -258,9 +318,9 @@ test fn a_divergent_second_writer_conflicts_and_changes_nothing_by_real_executio let verdict = match opened(root: root) { Absent => false Present { value: store } => { - let first = local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")) - let second = local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 fail")) - let read = local_store_lookup(root: root, store: store, req: bundle_req(module: "a")) + let first = local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")).commit + let second = local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 fail")).commit + let read = local_store_lookup(capability: store, req: bundle_req(module: "a")) let still_first = match store_lookup_payload(l: read, id: output_module_test_verdicts) { Present { value: v } => v == "a.t1 pass" Absent => false @@ -272,33 +332,30 @@ test fn a_divergent_second_writer_conflicts_and_changes_nothing_by_real_executio verdict && removed.success } -// provider unavailable: an uninitialized root is not a store. Open, batch and lookup all refuse, +// provider unavailable: an uninitialized root is not a store. Open refuses, so no capability exists for +// batch, lookup or commit to consume, // nothing is answered as a miss, and nothing is written anywhere to stand in for it. test fn an_unavailable_store_refuses_and_leaves_no_shadow_by_real_execution() -> Bool { let root_path = fresh_root_path() let root = WitnessScratchRoot { root: root_path } - let open_refused = match local_store_open(root: root) { + let open_refused = match local_store_open(root: root, budgets: witness_family_budgets) { LocalStoreUnavailable { root_path: _, cause: c } => match c { LocalStoreNotInitialized { marker_path: _ } => true _ => false } - LocalStoreOpened { root: _, store: _, durability: _ } => false + LocalStoreOpened { capability: _, durability: _ } => false } - let batch_refused = match local_store_lookup_batch(root: root, reqs: [bundle_req(module: "a"), bundle_req(module: "b")]) { - LocalStoreBatchUnavailable { root_path: _, cause: _ } => true - LocalStoreBatchLooked { store: _, lookups: _ } => false - } - let ungranted = match local_store_open(root: WitnessScratchRoot { root: "/tmp/not_a_gunbc_store_root" }) { + let ungranted = match local_store_open(root: WitnessScratchRoot { root: "/tmp/not_a_gunbc_store_root" }, budgets: witness_family_budgets) { LocalStoreUnavailable { root_path: _, cause: c } => match c { LocalStoreGrantRefused { verb: _, target: _, frame: _ } => true _ => false } - LocalStoreOpened { root: _, store: _, durability: _ } => false + LocalStoreOpened { capability: _, durability: _ } => false } let no_marker = shell.Test.IsFile(path: store_path(root_path: root_path, name: materialization_store_marker_name)) let no_object = shell.Test.IsFile(path: object_path(root_path: root_path, req: bundle_req(module: "a"))) let removed = shell.Remove.RecursiveForce(path: root_path) - open_refused && batch_refused && ungranted && !no_marker.is_file && !no_object.is_file && removed.success + open_refused && ungranted && !no_marker.is_file && !no_object.is_file && removed.success } // batch: one open, one exact read per request -- a committed request hits, an uncommitted one misses. @@ -308,9 +365,8 @@ test fn a_batch_answers_each_request_by_real_execution() -> Bool { let verdict = match opened(root: root) { Absent => false Present { value: store } => { - let committed = commit_tag(c: local_store_commit(root: root, store: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass"))) - match local_store_lookup_batch(root: root, reqs: [bundle_req(module: "a"), bundle_req(module: "b")]) { - LocalStoreBatchUnavailable { root_path: _, cause: _ } => false + let committed = commit_tag(c: local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a.t1 pass")).commit) + match local_store_lookup_batch(capability: store, reqs: [bundle_req(module: "a"), bundle_req(module: "b")]) { LocalStoreBatchLooked { store: _, lookups: ls } => committed == "committed" && (ls |> count()) == 2 && match get(xs: ls, index: 0) { Present { value: l } => store_lookup_is_hit(l: l), Absent => false } @@ -321,3 +377,702 @@ test fn a_batch_answers_each_request_by_real_execution() -> Bool { let removed = shell.Remove.RecursiveForce(path: root_path) verdict && removed.success } + + +// --------------------------------------------------------------------------------------------- +// THE BOUND, BY REAL EXECUTION. The WitnessScratchRoot fixture budget for the bundle family is 4096 +// bytes (witness_family_budgets); each commit below is ~1 KiB, so the store overruns its +// budget within a few commits and must evict to stay inside it. + +fn padded_verdicts(module: String) -> List { + verdicts(text: concat(module, repeat_string(s: " pass", n: 200))) +} + +fn bundle_family_budget_bytes(root: MaterializationStoreRoot) -> Int { + match local_store_family_budget(budgets: witness_family_budgets, family: native_module_verdict_bundle_artifact_kind) { + Present { value: b } => measure_count(m: b.budget) + Absent => 0 + } +} + +// What the store's objects actually occupy on disk: every object file in the root, read and measured +// with the store's own byte quantity. The index is not consulted, so this is an independent reading. +fn root_object_bytes(root_path: String) -> Int { + let listing = Filesystem.List(path: root_path) + match filesystem_listing_observation(directory: root_path, success: listing.success, entries: listing.entries, error: listing.error) { + FilesystemDirectoryListed(listed) => + fold(filesystem_listing_entry_names(listing: listed) |> filter(n => local_store_is_object_name(name: n)), init: 0, f: (acc, n) => + acc + measure_count(m: local_store_object_bytes(object: Filesystem.Read(path: store_path(root_path: root_path, name: n)).content))) + FilesystemDirectoryListingRefused { directory: _, error: _ } => 0 - 1 + FilesystemDirectorySubjectRefused { directory: _, cause: _ } => 0 - 1 + } +} + +fn root_names_with_prefix(root_path: String, prefix: String) -> Int { + let listing = Filesystem.List(path: root_path) + match filesystem_listing_observation(directory: root_path, success: listing.success, entries: listing.entries, error: listing.error) { + FilesystemDirectoryListed(listed) => filesystem_listing_entry_names(listing: listed) |> filter(n => starts_with(s: n, prefix: prefix)) |> count() + FilesystemDirectoryListingRefused { directory: _, error: _ } => 0 - 1 + FilesystemDirectorySubjectRefused { directory: _, cause: _ } => 0 - 1 + } +} + +type BoundStep { + within: Bool + evicted: Int +} + +// N = 12 commits into a 4096-byte family. After EVERY commit the on-disk object bytes stay within the +// budget; across the run some commit evicted; and the root's size before the run (0) and after it +// (positive, <= budget) are both measured, which is the disk-bound evidence the bound owes. +test fn the_family_budget_holds_on_disk_across_overrun_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let budget = bundle_family_budget_bytes(root: root) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let before = root_object_bytes(root_path: root_path) + let steps = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] |> map(i => { + let c = local_store_commit(capability: store, req: bundle_req(module: concat("m", i as String)), payloads: padded_verdicts(module: concat("m", i as String))) + BoundStep { within: commit_tag(c: c.commit) == "committed" && root_object_bytes(root_path: root_path) <= budget, evicted: count(c.evicted) } + }) + let after = root_object_bytes(root_path: root_path) + budget > 0 && before == 0 && after > 0 && after <= budget + && steps |> all(st => st.within) + && fold(steps, init: 0, f: (acc, st) => acc + st.evicted) > 0 + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// An artifact larger than its family's WHOLE budget refuses before any eviction: the entry already +// stored survives, and nothing is written for the oversized one. +test fn an_artifact_over_its_family_budget_refuses_and_evicts_nothing_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let small = local_store_commit(capability: store, req: bundle_req(module: "kept"), payloads: verdicts(text: "kept pass")) + let huge = local_store_commit(capability: store, req: bundle_req(module: "huge"), payloads: verdicts(text: repeat_string(s: "x", n: 5000))) + commit_tag(c: small.commit) == "committed" + && commit_tag(c: huge.commit) == "did_not_fit" + && count(huge.evicted) == 0 + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: bundle_req(module: "kept"))) + && store_lookup_is_miss(l: local_store_lookup(capability: store, req: bundle_req(module: "huge"))) + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// A family with no declared budget refuses; it never borrows another family's. +test fn a_family_without_a_declared_budget_refuses_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => + commit_tag(c: local_store_commit(capability: store, req: universe_req(), payloads: universe_payloads()).commit) == "budget_undeclared" + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// A corrupt occupancy index refuses the commit; it is never read as an empty store, so no bytes are +// published past it. +test fn a_corrupt_occupancy_index_refuses_and_never_reads_as_empty_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let first = local_store_commit(capability: store, req: bundle_req(module: "a"), payloads: verdicts(text: "a pass")) + let slot = store_path(root_path: root_path, name: concat(local_store_index_key(family: native_module_verdict_bundle_artifact_kind) as String, ".1")) + let planted = Filesystem.Write(path: slot, content: "not an occupancy index") + let second = local_store_commit(capability: store, req: bundle_req(module: "b"), payloads: verdicts(text: "b pass")) + commit_tag(c: first.commit) == "committed" && planted.success + && commit_tag(c: second.commit) == "occupancy_unavailable" + && store_lookup_is_miss(l: local_store_lookup(capability: store, req: bundle_req(module: "b"))) + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// An object no index names -- the residue of a crash between an eviction's index write and its +// delete -- is removed by the next open, and a named object and the marker are left alone. +test fn an_open_sweeps_an_object_no_index_names_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let kept = local_store_commit(capability: store, req: bundle_req(module: "kept"), payloads: verdicts(text: "kept pass")) + let orphan = object_name(req: bundle_req(module: "orphan")) + let planted = Filesystem.WriteCreateNew(path: store_path(root_path: root_path, name: orphan), content: "residue") + let swept = match local_store_open(root: root, budgets: witness_family_budgets) { + LocalStoreOpened { capability: c, durability: _ } => c.sweep.removed |> any(n => n == orphan) + LocalStoreUnavailable { root_path: _, cause: _ } => false + } + commit_tag(c: kept.commit) == "committed" && planted.success && swept + && !Filesystem.Read(path: store_path(root_path: root_path, name: orphan)).success + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: bundle_req(module: "kept"))) + && Filesystem.Read(path: store_path(root_path: root_path, name: materialization_store_marker_name)).success + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// The occupancy index is a windowed slot: after 12 commits its generation files number at most k+1 +// (k = 2), where a KeepAllGenerations slot would hold 12. +test fn the_occupancy_index_slot_stays_within_its_window_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let all_committed = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] + |> all(i => commit_tag(c: local_store_commit(capability: store, req: bundle_req(module: concat("w", i as String)), payloads: verdicts(text: "w pass")).commit) == "committed") + let slot_files = root_names_with_prefix(root_path: root_path, prefix: concat(local_store_index_key(family: native_module_verdict_bundle_artifact_kind) as String, ".")) + let hold_files = root_names_with_prefix(root_path: root_path, prefix: concat(local_store_hold_key(family: native_module_verdict_bundle_artifact_kind) as String, ".")) + all_committed && slot_files >= 1 && slot_files <= 3 && hold_files >= 1 && hold_files <= 3 + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// Recency is recorded once and is advisory: a served name is bumped, and a name the index does not +// hold records nothing rather than failing. +test fn recency_is_recorded_once_and_never_fails_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let c = local_store_commit(capability: store, req: bundle_req(module: "r"), payloads: verdicts(text: "r pass")) + let name = match evaluation_store_admitted_address(admission: evaluation_store_address(req: bundle_req(module: "r"))) { + Absent => "" + Present { value: addr } => store_object_name(address: addr) + } + let recorded = match first_recency(receipts: local_store_record_recency(capability: store, served: [bundle_req(module: "r")])) { + LocalStoreRecencyRecorded { family: _, bumped: b } => b == 1 + LocalStoreRecencyNothingToRecord { family: _ } => false + LocalStoreRecencyLost { family: _, refusal: _ } => false + LocalStoreRecencyRaced { family: _ } => false + LocalStoreRecencyHoldRefused { family: _, refusal: _ } => false + } + let nothing = match first_recency(receipts: local_store_record_recency(capability: store, served: [bundle_req(module: "never-committed")])) { + LocalStoreRecencyNothingToRecord { family: _ } => true + LocalStoreRecencyRecorded { family: _, bumped: _ } => false + LocalStoreRecencyLost { family: _, refusal: _ } => false + LocalStoreRecencyRaced { family: _ } => false + LocalStoreRecencyHoldRefused { family: _, refusal: _ } => false + } + commit_tag(c: c.commit) == "committed" && recorded && nothing + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// The host-wide ceiling is the declared sum of the durable root's family budgets, and today that is +// the typed-module policy figure alone. +test fn the_host_ceiling_is_the_declared_sum_of_family_budgets() -> Bool { + measure_count(m: materialization_store_host_ceiling_bytes) == measure_count(m: local_store_budget_sum(budgets: materialization_store_durable_family_budgets)) + && measure_count(m: materialization_store_host_ceiling_bytes) == measure_count(m: materialization_store_typed_module_budget_policy) +} + +// Two artifacts that each fit the 4096-byte witness family alone and do not fit together. +fn half_budget_verdicts(module: String) -> List { + verdicts(text: concat(module, repeat_string(s: " pass", n: 150))) +} + +fn root_object_names(root_path: String) -> List { + let listing = Filesystem.List(path: root_path) + match filesystem_listing_observation(directory: root_path, success: listing.success, entries: listing.entries, error: listing.error) { + FilesystemDirectoryListed(listed) => filesystem_listing_entry_names(listing: listed) |> filter(n => local_store_is_object_name(name: n)) + FilesystemDirectoryListingRefused { directory: _, error: _ } => [""] + FilesystemDirectorySubjectRefused { directory: _, cause: _ } => [""] + } +} + +fn hold_refusal_is_busy(r: LocalStoreHoldRefusal?) -> Bool { + match r { + Present { value: x } => match x { + LocalStoreFamilyBusy { holder: _, evidence: _ } => true + LocalStoreHolderUnobservable { holder: _, cause: _ } => false + LocalStoreHoldRecoveryRefused { refusal: _ } => false + LocalStoreHoldContended => false + LocalStoreHoldStoreUnavailable { cause: _ } => false + LocalStoreHoldSlotUndecodable { detail: _ } => false + LocalStoreHoldKeyNotAddressable => false + LocalStoreHoldSelfUnidentified { cause: _ } => false + LocalStoreHoldWindowTooNarrow { k: _ } => false + LocalStoreHoldUnsettled { settlement: _ } => false + LocalStoreHoldStale { observed: _ } => false + LocalStoreHoldWrongFamily { held: _, requested: _ } => false + } + Absent => false + } +} + +// ONE WRITER PER FAMILY. While the family is held -- here by this very process, which is observed live +// -- a commit refuses as busy and writes nothing: no index row, no object. Once released, the same +// commit lands. The interleavings per-operation CAS once had to handle (a row evicted under its +// publishing writer, a stale cleaner) need two writers in one family, and this is the wall between them. +test fn a_commit_while_the_family_is_held_refuses_and_writes_nothing_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let ry = bundle_req(module: "y") + match local_store_with_family(capability: store, of: ry, run: h => local_store_commit(capability: store, req: ry, payloads: half_budget_verdicts(module: "y"))) { + LocalStoreBracketRefused { refusal: _ } => false + LocalStoreBracketRan { value: blocked, hold_release: _ } => { + let y_absent = !Filesystem.Read(path: object_path(root_path: root_path, req: ry)).success + let landed = local_store_commit(capability: store, req: ry, payloads: half_budget_verdicts(module: "y")) + commit_tag(c: blocked.commit) == "occupancy_unavailable" && hold_refusal_is_busy(r: blocked.hold_refusal) + && count(blocked.evicted) == 0 && y_absent + && commit_tag(c: landed.commit) == "committed" + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: ry)) + } + } + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +fn hold_refusal_is_stale(r: LocalStoreHoldRefusal?) -> Bool { + match r { + Present { value: x } => match x { + LocalStoreHoldStale { observed: _ } => true + LocalStoreFamilyBusy { holder: _, evidence: _ } => false + LocalStoreHolderUnobservable { holder: _, cause: _ } => false + LocalStoreHoldRecoveryRefused { refusal: _ } => false + LocalStoreHoldContended => false + LocalStoreHoldStoreUnavailable { cause: _ } => false + LocalStoreHoldSlotUndecodable { detail: _ } => false + LocalStoreHoldKeyNotAddressable => false + LocalStoreHoldSelfUnidentified { cause: _ } => false + LocalStoreHoldWindowTooNarrow { k: _ } => false + LocalStoreHoldUnsettled { settlement: _ } => false + LocalStoreHoldWrongFamily { held: _, requested: _ } => false + } + Absent => false + } +} + +// RELEASE-THEN-MUTATE REFUSES. A bracket body returns the held value it was given, so it outlives its +// release. Committing under that leaked value re-observes the family's hold slot, finds it no longer +// held at that generation by that owner, and refuses as a stale hold; nothing is written. +test fn a_hold_leaked_out_of_its_bracket_is_refused_as_stale_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => + match local_store_with_family(capability: store, of: bundle_req(module: "s"), run: h => h) { + LocalStoreBracketRefused { refusal: _ } => false + LocalStoreBracketRan { value: leaked, hold_release: _ } => { + let rs = bundle_req(module: "s") + let c = local_store_commit_under(held: leaked, req: rs, payloads: verdicts(text: "s pass")) + commit_tag(c: c.commit) == "occupancy_unavailable" && hold_refusal_is_stale(r: c.hold_refusal) + && !Filesystem.Read(path: object_path(root_path: root_path, req: rs)).success + && store_lookup_is_miss(l: local_store_lookup(capability: store, req: rs)) + } + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// A HOLDER THAT DIED HOLDING THE FAMILY IS FREED BY OBSERVATION, NEVER BY AGE. The family is held by an +// owner naming this host's current boot and namespace but a pid with no process; the next commit looks +// the holder up, observes it dead, recovers the hold at the generation it saw, and lands. +test fn a_hold_left_by_a_dead_writer_is_recovered_by_observation_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => + match self_process_identity() { + SelfUnidentified { cause: _ } => false + SelfIdentified { process: me, pid_namespace: ns } => + match local_store_index_window() { + Absent => false + Present { value: window } => { + let dead = ProcessIdentity { boot_id: me.boot_id, pid: 4194303, start_time: "1" as NonEmptyStr } + let owner = concat(local_store_hold_owner_prefix, process_hold_text(p: dead, pid_namespace: ns)) as DurableHoldOwnerRef + let planted = match file_hold_acquire_windowed(root: root_path as NonEmptyStr, slot_key: local_store_hold_key(family: native_module_verdict_bundle_artifact_kind), requested_owner: owner, window: window) { + FileHoldAcquired { slot_key: _, owner: _, generation: _ } => true + FileHoldOccupied { slot_key: _, holder: _, generation: _ } => false + FileHoldAcquireLost { slot_key: _ } => false + FileHoldAcquireStoreUnavailable { slot_key: _, cause: _ } => false + FileHoldSlotUndecodable { slot_key: _, generation: _, detail: _ } => false + FileHoldKeyNotSlotAddressable { slot_key: _ } => false + } + let landed = local_store_commit(capability: store, req: bundle_req(module: "z"), payloads: verdicts(text: "z pass")) + planted && commit_tag(c: landed.commit) == "committed" + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: bundle_req(module: "z"))) + } + } + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// A DELETE THE HOST KEEPS REFUSING STOPS THE WRITES; IT NEVER GROWS THE DISK. X is committed, then its +// file is replaced by a directory, which Filesystem.Delete (a file delete) refuses on every attempt. +// Y's reservation evicts X, the delete is refused, and Y is NOT published (cleanup outstanding). A +// later Z refuses at reservation too, because X's bytes are still charged. Once the obstruction is +// gone, the next reservation confirms X gone and Z commits. Disk stays within budget at every step. +test fn a_persistently_refused_eviction_delete_stays_charged_and_stops_writes_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let budget = bundle_family_budget_bytes(root: root) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let rx = bundle_req(module: "x") + let cx = local_store_commit(capability: store, req: rx, payloads: half_budget_verdicts(module: "x")) + let x_path = object_path(root_path: root_path, req: rx) + let unlinked = Filesystem.Delete(path: x_path).success + let blocked = shell.Mkdir.NewOwnerOnly(path: x_path).success + let cy = local_store_commit(capability: store, req: bundle_req(module: "y"), payloads: half_budget_verdicts(module: "y")) + let y_within = root_object_bytes(root_path: root_path) <= budget + let cz_refused = local_store_commit(capability: store, req: bundle_req(module: "z"), payloads: half_budget_verdicts(module: "z")) + let cleared = shell.Remove.RecursiveForce(path: x_path).success + let cz = local_store_commit(capability: store, req: bundle_req(module: "z"), payloads: half_budget_verdicts(module: "z")) + commit_tag(c: cx.commit) == "committed" && unlinked && blocked + && commit_tag(c: cy.commit) == "cleanup_outstanding" + && cy.eviction_delete_refused |> any(t => t.name == object_name(req: rx) && delete_was_refused_by_the_host(t: t)) + && store_lookup_is_miss(l: local_store_lookup(capability: store, req: bundle_req(module: "y"))) + && y_within + && commit_tag(c: cz_refused.commit) == "cleanup_outstanding" + && cleared + && commit_tag(c: cz.commit) == "committed" + && cz.retired |> any(n => n == object_name(req: rx)) + && root_object_bytes(root_path: root_path) <= budget + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// THE DELETION AUTHORITY IS THE EXACT NAME RELATION. A traversal name and a foreign file that merely +// contains ".materialization." are not object names: the sweep leaves the foreign file alone, and an +// index row naming a traversal is a corrupt index that refuses, so it can never authorize a delete. +test fn only_canonical_object_names_are_indexed_or_swept_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let canonical = object_name(req: bundle_req(module: "c")) + let names_judged = local_store_is_object_name(name: canonical) + && !local_store_is_object_name(name: concat("../", canonical)) + && !local_store_is_object_name(name: "notes.materialization.v2") + && !local_store_is_object_name(name: concat(canonical, ".bak")) + && !local_store_is_object_name(name: concat("x", canonical)) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let first = local_store_commit(capability: store, req: bundle_req(module: "c"), payloads: verdicts(text: "c pass")) + let foreign = "notes.materialization.txt" + let planted = Filesystem.WriteCreateNew(path: store_path(root_path: root_path, name: foreign), content: "not the store's").success + let swept_alone = match local_store_open(root: root, budgets: witness_family_budgets) { + LocalStoreOpened { capability: c, durability: _ } => !(c.sweep.removed |> any(n => n == foreign)) + LocalStoreUnavailable { root_path: _, cause: _ } => false + } + let slot = store_path(root_path: root_path, name: concat(local_store_index_key(family: native_module_verdict_bundle_artifact_kind) as String, ".1")) + let traversal_row = Filesystem.Write(path: slot, content: concat("gunbc-store-occupancy v2\nnext 2\n../outside", ".materialization.v2 10 1")).success + let refused = match local_store_read_index(root_path: root_path, family: native_module_verdict_bundle_artifact_kind) { + LocalStoreIndexUnavailable { refusal: _ } => true + LocalStoreIndexAt { index: _, expected: _ } => false + } + commit_tag(c: first.commit) == "committed" && planted && swept_alone + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: bundle_req(module: "c"))) + && Filesystem.Read(path: store_path(root_path: root_path, name: foreign)).success + && traversal_row && refused + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + names_judged && verdict && removed.success +} + +// A RETRY IS NOT A WAY AROUND UNDELETED BYTES. Y's first attempt evicts X, whose delete the host +// refuses, so Y's row is reserved but Y is never published. Retrying the SAME request finds its own row +// live -- the arm that would converge -- but the object is absent, so the retry is an unfinished +// reservation and stays cleanup outstanding while X's bytes remain; Y is still absent and disk stays +// within budget. Once X is cleared, the same retry publishes. +test fn a_retry_of_an_unpublished_request_stays_refused_until_the_doomed_bytes_are_gone_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let budget = bundle_family_budget_bytes(root: root) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let rx = bundle_req(module: "x") + let ry = bundle_req(module: "y") + let cx = local_store_commit(capability: store, req: rx, payloads: half_budget_verdicts(module: "x")) + let x_path = object_path(root_path: root_path, req: rx) + let unlinked = Filesystem.Delete(path: x_path).success + let blocked = shell.Mkdir.NewOwnerOnly(path: x_path).success + let first = local_store_commit(capability: store, req: ry, payloads: half_budget_verdicts(module: "y")) + let retry = local_store_commit(capability: store, req: ry, payloads: half_budget_verdicts(module: "y")) + let y_absent_after_retry = !Filesystem.Read(path: object_path(root_path: root_path, req: ry)).success + let within_after_retry = root_object_bytes(root_path: root_path) <= budget + let cleared = shell.Remove.RecursiveForce(path: x_path).success + let settled = local_store_commit(capability: store, req: ry, payloads: half_budget_verdicts(module: "y")) + commit_tag(c: cx.commit) == "committed" && unlinked && blocked + && commit_tag(c: first.commit) == "cleanup_outstanding" + && commit_tag(c: retry.commit) == "cleanup_outstanding" + && y_absent_after_retry && within_after_retry + && cleared + && commit_tag(c: settled.commit) == "committed" + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: ry)) + && root_object_bytes(root_path: root_path) <= budget + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +fn sized_verdicts(module: String, n: Int) -> List { + verdicts(text: concat(module, repeat_string(s: " pass", n: n))) +} + +// AN UNFINISHED RESERVATION BINDS NOTHING, INCLUDING ITS SIZE. X (oldest) and W are live; a SMALL Y +// evicts X, whose delete is refused, so Y is refused unpublished with its small row live. X is +// cleared. The same request Y is retried with a LARGER result: it must be charged at its current size +// -- which no longer fits beside W, so W is evicted -- and disk stays within budget. Admitting the +// retry on the first attempt's smaller row would publish Y beside W and settle over the ceiling. +test fn a_retry_with_a_larger_result_is_charged_at_its_current_size_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let budget = bundle_family_budget_bytes(root: root) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let rx = bundle_req(module: "x") + let rw = bundle_req(module: "w") + let ry = bundle_req(module: "y") + let cx = local_store_commit(capability: store, req: rx, payloads: sized_verdicts(module: "x", n: 70)) + let cw = local_store_commit(capability: store, req: rw, payloads: sized_verdicts(module: "w", n: 50)) + let x_path = object_path(root_path: root_path, req: rx) + let unlinked = Filesystem.Delete(path: x_path).success + let blocked = shell.Mkdir.NewOwnerOnly(path: x_path).success + let small = local_store_commit(capability: store, req: ry, payloads: sized_verdicts(module: "y", n: 100)) + let cleared = shell.Remove.RecursiveForce(path: x_path).success + let large = local_store_commit(capability: store, req: ry, payloads: sized_verdicts(module: "y", n: 190)) + commit_tag(c: cx.commit) == "committed" && commit_tag(c: cw.commit) == "committed" && unlinked && blocked + && commit_tag(c: small.commit) == "cleanup_outstanding" + && cleared + && commit_tag(c: large.commit) == "committed" + && large.evicted |> any(n => n == object_name(req: rw)) + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: ry)) + && root_object_bytes(root_path: root_path) <= budget + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +fn index_row_count(root_path: String) -> Int { + match local_store_read_index(root_path: root_path, family: native_module_verdict_bundle_artifact_kind) { + LocalStoreIndexAt { index: i, expected: _ } => count(i.rows) + LocalStoreIndexUnavailable { refusal: _ } => 0 - 1 + } +} + +// PRESENCE THAT CANNOT BE ESTABLISHED REFUSES BEFORE ANY INDEX WRITE. Y's object path is occupied by +// something the store cannot read (a directory): it is neither verified present nor established absent, +// so the commit refuses, and the index is exactly what it was -- no row reserved, nothing evicted. +test fn an_object_whose_presence_cannot_be_established_refuses_before_any_index_write_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let cx = local_store_commit(capability: store, req: bundle_req(module: "x"), payloads: verdicts(text: "x pass")) + let before = index_row_count(root_path: root_path) + let blocked = shell.Mkdir.NewOwnerOnly(path: object_path(root_path: root_path, req: bundle_req(module: "y"))).success + let cy = local_store_commit(capability: store, req: bundle_req(module: "y"), payloads: verdicts(text: "y pass")) + commit_tag(c: cx.commit) == "committed" && blocked + && commit_tag(c: cy.commit) == "occupancy_unavailable" + && count(cy.evicted) == 0 + && before == 1 && index_row_count(root_path: root_path) == before + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +fn opening_refused_for_catalog(o: LocalStoreOpening) -> Bool { + match o { + LocalStoreOpened { capability: _, durability: _ } => false + LocalStoreUnavailable { root_path: _, cause: c } => match c { + LocalStoreCatalogMismatch { marker_path: _, offered: _ } => true + LocalStoreCatalogInadmissible { detail: _ } => true + LocalStoreGrantRefused { verb: _, target: _, frame: _ } => false + LocalStoreNotInitialized { marker_path: _ } => false + LocalStoreMarkerUnrecognized { marker_path: _ } => false + LocalStoreMarkerUnavailable { fault: _ } => false + LocalStoreMarkerMalformed { marker_path: _, detail: _ } => false + LocalStoreSweepIncomplete { sweep: _ } => false + } + } +} + +// THE SWEEP RUNS ONLY AGAINST THE STORE'S OWN CATALOG. The store was initialized with two families and +// holds an object of one of them. Reopening it with a roster that omits that family -- a partial one, +// or none -- would make its object look like an orphan to the sweep; instead both opens refuse before +// sweeping, and the object still serves. +test fn opening_with_a_partial_family_catalog_refuses_and_sweeps_nothing_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let c = local_store_commit(capability: store, req: bundle_req(module: "b"), payloads: verdicts(text: "b pass")) + let partial = local_store_open(root: root, budgets: witness_family_budgets |> filter(b => b.family != native_module_verdict_bundle_artifact_kind)) + let empty = local_store_open(root: root, budgets: []) + commit_tag(c: c.commit) == "committed" + && opening_refused_for_catalog(o: partial) && opening_refused_for_catalog(o: empty) + && Filesystem.Read(path: object_path(root_path: root_path, req: bundle_req(module: "b"))).success + && store_lookup_is_hit(l: local_store_lookup(capability: store, req: bundle_req(module: "b"))) + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// A WINDOWED SLOT STOPS ADVANCING WHILE ITS CLEANUP IS OUTSTANDING. The occupancy index's first +// generation is replaced by a directory, which no reclamation can delete. Once the window has moved +// past it, the next write finds a stale generation it cannot reclaim and refuses; every later write +// refuses too, and the slot's generation files do not grow with them. +test fn an_unreclaimable_index_generation_stops_further_writes_and_bounds_the_slot_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let slot_prefix = concat(local_store_index_key(family: native_module_verdict_bundle_artifact_kind) as String, ".") + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let c1 = local_store_commit(capability: store, req: bundle_req(module: "g1"), payloads: verdicts(text: "g pass")) + let c2 = local_store_commit(capability: store, req: bundle_req(module: "g2"), payloads: verdicts(text: "g pass")) + let first_gen = store_path(root_path: root_path, name: concat(slot_prefix, "1")) + let unlinked = Filesystem.Delete(path: first_gen).success + let blocked = shell.Mkdir.NewOwnerOnly(path: first_gen).success + let c3 = local_store_commit(capability: store, req: bundle_req(module: "g3"), payloads: verdicts(text: "g pass")) + let after_c3 = root_names_with_prefix(root_path: root_path, prefix: slot_prefix) + let later = [4, 5, 6, 7, 8] |> map(i => commit_tag(c: local_store_commit(capability: store, req: bundle_req(module: concat("g", i as String)), payloads: verdicts(text: "g pass")).commit)) + let population = root_names_with_prefix(root_path: root_path, prefix: slot_prefix) + commit_tag(c: c1.commit) == "committed" && commit_tag(c: c2.commit) == "committed" && unlinked && blocked + && commit_tag(c: c3.commit) == "committed" + && later |> all(t => t == "occupancy_unavailable") + && population == after_c3 && population <= 4 + } + } + let _cleared = shell.Remove.RecursiveForce(path: concat(root_path, concat("/", concat(local_store_index_key(family: native_module_verdict_bundle_artifact_kind) as String, ".1")))) + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// A REOPEN WITH A DIFFERENT BUDGET IS A DIFFERENT CATALOG. The store was created with the fixture +// budgets; reopening it with the same family at a larger budget refuses, so no capability exists that +// would enforce the larger figure, and the root's bytes stay within the budget it was created with. +test fn reopening_with_a_larger_budget_than_the_store_was_created_with_refuses_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let budget = bundle_family_budget_bytes(root: root) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let c = local_store_commit(capability: store, req: bundle_req(module: "b"), payloads: verdicts(text: "b pass")) + let larger = witness_family_budgets |> map(b => LocalStoreFamilyBudget { family: b.family, budget: byte_size(count: measure_count(m: b.budget) * 4) }) + commit_tag(c: c.commit) == "committed" + && opening_refused_for_catalog(o: local_store_open(root: root, budgets: larger)) + && root_object_bytes(root_path: root_path) <= budget + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +fn opening_refused_for_sweep(o: LocalStoreOpening) -> Bool { + match o { + LocalStoreOpened { capability: _, durability: _ } => false + LocalStoreUnavailable { root_path: _, cause: c } => match c { + LocalStoreSweepIncomplete { sweep: _ } => true + _ => false + } + } +} + +// AN OPEN WHOSE SWEEP LEAVES AN ORPHAN ON DISK MINTS NO CAPABILITY. A canonical object name the index +// does not charge is planted as a directory holding exactly the bytes the budget has left, so its +// delete is refused. Reopening refuses with the incomplete sweep, and nothing is written: the root +// holds the charged object plus that orphan and stays at the budget. Red when the capability is minted +// regardless of the sweep: the reopened store commits a second half-budget object beside the orphan +// and the root exceeds the budget. +test fn a_refused_orphan_delete_leaves_the_store_unwritable_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let budget = bundle_family_budget_bytes(root: root) + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let kept = local_store_commit(capability: store, req: bundle_req(module: "kept"), payloads: half_budget_verdicts(module: "kept")) + let orphan_dir = store_path(root_path: root_path, name: object_name(req: bundle_req(module: "orphan"))) + let orphan_bytes = budget - root_object_bytes(root_path: root_path) + let planted = shell.Mkdir.NewOwnerOnly(path: orphan_dir).success + && Filesystem.WriteCreateNew(path: concat(orphan_dir, "/residue"), content: repeat_string(s: "x", n: orphan_bytes)).success + let reopened = local_store_open(root: root, budgets: witness_family_budgets) + let wrote = match reopened { + LocalStoreOpened { capability: c, durability: _ } => + commit_tag(c: local_store_commit(capability: c, req: bundle_req(module: "late"), payloads: half_budget_verdicts(module: "late")).commit) == "committed" + LocalStoreUnavailable { root_path: _, cause: _ } => false + } + commit_tag(c: kept.commit) == "committed" && orphan_bytes > 0 && planted + && opening_refused_for_sweep(o: reopened) && !wrote + && root_object_bytes(root_path: root_path) + orphan_bytes <= budget + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} + +// AN OPEN WHOSE SWEEP CANNOT RUN MINTS NO CAPABILITY. While a bracket holds the family, a reopen cannot +// hold every family, so its sweep is unavailable and the open refuses; the planted orphan is still on +// disk. Once the bracket has released, the same reopen sweeps the orphan and opens. +test fn an_unavailable_sweep_leaves_the_store_unwritable_by_real_execution() -> Bool { + let root_path = fresh_root_path() + let root = WitnessScratchRoot { root: root_path } + let verdict = match opened(root: root) { + Absent => false + Present { value: store } => { + let orphan = object_name(req: bundle_req(module: "orphan")) + let planted = Filesystem.WriteCreateNew(path: store_path(root_path: root_path, name: orphan), content: "residue").success + match local_store_with_family(capability: store, of: bundle_req(module: "o"), run: h => local_store_open(root: root, budgets: witness_family_budgets)) { + LocalStoreBracketRefused { refusal: _ } => false + LocalStoreBracketRan { value: inside, hold_release: _ } => { + let still_there = Filesystem.Read(path: store_path(root_path: root_path, name: orphan)).success + let after = local_store_open(root: root, budgets: witness_family_budgets) + planted && opening_refused_for_sweep(o: inside) && still_there + && match after { + LocalStoreOpened { capability: c, durability: _ } => c.sweep.removed |> any(n => n == orphan) + LocalStoreUnavailable { root_path: _, cause: _ } => false + } + && !Filesystem.Read(path: store_path(root_path: root_path, name: orphan)).success + } + } + } + } + let removed = shell.Remove.RecursiveForce(path: root_path) + verdict && removed.success +} diff --git a/dag/test/claim/materialization_store_witness_test.dag b/dag/test/claim/materialization_store_witness_test.dag index a767a81c699..1914f4ca73c 100644 --- a/dag/test/claim/materialization_store_witness_test.dag +++ b/dag/test/claim/materialization_store_witness_test.dag @@ -1,5 +1,6 @@ module test.claim.materialization_store_witness +import std.cache_identity { native_module_verdict_bundle_artifact_kind } import std.types { Bool, Int, List, String, NonEmptyStr } import std.content_hash { ContentHash, Fnv1a64Structural, content_hash_atom, content_hash_of_value, serialize_content_hash, parse_content_hash, @@ -48,9 +49,11 @@ import extdeps.filesystem.filesystem_io { filesystem_exact_read, filesystem_create_new, } import extdeps.realization.materialization_store_local { - LocalStoreOpened, LocalStoreUnavailable, + LocalStorePreOpened, LocalStorePreOpenRefused, LocalStoreCatalogInadmissible, + store_marker_decode, StoreMarkerDecoded, StoreMarkerForeign, StoreMarkerMalformed, + store_marker_head, store_marker_catalog_field, store_marker_catalog_admissible, LocalStoreNotInitialized, LocalStoreMarkerUnrecognized, LocalStoreMarkerUnavailable, - local_store_open_decide, store_marker_content, materialization_store_local_id, + local_store_open_decide, store_marker_content, materialization_store_local_id, LocalStoreFamilyBudget, store_read_observation, } import v2.std.live_tree { LiveTreeDisposition, SubstrateInputsOnly } @@ -401,23 +404,97 @@ test fn the_error_kind_channel_separates_absent_from_unreadable() -> Bool { absent && unreadable && unrecognized && occupied && refused } -// a durable store's marker is not an ephemeral store's marker: a scratch root holding it refuses +// a durable store's marker is not an ephemeral store's marker: a scratch root holding it refuses. +// The marker decision yields a PRE-OPEN whose budgets are the catalog DECODED from the marker: offered +// out of order, the bound catalog still comes back in the marker's canonical order, so the binding is +// read from the marker and not copied from the caller's list. test fn a_store_opens_only_on_its_own_marker() -> Bool { let root = WitnessScratchRoot { root: "/tmp/gunbc_matstore_witness" } - let marker = store_marker_content(root: root, instance_label: "witness") - let opened = match local_store_open_decide(root: root, marker_read: FilesystemExactPathRead { path: "/m", content: marker }) { - LocalStoreOpened { root: _, store: s, durability: _ } => s.provider == materialization_store_local_id - LocalStoreUnavailable { root_path: _, cause: _ } => false + let two = [budget_row(family: "zeta", bytes: 7), budget_row(family: "alpha", bytes: 4096)] + let marker = store_marker_content(root: root, instance_label: "witness", budgets: two) + let opened = match local_store_open_decide(root: root, marker_read: FilesystemExactPathRead { path: "/m", content: marker }, budgets: two) { + LocalStorePreOpened { pre: p, durability: _ } => + p.binding.store.provider == materialization_store_local_id + && (p.binding.budgets |> map(b => b.family as String)) == ["alpha", "zeta"] + && (p.binding.budgets |> map(b => measure_count(m: b.budget))) == [4096, 7] + LocalStorePreOpenRefused { root_path: _, cause: _ } => false } - let uninitialized = match local_store_open_decide(root: root, marker_read: FilesystemExactPathAbsent { path: "/m" }) { - LocalStoreUnavailable { root_path: _, cause: c } => match c { LocalStoreNotInitialized { marker_path: _ } => true, _ => false } - LocalStoreOpened { root: _, store: _, durability: _ } => false + let uninitialized = match local_store_open_decide(root: root, marker_read: FilesystemExactPathAbsent { path: "/m" }, budgets: two) { + LocalStorePreOpenRefused { root_path: _, cause: c } => match c { LocalStoreNotInitialized { marker_path: _ } => true, _ => false } + LocalStorePreOpened { pre: _, durability: _ } => false } - let foreign = match local_store_open_decide(root: root, marker_read: FilesystemExactPathRead { path: "/m", content: store_marker_content(root: DurableHostVolumeRoot, instance_label: "witness") }) { - LocalStoreUnavailable { root_path: _, cause: c } => match c { LocalStoreMarkerUnrecognized { marker_path: _ } => true, _ => false } - LocalStoreOpened { root: _, store: _, durability: _ } => false + let foreign = match local_store_open_decide(root: root, marker_read: FilesystemExactPathRead { path: "/m", content: store_marker_content(root: DurableHostVolumeRoot, instance_label: "witness", budgets: two) }, budgets: two) { + LocalStorePreOpenRefused { root_path: _, cause: c } => match c { LocalStoreMarkerUnrecognized { marker_path: _ } => true, _ => false } + LocalStorePreOpened { pre: _, durability: _ } => false } - opened && uninitialized && foreign + let duplicate_offer = match local_store_open_decide(root: root, marker_read: FilesystemExactPathRead { path: "/m", content: marker }, budgets: [budget_row(family: "alpha", bytes: 4096), budget_row(family: "alpha", bytes: 4096), budget_row(family: "zeta", bytes: 7)]) { + LocalStorePreOpenRefused { root_path: _, cause: c } => match c { LocalStoreCatalogInadmissible { detail: _ } => true, _ => false } + LocalStorePreOpened { pre: _, durability: _ } => false + } + opened && uninitialized && foreign && duplicate_offer +} + +fn budget_row(family: String, bytes: Int) -> LocalStoreFamilyBudget { + LocalStoreFamilyBudget { family: family as ArtifactKindId, budget: byte_size(count: bytes) } +} + +fn catalog_text(c: List) -> List { + c |> map(b => concat(concat(b.family as String, " -> "), measure_count(m: b.budget) as String)) +} + +fn decodes_to(root: MaterializationStoreRoot, content: String, expected: List) -> Bool { + match store_marker_decode(root: root, content: content) { + StoreMarkerDecoded { catalog: c } => catalog_text(c: c) == catalog_text(c: expected) + StoreMarkerForeign => false + StoreMarkerMalformed { detail: _ } => false + } +} + +fn is_malformed(root: MaterializationStoreRoot, content: String) -> Bool { + match store_marker_decode(root: root, content: content) { + StoreMarkerMalformed { detail: _ } => true + StoreMarkerDecoded { catalog: _ } => false + StoreMarkerForeign => false + } +} + +fn hand_marker(root: MaterializationStoreRoot, after_head: String) -> String { + concat(store_marker_head(root: root), after_head) +} + +// THE MARKER CODEC IS EXACT. Family names that carry every delimiter the old text form used +// (",", "=", ";", ":", a line feed, a non-ASCII scalar) round-trip to exactly their catalog, and the +// catalog [x=1, y=2] and the single family "x=1,y" at 2 -- one text under the old encoding -- encode +// differently and each decodes to itself. +test fn the_marker_catalog_round_trips_and_no_two_catalogs_share_an_encoding() -> Bool { + let root = WitnessScratchRoot { root: "/tmp/gunbc_matstore_witness" } + let awkward = [budget_row(family: "a=1,b;2:c", bytes: 10), budget_row(family: "line\nfeed", bytes: 0), budget_row(family: "é€", bytes: 99)] + let sorted_awkward = awkward |> sort_by(b => b.family as String) + let pair = [budget_row(family: "x", bytes: 1), budget_row(family: "y", bytes: 2)] + let fused = [budget_row(family: "x=1,y", bytes: 2)] + decodes_to(root: root, content: store_marker_content(root: root, instance_label: "w", budgets: awkward), expected: sorted_awkward) + && decodes_to(root: root, content: store_marker_content(root: root, instance_label: "w", budgets: pair), expected: pair) + && decodes_to(root: root, content: store_marker_content(root: root, instance_label: "w", budgets: fused), expected: fused) + && store_marker_catalog_field(budgets: pair) != store_marker_catalog_field(budgets: fused) +} + +// THE DECODER ACCEPTS ONLY THE ENCODER'S IMAGE. A duplicated family refuses in either order (so no +// "first row wins" can pick a budget), two catalog fields refuse, and so do a leading zero, a length +// that runs past its name, an unsorted pair, an empty catalog and a missing or doubled instance line. +test fn a_marker_outside_the_canonical_encoding_is_malformed() -> Bool { + let root = WitnessScratchRoot { root: "/tmp/gunbc_matstore_witness" } + is_malformed(root: root, content: hand_marker(root: root, after_head: "1:F=10;1:F=20;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "1:F=20;1:F=10;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "1:x=1;\nfamilies 1:x=9;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "01:x=1;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "1:x=01;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "9:x=1;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "1:y=2;1:x=1;\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "\ninstance w\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "1:x=1;\n")) + && is_malformed(root: root, content: hand_marker(root: root, after_head: "1:x=1;\ninstance w\ninstance v\n")) + && decodes_to(root: root, content: hand_marker(root: root, after_head: "1:x=1;1:y=2;\ninstance w\n"), expected: [budget_row(family: "x", bytes: 1), budget_row(family: "y", bytes: 2)]) + && !store_marker_catalog_admissible(budgets: [budget_row(family: "F", bytes: 10), budget_row(family: "F", bytes: 20)]) } // A part's size is its UTF-8 BYTE count, not its code-point count: "é€" is 2 code points and 5 diff --git a/src/v2/workflow/floor_route_gap.dag b/src/v2/workflow/floor_route_gap.dag index 5e347133cd3..c3636ae3849 100644 --- a/src/v2/workflow/floor_route_gap.dag +++ b/src/v2/workflow/floor_route_gap.dag @@ -1067,7 +1067,67 @@ fn floor_route_gap_expectation_chunk_09() -> List { head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.an_unavailable_store_refuses_and_leaves_no_shadow_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, tail: Cons { head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_batch_answers_each_request_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.the_family_budget_holds_on_disk_across_overrun_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.an_artifact_over_its_family_budget_refuses_and_evicts_nothing_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_family_without_a_declared_budget_refuses_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_corrupt_occupancy_index_refuses_and_never_reads_as_empty_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.an_open_sweeps_an_object_no_index_names_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.the_occupancy_index_slot_stays_within_its_window_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.recency_is_recorded_once_and_never_fails_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_commit_while_the_family_is_held_refuses_and_writes_nothing_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_persistently_refused_eviction_delete_stays_charged_and_stops_writes_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.only_canonical_object_names_are_indexed_or_swept_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_retry_of_an_unpublished_request_stays_refused_until_the_doomed_bytes_are_gone_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_retry_with_a_larger_result_is_charged_at_its_current_size_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_hold_left_by_a_dead_writer_is_recovered_by_observation_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.an_object_whose_presence_cannot_be_established_refuses_before_any_index_write_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.opening_with_a_partial_family_catalog_refuses_and_sweeps_nothing_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.an_unreclaimable_index_generation_stops_further_writes_and_bounds_the_slot_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_hold_leaked_out_of_its_bracket_is_refused_as_stale_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.reopening_with_a_larger_budget_than_the_store_was_created_with_refuses_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.a_refused_orphan_delete_leaves_the_store_unwritable_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.materialization_store_local_wet_witness.an_unavailable_sweep_leaves_the_store_unwritable_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, tail: Empty {} + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } + } } } } @@ -1174,6 +1234,12 @@ fn floor_route_gap_expectation_chunk_12() -> List { tail: Cons { head: FloorRouteGapExpectation { identity: "test.claim.durable_cas_file_store_wet_witness.an_established_empty_root_reads_the_slot_absent_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.durable_cas_file_store_wet_witness.a_window_slot_holds_at_most_k_plus_one_generations_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.durable_cas_file_store_wet_witness.the_unwindowed_compare_and_set_keeps_every_generation_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { + head: FloorRouteGapExpectation { identity: "test.claim.durable_cas_file_store_wet_witness.a_racing_writer_under_a_window_retries_and_loses_no_commit_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, + tail: Cons { head: FloorRouteGapExpectation { identity: "test.claim.filesystem_link_create_new_wet_witness.a_link_publishes_the_source_bytes_and_keeps_the_source_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, tail: Cons { head: FloorRouteGapExpectation { identity: "test.claim.filesystem_link_create_new_wet_witness.an_existing_target_refuses_as_occupied_and_is_unchanged_by_real_execution", operation: "DirWithTemplate", ground: NoMockResponse {} }, @@ -1186,6 +1252,9 @@ fn floor_route_gap_expectation_chunk_12() -> List { } } } + } + } + } } } } diff --git a/src/v2/workflow/local_repo_wet_terminal.dag b/src/v2/workflow/local_repo_wet_terminal.dag index 649bfaeba64..2989382c68f 100644 --- a/src/v2/workflow/local_repo_wet_terminal.dag +++ b/src/v2/workflow/local_repo_wet_terminal.dag @@ -1806,6 +1806,24 @@ fn local_repo_wet_schedule() -> List { function: "an_established_empty_root_reads_the_slot_absent_by_real_execution", expectation: ExpectedToHold {} }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.durable_cas_file_store_wet_witness", function: "a_window_slot_holds_at_most_k_plus_one_generations_by_real_execution" }, + entry: "dag/test/claim/durable_cas_file_store_wet_witness_test.dag", + function: "a_window_slot_holds_at_most_k_plus_one_generations_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.durable_cas_file_store_wet_witness", function: "the_unwindowed_compare_and_set_keeps_every_generation_by_real_execution" }, + entry: "dag/test/claim/durable_cas_file_store_wet_witness_test.dag", + function: "the_unwindowed_compare_and_set_keeps_every_generation_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.durable_cas_file_store_wet_witness", function: "a_racing_writer_under_a_window_retries_and_loses_no_commit_by_real_execution" }, + entry: "dag/test/claim/durable_cas_file_store_wet_witness_test.dag", + function: "a_racing_writer_under_a_window_retries_and_loses_no_commit_by_real_execution", + expectation: ExpectedToHold {} + }, WetScheduledClaim { identity: WitnessIdentity { module_path: "test.claim.filesystem_link_create_new_wet_witness", function: "a_link_publishes_the_source_bytes_and_keeps_the_source_by_real_execution" }, entry: "dag/test/claim/filesystem_link_create_new_wet_witness_test.dag", @@ -1968,6 +1986,126 @@ fn local_repo_wet_schedule() -> List { function: "a_batch_answers_each_request_by_real_execution", expectation: ExpectedToHold {} }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "the_family_budget_holds_on_disk_across_overrun_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "the_family_budget_holds_on_disk_across_overrun_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "an_artifact_over_its_family_budget_refuses_and_evicts_nothing_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "an_artifact_over_its_family_budget_refuses_and_evicts_nothing_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_family_without_a_declared_budget_refuses_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_family_without_a_declared_budget_refuses_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_corrupt_occupancy_index_refuses_and_never_reads_as_empty_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_corrupt_occupancy_index_refuses_and_never_reads_as_empty_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "an_open_sweeps_an_object_no_index_names_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "an_open_sweeps_an_object_no_index_names_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "the_occupancy_index_slot_stays_within_its_window_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "the_occupancy_index_slot_stays_within_its_window_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "recency_is_recorded_once_and_never_fails_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "recency_is_recorded_once_and_never_fails_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_commit_while_the_family_is_held_refuses_and_writes_nothing_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_commit_while_the_family_is_held_refuses_and_writes_nothing_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_persistently_refused_eviction_delete_stays_charged_and_stops_writes_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_persistently_refused_eviction_delete_stays_charged_and_stops_writes_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "only_canonical_object_names_are_indexed_or_swept_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "only_canonical_object_names_are_indexed_or_swept_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_retry_of_an_unpublished_request_stays_refused_until_the_doomed_bytes_are_gone_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_retry_of_an_unpublished_request_stays_refused_until_the_doomed_bytes_are_gone_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_retry_with_a_larger_result_is_charged_at_its_current_size_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_retry_with_a_larger_result_is_charged_at_its_current_size_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_hold_left_by_a_dead_writer_is_recovered_by_observation_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_hold_left_by_a_dead_writer_is_recovered_by_observation_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "an_object_whose_presence_cannot_be_established_refuses_before_any_index_write_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "an_object_whose_presence_cannot_be_established_refuses_before_any_index_write_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "opening_with_a_partial_family_catalog_refuses_and_sweeps_nothing_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "opening_with_a_partial_family_catalog_refuses_and_sweeps_nothing_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "an_unreclaimable_index_generation_stops_further_writes_and_bounds_the_slot_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "an_unreclaimable_index_generation_stops_further_writes_and_bounds_the_slot_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_hold_leaked_out_of_its_bracket_is_refused_as_stale_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_hold_leaked_out_of_its_bracket_is_refused_as_stale_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "reopening_with_a_larger_budget_than_the_store_was_created_with_refuses_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "reopening_with_a_larger_budget_than_the_store_was_created_with_refuses_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "a_refused_orphan_delete_leaves_the_store_unwritable_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "a_refused_orphan_delete_leaves_the_store_unwritable_by_real_execution", + expectation: ExpectedToHold {} + }, + WetScheduledClaim { + identity: WitnessIdentity { module_path: "test.claim.materialization_store_local_wet_witness", function: "an_unavailable_sweep_leaves_the_store_unwritable_by_real_execution" }, + entry: "dag/test/claim/materialization_store_local_wet_witness_test.dag", + function: "an_unavailable_sweep_leaves_the_store_unwritable_by_real_execution", + expectation: ExpectedToHold {} + }, WetScheduledClaim { identity: WitnessIdentity { module_path: "test.claim.pre_os_capture_replay_witness", function: "w_early_exit_refuses_absence_on_real_bytes" }, entry: "dag/test/claim/machine_intake/pre_os_capture_replay_witness_test.dag",