Skip to content

G1: belt verify through the materialization provider; compute outcomes and binaries in the bounded store - #13097

Open
gunbai-bot[bot] wants to merge 80 commits into
mainfrom
session/bold-dove-431-cutover
Open

gunbai-bot[bot] wants to merge 80 commits into
mainfrom
session/bold-dove-431-cutover

Conversation

@gunbai-bot

@gunbai-bot gunbai-bot Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Dogfood G1, part 2 (node adhoc-cfe036d0-023): the shared-computation cutover. This PR is stacked on #13095 (C1b byte parts), which sits on #13081, #13082 and #13091, and it lands after that stack. It includes #13072, the compute request family with the ruling's fixes, which lands first.

What moves onto the real route

  • One store for compute outcomes. They realize only through extdeps.realization.materialization_store_local, in the bounded per-host durable root.
    • The provider's private outcome.json is deleted.
    • The per-identity store_dir is deleted.
    • Built binaries are byte parts of the same record (C1b): staged beside the root, linked create-only, charged to the family budget, and evicted with the record.
  • Family budget, a named policy. gunbc.materialization_store_budgets materialization_store_compute_work_budget_policy is 1 GiB per host. The store transport receives budgets as a parameter, so the policy lives in that module beside the typed-module row. The budget must hold one build record per compute_pool_slots, with headroom. It is chosen rather than derived, because no instrument re-derives a build's size. The revision trigger (a build record refused for not fitting, or evicted while a dependent is in flight) is stated beside the row.
  • The ladder decides store versus recompute. compute_work_ladder_verdict judges against the store's own family provider row. A build, a compile or a hermetic claim run is Discharged, so it is memoized. A wet claim run is ExemptFreshEffect, so it is recomputed.
  • The demand identity and the execution identity are separate (ruling item 2). Memoized work runs once per request key, and a second identical request attaches. A recomputed (wet) run executes per occurrence, never reads the store and is never committed.
  • A refused lookup refuses; only established absence produces. The store's own StoreLookup is routed directly. realize_route's only recompute-on-rejection arm is FreshnessExpired, and this store has no freshness window to expire. That is a stated divergence; the rule itself is kept whole.
  • Dependents execute only verified readings. compute_dependency_read resolves the build's binaries through local_store_lookup_parts. A build record that is evicted or tampered refuses; it never triggers a rebuild.
  • Belt verify. Each GunbcClaimValidation now runs as RunClaims through compute_provide, on the verified head's tree, in a capped unit.
    • The affected-set diff window is a keyed declared input: its identity is in the request key, and its path is bound into the unit's environment.
    • The run is wet, matching the host --claim-run it replaces. The argv now spells --wet. Before this, claim_batch's hermetic default ran "wet" requests hermetic.
  • Receipt (G2). ValidationExecution.computation cites the request key, the outcome identity, the verified head's ExactTree, and whether the run attached or computed. A present-but-broken citation makes the receipt unreadable.
  • Reader for bold-bee-114. gunbc.roadmap_belt_actuate shared_computation_observe_for_attempt(instance, attempt, head_sha) returns SharedComputationObserved{head_sha, window_identity, citations}, Absent{reason} or Refused{cause}.
  • Deploy. gunbc.live_deploy.spec gets a MaterializationStoreDirectory row, which discharges the std.materialization_store_grant trigger. Its path comes from the grant's datum.

Evidence (claim_batch, remote; wet files run with --wet)

Witness file Passed
compute/work_request_witness_test 23/23
materialization_provider_witness_test 48/48 (stale get roster row retired on the runner only; #13064 retires it)
materialization_store_witness_test 28/29
materialization_store_local_wet_witness_test 21/21
compute/attempt_lifecycle_wet_witness_test 13/13
roadmap/roadmap_verification_receipt_witness_test 8/8
roadmap/roadmap_validation_oracle_witness_test 38/38
roadmap/roadmap_belt_actuate_witness_test 133/140
devboot_subject_identity_witness_test 20/20
compute/work_class_grant_witness_test 20/20

The one store failure (a_tampered_part_size_is_an_integrity_refusal) and the 7 belt failures (witness_exec_*, hermetic route gaps) fail identically on main 6d87482.

attempt_lifecycle_wet's a_build_record_round_trips_and_a_dependent_resolves_verified_binaries is the real-route claim the pure controls are paired with. It runs commit with staged parts, then attach, then dependent resolution, on a scratch root.

Not in this PR

🤖 Generated with Claude Code

Brian Searls and others added 30 commits October 3, 2026 04:21
…k_identity is its derived request_key

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ReleasePolicy gains ReleasedAfterInterval { after_write } -- an object lifecycle
rule or artifact retention period reclaims the entry, after which a lookup is an
established Miss. It is distinct from InvalidationTrigger TtlExpiry (freshness).

realize_route no longer recomputes over every rejected hit: only FreshnessExpired
(a declared invalidation) recomputes; integrity and availability rejections
refuse, so a corrupt or unreachable store is never absorbed by a rebuild.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… the built bytes

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…dowed occupancy index, self-clean on open

Each request family's store is constructed by std.artifact_store store_over_provider from its own
CapacityBounded/ExactLimit/LeastRecentlyUsed provider row (typed-module 4 GiB as named policy); the
host ceiling is their declared sum. Occupancy lives in a per-family CAS slot retained as a
generation window (new opt-in gunbc.durable_cas_file_store CasSlotRetention; every existing slot
declares KeepAllGenerations). Commits reserve before publishing, evict LRU within the family, and
refuse typed on did-not-fit / undeclared budget / unreadable index. Open sweeps unindexed objects;
recency is advisory, one CAS per process.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ects, sealed FabricBlobReading, put plan with size bound)

Split out of session/royal-moth-86 (#13080) so the bounded local store (C1b) can realize byte parts on it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…are module-item grain); retire stale std.materialization_object#get roster row

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e; belt claim verify through compute_provide; receipt cites the shared computation

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…im run spells --wet

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s address fails closed instead of substituting a digest

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… no mock) and scheduled on the local-repo wet lane, as their file's existing wet witnesses are

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…s only at the occupancy codec

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e refusal, one realization for seed and emitted programs

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…, not-found; classifier: cross-device, unrecognized), enrolled as held route gaps and on the wet lane

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… third enrolment beside the route-gap and wet-schedule rows)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… session/quick-gull-60-c1b

# Conflicts:
#	src/v2/workflow/floor_route_gap.dag
#	src/v2/workflow/local_repo_wet_terminal.dag
…rd v3), staged files linked create-only and verified on every read, charged to the family budget and evicted with their record

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…d; dependents execute verified readings; store_dir deleted

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… build commit, attach and dependent resolution on a scratch root

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… identity (triple, rustc -vV, cargo, sccache, provider revision); wet claim runs refused on the legacy provider; honest native provenance rung

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…m, link into /tmp, asserting the st_dev precondition (coreutils.Stat.PathDevice) so an unobservable runner reds rather than greens

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Brian Searls and others added 5 commits October 3, 2026 21:06
…ared frontier at #13095, not a present fact

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ts trigger (local_store_link_part consumes LinkCreateNew); the seed-growth receipt names the present consumer

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ve a body-position annotation to its declaration

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Merged the reworked C1 base (session/quick-gull-60-c1b bf9ade3), with follow-ups in 9e1919d:

  • The only textual conflict was the import line, and I kept both sides.
  • The merged head did not resolve until two fixes:
    • compute_store_commit_refusal_detail now covers the base's new StoreCommitCleanupOutstanding and StoreCommitEvictedDuringPublish.
    • An annotation inside a test fn body is moved to its declaration (DESIGN 4c).

Remote runs on this head:

  • store_local wet: 24/24
  • attempt_lifecycle wet: 14/14
  • work_request: 23/23
  • verification_receipt: 8/8
  • materialization_store: 28/29, where the one failure (a_tampered_part_size_is_an_integrity_refusal) fails on main too

— sent from bold-dove-431

gunbc-ci-auto-heal and others added 7 commits October 3, 2026 21:49
- LocalStoreDeleteAttempt carries the FilesystemDelete outcome itself; confirmed and refused
  deletes are split by matching it once, and receipts (eviction_delete_refused, the sweep's
  delete_refused) carry the refused attempts with their typed outcomes.
- LocalStoreIndexRefusal carries the CAS layer's causes as they are (CasAttemptAdmission,
  CasStoreFailure, CasUnreadableSlot, window, decode, contention) on the index read, the index
  write, the reservation, the post-publish re-read, the sweep and recency. The one rendering to
  text is local_store_index_refusal_text, used only where std.materialization_object's
  realization-agnostic refusal takes a String; the commit receipt also carries the typed refusal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
LocalStoreRecordDelete says which file the host refused (the record, with its typed attempt;
or its parts, with theirs) or that the root could not be listed, so no record is assumed
partless. Receipts carry the record-level attempts. 24/24 store wet controls pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…t named for its unit

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…75099)

- The policy figure, the durable rows and the host ceiling move to
  gunbc.materialization_store_budgets (with a C2 consumer frontier); the transport keeps the
  budget shape, its sum and its enforcement, and every admitting/sweeping operation takes the
  rows as a parameter. Witness fixture rows live in the witness file. The catalog row is
  materialization_store_local_facts_at(ceiling) applied by the deploying layer.
- local_store_doomed_bytes returns ByteSize and is the one doomed-bytes sum (reserve and
  publish); the reservation carries its budget, replacing a zero fallback; an unprepared
  commit has its own reservation arm.
- design_argument leasing row: sentence boundary and doubled period fixed; DESIGN.md regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… through it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…moves to gunbc.materialization_store_budgets; the provider opens its store with root + budgets

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…tier naming gunbc#13077 (review 75127)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Two changes in b37307b.

1. Merged the reworked C1 base (248bd5e074). Budgets are now the consumer's policy:

  • The compute budget row moves to gunbc.materialization_store_budgets (materialization_store_compute_work_budget_policy, a ByteSize derivation) beside the typed-module row.
  • The provider opens its store as ComputeStore { root, budgets }: compute_durable_store in production, and fixture rows in the wet witness.
  • That module's frontier note now names the compute provider as the durable root's executing consumer. The typed-module row keeps its declared C2 frontier.
  • The host-ceiling control expects the sum of the typed-module and compute budgets.

2. Review 75127 is right, and fixed. shared_computation_observe_for_attempt has no caller in this change. Its consumer is gunbc.roadmap_dogfood_route DogfoodRouteReceipt (#13077), which is not on this base and does not call it yet. So:

  • shared_computation_reader_consumer_frontier declares that frontier with its trigger: roadmap: the dogfood route as one acceptance case (factory-dogfood-route) #13077's shared-computation stage calls the reader. It is retired by that and nothing else.
  • SharedComputationAbsent now carries a closed SharedComputationAbsence (NoVerificationSelectionPublished | SelectedReceiptNotWritten | ReceiptRecordsNoSharedComputation) instead of prose.
  • SharedComputationRefused { cause } stays a String, because it carries the upstream observation's own refusal text.

Remote runs:

  • store_local wet: 24/24
  • attempt_lifecycle wet: 14/14
  • work_request: 23/23
  • verification_receipt: 8/8
  • devboot identity: 20/20
  • belt_actuate: 134/141. The 7 failures are the witness_exec_* hermetic route gaps, which fail identically on main.

— sent from bold-dove-431

…sion trigger, not a transcribed measurement (review 75170)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

Review 75170: fixed in c6c7c40. gunbc.materialization_store_budgets materialization_store_compute_work_budget_policy is now a named policy (1 GiB per host), the same form as the typed-module row beside it.

  • Reasoning: the budget must hold one build record per compute_pool_slots with an order of magnitude of headroom.
  • Not a reading: it is stated as chosen, not derived from a measurement.
  • Revision trigger: a build record refused StoreCommitDidNotFit, or evicted while a dependent of its tree is in flight, on the durable root.

The transcribed per-binary sizes and the record_size_measured and retained_records rows are deleted; no figure of a build's size is carried, because no instrument re-derives one. materialization_store_local_wet_witness_test (--wet) passes 24/24, including the host-ceiling sum.

— sent from bold-dove-431

Base automatically changed from session/quick-gull-60-c1b to main October 6, 2026 14:31
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Retargeted to main after the store stack landed: merge commit 3a08c88, with parents the previous cutover tip and main a453be9.

Merge. The stack was squash-merged, so a textual three-way merge carried stale stack content and differed from main in about 1,000 files. I rebuilt the merge as main's tree plus only this PR's own diff (248bd5e074..c6c7c40954). It now differs from main in exactly this PR's files.

Adapted to main's final store API:

  • lookup, commit and parts lookup take the opened-store LocalStoreCapability;
  • LocalStoreOpened { capability, durability };
  • the four new LocalStoreUnavailableCause arms are rendered;
  • StoreCommitEvictedDuringPublish (gone on main) is dropped.

Also moved the live_deploy deploy-row annotation to module-item grain.

Main is currently broken for every closure reaching the roadmap modules: #13388 removed v2.std.optional while #13367's claude_code modules still import it. bold-bee-114 owns the fix as #13480. Until it lands this PR's CI will fail on that import, not on this change. With those three imports patched on the runner only, remote runs:

  • work_request: 23/23
  • attempt_lifecycle wet: 14/14
  • store_local wet: 39/39
  • provider: 48/48
  • verification_receipt: 8/8
  • work_class_grant: 20/20
  • devboot: 20/20
  • materialization_store: 30/31 (a_tampered_part_size_is_an_integrity_refusal, which fails on main too)
  • belt_actuate: 137/144 (the 7 witness_exec_* hermetic route gaps, which fail on main too)

I'll merge main again once #13480 lands.

— sent from bold-dove-431

…ncy binaries join by typed part id and digest, not path shape (review 77120)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Review 77120: both findings fixed in 74a8846.

  1. A refused commit keeps its unstage failure. The StoreCommitRefused arm of compute_store_commit now joins compute_unstage's detail into the refusal when a staged part could not be removed. An unremoved stage is reported on every ending, as the provider's own rule says.
  2. Binaries join by typed part identity. compute_dependency_decide binds the hit's blobs, takes each binary's part by its ArtifactOutputId (compute_work_binary_output), and selects the verified reading whose SHA-256 is that part's digest. The empty-record local_store_part_name suffix probe and its import are deleted.

Remote runs, with the v2.std.optional imports patched on the runner only until #13480 lands:

  • attempt_lifecycle_wet (--wet, including the real commit → attach → dependency-resolution route): 14/14
  • work_request (including a_dependency_read_refuses_an_absent_or_refused_build): 23/23

— sent from bold-dove-431

@gunbai-bot

gunbai-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

The generated failure at 74a8846 is main's break, not this change. Run 37496538580's required-ci: generated-artifact CarrierRefused is unresolved import: module 'v2.std.optional' not found in dag/extdeps/llm/claude_code_stream_json.dag, dag/gunbc/claude_code_credential.dag and dag/gunbc/claude_code_limit_standing.dag. Those files are identical on main at a453be9: #13388 re-homed the module and #13367's imports crossed it. The repoint is #13487, which superseded #13480. I'm not duplicating it here. Once it lands I'll merge main and re-run CI. This PR's own witnesses are green with those imports patched on the runner only (see the comment above).

— sent from bold-dove-431

Brian Searls and others added 2 commits October 6, 2026 18:10
… binaries are one verified reading per GunbcBinary; an uncitable compute claim run refuses

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Review 77173: all three findings fixed in 87ec0f8.

  1. Stale DeclRef. gunbc.harness.harness_guidance RunRebuildsOnTreeChange is now grounded on gunbc.compute.work_provider_local compute_work_identity. I checked every decl_ref into gunbc.compute.work_request and work_provider_local across the corpus, and this was the only one naming a deleted symbol.
  2. Closed set reopened as strings. ComputeDependency is now VerifiedBuildBinaries { gunbc: FabricBlobReading, claim_batch: FabricBlobReading }, so there is one verified reading per GunbcBinary and no list or name lookup. compute_dependency_program(dependency, bin: GunbcBinary) is a total match over the variant, and the argv call sites pass GunbcDriver / ClaimBatchRunner. A missing binary has no constructor, and resolution refuses unless both verify (compute_verified_binary, joined by part id and digest).
  3. Pass without a citation. belt_run_claim_through_compute now refuses first when the SharedComputationCitation cannot be built, and records no execution. A compute-routed claim run therefore never passes, fails, or appears in the receipt without its citation, which matches ValidationExecution.computation's contract.

Remote runs on main with #13487 (no runner-side patches now):

  • attempt_lifecycle_wet (--wet, real commit → attach → dependency resolution): 14/14
  • work_request: 23/23
  • roadmap_belt_actuate: 137/144. The 7 failures are the witness_exec_* hermetic route gaps, which fail on main too.

— sent from bold-dove-431

…ts display word (review 77184 note)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@gunbai-bot

gunbai-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

The floor red at a9b217a (run 37516531520) is not this change. The floor ran on the merge ref against main 580f66d (#13419), and its single blocking diagnostic is:

src/v2/test/claim/long/mandatory_tag_gate_witness_test.dag:78:77: value does not inhabit its declared type at the record literal field 'root': declared 'Primitive(Node)', produced 'Product(NormalizedTree)'

This PR does not touch that witness or NormalizedTree, and the file is identical to main. The other three lanes are green: emit-build, generated, rust-unit-tests. I'm reporting it upstream rather than patching another lane's file here, and I'll re-run once main is fixed.

— sent from bold-dove-431

@gunbai-bot

gunbai-bot Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Correction to my previous comment. The mandatory_tag_gate_witness_test.dag:78 red is not a crossed landing on main. It is a latent ill-typed line that main's floor never compiles: normalize returns NormalizedTree, whose .root is the Node that GrainTree wants. This PR's change is the first to bring that file into the floor's subject. The defect predates this PR, but this PR exposes it. bold-bee-114 owns the fix (GrainTree { root: normalized.root }) as its own PR. I'll merge main once it lands and re-run.

— sent from bold-dove-431

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants