Repository navigation
The exact work contract and the canonical test process: one build per tree, gunbc test over witness patterns, and a worker told the one command - #10636
Conversation
…srv6 and srv9, not .232 The heads moved. .232 (srv7) was the first pair's front door and stayed in harness_engine_replica_addresses as a bare literal after Group B's head became srv9 (.236); nothing observed it until a dispatch on srv1 refused against it on 2026-09-05 while .225 and .236 both advertised the model. The roster now names both heads through gunbc.fleet_intent_network endpoint rows, and its annotation cites the authority it derives from once that module is on main: gunbc.spark.pair_serving_desired spark_pair_serving_head (spark-pair-serving, #10540). The srv9/srv10/srv11 identity slots and the srv7..srv11 LAN endpoint rows are ported from that branch verbatim so it merges onto them unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
…opology; no address is authored #10540 put gunbc.spark.pair_serving_desired on main, so the roster stops naming endpoint rows and asks spark_pair_serving_head for each serving group, then gunbc.fleet_intent_network host_lan_address for that host's LAN endpoint. A head with no endpoint row makes the roster HarnessReplicasUnresolved, which the backend observer, dispatch selection, the belt preflight and every harness entry refuse on naming the host -- never a shorter list. The seed helpers over the authored list (replica_index / address / engine_backend_url) are deleted; the pick is harness_replica_pick over whatever roster is handed in. The LAN locality is still this module's choice and is annotated as the next move: the fabric owes a locus-to-locus resolution (dispatching host -> head front door), the relation fleet_locus_ssh_target already computes for ssh. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
…on, toolchain), one local provider, dedup by store gunbc.compute.work_request (pure): a subject is a git tree, an operation is build / compile / claims with its declared outputs, the toolchain half of the identity is observed, and the outcome has exactly the five endings the roadmap row names. gunbc.compute.work_provider_local: resolves the commit's tree, observes rustc and sccache, attaches to a stored outcome or takes a lease under a per-host cold-build cap, checks the commit out detached, runs the operation as a transient user unit under the worker memory ceiling with a shared per-host CARGO_TARGET_DIR and RUSTC_WRAPPER, returns declared outputs by blob identity into a store, writes the outcome beside them and releases the lease. compile and claims consume the SAME subject's build from its store, so a worker never checks its .dag with a compiler built from another tree. compute_request_cli is the worker-facing entry; the receipt is written where the caller asks because no stdout channel is modeled. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
…d outcome is attached to The first wet run on srv2 ran the whole path -- lease, detached checkout, transient unit, cargo build (16 CPU-minutes, 4.5 GiB peak), outcome written, lease released -- and reported output_mismatch on a build that had succeeded: the binaries were declared under the checkout while cargo writes them under the shared per-host CARGO_TARGET_DIR. DeclaredOutput now names its root. A stored failure, mismatch or refusal is a record that a new request supersedes; only a stored success is attached to. Wet receipts (srv2-deploy instance, tree bc938a2): build -> succeeded with two blob-identified outputs; identical build -> attached, same identity, same blobs, no unit run; claims over the compute witnesses -> succeeded through the stored claim_batch of the same subject. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a61124d06e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| match run { | ||
| ComputeExecFailed { exit_code, stdout: _, stderr: _ } => WorkFailed { identity: identity, exit_code: exit_code, log_path: layout.log_path } |
There was a problem hiding this comment.
When systemd-run cannot launch the unit, cannot reach the user manager, or terminates it at MemoryMax, compute_run_unit returns ComputeExecFailed, and this arm unconditionally records WorkFailed. Even the explicit “no systemd-run” path therefore reports that the requested work failed, although no work ran. Preserve launch/resource failures as WorkRefusedByInfrastructure or WorkCancelled so infrastructure failures cannot corrupt experiment verdicts.
Useful? React with 👍 / 👎.
| let leases = Filesystem.List(path: layout.leases_dir) | ||
| let in_flight = if leases.success { compute_count_lines(text: leases.entries) } else { 0 } | ||
| if in_flight >= compute_cold_build_cap { |
There was a problem hiding this comment.
For concurrent requests with different identities, listing the lease directory and creating an identity-specific lease are separate operations. For example, with one active lease and two simultaneous callers, both can observe in_flight == 1, pass this check, and then successfully create different lease files, exceeding the cap of two. Use an atomic slot/semaphore acquisition so the memory-safety admission bound cannot be raced.
Useful? React with 👍 / 👎.
| leases_dir: join([root, "/leases"], ""), | ||
| lease_path: join([root, "/leases/", identity_hex], ""), | ||
| store_dir: join([root, "/store/", identity_hex], ""), | ||
| target_dir: join([root, "/target"], ""), |
There was a problem hiding this comment.
Every identity uses the same compute/target directory, while the provider explicitly admits multiple distinct builds. Cargo's build-directory lock is released when systemd-run returns, before compute_return_outputs hashes and copies target/release/gunbc; another admitted build can therefore replace that path during the post-build hash/copy window, storing tree B's binary under tree A's successful identity. Keep identity-specific final-output paths or copy artifacts while exclusive ownership is still held.
Useful? React with 👍 / 👎.
| if stored_outcome_is_success(r: attached) { | ||
| match attached { | ||
| StoredOutcomeFound { ending, identity: stored_identity, document } => ProvideAttached { identity: stored_identity, ending: ending, document: document } |
There was a problem hiding this comment.
A cached attachment validates only that the outcome document says succeeded; it does not parse the output manifest or check that each stored file still exists and matches its recorded blob. If an artifact is deleted, truncated, or replaced while outcome.json remains, every subsequent build request returns success indefinitely, and dependent compile/claim requests execute the missing or wrong binary. Revalidate the declared outputs before returning ProvideAttached, otherwise return a store/output mismatch.
Useful? React with 👍 / 👎.
| if in_flight >= compute_cold_build_cap { | ||
| compute_fresh(outcome: WorkRefusedByInfrastructure { identity: identity, cause: HostAtColdBuildCap { in_flight: in_flight, cap: compute_cold_build_cap } }, subject: subject, op: op) | ||
| } else { | ||
| let lease = Filesystem.WriteCreateNew(path: layout.lease_path, content: join([commit, " ", clock_now_probed_at_or_unknown() as String, "\n"], "")) |
There was a problem hiding this comment.
The lease records a timestamp but has no expiry, owner-liveness check, or startup reaper; it is deleted only after compute_run_leased returns. If the provider process is killed, crashes, or the host reboots during a request, that lease remains forever, permanently refusing the same identity and consuming one of the two host slots until an operator manually removes it. Add stale-owner recovery or a lease mechanism that is released automatically when its owner dies.
Useful? React with 👍 / 👎.
| fn compute_observe_toolchain(instance: HostDashboardInstance) -> ToolchainObservation { | ||
| match compute_exec(program: compute_program(instance: instance, cap: RustcCapability), workdir: instance.repo_root as String, args: ["--version"]) { |
There was a problem hiding this comment.
The Linux dashboard binds rustc through the rustup proxy, but this version probe runs in instance.repo_root, whereas Cargo later runs in layout.checkout. For a requested commit whose rust-toolchain.toml differs from the deployed provider tree, the identity records the provider tree's compiler rather than the compiler Cargo actually selects for the request; provider upgrades can consequently force duplicate identities for unchanged work and the receipt misstates its toolchain. Probe in the requested checkout or explicitly bind Cargo to the observed toolchain.
Useful? React with 👍 / 👎.
…es, through the exact work contract extdeps.bazel.target_pattern admits the three set forms the cited authority names (//pkg:name, //pkg:all, //pkg/...) as a type distinct from Label, which keeps refusing them for the single-target verb. gunbc.compute.test_selection maps repository paths to witness labels, selects by pattern, reads `test fn` declarations and the module line, and classifies each function by the floor's own home policy (long and fixture home prefixes, the cost-debt roster; the strict gate's required_gate_prefixes is a different question and is not consulted). gunbc.compute.test_run snapshots the worker's worktree as a dangling commit that moves no ref, lists the snapshot's tree with git, and runs each selected module as one RunClaims request through the local provider, so the build is shared and the run deduplicated; the report is one document per pattern with one row per module. test_pattern_cli is the worker-facing entry until the CLI verb admits patterns. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
…other way is a withheld capability
harness_guidance gains TestProcessIs { command, receipt_path } as a grounded premise (its ground is the
process's own CLI, gunbc.compute.test_run test_pattern_cli) and BuildOrTestOutsideTheProcess as a withheld
capability; with an acceptance term present the roster now entails TestWithTheProcess, the one acceptance
command verbatim. The command text is spelled once, by gunbc.compute.test_run test_process_command, and
harness_cli renders it from the repo root and gunbc binary the dispatcher now passes as --arg rows, so the
prompt and the CLI cannot drift apart. The guidance witnesses cover the entailment and its absence.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
briansrls
changed the title
|
Wet receipt for the canonical test process, srv2-deploy instance, branch head {"pattern": "//dag/test/claim/compute/...", "snapshot_commit": "cd589493f0…",
"selected": 2, "succeeded": 2, "not_succeeded": 0,
"modules": [
{"label": "//dag/test/claim/compute:test_selection_witness_test", "module": "test.claim.compute.test_selection_witness_test",
"planned": ["the_pattern_grammar_admits_three_set_forms_and_refuses_the_rest", "selection_follows_the_pattern_and_never_a_shared_prefix", "functions_and_module_are_read_from_declarations_and_classified_by_the_floor"],
"declined": [], "ending": "succeeded", "identity": "35ca283915995556"},
{"label": "//dag/test/claim/compute:work_request_witness_test", "module": "test.claim.compute.work_request_witness_test",
"planned": ["identical_requests_share_an_identity_and_every_difference_separates_them", "declared_outputs_follow_the_operation", "the_stored_outcome_round_trips_and_a_broken_one_is_unreadable"],
"declined": [], "ending": "succeeded", "identity": "93af5095bc442d8f"}]}The snapshot commit is dangling (no ref moved); the 634 s wall is one build of this tree (a new tree, so a real cargo run, shared with any later request for it) plus two hermetic |
briansrls
changed the title
…he worktree and typechecks a closure through the contract gunbc.compute.test_run gains compile_entry_cli and compile_process_command, the same snapshot-then-request shape as the test process with CompileEntry as the operation, so a worker typechecks its change with the compiler built from the same tree. The worker guidance carries it as CompileProcessIs, grounded on that CLI, stating the compiler's own bar: zero blocking diagnostics, advisories are not the bar. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY
|
Wet receipt for the compile process, srv2-deploy instance, branch head {"ending": "succeeded", "identity": "41ce0d9b84c33ef6", "subject_tree": "e5ef98d8c0…",
"operation": "compile:dag/gunbc/compute/test_run.dag", "outputs": [], "log_path": ".../compute/work/41ce0d9b84c33ef6/log"}The compile ran with the |
…0 s so a dispatch click can complete (#10644) * The compute root is a deployment member, named once for the deployment and the provider The first srv1 converge after #10636 landed left /opt/gunbc root-owned with no compute member, so a worker's first build/test request there would have refused StoreUnavailable at `install -d` -- the path was created by the provider on demand, which the attempt-state root's own member note already records as "a written path outside the roster is a path nothing creates". gunbc.roadmap_dashboard_instance dashboard_instance_compute_root is the one authority; gunbc.live_deploy.spec carries ComputeRoot as an owned member installed by the service principal after the attempt-state root and torn down recursively (its contents are a regenerable cache and store); the provider's layout reads the same fn. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY * The serve wall cap is 120 s: a dispatch is an operator click that observes every attempt, not a page render Three POST /dispatch requests on srv1 (2026-09-06 03:57Z) were refused at 30.06-30.21 s by the wall cap declared for page renders, after paying the whole cost and spawning nothing. The journal shows the route is wait rather than CPU -- about twenty host process spawns near 0.8 s each, six engine GETs near 0.5 s, a worktree add -- which is the clock the cap bounds. The declared wall cap is raised to 120 s with that journal as its basis; the CPU cap stays at 30 s. Shrinking the per-spawn cost is the next cost row. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY --------- Co-authored-by: gunbc-ci-auto-heal <gunbc-ci-auto-heal@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 participant
Stacks on #10627 (branch base). Three slices in one branch, each with its own commit and witnesses.
Slice 1 — the exact work contract, locally
gunbc.compute.work_request(pure):WorkSubject = ExactTree { tree };WorkOperation = BuildGunbcBinaries | CompileEntry | RunClaimswith declared outputs under a named root;WorkToolchainIdentity(observed rustc + sccache);work_identity= tagged fnv1a64 over tree × operation × toolchain;WorkOutcomewith exactly five endings; typed infrastructure refusals; JSON wire + decode.gunbc.compute.work_provider_local: resolves the commit's tree, observes the toolchain, attaches to a stored succeeded outcome, else refuses at the per-host cold-build cap or behind a lease, takes the lease, checks out detached under<instance_root>/compute/work/<id>/src, runs the operation as a transient user unit (systemd-run --user --wait --pipe --collect, workerMemoryMax, sharedCARGO_TARGET_DIR,RUSTC_WRAPPER=sccache), returns outputs by git blob id into a store, writesoutcome.json, releases the lease.compile/claimsconsume the same subject's build.compute_request_cli(repo_root, commit, operation, entry, functions, hermetic, receipt_path).Slice 2 — the canonical test process (
gunbc test <pattern>)extdeps.bazel.target_pattern: the three set forms the cited authority names (//pkg:name,//pkg:all/:*,//pkg/...,//...) as a type distinct fromLabel, which keeps refusing patterns for the single-target verb. Cited to bazel.build/run/build#specifying-build-targets.gunbc.compute.test_selection(pure): path → witness label (//dag/test/claim/roadmap:x_test), selection by pattern (no shared-prefix sweep),test fnandmoduleextraction from declarations, classification by the floor's real home policy: long/fixture home prefixes and the cost-debt roster fromv2.workflow.required_floor/floor_cost_debt. (required_gate_prefixesis the strict compile gate's roster and is deliberately not consulted; the first probe used it and it declined the whole roadmap package.)gunbc.compute.test_run: snapshots the worker's worktree as a dangling commit (add -A,write-tree,commit-tree -p HEAD; moves no ref), lists the snapshot's tree withgit ls-tree, runs each selected module as oneRunClaimsrequest through the provider, writes one report per pattern with one row per module.test_pattern_cli(repo_root, worktree, pattern, receipt_path); exit 0 iff at least one module selected and all ran green or were wholly declined.test_process_commandspells the invocation once.Slice 3 — the worker is told the one command
harness_guidance:TestProcessIs { command, receipt_path }premise (grounded ontest_pattern_cli),BuildOrTestOutsideTheProcesswithheld capability,TestWithTheProcessconclusion entailed only beside an acceptance term. Worker prose now says: "The canonical test process is one command:…… You may not build or test any other way: no cargo, no gunbc compile, no claim_batch by hand."harness_clirenders the command fromrepo_rootandgunbc_program, whichroadmap_dispatch_actuatornow passes as--argrows.Verified
gunbc compile --target dag: 0 blocking onwork_provider_local,test_run,harness_cli,roadmap_serve.claim_batch --hermetic:work_request_witness_test3/3,test_selection_witness_test3/3,harness_guidance_test26/26.succeeded(two blob-identified outputs); identical build → attached;claims→succeededthrough the storedclaim_batch. Thetest_pattern_cliwet run over//dag/test/claim/compute/...is recorded in a comment below.Known cost, not hidden
Each
gunbc runof these CLIs pays ~2.5 min compiling the closure before the request is looked at (instance lookup drags the roadmap modules in). The attach itself is milliseconds. Shrinking that closure or running from an emitted binary is the next cost row.Not here
Verify (the belt) switching to the request; placement across hosts over ssh; affected-set selection (the pattern type has the arm to grow into); the
gunbc testverb admitting patterns in the generated CLI dispatch.🤖 Generated with Claude Code
https://claude.ai/code/session_019iXPtAqzPaNBZbaaM5rxrY